The tragedy of commons in OSS

A data essay exploring symptoms, remedies, and opportunities Recently GitHub released For Good First Issue, a special index that tracks Digital Public Goods (DPGs) repos. This includes open-source software, open data, open AI system, or open content collection that supports the United Nations sustainable Development Goals (SDGs). Currently, the official registry counts ~132 projects which includes a variety of social and impact-focused projects like a webapp to track hypertensive patients across a population, an app for citizen participation and open government, and software to help social workers manage data on vulnerable children. For Good First Issue addresses a significant need in the open-source ecosystem: directing dedicated and socially conscious talent to projects that positively impact society and would otherwise struggle to attract contributors. Initiatives like this one can help raise the collective consciousness of contributors and ameliorate the negative externalities that arise from the tragedy of the commons. The tragedy of the commons In economics, the tragedy of the commons refers to the situation in which individuals with access to a public resource act in their own interest and, in doing so, ultimately deplete the resource. Though this concept is traditionally associated with shared environmental resources, it can also extend to the domain of open source software. While resources can’t be depleted in the context of software, they can be regarded as abused when users do not donate time, work, or resources to the communities they take from. In these situations one can observe a number of negative consequences like slow development, weakened community support, or even legal and licensing risks. The famous monetisation and maintainer problems in open source are nothing less than a manifestation of this economic pathology. A number of initiatives have tried to ameliorate the tragedy, but no definite solution exists yet. In light of GitHub’s recent initiative it is important to understand in what ways the tragedy of the commons manifests in open source, and whether curated contributor indices like DPGs can help break it. In particular, we investigate whether initiatives like DPG projects present a real opportunity to current or aspiring open source contributors. To do this, we compare the set of DPG projects on GitHub against three other sets of open source repos, 2,395 projects from the Apache foundation, 8,129 projects under the Linux Foundation umbrella, 6,805 projects maintained by RedHat. 1,255 repos belonging to Digital Public Goods, To perform this comparison we derive traction and engagement metrics using publicly available GitHub data. Specifically, we look at three fundamental metrics: issue backlog, contributor retention, and maintainer responsiveness. These metrics will help us understand the overall health of each group of projects, as measured by the activity metrics that most strongly correlate either with the resources available to a project or with the community’s support to the project. This will help us understand whether DPGs are particularly affected by the tragedy of the commons and the type of developer experience (DX) that new contributors can expect. #1: DPGs have (on average) less contributors per issue than other groups An important indicator of the “tragedy of the commons” in open source is the size of the of the issue backlog a project has. In other words, the number of unresolved open issues in the repository. Although not every open issue represents critical technical debt, an imbalance between issue creations and number of merged PRs suggests a lack of resources to manage them effectively. Public data shows that DPGs have the highest concentration of open issues per repository, but also the fewest number of contributors per issue. While this indicates a scarcity of contributor resources, the relative high number of open issues also suggests an opportunity for contributors who want to make a significant impact by applying their software skills to for-good projects. #2: DPGs have less long-term contributor retention than RedHat & Apache Contributor retention is important to keep a project’s issue backlog under control. To better understand contributor engagement we compute “On or After Contributor Retention” which we define as the percentage of contributors who merge a Pull Request (PR) at a specific time or any time thereafter after the first contribution. Interestingly, DPGs show the highest “On or After Contributor Retention” contributor within a three month period. However, this engagement drops significantly after month 3 and stays below the retention shown by Apache’s and RedHat’s projects. It’s worth noting that DPGs have the hightest short-term retention, but that six months after the first contribution 60% of them don’t contribute again. #3: DPG maintainers are as responsive as maintainers from RedHat and the Linux Foundation. Another strong indicator of resource scarcity in a project is the waiting time a contributor experiences after submitting a PR, specifically, the average time they wait to see their PR merged into the codebase. If a maintainer has more bandwidth, they can review and give feedback more quickly, thus decreasing the average merge time. Notably, public data shows that DPG repositories exhibit a level of maintainer responsiveness comparable to that of RedHat and the Linux Foundation and superior to that of the Apache Foundation. This indicates that the level of Developer Experience (DX) when contributing to these repos is comparable to that of more established and popular projects. Breaking the tragedy of the commons Let’s get contributing! Our analysis shows that DPG repos are rich in open issues and that in some respects (like maintainer responsiveness) one can expect a similar level of DX than repos under the RedHat or the Linux Foundation patronage. As the number of DPG projects grow, it’s important for aspiring contributors to use an index to browse for the most relevant communities. Considering that open source is the most secure model for developing AI systems for the public, we anticipate a significant growth in DPGs indices over the next few years. GitHub’s For Good First Issue maintains a special index of “For Good” repos, while platforms like Quira index repos across the ecosystem and helps you discover and filter issues that align to your language and topic preferences. In fact, we just released a new interface that allows you to not only select DPG repos, but also repos in other categories mentioned in this essay like the Linux Foundation. Try it out 👇 However, to fully break the tragedy of the commons it’s also important to make sure that projects are well funded and that part of those funds are distributed equally to core maintainers and casual contributors. Funding a project is now straightforward thanks to products like GitHub Sponsors, BuyMeACoffee and Patreon. However, distributing funds to casual contributors is less common, but platforms like Quira are launching ML-powered products that enable organisations to pay their contributors when they merge meaningul and valuable PRs. This can help organisations retain contributors and encourage them to return to contribute in situations where the opportunity cost of contributing is high for the developer. Do you have an open source project and would like to create a reward scheme for your contributors? Get in touch!📱 Thanks to Alex Bird for his help in preparing this essay and to Kasiasun for her feedback and comments — Artwork from asciiart.eu — Written with ❤️ by Quira

By Rodrigo Mendoza-Smith

•

Mar 21, 2024

•

9 min read

Developer reputation in the era of LLMs

Part I: The era of infinite code content 📺 Nearly a year has passed since the introduction of GPT-4. One of the public discussions we followed at the time was the one concerned with the impact LLMs would have on software development and, in particular, on jobs in this field. ChatGPT felt like magic and it was easy to extrapolate the impact of these technologies, even to the extent of rendering the software engineering profession redundant. One year later, it’s clear that LLMs are not replacing developers, but helping them focus less on the grunt work and more on the big picture. This transition has already had a measurable impact on unit economics. For example, last year, Microsoft reported that developers using GitHub Copilot are 55% more productive than those not using it. Interestingly, they also noted that 40% of the code checked in by these developers was AI-generated and remained unmodified. Statistics like these suggest a new inflection point in the software economy and an impending shift in our path towards the future of work. Specifically, they prompt us to question, in an era where code is automatically generated, how do we measure the quality of software talent? Computational Talent Validation Talent validation inherently depends on social consensus. For developers, this consensus can be reached through various means like earning a diploma, interviewing for a job, or achieving a high score in LeetCode quizzes. However, with the accelerated production of software talent that LLMs have catalysed, it’s uncertain whether these traditional consensus-building methods can truly scale or remain relevant. At Quira, we believe that in the era of Generative AI, scalable consensus can only be achieved through experience quantification. Under this approach, a developer’s skills and abilities are implicitly inferred through data and computation. Specifically, by semantically understanding the code and contribution metadata that developers generate when creating software. The evolving dynamics of open source are accelerating this potential and accentuating the need for this technology. Indeed, we are observing a shift in attitude towards open source, where a new generation of eager and ambitious developers are embracing it not only to uphold its ethos of community-driven development, but also as a means to validate their work socially. However, even in this new dynamic, the same problems persist. When code can be automatically generated from natural language, analysis of code content alone is insufficient to achieve robust consensus. To automatically quantify experience, one must incorporate additional context regarding the relevance and merit of the software created. In other words, we need to understand the social proof of code and how it has been “validated” by other members in the network. Measuring the social proof of OSS contributions Today we’re excited to release a new version of DevRank, a metric that measures the social proof of open source contributions by quantifying reputation flows in the GitHub network. In a nutshell, DevRank models the ecosystem as a network of contributors and repositories linked together by a directed edge representing a reputation-inducing event on GitHub. The current version considers three types of edges. Developer → Repo, to represent events where a developer stars a repo. Repo → Developer, for events where a developer commits to a repository. Repo → Repo, to incorporate events where a repository imports a package or a dependency. Considering all such edges within GitHub, we can construct a large directed network where reputation flows from one node into another through stargazer, commit, and import events. By applying a customised version of the PageRank algorithm on this resulting graph we can compute the stationary state probabilities of a random walker in the network. These raw probabilities indicate the importance of a developer within the open source ecosystem in the same way that PageRank calculates the importance of a webpage on the internet. The higher the DevRank of a developer, the more their contributions have been socially validated by other developers through star, commit, and import events on GitHub. If you’re interested in the technical details of DevRank, you can read more about it in our documentation. Rethinking open source status One of the coolest applications of DevRank is that it gives us a new framework to think about status & reputation in open source, not only for individual developers, but also for communities. Developer reputation is already a valuable intangible that can have a positive financial impact on the lives of developers, say, by giving them access to work opportunities or, in some cases, making them eligible for grants or venture capital funding. For these reasons, quantifying reputation in a principled and transparent way is critical to foster a meritocratic ecosystem. Think, for example, of stargazer traction, which is widely used as a measure of the success of a project in the ecosystem and that, according to Wired, is now being artificially inflated and is no longer a reliable indication of traction. Let’s look at how DevRank can change our perspective of success and status in open source. Stargazers Let’s start with stargazers, which are by far the most popular way to assess traction in open source. To do this, let’s first ask ourselves, How would the list of most popular repos on GitHub change if instead of ranking them by number of stars, we ranked them by the DevRank of their stargazers? It turns out that this adjustment would significantly alter the rankings. Certain repositories, such as FreeCodeCamp/freecodecamp, public-apis/public-apis, or jwasham/coding-interview-university would not only lose their place in the top 5 of the list… they wouldn’t even feature in the top 30! On the other hand, some repositories like facebook/react, twbs/bootstrap, or electron/electron would rise from the 10th, the 20th, and 30th, to the 1st, 3rd, and 4th place in the list. In other words, the list of top 10 repos would see a demotion of educational repos in favour of some popular development frameworks. But we don’t need to stop there… We can go one level further and look at the repos that have been starred the most by developers in the top and bottom 10% (see table below). The results are qualitatively more pronounced and show that the top 10% tends to star repos focused on developer tooling, while the bottom 10% tends to star beginner resources and educational projects. Contributor Growth Another popular way to assess traction in open-source communities is contributor growth. Contributor growth is already a purer metric than GitHub stars because an increase in the contributor growth counter requires the submission and acceptance of a PR. However, similarly to stargazers, this metric is not resistant to noise. For example, some “first contribution repos” like firstcontributions/first-contributions trivialise merging requirements for PRs, making contribution events almost as frictionless as stargazing. DevRank can help us filter out the noise and add context to these metrics. To illustrate this, consider the top repositories in the ecosystem ranked by the number of new contributors since January 2023. We can see that in the top 20 of the list, there are projects like firstcontributions/first-contributions, mouredev/retos-programacion-2023, Syknapse/Contribute-To-This-Project, which have very low merging requirements. We can derive an alternative ranking if we restrict the counts to contributions by developers in the Top 1% of developers based on DevRank (table below). When doing this, we immediately see that 30% of places in the top 20 are replaced by top AI projects and developer tools like ggerganov/llama.cpp, run-llama/llama_index, langchain-ai/langchainjs. Grudge matches DevRank can also aid in determining a winner in classic rivalries between developer communities. For example, a frequently debated topic among AI researchers is whether PyTorch is better than Tensorflow. There are many non-violent ways to settle this matter, with our preferred option being to appeal to open-source data. Unfortunately, in this particular case, vanilla statistics provide limited insight and fail to give a definite conclusion: TensorFlow leads in the stargazer count with 180.8k stars, but PyTorch is imported by 265.9k repositories. To gain true insight into this question we need to increase the resolution of the data. For example, we can use DevRank to understand which repo is preferred by developers with higher reputation. To do this, we can compute the median DevRank of the contributors of each dependent repository. The results are fascinating. The median DevRank of PyTorch users turns out to be 140.3, whereas for TensorFlow users, it stands at 56.9. Moreover, the total DevRank sum of repos that use PyTorch is 2.65 bn, while the DevRank sum of repos that import Tensorflow is 1.75 bn. This experiment shows that even though PyTorch has significantly fewer stargazers than Tensorflow, it is generally used and preferred by repositories and developers with higher DevRank. Go PyTorch! 🔥 Outro The future of developer work and education is evolving quickly and LLMs playing a significant role in accelerating this transformation. As the software engineering profession becomes democratised, the need to assess the quality of software talent in a scalable and systematic way becomes paramount. At Quira, we are leveraging DevRank along with other signals to algorithmically gauge the expertise of our users. This allows us to rank developers in our community across languages, frameworks, topics, and industries to ultimately match them to paid contribution opportunities in open source. Our technology is already helping a number of Commercial Open Source Organisations (COSS) receive PRs from top contributors outside of their orbits. If you’d like to use Quira to build a dynamic and sustainable community of high-quality contributors, drop us a line, we’d love to chat. Alternatively, if you’re a developer looking to build or monetise your reputation in open-source, then log in to Quira and start your journey with us.

By Rodrigo Mendoza-Smith

•

Mar 15, 2024

•

11 min read