Add as preferred on Google
Breaking the 20-Field Limit: How We Built Histora for Salesforce Field History TrackingRead More
How a Repository Health Dashboard Helped Open edX Patch certifi in 48 Hours
In October 2026, a compromised certificate authority forced every Python project that bundles certifi to upgrade, fast. For Open edX, that meant finding every one of its roughly 180 repositories that pinned certifi, 85 in all, and patching each one. The fix was one line. Knowing where to apply it was the hard part. This is how OSSvitals, an open-source repository health dashboard, turned that question into a live tracker in 27 minutes, and how the community was ready to ship Open edX security releases within 48 hours.
What Happened in the eMudhra Certificate Authority Compromise
Every time a browser shows a padlock, it is trusting a stranger. The website hands over a certificate, a kind of digital ID card, and the browser checks who signed it. The signers are certificate authorities, a few dozen organisations the world has agreed to believe. Each one holds secret signing keys, and the whole system rests on those keys staying secret. On the evening of Saturday 17 October, a researcher wrote to Google's Chrome team with bad news about one of them. Eight signing keys belonging to eMudhra, an Indian certificate authority, had been generated with too little randomness, which meant they could be worked out from public information.
Within four hours, Mozilla's engineers had done exactly that, recovering most of the secret keys from nothing but certificates anyone can download.
A key like that is a master stamp. Whoever holds it can issue a valid-looking ID card for any website on the internet: a bank, an email provider, a university's login page. That is what a certificate authority compromise means in practice.
The browsers moved fast. Chrome blocked every certificate signed by those keys less than two hours after the report, without anyone installing an update. Firefox followed. Google then removed the affected eMudhra root from Chrome's trust store, a change scheduled to reach every device by 27 October, and on 22 October eMudhra announced it was leaving the global web certificate business altogether.
For browsers, that was the end of the story. For most of the software that runs the world, it was only the start.
Why Python Software Depends on certifi for Certificate Trust
Here is the part most people never see. A browser carries a list of the certificate authorities it trusts, and the browser's maker updates that list for you. A lot of software is not a browser, though. Web services, background jobs, data pipelines, the scripts that build and test code: they talk to the internet too, and many of them carry their own copy of the list.
In the Python world that copy is usually a small package called certifi, and a large share of Python software, including anything built on the popular requests library, depends on it for certificate trust. On 20 October its maintainers published a critical security advisory and a new version without the affected root. Every program that pins an older certifi keeps trusting the compromised root until someone changes one line, gets it reviewed and ships it.
For one application, that is a five-minute job. For an open-source ecosystem, it is a very different problem.
Finding Every certifi Dependency Across 180 Open edX Repositories
Open edX is the open-source learning platform behind edx.org and a large number of universities and organisations that run their own sites. It is not one program but around 180 code repositories on GitHub, looked after by Axim Collaborative, 2U, edly Arbisoft, OpenCraft, eduNEXT and many individual contributors.
On Wednesday 21 October, Kyle McCormick at Axim opened the tracking issue for the certifi upgrade. The fix itself was obvious. The question underneath it was not: which of those repositories carry a certifi dependency at all, in which files, and which of those copies run in production rather than only in tests?
Nobody had that list. Nobody ever had.
How Past Dependency Upgrades Led to OSSvitals
On 25 September 2020, a new release of another Python package, importlib-metadata, broke the tests in every Open edX repository that still supported two older Python versions. There was no list of which repositories those were. I knew because I had worked in them. I fixed six that day from memory and a seventh five days later, and I was never sure there were no more.
That became the pattern for years: fix a failure in one place, then hunt for the same failure everywhere else from memory. In 2022 a one-line change I made to a shared constraints file altered what more than a hundred repositories installed that week, and nobody, me included, had a picture of that reach.
So I started drawing the map, one dependency upgrade wave at a time. Thirty repositories moved from one CI system to another in six weeks. The Django and Python upgrade waves.
Then I took over edx-repo-health, a set of checks that clone every repository and write down what they find, and moved it off a Jenkins server and a Google Sheet into a daily GitHub job. In 2024 Soban and I presented it at the Open edX Conference in Stellenbosch, South Africa. The talk went well, and then not much happened. A spreadsheet in a repository is not something anyone opens in the morning.
2026 changed that. In March, Bitergia, the vendor that hosted the community's contribution analytics, shut down its Open edX instance, and the following month the Maintenance Working Group asked for a replacement.
Over the summer I rebuilt the checks and put a dashboard on top of them. Since September, OSSvitals has graded every Open edX repository daily, and its collector reads every repository's requirements files and lockfiles along the way. On the day the certifi issue opened, the raw material for the list already existed. The dashboard just could not answer the question yet.
Building a Live certifi Vulnerability Tracker in 27 Minutes
The dashboard knew what every repository depended on. It could not yet answer "who is still below the safe version?", so I wrote that piece then and there.
The new layer did three things. It compared each repository's certifi version with the fixed release. It looked up any open pull request already changing it. And it put every repository on one page with one of three labels: done, pull request open, or not started.
Twenty-seven minutes after Kyle opened the issue, I posted a link to the live certifi vulnerability tracker on it. 85 repositories pinned certifi, and each one had a status and, where there was one, a link to its fix. The code behind it was merged within the hour, and the page refreshed itself every hour for the next two days.
F1. The certifi campaign tab on 27 October: 77 of 85 repositories done, 8 with a pull request open.
Under the hood. The campaign layer is a few hundred lines on top of a scan that already ran: the daily collector parses every requirements file and uv.lock across 182 repositories (a wider set than the 169 that get a grade). A campaign is one block of configuration naming the package, the minimum safe version, the advisory, and which files count as runtime. The uv and pip upgrades that followed each got a campaign from one more block.
F2. What a campaign is: one configuration block, the daily scan, a comparison against the safe version, a pull request cross-reference, and one of three statuses per repository.
Shipping the certifi Fix and Open edX Security Releases in 48 Hours
F3. Repositories with the fix merged, by hour, reconstructed from GitHub pull request merge times. 56 by the end of 21 October, 72 by the end of the 22nd.
What followed was a lot of people doing unglamorous work at once. The fixes came from three places: the community's bulk upgrade job, which I ran across the organisation; Dependabot, GitHub's automated update bot; and Kyle himself, who wrote a large share of the pull requests by hand. My part was running the job, reviewing the batch, hand-fixing the repositories the job could not patch, and keeping the tracker honest, so that anyone in any timezone could see what was left without having to ask.
By the end of the first day, 56 of the 85 repositories had the fix merged. By the end of the second, 72. On 22 October the Open edX Verawood security release was cut.
Kyle summed it up a few days later in the Open edX Slack:
"The disclosure affected every backend Open edX repository, but thanks to Usama, we were ready to tag security releases for master and Verawood within 48 hours of the public disclosure."
As of 27 October, 77 of the 85 are done and the last 8 have pull requests open.
How OSSvitals Monitors Open Source Repository Health Every Day
It would be easy to tell this as a story about one fast afternoon. It is really a story about the quiet days before it.
F4. The overview on 27 October: 169 repositories, organisation average 72.6, grade B.
On an ordinary day, OSSvitals monitors open-source repository health by grading 169 Open edX repositories on nine signals, from how recently anyone committed to how quickly a pull request gets a human reply, and it publishes every weight and threshold so anyone can check the maths. On 27 October, 139 of the 169 are an A or a B.
Most days nobody reads the grades with any urgency, and that is fine. The page the Maintenance Working Group cares about is At Risk: production repositories that nobody owns any more, or whose owner has gone quiet, slowly getting worse. Those are the ones the next emergency will find first.
The dashboard also watches the 89 automated jobs that keep each repository's dependencies up to date. On 26 October, 42 of them had a failing latest run. That is where the next incident is waiting: the repositories where, next time, no bot will open the fix.
Under the hood. Missing data scores 50, not zero, so a finished library is not punished for being finished. The pipeline writes plain files on a schedule, and the website and the MCP server only read them, so nothing calls an API when a page loads and anyone can rebuild the numbers from public data.
F5. The ordinary day. Files are written on a schedule, and the dashboard and the MCP server only read them.
The response was fast because the ordinary day already existed.
Why Dependency Mapping Speeds Up Security Incident Response
Nothing in OSSvitals is specific to Open edX. It runs for any GitHub organisation, each with its own scoring rules, and a second public dashboard already tracks how far a set of forks have drifted from their upstream projects. The same data is served over the Model Context Protocol, so an AI assistant can ask "which of our repositories still pin the vulnerable version?" and get the dashboard's answer instead of a guess.
The lesson I keep relearning, from 2020 to this October, is the same one. In an incident, the fix is rarely the hard part. Knowing where to apply it is. If your organisation has more repositories than fit in one person's head, draw the dependency map before you need it.
The code, the scoring method and the live dashboards are at ossvitals.org. If a grade looks wrong to you, I would like to hear about it.
OSSvitals is an open-source community project under AGPL-3.0. It is not affiliated with or endorsed by Axim Collaborative. Open edX is a registered trademark of Axim Collaborative.
...Loading Related Blogs
