It's time to look back on S3's 20-year history
Fantastic write up, but I still have doubts. As the author states: Serving git at scale is hard. I can only concur.
I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.
Anyway, I'd love to have a friendly chat with the author.
This is an excellent post that explains why their version of git hosting is more scalable. But it seems like most people won't need it, assuming Github remains free to use and they get their availability problems fixed.
Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
Reading this, I can't come to a conclusion other than "git rapidly starts sucking balls for use cases outside of Linux development". Github started with ze_current_fastness but then ended up recurrently smashing into walls because of it.
They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.
Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.
> The productivity cost of five minutes of downtime in your CI system is hard to quantify in dollars, but it is, by any measure, a humongous amount.
Not sure the last month GH had fewer than five minutes of downtime - but I think it's been a while
The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.
Any ideas how these interactive animated diagrams were made?
Very cool write up. But I do have to wonder, is Git the right solution for version control given where we are heading? The commands are all muscle memory for me now at this point, but all the projects I've been working on lately have just all moved to trunk based development, rebasing, and squashing merging feature branches onto main.
I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.
> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
Not just durability. But read after write guarantee on a distributed object store.
How many startups and corporations built on top of S3 semantics?
Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.
I don’t see a button to switch to English and the German text is a pain to read, comparable to Reddit auto translate..
The button is apparently in the footer: Another comment earlier has the english-only link: https://news.ycombinator.com/item?id=49376960
Remember, don't put large objects in cnt, it makes it a real pain later.
cnt?
Continuity system of Cursor. Brilliant name, just like the octo"cat"! Elon will approve.
how does git lock the repo on extreme concurrency? let's say 1000 devs writing to the same repository? eventually you'll have saturation of the resource locking everyone out.
When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.
If those 1000 devs write to different branches then you only take a light lock on the branch, usually that is not the problem.
The problems start when there is a lot of changes on one branch and usually the first band-aid is to put a commit queue in place. The few very big companies that need even more than that probably aren't going to put their code on a new git hosting provider tomorrow.
that sounds like a practical solution once you reach that size. yeah you're right i think only the refs are locked, the rest of the objects can all be submitted concurrently, so it would be high contention refs/branches that would be a pain point.
> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.
Doesn't 3PC require all nodes to agree, not just a majority?
I would think so - outside this sentence the author indicates they're talking about consensus across all members.
IE - the interactive diagram and note about latency being bounded by the slowest replica
https://cursor.com/en-US/blog/git-at-any-scale to avoid automatic translations (couldn't find a button to switch)
Thank you so much, what an obnoxious behavior!
The language selection is in the footer. Interestingly, the URL for English is the original post without en-US.
WAL on S3 is the new normal.
Interesting, wonder how Cursor will scale with Git with this approach.
There's nothing Cursor can do that GitHub/Microsoft can't in 2026... And vice versa... after several years of Cursor vibecoding a GitHub clone while catching up with GitHub's new features. Git's performance it's not the issue; it's GitHub Actions, PRs, etc. The moment Cursors steals paid GitHub customers and gets the same workloads, they will start having similar issues! Stealing engineers who couldn't fix this at GitHub won't make much of a difference. GitHub is not just source control; everybody can set up Gitolite and have an amazingly configurable and secure Git setup. I did this 10 years ago, and CODEOWNERS, protected branches, and tags can't do 1% of what I had done before. Cursor is going all over the place as it can't compete with their model. So, this is a defeat, a pivot, not something glorious!
I'd argue that for whatever other faults they have, Composer 2.5 is pretty competitive as an implement-planned-work model. It's several times faster than Sonnet 5, cheaper, and performance is comparable.
Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.
Perhaps you didn't read the article, but the implementation they describe is interesting and perhaps easier implemented in a new project from scratch.
It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.
Took me some thinking to figure out where this design is not optimal.
The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.
Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).
For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.
Part of the problem is that the API you're supposed to provide externally depends on git packs which you might have to reconstruct on the fly from such a system.
Exactly. I would love to see that design work, but it doesn't for any realistic workload.
Then the local cache system needs to provide or cache those git packs
It doesn't matter how many technical blog posts they put out, I will still never host my code with Elon Musk.
I missed the news about SpaceX acquiring Cursor for 60 billion. Good to know.
Agreed. The recent privacy kerfuffle with Grok CLI (plus the untrue "it's your fault" response, although they then changed the behavior) is a really bad sign for how your code and data will be the treated.
Same. Sometimes you have to live with inconvenience to keep a moral stand. Those down-voting don't understand why any sacrifice of convenience or tech is worth it.
I agree, although I don't think their offering is even convenient. I also don't use Cursor (I think most people have moved on to Claude Code/Codex), so also not helpful from that point of view. Regardless, not only do I find it morally wrong to give Musk money, I think it's a bad business idea. He can't be trusted and your code is at risk on his platform.
> Those down-voting don't understand why any sacrifice of convenience or tech is worth it.
FWIW my comment was actually highly upvoted but it looks like X employees and/or Musk fans showed up and downvoted all criticism in this thread.
There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.
at this point, s3 just means object storage and doesnt mean it actually has to be hosted on aws. theres plenty of other companies that provide s3 compatible storage apis.
APIs, sure, but do they provide the semi-magical consistency, atomicity, durability, and latency guarantees that this depends on?
Yes a good object storage service provider would. It all depends on what you are optimizing for. If you are only optimizing for cost then there are several cheap options that probably don't make sense for hot or warm storage. But if you care about availability, consistency, and latency guarantees then there are different service providers available.
That was a very well written and interesting article. I really enjoyed it. Some good background for people telling GitHub to "just scale".
GitHub issues are mostly with PRs, CI, etc, not with git
Source?
When I look at one of the recent blog posts (https://github.blog/news-insights/company-news/an-update-on-...) I see crazy charts of PR merged, Commits, Repositories created. These are all git operations and not just some slow Rails app.
Have you read the article?
> Short term, we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL), redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load. We also leveraged our migration to Azure to stand up a lot more compute.
The scaling issues they have don’t have git as the bottleneck from what I understand
In the same article it says:
> The number of repositories on GitHub is growing faster than ever, but a much harder scaling challenge is the rise of large monorepos. For the last three months, we’ve been investing heavily in response to this trend both within git system and in the pull request experience.
They also mention large monorepos and git read operations as a core issue in the other incident post that's on the HN front page right now:
https://github.blog/news-insights/company-news/the-august-17...
It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
Came here to share a related fact :)
This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.
I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.
So no surprise these two contexts resulted in the design of Continuity as it is.
https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.
Is this enough to counteract the reputation of his new boss, Elon Musk?
<edit>: there is lots of demand for a GitHub alternative due to poor uptime, poorly implemented components (CI), and Microsoft stewardship. Switching, especially at this early stage requires a huge component of trust.
Cursor isn't covering itself in glory regardless. The flagship app is getting enshittified at a surprising clip. It constantly pops up and interrupts your work pushing new features, changes your model to whatever the latest Grok is without prompting, has this mystery meat UI that is constantly changing, pushes cloud agents in ways that are definitely designed to trick you. We're actively looking at alternatives, I wouldn't touch anything this company produces from here on out.
[flagged]
Did you really create an account for this?
its literally elon getting mad and making an account calling people cock suckers
But OP is the loser for leaving a comment.
* spent tokens on this
An LLM would have produced a more interesting and better formatted insult.
tired of this
There's a reason you say that with a throwaway account. Even you know it's shameful.
Well, there's going to be a lot more of it. You can't be an outright ultra-right wing racist and expect that it doesn't cause brand damage. It never made business sense for Musk to act the way he does, but he couldn't help himself. Do not expect people to just be ok with it.
tired of you
Just like with climate change, the problems Elon creates don’t go away just because people are reminded of them.
He still pushes his agenda, he still has lots of followers who believe his nonsense
An agenda to improve the future for humanity? What do you think his agenda is?
Why don’t you just go read his twitter account and take a guess. He hasn’t been focused on improving the future for anyone but himself for years.
Racism? Here's Elon Musk today whining about George Floyd.
His agenda is to improve the stock price of TSLA and SPCX via a) overpromising and memeing on X and b) political deals.
Members of he Kushner family got rich via the outrageous $60B payment for Cursor, and I'm sure Musk will be rewarded later with government deals.
He can do that without the "Roman salute" part. Or the "methane reactors without emission controls" part. Or the "whining about the great replacement" part.
To get racist murderer cops out of jail, is the story I read today.