Ten days to Orochi: how we took our monorepo off GitHub's merge queue
A fleet of AI agents writes most of our code, and GitHub's merge queue was the slowest part of the company. So we replaced it with Orochi, our own version control and CI. Every change now lands through Orochi, every check and release runs on Orochi CI, and GitHub is a read-only mirror.
Why: the merge queue
Goliath Data is a small team. Most of our code is written by Claude and Codex sessions on a fleet of dev boxes: a person files a card, a session writes the change, answers its review and asks to land it. On a busy day that’s hundreds of pull requests.
We knew where the pain was. CI runs took about ten minutes, which is fine. GitHub’s merge queue was not. It merged in lock step, so one failure threw away every speculative build behind it, and 41% of its runs ended up cancelled. We’d bolted on an hourly batching rollup to cope, and the queue still backed up to 188 entries. A PR that finished review in the morning landed after lunch.
Anthropic’s Redesigning source control described the fix: drop the global merge lock, make landing a compare-and-swap on main, test after landing, and protect what you deploy. Nothing off the shelf did that for a repo like ours, so we built it. We call it Orochi.
Why “Orochi”
Yamata no Orochi is the great serpent of Japanese myth, told in the Kojiki, Japan’s oldest chronicle: one body with eight heads and eight tails, so long it stretched across eight valleys and eight hills. The storm god Susanoo beat it by setting out eight vats of strong sake, waiting for every head to drink, and cutting it apart while it slept. In one of its tails he found a sword, Kusanagi, that became one of Japan’s three imperial regalia.
The shape fit what we needed. Our work comes from many heads at once, engineers and dozens of agent sessions each pushing their own Change, and it all has to end up in one body, main. A merge queue makes the heads take turns. Orochi lets them all move and keeps the body whole. We kept the serpent in the details too: Orochi’s mark is the coil, a serpent wound around main, and the graph of every day’s lands is drawn in scales.
What we built
Orochi is about 158,000 lines of Rust, most of it written by the same agent fleet it now serves. A few choices shaped it:
- Git’s object format, Postgres for refs. Objects are keyed by git SHA in cloud storage, so mirroring to and from GitHub needs no translation. A land is a single compare-and-swap on the
mainref. Ifmainmoved, rebase and retry. - No approval belongs to a person. A change needs a passing verdict from any reviewer that isn’t its author, agent or human. Protected paths (CI, auth, billing, migrations, Orochi itself) need two, from different reviewers. Neither has to be a human.
- Everything leaves GitHub Actions, releases and production deploys included. GitHub keeps one job: a read-only mirror of
main.
How a change lands
A change goes from a session’s worktree to production without touching GitHub and without waiting on a person.
- Push a Change. There is no pull request. Each push is a changeset bound to a sha.
- Review. Native reviewers post verdicts bound to that sha. If one review provider is down, the next takes over.
- Pre-screen. Orochi CI runs every job the Change could affect, including the heavy ones.
- Land. A compare-and-swap onto
main. With no queue, that takes seconds. - Check after landing. The full suite runs on
main. A red is bisected, the culprit’s check is re-run, and only a confirmed culprit is reverted. - Release. The release lane cuts a candidate from
mainand Promote ships it to production. A hotfix is a Change on a release line that goes through the same lanes.
The lawn and the coil
Orochi’s mark is the coil: two tapered crescents nested into one inward spiral, the serpent wound around main. It sits on everything Orochi draws, including the lawn.
The lawn is a contribution graph of every land on main. Orochi reads main’s first-parent history every minute and serves the last year as a card anyone can embed in a README, no token needed. orochi lawn prints the same grid in a terminal, with a diamond on each day that landed through Orochi, and orochi streak prints the current run. On the day we wrote this, the card read 31,316 lands in the last year, a 123-day streak, and a busiest day of 723 lands on October 16.
Inside Orochi’s console, the Scales tab draws that year two ways: as the lawn’s grid, or as the coil, which winds the same days into the mark itself, starting at the centre and growing outward as main grows, each scale sized by its day’s lands.
The same year, raised. Each day is a column as tall as its lands, so the climb through September and the October cliff read at a glance. Days that landed through Orochi carry a bright cap.
And wound into the serpent itself: the year as a body that thickens on busy days, ending in eight heads, each grown to one group of authors’ year of lands.
What we learned building it
Slower before faster. On its first evening Orochi landed a quarter as many PRs an hour as GitHub had, at five times the CI minutes per PR. Most of that was our own tooling running the full suite before handing a PR over, then Orochi running it again. Removing the duplicate was the first of many speed fixes.
Humans were the next queue. Once CI was quick, changes sat waiting for reviews, and some waited on a human approval. We made it a rule that no change ever waits on a person. Every gate has a bot path, and a reviewer outage falls through to the next provider for every Change at once.
Batch the lane, not the PRs. The old hourly rollup, which merged many PRs into one so heavy CI ran once, only added waiting, so it’s gone. Inside the lane, Orochi stacks a few queued changes, checks the top tree once, and bisects if it’s red. The batch size follows the lane’s recent red rate: about 4 on a clean day, singles on a bad one.
Land-first needs a real pre-screen. Landing first and checking after only works if reverts are rare. Our first replays reverted almost 13% of lands. When we looked at why, every pre-land failure was on a heavy job that a PR’s own CI never ran. The fix wasn’t more review. It was a pre-screen that runs every affected job before landing, which brought replayed reverts down to about 4%. We still ramped land-first by path risk, low-risk code first, with a shadow mode that logged what it would have done.
A flaky test can revert someone else’s work. Under land-first, a test that fails at random gets bisected to an innocent change and reverts it. That happened to us before we’d built the guard against it, which is why Orochi now re-runs a culprit’s check on its exact tree and reverts only if it fails again. Since then it has stopped several false reds without reverting anything.
Map every gate before you replace a system. Each red main after the switch was a check the old system ran that the new one didn’t: a lint that only ran on PRs, a test missing from its path filter. Now, before replacing anything, we list every job the old system runs and where it runs next.
Judge a replacement on quality, not agreement. Our native reviewer agreed with GitHub’s Codex on only 19 of 50 replayed PRs, and every mismatch was it being stricter. Agreement only proves imitation. A replacement now has to catch every real bug the old one caught, block no more good changes, and answer within five minutes. The same bar applies to every CI job we move.
Cut over one tier at a time. PR checks moved first, then the land gate, then change intake, scheduled jobs, and finally the production writers: releases, Promote, Terraform and hotfixes. Each move passed a shadow bar first: 95% agreement with Actions, every mismatch explained, and no more than 1.2 times the wall time. We started shadows on demand and replayed history instead of waiting for natural runs, which turned days into hours.
Results
The first week was a steady climb down. The median Change took 213 minutes from push to main on October 23, Orochi’s first full day. By October 28 it was 17 minutes, on the busiest day yet, and through 3:40 PM on October 29 it was 11.9, across 378 changes. On GitHub, the median pull request took 4.4 hours from opening to merge, so Orochi lands the typical change about 22 times faster. Once a Change is approved, the median wait to land fell from 69 minutes on October 24 to 2 seconds.
| GitHub merge queue | Orochi | |
|---|---|---|
| Queue | up to 188 deep, 41% of runs cancelled | none |
| Waiting to land once approved | median 1.4 h | about 2 seconds |
| Push to landed | 4.4 h median (PR opened → merged) | 11.9 min median |
| Busiest hour | not measured | 58 changes |
| Automatic reverts | none (pre-land gate) | about 1% of lands |
In its first nine days Orochi landed 2,397 changesets, 2,114 of them real changes from six engineers and the agent fleet. On the afternoon of October 29 it landed 58 changes in a single hour, and it is still getting faster. The lane’s queue sits at zero, so landing speed is now set by how fast work arrives, not by the system.
What’s next: Orochi dreams
Looking back, every incident on the way had one thing in common: someone found it by hand. A setting leaked into tests, reviews didn’t converge, a path filter had a gap, spend crept up. So Orochi now looks for these itself.
Every night it reviews its own records: lands, reds, reverts, review rounds, stalls, settings changes, and CI speed and cost by runner pool and job. It groups what it finds into recurring failure classes and ranks them by minutes and dollars lost. A safe fix becomes an Orochi Change reviewed like any other. A risky one, like a runner size or a gate, runs in shadow first and ships only if it measures better. A day later Orochi re-measures and keeps the fix or reverts it. Every step is visible in the Orochi console and in a nightly thread.
Dreaming never approves its own work, and it can’t touch revert guards, production promotion or secrets. It’s how the system that replaced our merge queue keeps getting faster and cheaper without anyone tending it.