GoliathDevelopersGet an API key
Blog / Version control

785 lands a day: Orochi's second week

A full week at record volume: where we traded speed for correctness, the ideas we borrowed, a simulator for the land lane, and Argus, our nightly AI QA.

A week ago we wrote about Orochi, the version control and CI system that replaced GitHub’s merge queue for our monorepo. Launch numbers only tell you so much. The real test is a full week of production load, so here’s that week.

Orochi landed 3,665 Changes between Oct 30 and Nov 5, and 785 of them on Nov 4 alone, a new record. It peaked at 77 lands in an hour. Seven of those Changes were reverts, about 0.2%. The median from push to main over the week was 18.4 minutes, 14× faster than the 4.4 hours we had on GitHub’s merge queue.

That’s slower than the 11.9 minutes we reported last week, and much of the difference is a decision we made on purpose. This post covers that decision, the ideas we borrowed from other teams, how we now prove the lane correct, and Argus, the new system watching the product itself.

Trading seconds for a greener main

Land-first version control has one known tradeoff: two Changes can each pass alone and fail together. On Oct 29, two Changes each passed our 500-line file limit and together took a file to 501.

So on Oct 30 we made the land lane stricter. It now runs whole-tree jobs, like file length limits, bundle budgets and unused exports, on the exact tree each Change lands as. If a job goes red, Orochi runs it again on the tree ahead of the Change to tell a real composition failure from a main that was already red.

Two charts for Oct 28 to Nov 5. Changes landed per day rise to 785 on Nov 4. Median push to land rises from 11.9 minutes on Oct 29 to 39.2 on Nov 3 and is 15.8 on Nov 5 so far. Median queue wait rises from 2 seconds to 19.8 minutes and is 5.1 so far.
The re-check moved the queue wait from seconds to minutes. The Nov 2 and Nov 3 peaks were two separate incidents, both fixed.

That re-check runs about 5 minutes at the median, and it’s now most of the normal queue wait. It’s a good trade. Skipping it would save about 7 minutes, and on one measured day it caught about 18 distinct failures before they reached main.

Two incidents at record volume

Nov 2: a slow recovery from a red main. One bad land held main red for 86 minutes, because its revert sat in a normal batch and failed a format check. We changed the rule: a revert now lands first and alone, and the fix for a red main goes to the front of the lane.

Nov 3: a deadlock in CI. A read parked mid-query held one of only 16 database connections while the dispatcher waited on the same run. CI stalled three times that afternoon, and queue wait hit its peak of 19.8 minutes at the median. We removed the deadlock and tripled the pool to 48. The red-main guard held throughout and Orochi never fell back to check-first, so the design did what it was built to do.

By midday on Nov 5, the median was back to 15.8 minutes, with a p90 of 24.2.

Hardening

A system this new finds its edges under load. This week’s fixes made a few things stricter:

  • Whole-tree reds open incidents. A red in a job that post-land checks don’t cover now opens an incident like any other.
  • Budgets judge a Change’s own growth. A bundle budget now fails a Change only for what it added. Replayed against Nov 4, that avoids all 19 land failures it caused that day.
  • A hotfix is only a hotfix. A hotfix press is now refused if it carries more than the fix or any migration.
  • Orochi got its own database. We moved it off the shared instance on Oct 30 in a 1 minute 48 second cutover.
  • Runner checkouts. A revert brought back objects our runner mirrors had pruned, and about 20% of jobs failed at checkout for 77 minutes. We paused auto-revert during the outage, so no good work was reverted, and fixed checkout.

Tuning with real history

Land-first makes tuning cheap, because you can replay an idea against real lands before you turn it on, and put it in shadow the moment a guardrail trips.

  • Check every third land. Replay showed about $48 a day in savings. Live, the first red’s bisect took 26.9 minutes against a 13-minute guardrail, and it was back in shadow within 50 minutes with no red missed. We closed the experiment on Nov 5. A faster culprit is worth more than $48 a day, so every land still gets checked.
  • Bigger batches. Replay showed caps of 4, 6 and 8 land the same Changes, with a worse p99: 38 minutes against 19. Adaptive batching stays on for surges while we keep measuring it.
  • Smaller wins. Resizing five jobs should save about $6.70 a day, and runner cost was about $0.58 per land on Nov 3. Every land now records where its time goes, phase by phase, so the next tuning argument starts from data.

Ideas we borrowed

Some of the week’s best ideas came from outside.

Merge rules. An open-source merge service gave us the idea of declaring how certain files merge, and it solves a problem we’d measured: in the 48 hours to Nov 4, colliding migration numbers caused 118 land refusals across 70 Changes, each sent back to be re-stamped by hand. Orochi can now renumber a colliding migration and merge list files itself as a land builds its tree. The rules are read only from main, so a Change can’t grant itself a merge. It’s measuring in shadow.

Footprints. The same project models a change as a footprint: what it writes plus what it reaches through interfaces. We adapted that to a single file, computed from git trees with no node_modules: a body-only edit stops at its direct importers, and an interface change reaches every dependent. Across 1,306 lands in shadow, it would have skipped about 10% of the tests we selected. We also tried it as a way to skip the whole-tree re-check, and it saved only 0.2 minutes, because nearly every land’s reads overlap by name. So the re-check stays and nothing narrows yet.

Five levers, from a call with another team. A conversation on Nov 3 gave us a list of CI costs to go measure, at about 110 runner minutes per land. Fetch round trips turned out not to matter, with a warm checkout at 2 to 6 seconds. Spot reclaims were 0.8%. Post-land full-suite runs are about 25,000 minutes a week, and we’re keeping them because bisect proofs depend on them. Hydration was the real one, at 15% of CI minutes, and restoring setup outputs has already cut that step from 19 seconds to 8.

Proving the lane

A land lane is concurrent, and the bugs that matter only show up in rare orderings. This week it got two layers of proof.

The first is a TLA+ model of the lane, its heal and promote qualification. TLC checks it on every Change that touches the spec or a file it models, with larger models nightly, and CI fails a Change that edits a modelled file without updating the model or naming a waiver.

The second is deterministic simulation. It runs the real lane, bisect, revert and batch sizing against a fake clock, a seeded world and a fake CI, on a real Postgres. Same seed, same run. The TLA+ properties are checked after every step, and a failure prints its seed and a minimal trace, so any bug it finds can be replayed from cargo test.

We also started scoring our reviewers instead of trusting them. A nightly replay runs 28 Changes that broke main and 25 that didn’t through each reviewer’s real prompt and model. Together our three Claude reviewers caught 50% of the breaking Changes for $1.29 a Change, and they flagged something on all 25 clean ones. Now that we have a score, we can tune against it.

Argus: watching the product

Orochi can prove a Change passes its checks. It can’t tell you the page looks wrong. So we built Argus, named for the hundred-eyed watchman of Greek myth. Every night it uses staging like a first-time customer, follows flows written in plain English, and sorts each red into a real bug, a flaky test or an environment blip.

In its first days it filed seven bugs, including unread counts that disagreed across the inbox and a contact list that couldn’t be searched by name.

We measured Argus the way we measure everything else, and it changed how it runs. From Nov 2 to Nov 5, per-Change sessions took $19.47 of the $24.35 we spent and found no bugs, while the nightly pass runs at about $26 a month. So per-Change sessions now run only while the newest release is red.

Three more pieces shipped alongside it:

  • A staging gate. It reads 14 pages on every release candidate at no AI cost. Its negative control strips all GraphQL data, and all 14 fail, so we know a pass means something. It’s in shadow and doesn’t hold a promote yet.
  • Its own dream. Each night Argus reads its record and proposes edits to its flows, and nothing changes until a person approves.
  • A red-team track. Security probes run through the same pipeline against staging only. The pipeline is built, and each probe gets armed one at a time, with a test proving it goes red when its control is removed.

For now Argus stops at every confirm step, and flows that take real actions stay off until we’ve proven its test orgs are sealed.

What dreaming found

Dreaming, Orochi’s nightly review of its own records, finished its first full week: nine runs, about $0.15 in total on a small model.

Part 1 described the full loop, where dreaming opens safe fixes and re-measures them a day later. So far it runs the first half: it finds and ranks problems, files a card, and our sessions do the fix. It can’t open a Change yet, because only a narrow set of problems is marked safe to act on, and none has appeared since its third run. In the latest run, 0 of 564 classes were safe. So re-measure has never run either.

It filed 23 cards, and 17 are merged. Merged isn’t the same as fixed, though, and no merged card has been re-measured.

Dreaming’s minutes are estimates from fixed formulas, not measurements, so read them as a ranking. Review rounds, for example, are rounds past the second, minus those that caught a blocker, times five minutes. Totals also swing with volume and alert noise, from 3,153 to 32,465 estimated minutes a night. In the latest run, 494 of the 564 classes were Watch alerts.

Estimated minutes per day from dreaming's nightly formulas, Oct 30 to Nov 5. The false-red class falls from 6,797 to 0, with a 1,746 spike on Nov 4 during the checkout outage. Review rounds fall from 5,300 to about 640 over the weekend, return to 4,810 on Nov 4 and are 2,450 on Nov 5.
Dreaming's estimates, not measurements. The weekend dip in review rounds is low volume, not a fix.

The false-red class, a red on a land that was actually good, went from the top of the list to zero, with one spike during the checkout outage. That’s the class and not every red: post-land checks still go red, and 72 of 649 lands checked red on Nov 3, some only because main was already red.

Review rounds are the open one. The dip to about 640 on the chart was the weekend, when we landed about half as many Changes. Measured directly, the share of Changes past two rounds was 33% on Oct 29 and 25% on Nov 4, moving between 20% and 33% all week. That’s flat. The analysis behind one of dreaming’s cards showed that 49% of rounds past the cap came right after a clean round, so the next step is to stop rereading work that already passed. A change that skips the second read on no-op rebases landed Nov 5, and it’s too early to see in these numbers.

GitHub’s remaining role

The last move finished this week: the AWS Terraform stacks went to Orochi on Nov 4, and the old Actions copies of the promote pipeline are gone. Releases, hotfixes, promotion and Terraform all run on Orochi now.

GitHub decides nothing. It mirrors main and release tags, and it builds one Mac binary, because the Orochi daemon needs a macOS runner.

The numbers

Oct 29 Oct 30 to Nov 4
Median, push to main 11.9 min 18.4 min
p90, push to main 23.2 min 76.1 min
Median queue wait 2 s 1.8 to 19.8 min by day
Busiest day 726 lands 785 lands
Busiest hour 65 lands 77 lands

For scale, GitHub’s merge queue took a median of 4.4 hours from PR opened to merged, with a 28.7-hour p90, on 2,630 PRs in a week. Orochi landed 3,429 Changes in those six days, and on its slowest day it was still almost 7× faster.

What’s next

  1. Closing dreaming’s loop. Work has started on re-measuring merged cards, normalizing its estimates by land, grouping alerts, and getting its first real Change through review.
  2. Review rounds. They’re at the top of dreaming’s list and haven’t moved yet, and the replay gives us a score to tune against.
  3. The pre-screen. At 9.2 of a 17.3-minute median, it’s now the biggest phase of a land.
  4. Merge rules and Argus’s gate out of shadow, once their numbers earn it.