Tutorials ↗

Anthropic Scion explained: source control architecture and an OpenScion walkthrough

Follow a CI job that needs one directory to understand Anthropic Scion's data model and object caches, then run a sparse clone and commit with a pinned OpenScion version.

Scion retrieves the branch of a source tree that a task needs

A CI job only needs to run tests in services/api/. An AI agent only needs to change one file in that directory. Both need a small slice of the current code, and their working directories may be discarded as soon as the task ends. If every start involves fetching history, preparing an index, and processing objects across the repository, source control consumes time before the actual work begins.

On October 13, 2026, Anthropic published the design of its internal source control system, Scion. It moves history queries, commits, and merges to the server, records work as file changes, and lets clients fetch trees and file objects on demand. The system now serves Anthropic’s internal monorepo. The public OpenScion project, maintained by tinyhumansai, is an early implementation following that design. It offers code to inspect and small examples to run; it is not a release of Anthropic’s Scion source code.

The useful starting point is the small task: if a client only needs services/api/, what must the server do so the client can avoid preparing a complete repository?

The workload explains the design

In its engineering article, Anthropic reports that average daily merged changes grew from about 2,500 in May 2026 to more than 30,000 in September. The main branch’s history was almost 100 GB, and a compressed working copy exceeded 25 GB. CI alone performed roughly 25 million code checkouts per day, excluding developer environments, agent sandboxes, and research jobs.

The article estimates that fetching those checkouts naively would produce over 600 PB of traffic per day. That is a conditional estimate without the relevant optimizations. In practice, Anthropic had already used shallow clones, sparse checkouts, promisor remotes, internal mirrors, and repository snapshots held in object storage. At its peak, it operated several hundred Git mirrors.

Those techniques reduced the cost of obtaining code but introduced continuing maintenance work. Snapshots quickly fell behind main, and operations such as diff and blame still had to handle missing history and objects correctly in partial clones. As the merge rate increased, the mirrors’ disk, memory, and synchronization costs continued to grow.

A complete Git repository supports offline history, diffs, commits, and branch operations. Git also supports partial clones and fetching objects on demand. Scion goes further in moving computation to the server because of Anthropic’s particular conditions: most clients are agents and jobs that remain online, the repository is enormous, there are many clients, and working directories are short-lived.

Those conditions may not apply to a small team’s ordinary repository. Moving more functionality to a server adds online dependencies, operational work, and migration costs. Anthropic itself says Scion is unsuitable as a general replacement for Git.

Changesets record file changes; the server derives snapshots

Suppose an agent edits services/api/app.py and deletes services/api/legacy.py. A Scion changeset records its parent changeset, author, message, and the changes to those two paths. The updated file contents live in a blob, referenced by content address; the deletion records the removed path.

These are file-level operations. Even when only three lines change, the changeset points to the updated file contents.

Anthropic describes a changeset as an entry in a write-ahead log, storing file changes relative to its first parent. A Git commit, by contrast, logically points to a tree that represents the files at that revision. Git reuses unchanged files and objects, so a snapshot does not mean copying all repository bytes on every commit. Delta compression within packfiles is a separate physical storage concern. The official Git book explains this distinction.

Scion still needs a complete directory view for each revision. Otherwise, a client cannot discover which files are in services/api/. The difference is that the root tree is derived by the server from changesets, rather than logically being part of the changeset itself.

For the example above, the server can start from the parent’s root tree, rebuild the directory path through services/ and api/, and reuse unaffected subtrees. Two questions therefore have separate representations:

  • The changeset describes what this piece of work changed.
  • The derived tree describes which files exist at this revision.

The first parent also establishes a choice about history. A merge changeset can have multiple parents, but its file changes are relative to the first. Anthropic explains that in its trunk-based monorepo workflow, everyday history and blame primarily follow main. Projects with extensive branching should assess whether that history view meets their needs.

A changeset records a file change; the server derives a root tree and reuses unchanged subtrees

The changeset records what changed; the tree describes the files at that revision. Simplified illustration.

Make history, diff, and merging into services

Recording changes alone does not make every query fast. To answer who last edited a line, the server needs a blame structure. To find the changes that affected a path, it needs path history. Fast queries about relationships between commits require graph indexes.

Scion groups these structures under derived data. Each type is recursively computed from the corresponding data of a changeset’s parents. It can be calculated eagerly or when needed. Keeping those indexes on the server lets their implementation improve without requiring clients to understand the internals or upgrade a repository format.

For CI, this changes where a task begins. A job deciding which build targets a change affects can request the diff against its merge base directly from the server. A job analyzing commit ancestry can also call an API. Only tasks that actually execute, compile, or read code require a working copy.

Writes use the server too. A client supplies changes and content objects; the server creates changesets and performs rebases or merges. Scion has no separate local-commit-then-push stage for sharing commits.

Creating a piece of work and updating main remain separate operations. Creating a changeset does not move a ref. Updating a ref uses compare-and-swap on a single Spanner record: the request supplies the old value it observed, and the replacement succeeds only if that value still matches. Concurrent writers can save their changesets without sharing a global commit lock; movement of main still has an explicit consistency check.

This separation reduces globally mutable state, but review, landing, and testing must work with it. A changeset’s existence does not establish that it has been reviewed or landed on main.

Fetch objects and share caches across jobs

Return to services/api/. The client obtains the target revision’s root tree, requests trees along the path through services and api, and then fetches the blobs in that directory. It need not traverse web/ or download all the history.

Scion’s protocol revolves around the objects a client wants. The client checks its local cache and requests missing trees or blobs by ID. Objects are immutable and content-addressed, so any cache tier can serve the same ID. A server need not compute a different giant packfile for each client or perform delta compression based on everything the client already possesses.

Anthropic’s deployment separates object reads into several tiers:

  1. Jobs on a compute node share objects through scion-daemon. A daemon starts from a periodic snapshot, reducing cold-start fetches.
  2. Node daemons request trees and blobs from regional scion-cache servers. These caches can stream objects directly from memory.
  3. On a cache miss, the request reaches the authoritative scion-server, which can read from Spanner or cloud object storage.

Scion object reads pass from client to node daemon, regional cache, and authoritative server

Only a cache miss continues downstream; the object returns along the request path.

These tiers support the same object RPCs. Anthropic nevertheless enforces the hierarchy so a regional cache failure cannot send every request directly to the authoritative servers. Content addressing enables cache reuse; failure isolation and traffic controls still require their own design.

This also separates two forms of checkout: a sparse checkout selects paths, while a virtual checkout delays materializing files. The first limits the task to services/api/. The second materializes files within that selection when they are accessed. The article describes files and objects as the units of materialization.

FUSE offers one way to connect such on-demand reads to a filesystem interface. OpenScion’s architecture roadmap lists FUSE-based virtual checkouts as a future phase. Anthropic describes Scion’s virtual checkout behavior without confirming that it uses FUSE. The OpenScion example below runs an ordinary sparse clone.

A server holding more data must also handle more access control. Scion supports repository and directory permissions, and Antique’s approval rules can distinguish comment changes from sensitive logic. A cache’s decision to return content must remain consistent with the permission model. A content hash establishes object identity; it does not grant permission to read it.

Where the time savings occur

Anthropic reports cache hit rates above 99% at each tier, a CPU and memory footprint 200 times smaller than its previous Git mirroring infrastructure, and a 25% CPU capacity gain for CI. The 200-fold figure describes infrastructure footprint, while 25% describes CI capacity. Individual operation times need a separate comparison.

The article’s performance chart gives median times for different environments. Several operations tied to preparing working directories are:

Environment and operationGitScion
Agent: create a new worktree7.1 minutes5.1 seconds
Agent: total source control time per change request34 minutes3.7 minutes
Checkout at the start of a compute job175 seconds22 seconds
CI checkout31 seconds3.5 seconds

These are measurements reported from Anthropic’s internal environment. Large gains occur in tasks that prepare substantial file and repository state, while other operations have different gains. In the same chart, a developer-environment commit falls from 1.4 seconds to 1.1 seconds, about 1.3x.

The engineering workflow changed as well. Anthropic moved away from a global merge queue lock and invested in post-merge and pre-deploy automated testing, accepting that concurrent changes can conflict. That requires dependable detection and repair; removing a lock does not automatically preserve every previous acceptance condition.

Human judgment remains part of the process. A seven-person team built and migrated to Scion, Antique, and several related systems in five months, with Claude participating extensively in implementation. One specific example in the article is a rewritten Git-to-Scion ingestion pipeline that used 20 times less memory and ran four times faster. The team still reviews every change request, focusing on intent, system constraints, and design tradeoffs, and steers agents away from excessive complexity introduced to handle hypothetical concerns.

Use OpenScion to turn a repository slice into a changeset

OpenScion makes some of these ideas available as public code. This example pins commit 00e92ab9c2a9985f39a64636b4b6c5c4bb583916, workspace version 0.2.1. It requires Rust 1.88 or newer and uses the GPL-3.0-only license.

This version implements an in-memory object store, a changeset log, ref compare-and-swap, HTTP endpoints, and opnscn import, commit, clone, and directory-limited clones. The repository does not survive a server restart, so temporary files are appropriate for exploring the mechanism.

We ran a local example with three files:

README.md
services/api/app.py
web/index.html

With Rust available, fetch and build the pinned version:

git clone https://github.com/tinyhumansai/openscion.git
cd openscion
git checkout 00e92ab9c2a9985f39a64636b4b6c5c4bb583916
cargo build --locked -p openscion -p openscion-server

Start the server in one terminal and keep it running. This example binds to the loopback interface on port 17878:

./target/debug/openscion-server --listen 127.0.0.1:17878

In a second terminal, still at the OpenScion repository root, create a temporary fixture, import it, and clone only the API directory:

SCION_DEMO=$(mktemp -d)
mkdir -p "$SCION_DEMO/tree/services/api" "$SCION_DEMO/tree/web"
printf '# Small monorepo\n' > "$SCION_DEMO/tree/README.md"
printf 'print("hello")\n' > "$SCION_DEMO/tree/services/api/app.py"
printf '<h1>Web</h1>\n' > "$SCION_DEMO/tree/web/index.html"

./target/debug/opnscn --server http://127.0.0.1:17878 \
  import "$SCION_DEMO/tree" --author demo -m 'initial tree'
./target/debug/opnscn --server http://127.0.0.1:17878 \
  clone "$SCION_DEMO/api-copy" --path services/api

Our run returned imported 3 files and cloned 1 files. In api-copy, the code remained at services/api/app.py; it was not moved to the root. Neither web/index.html nor README.md was materialized in that copy.

Edit one file in the sparse copy, commit it, and fetch a full copy to check the result:

printf 'print("hello, Scion")\n' \
  > "$SCION_DEMO/api-copy/services/api/app.py"
./target/debug/opnscn commit \
  --dir "$SCION_DEMO/api-copy" --author demo -m 'edit one API file'
./target/debug/opnscn --server http://127.0.0.1:17878 \
  clone "$SCION_DEMO/full-copy"

cat "$SCION_DEMO/full-copy/services/api/app.py"
cmp "$SCION_DEMO/tree/README.md" "$SCION_DEMO/full-copy/README.md"
cmp "$SCION_DEMO/tree/web/index.html" \
  "$SCION_DEMO/full-copy/web/index.html"

The commit command uses the server address stored in the working copy, so it does not need another --server argument. Our run returned committed 1 changes, followed by a full clone of three files. The API file contained the new text, and the other two files retained identical bytes.

Reading the resulting changeset also showed a single upsert in its changes array, for services/api/app.py. This confirms that committing from the sparse copy did not treat unrequested directories as deleted. Stop the server with Ctrl-C in its terminal when finished; its in-memory repository ends with the process.

Follow three parts of the implementation

The change payload definitions represent file changes as Upsert { path, kind, blob } or Delete { path }. A request also includes parents, author, message, and metadata. Updated files use blob IDs, allowing clients and servers to verify that returned content matches its ID.

The clone tree traversal fetches trees level by level. Before it reaches services/api, it retains only matching entries along that path. Once inside the selected directory, it enumerates its files. It then fetches blobs in batches, verifies their hashes, and writes them into the working copy. This directly implements requesting only the objects needed for the selection.

When the server creates a changeset, it validates parents and content, derives a root tree, and appends the record to the log. The tree derivation loads subtrees along changed paths while reusing other subtree IDs. It avoids scanning every subtree, although cost still depends on path depth, affected directory sizes, and file contents.

The CLI commit then attempts to move the ref with its old base as the expected value. If another writer has advanced main, the server reports a conflict. We separately sent a ref update using the old parent as expected and received HTTP 409 RefConflict. This version reports the conflict; it does not yet provide server-side rebase and automatic landing retries.

What remains between this example and the full system

OpenScion’s README describes the intended server, node daemon, and regional cache architecture. In the pinned version’s roadmap, history, diff, and ancestry queries, followed by rebase and landing, are next steps. Persistent storage, Git history ingestion, the daemon, network cache tiers, blame, virtual checkouts, and access control still require subsequent work.

OpenScion roadmap showing Done, Next, and Later

The OpenScion roadmap separates working components from the next planned stages.

The current HTTP server protects a single in-memory repository with Arc<RwLock<Repository>>. It separates changeset creation from ref updates, but it cannot yet be described as the distributed deployment without a global commit lock that Anthropic operates. This example listens only on the loopback interface. TLS and access control remain future work.

The repository’s benchmark comparison uses synthetic repositories of 1,000 and 10,000 files, each 4 KiB. The author explains that OpenScion keeps data in memory without persistence or compression, while Git writes to disk, compresses objects, and produces packs. The OpenScion backend also runs its client in-process, whereas Git commands incur process startup costs. Those timings illustrate how directory selection changes the work performed; they cannot establish a fair production ranking. Our small example likewise verifies file and changeset behavior without testing throughput.

For an existing Git workflow, the most accessible lesson is to reduce unnecessary work first: measure how long CI spends preparing repositories, determine which paths tasks actually read, move metadata-only tasks to query services, and reuse immutable objects across repeated jobs. If that still cannot meet the needs of an enormous monorepo, assess a different data model together with the review, search, permissions, and CI migrations it would require.

References

← Back to the list