Each task is the subject line of a real commit from one of 9 open-source repositories (JavaScript, TypeScript, Vue, Python, Go, Rust, Java). The ground truth is the set of source files that commit changed, evaluated at its parent. Exact js-tiktoken counts at a 12,000-token pack budget.
Naive is every text file matching a task keyword, read in full. Savings grow with repository size.
fastify/fastify-example-todo
fastapi/full-stack-fastapi-template
pallets/flask
spf13/cobra
encode/httpx
gin-gonic/gin
google/gson
BurntSushi/ripgrep
hoppscotch/hoppscotch
These figures replace the earlier 12.2× claim, which is retracted.
On the small repository a pack saves little: in 3 of its 9 tasks the pack was larger than the naive read.
Every matching file, read in full, holds the needed files 96% of the time against 87% for the pack, at many times the tokens. The 10 best-matching files hold them 76% of the time, at a median 7.5× more tokens than the pack.
On average 73% of the exact lines a commit edited are in the pack; it is lowest on repositories with very large files.
This measures whether the context contains what the task needs, not the quality of a model's answer.
50 tasks were used while changing the code and 50 were not. The held-out half scored 88% needed files and an 18.4× median saving.
Measured on the task pack built by tokencap ask "<task>" and the Companion bridge, at a 12,000-token budget. The tokencap make capsule is not measured here.
From a TokenCap checkout. Method, task list and raw per-task results: benchmarks/fair · Honest Numbers
node benchmarks/fair/fetch-repos.js <dir>node benchmarks/fair/run.js --repos <dir> --set all{
"profile": "balanced",
"naiveTokens": 146702,
"naiveCost": 0.3668,
"tokencapTokens": 49608,
"tokencapCost": 0.1240,
"savedTokens": 97094,
"savedCost": 0.2428,
"reductionRatio": "3.0x"
}