Query-adaptive context assembly for code retrieval. Tobler's first law for your context window: near things are more related than distant things, so let the score distribution decide how much context a query deserves, instead of a fixed top-5.
If you drive an AI coding agent, tobler feeds it the right code in fewer tokens, and on large repos it finds the target chunk where a fixed top-5 misses it two-thirds of the time. It sits on top of semble retrieval and ships as a library and a FastMCP server.
Requirements: Python ≥ 3.12 and an MCP client (e.g. Claude Code). semble is pulled in automatically (pinned). Published on PyPI as tobler-search (the import name stays tobler).
- Sharp query ("what does BACKOFF_BASE_MS do") → N_eff = 3, a few full bodies + a map, ε honest.
- Diffuse query ("how does real-time collaboration work") → N_eff = 550, wide map of signatures and stubs.
- Same token budget both times. Fixed top-k is wrong in both directions on the same repo.
τ was calibrated once on two repos and never touched again; the first three rows are repos it never saw:
| repo | semble top-5 | tobler B=1000 | tobler B=2000 |
|---|---|---|---|
| numpy (16,465 chunks) | 32% | 88% | 92% |
| ripgrep (Rust) | 48% | 92% | 95% |
| zod (TypeScript) | 80% | 95% | 98% |
| 8 smaller repos (247 queries) | 76% | 99% | 100% |
| calibration repos (80 q, not held out) | 94% | 98% | 99% |
Fixed top-5 degrades hard as repos grow: on numpy it finds the right chunk a third of the time. Tobler holds ≥88% everywhere with zero per-repo tuning.
| repo | semble top-5 | tobler B=1000 | tobler B=2000 |
|---|---|---|---|
| numpy | 1463 | 995 | 1940 |
| ripgrep | 1544 | 993 | 1944 |
| zod | 1453 | 981 | 1940 |
| 8 smaller repos | 1395 | 992 | 1966 |
Semble's spend is uncontrolled (whatever five chunks happen to cost, observed range ~1,000–2,800/query); tobler's budget is a hard ceiling that was never exceeded on any of ~370 benchmark queries. Tokens per correct answer (mean tokens ÷ hit rate): on numpy semble costs ~4,570 tokens per hit vs tobler's ~1,130 at B=1000, so it is 4× more token-efficient; ~3× on ripgrep, ~1.8× on small repos. When the budget genuinely can't cover the relevant code, ε reports it instead of silently truncating.
- Center/standardize raw cosine scores over all N chunks:
s̃ = (s − s̄)/σ, which removes the "everything resembles everything" anisotropy baseline. - Softmax at temperature τ:
ŵᵢ = softmax(s̃/τ). - Effective count
N_eff = 1/Σŵᵢ²: how many chunks genuinely matter for this query. - Nucleus cut: smallest set S with mass ≥ p (default 0.9), so k adapts per query.
- Residual tail
ε = 1 − Σ_S ŵᵢ, returned to the caller so the agent knows when it's reasoning inside a truncation. - Budget allocation
bᵢ = B·ŵᵢ/Σ_S ŵⱼ, mapped to level-of-detail tiers (full body → signature+docstring → path+symbol), with greedy repair rolling unspent budget down by weight. - Session smoothing:
m_t = λ_t·m_{t−1} + (1−λ_t)·s̃_twithλ_t = λ_max·max(0, corr(s̃_t, m_{t−1})). An anaphoric follow-up ("what are the arguments to that") gets pulled toward the session's focus; a topic pivot has corr ≈ 0, so memory vanishes with no reset heuristic. The gate experiment calibrated λ_max* = 1.0. The knob removed itself: the correlation is the blend. Follow-up hit rate 9% → 46%, pivots unharmed at 96% (seeexperiments/RESULT.mdAddendum 3).
τ is the one calibrated free parameter (found by maximizing var(log N_eff) across real queries); p, k_min, and the budget B are fixed defaults you can override. On two real repos, τ* = 1.28 on the hybrid channel and N_eff spans two orders of magnitude (p90/p10 = 215×); see experiments/RESULT.md.
Scores come from a hybrid channel (z-scored model2vec cosine blended with z-scored raw BM25) because static embedders can't see rare identifiers (mulberry32 ranks ~200th semantically, 3rd lexically). A k_min=5 floor on the nucleus cut guarantees coverage never drops below the old fixed top-5. Warm-index latency: numpy 54 ms/query, ripgrep 15 ms, zod 14 ms.
Install:
pip install tobler-search # or: uv pip install tobler-searchRegister the MCP server with Claude Code (uvx needs no separate install):
claude mcp add --scope user tobler -- uvx --from tobler-search tobler-mcpTool: search(query, repo, budget=2000, content="code", fresh=False) → markdown with an honesty header (N_eff=… k=… ε=… tokens=…/…, plus mem=… when session memory shaped the result) followed by tiered context blocks. Related follow-up queries are smoothed toward the session's focus; pass fresh=true to drop the memory for a query.
Run the server directly (stdio) with tobler-mcp.
pip install tobler-search puts the tobler command on your PATH. For shells, scripts, or agents without MCP access, the same search runs from the command line:
tobler index ./repo # warm the semble index first (optional)
tobler search "authentication flow" ./repo # print the adaptive context
tobler search "save model to disk" ./repo --budget 3000
tobler search "deployment guide" ./repo --content docsOutput is the same honesty header + tiered blocks the MCP tool returns. (Session smoothing is MCP-only; the CLI is stateless per call.)
skills/tobler-search/ ships an agent skill that teaches a coding agent when and how to reach for tobler (MCP first, CLI fallback) and how to read the N_eff / k / ε / mem header. Point your agent's skill loader at it, or copy it into your skills directory.
From a source checkout (git clone https://github.com/datacompose/tobler):
uv run pytest # 322 tests
uv run experiments/neff_sweep.py <repo1> <repo2> # the τ calibration
uv run experiments/benchmark.py <repo1> [...] # hit rate + tokens vs semble
uv run experiments/session_sweep.py <repo1> [...] # the session-smoothing gatesrc/tobler/core.py: the pipeline math, pure numpysrc/tobler/bridge.py: the only module touching semble private attrs (pinnedsemble==0.3.4)src/tobler/render.py: tier rendering + token cost estimatessrc/tobler/assemble.py: steps 1–7 composed;Sessionholds the smoothing memorysrc/tobler/mcp.py: FastMCP server, one session stream per (repo, content)src/tobler/cli.py: thetoblercommand (search+index)skills/tobler-search/: shippable agent skill wrapping the MCP tool + CLIexperiments/: the validation gates (Phase 0, hybrid benchmark, session smoothing: all passed)
MIT. See LICENSE.
Built in a night, but the numbers aren't hand-wavy: τ calibrated once on held-out repos, every claim gated by an experiment in experiments/. Kick the tires and file issues.