corpora-builder
ai-labs/corpora-builder/context-v/
32 files here
specs/15
Header Chrome
The workspace becomes a switcher, not a label. The mode toggle shows one icon and says what the next click gets you.
Ranked Search — Pagefind over the manifest
Substring matching cannot rank, stem, or take two words in any order. Pagefind can. It indexes the manifest, and never replaces it.
Search Index — the manifest that stops search reading the corpus
A search opens every file: 845 reads, up to 5.8 seconds. A manifest in the corpus makes it one, and closes the gap FOCUS-07 named.
Strategy Focus
A strategy chip narrows the list and compounds with search. The corpus count rides alongside, so a subset always says what it is a subset of.
Binary Ingest and the bin/ Store — one copy, optimized, fetched on demand
78 files are 90.5% of the corpus bytes. Compress once, store once by hash, fetch when asked, delete locally without fear.
Corpus Change Feed — what changed in a client's corpus, and why
A change record any history engine can emit, and a feed that renders it for someone who will never read a diff.
Corpus Tree
A corpus is a folder the client can't see. The tree is built from keys alone, so showing 944 files costs zero reads.
Domain Navigation
112 folders in a dropdown is a list you scroll, not a list you use. Type any near-match; backspace walks back a segment at a time.
Browse Corpus — a screen for the work that already exists
The first surface: read-only over reach-edu's 892 existing files. Pulled out of Phase 7 because Tauri was never the cheapest first screen.
Capture From The Screen — the surface stops being a viewer
Paste a URL and watch it get filed. Writing is opt-in at the server (`--writable`) because the first corpus this screen saw was a client's.
Capture, Link-First — the first command that pays for itself
`corpora add <url>` fetches, names, dedupes, and files a link as a source — metadata-only by default, because bulk enrichment goes haywire.
Loop Harness — the ledger that lets an agent run unattended
Spec IDs bind to test functions, status is derived by running the suite, and a promise with no test is a build failure.
Source-File Model — the schema becomes code, and two silent failures become loud
The schema as a round-tripping dataclass — the stray `---` that hid 13 of ImmuneCo's 93 sources is now a hard error, not a silent one.
Storage Seam — one interface, three implementations, and the workspace that names the bucket
A CorpusStore over local, R2, and cache, proven by one conformance suite that runs against every backend with no test-side branching.
corpora-builder/specs
plans/3
didi.sh Auth and Multi-Org Corpora
Log in once, hold several organizations at once, and read one federated corpus across them. Three catches, and two specs that disagree.
Search Index, Then Pagefind
Search reads 845 files. Phase 1 writes a manifest so it reads one. Phase 2 puts Pagefind on top of that manifest, not in place of it.
corpora-builder/plans
blueprints/1
reminders/2
explorations/3
Corpora-Builder System Design
Corpus building has been rebuilt as a sub-feature three times across ai-labs. This exploration surveys what those iterations actually taught us, and sketches the system that deserves its own repo.
Design Front-Loading and the Fable Build Loop
If frontier models can now carry complex builds end-to-end, the leverage moves to the documents. Can we front-load design so hard that the whole system gets built in a TDD loop — and impose design-system discipline from commit one?
corpora-builder/explorations
issues/3
Need an Elegant Resolution to Source Pointers vs. Master
One source, many folders, is the design. But for 75 sources the body sits in an untriaged inbox file the scoped read never reaches.
24 Orphaned bin/ Objects From a Half-Migration
A test that checked the object existed but never the pointer. Ghostscript's nondeterminism turned a sequencing mistake into 24 untraceable objects.
corpora-builder/issues