Source Content Storage — SurrealDB-Primary, Local as a Per-User Toggle

Why Care?

Today the storage model is the inverse of what it should be. The markdown body lives only on the filesystem (CLIENTS_ROOT); SurrealDB is “the rebuildable index” (domains.ts:16) and holds no content. There’s no local toggle (the file write is unconditional) and the write order is local-first, then DB, non-atomic.

That shape is a fossil of an ambition never shipped: corpus files on a remote filesystem. We explored it ([[JuiceFS-Pinned-Path-Off-Local-Substrate]]), rejected a live mount, and landed on R2 via rclone for backup — but the content itself never left the filesystem, so local-first bought nothing but drift. The body is just a markdown string; SurrealDB stores that trivially.

Decision (operator, 2026-08-02): store the content in SurrealDB; make the local write an explicit per-workspace + per-user toggle.

The North Star this serves

Enable RAG/KAG for a whole team, by topic. A topic = a domain/corpus. Every collaborator should be able to retrieve over the corpus — some through the hosted app, some by syncing a topic’s content down to their own local folder for local retrieval. For that, the canonical content has to live somewhere every authed collaborator can reach: the DB, not one operator’s laptop.

The key structural win

The [[Syncthing-For-Collaborator-Access-To-The-Corpus]] exploration drew a hard line: Syncthing only solves the filesystem-corpus leg; the SurrealDB canonical layer (persons/orgs/observations) is a separate access problem needing didi.sh auth. Putting content in the DB erases that line — content access becomes canonical-layer access, one auth path (didi.sh) for both. Collaborators no longer need a filesystem mirror to read content; the mirror becomes an optional local-RAG convenience.

The decision, stated

1. SurrealDB is the primary content store. The fetched body is a content field on the canonical sources row (client-agnostic — same URL, same body). Analyst Extracts are stored per source_usages (each client’s own curation of that source). Reads serve both from the DB.
2. Local filesystem write is an explicit toggle, off by default, scoped per-workspace AND per-user — each collaborator names their own local FS location for a topic-scoped sync of the content, for their local RAG. (This is the export/sync leg the Syncthing exploration was circling; content-in-DB makes it optional, not load-bearing.)
3. Write order is DB-first. DB success = “saved.” When a local destination is enabled, that write must also succeed for the operation to report done — a failure surfaces explicitly (“saved to db, local export failed”), never a silent half-write.
4. corpus_path becomes optional — populated only where a local mirror exists. R2 backup (rclone) and any Syncthing FS mirror operate on that optional local tier, unchanged.

Resolved forks (operator, 2026-08-02)

Still open (need a call before/at build)

• D — Migration. Bodies + Extracts currently live only in files, across three diverging copies: Michael’s laptop (clients/*/corpus), the Railway volume (/data/clients), and git (committed corpus). Backfill has to read existing corpus .md, split body vs Extracts, write into the new columns — and reconcile which of the three copies is source-of-truth per record.
• Collaboration transport. With content in the DB, is the per-user local sync a DB→FS export (new, simple, one-directional) or does it reuse the Syncthing Receive-Only mesh from the exploration? Leaning DB→FS export now that content is canonical in the DB; Syncthing/R2 remain for FS-tier mirror/backup, not the primary access path.
• Blob size in Surreal. Bodies are ~tens of KB (the Springer one was ~83KB); fine as a field, but confirm query/storage patterns don’t regress on large rows (drives the “project away on list reads” note in Fork A).

Write contract (target)

fetch/save(source):
  1. resolve content + metadata (Jina, two-profile parser)
  2. WRITE DB:  sources.content (body) + source_usages (Extracts, status='fetched', bib)
       └─ fail → abort, report "save failed (db)"; nothing half-committed
  3. FOR each enabled local destination (per-workspace and/or per-user):
       WRITE file (dest/…/<slug>.md), preserving Extracts
         └─ fail → report "saved to db, local export failed at <dest>" (explicit)
  4. return { ok, source_uuid, stored: ['db', ...enabledLocalDests] }

Reads serve body + Extracts from the DB; local files are mirrors, and R2 backs up the local tier via rclone as already decided.

Cross-references

• [[Syncthing-For-Collaborator-Access-To-The-Corpus]] — the FS-mirror leg; this spec makes it optional.
• [[Per-Client-Privacy-and-the-Path-Off-Local]] — isolation posture that shapes per-user/per-client scoping.
• [[Best-Way-to-RAG-Over-the-Corpus]] — the retrieval design this storage model feeds.
• [[JuiceFS-Pinned-Path-Off-Local-Substrate]] — the R2/rclone backup decision, not re-litigated here.
• [[Fetch-Full-Content-Clobbers-Operator-Metadata]] · [[Jina-Metadata-Parser-Is-Blog-Only-Needs-Two-Profiles-And-Routing]] — the fetch/parse steps feeding step 1.
• [[Corpora-Curator-Entry-Point-for-Augment-It]] — the surface.