Augment Transformations Not Reliably Persisting

Audit findings — what the row-store actually contains

Read the backup rows.json end-to-end. The store holds 3 record sets:

231 = 39 (Reach DC RSVP) + 96 (v4) + 96 (v5). That’s the row- multiplication-across-versions pattern named in [[../reminders/Record-Count-Stays-Stable-Across-Versions]]. The user’s dataset is 96 rows; the store keeps a separate copy per version.

v3 is gone, but its data lives on in v4

The chain points back to rs_promoted_mphy26mz_9lczxp — that’s v3. It is not in the store. Deleted at some prior point. The user’s hand-curation of homepage URLs happened on v3.

But — v4 was promoted from v3, and the promote merge carried the v3 field values onto v4’s rows. So v4 has the hand-edits even though v3 itself is gone.

Sampled three rows — every field made the trip

Spot-check of v4 vs v5 for Annie E. Casey, Arthur Blank Foundation, Britebound/American Student Assistance (ASA):

Plus v5 has the user’s TODAY work:

• official_updates_index_urls array (today’s Records Surface picks)
• legacy singular official_updates_index_url on some rows (from earlier in today’s session before I changed it to plural)

Full-set diff — v4 vs v5

Programmatic compare across all 96 shared identities:

shared identities (v4 ∩ v5):       96
same `url`:                         96
different `url`:                     0

Zero rows changed value between v4 and v5. Promote is preserving the url field across the full set, not just the sampled three.

What was actually wrong (and what wasn’t)

NOT actually wrong: data persistence through promote

The user’s stated belief — “v5 has none of the homepage URL fetching I did” — is inverted from reality. v5 has every URL the user curated on v3 → v4. The promote merge works. The fields propagate. Issue #1 in its original framing is not a bug.

What the user actually saw — a UI visibility problem

Working hypothesis (cannot verify from outside the user’s head): the user looked at one of the augment surfaces and didn’t see the url column displayed prominently per row. The Pack Runner / Augment surface shows entity name + a has-url status chip but not the URL itself. The Records Surface shows the URL inline on each row. If the user spot-checked from a surface that doesn’t render the URL, they would observe “no URLs here” and conclude the data was lost.

Whatever the source of the misread: the data is intact. The investigation closed issue #1’s “data was lost” framing.

The Griffin Catalyst exemplar — the real persistence failure mode

User correction after the initial audit: “Just take Griffin Catalyst, it had no url.”

Full row contents for Griffin Catalyst (identical in v4 and v5):

Reconstructing the history:

1. The source CSV had no URL for Griffin Catalyst.
2. At some point a fan-out / search filled url with https://www.citadel.com — wrong; Citadel is the parent firm. Griffin Catalyst lives at griffincatalyst.org.
3. The user spotted that and opened the by-record card on Response Reviewer (or whichever surface was offering URL editing).
4. The user added the correct URL: https://www.griffincatalyst.org/....
5. That edit landed in helpful_links (additive — a side list of manually-added links) and did NOT update the url column.
6. From the user’s perspective: I fixed the URL.
7. From the system’s perspective: the user added a helpful link alongside the existing wrong URL.

This is the real persistence failure mode. Not “promote loses data.” It’s:

The user’s intent to correct the primary URL field didn’t update the field the system reads as the primary URL.

When the user opens Records Surface today and clicks Firecrawl scan on Griffin Catalyst’s row, the connector fires against citadel.com — Citadel’s homepage — not against griffincatalyst.org. So every candidate it returns is a Citadel page, not a Griffin Catalyst page. The user can’t use this row at all until the url is fixed in the column the connector reads from.

How widespread is this?

Per row, possible patterns:

• A. url is correct, helpful_links empty or supplementary. → Healthy.
• B. url is correct, helpful_links carries SUB-pages (/grants, /priorities/education, etc.). → Healthy — helpful_links is doing what it was designed for.
• C. url is wrong (fan-out got it wrong), helpful_links carries the right homepage. → The Griffin Catalyst case. The user’s hand-curation didn’t replace the bad value.
• D. url is 'unknown' or empty, helpful_links carries the right homepage. → Same as C but with a sentinel instead of a wrong URL.

Full v5 sweep — all 96 rows:

~23 of 96 rows (24% of the dataset) have the persistence bug. The user’s hand-curation is preserved in the data but in the wrong column.

Clear-cut D-pattern recoveries (user added the correct homepage, the system kept the bad/empty url):

• Howard Schultz Foundation → https://schultzfamilyfoundation.org/
• Lumina → https://www.luminafoundation.org/
• McGovern → https://www.mcgovern.org/
• Tepper Family Foundation → https://tepperfoundation.org/
• The DeLaski Family Foundation → https://www.delaskifamilyfoundation.org/
• Laura & Gary Lauder Family Foundation → https://www.lauderfamilyfund.org/
• Mandelblatt Foundation → (a college board scholarship link, debatable)
• Pyramid Peak → (LinkedIn profile — debatable)
• Toolbox Family Fund (Joshua Biber) → https://www.onegoal.org/

C-pattern Griffin-shaped (wrong url, right helpful_link):

• Griffin Catalyst: url = citadel.com (wrong), helpful = griffincatalyst.org (right)
• Pinterest: url = pinterest.com (the consumer site, wrong target), helpful = newsroom.pinterest.com/impact (the foundation context, right)
• General Motors: url = gm.com (corporate, ambiguous), helpful = gm.com/impact/corporate-giving (right for this dataset’s purpose)

Root cause

Either:

• The URL-editing affordance the user used wrote to the wrong column. If the by-record card’s inline URL edit calls row.helpful_links.add instead of row.update({ url: ... }), every “fix the URL” edit lands as an additive helpful link, not as a column replacement.
• There was no URL-editing affordance — only an add-helpful-link affordance. If the user was offered “add a link” but not “fix the URL,” the user used what was available and the helpful_links capture is the best the system could do.

Either is recoverable. Look at the by-record card’s actual code to see what inline URL edit actually fires.

Remediation candidates

1. Promote helpful_links to url when the user marks one as canonical. Add a ”✓ make this the canonical URL” button next to each helpful_link in the by-record card; clicking it calls row.update({ url: <that link> }) and clears the helpful_links entry (or keeps it tagged as “now-canonical”).
2. Auto-promote on first add. When a row’s url is 'unknown' or empty AND the user adds a helpful_link, treat that add as a url replacement instead of an additive entry. Surface a small “this became the row’s URL” confirmation.
3. Display merged URL column. Records Surface (and any other surface that surfaces “this row’s URL”) reads url first, then falls back to helpful_links[0].url. The user sees the right URL even if it’s stored in the wrong column. Plus a “promote helpful link to URL” affordance to write through.
4. One-time data fixup. Run a script over v5 that detects pattern C and D (url wrong/unknown + helpful_links non-empty) and promotes the helpful_link to url. With user confirmation, per row, before writing.

Lean: (3) for immediate read-side correctness across all surfaces. (1) or (2) for write-side once the user signals which pattern feels right. (4) as a one-time post-fix to clean up the C-shaped rows.

What is actually wrong (still)

Two real problems remain, both reframed from the original draft:

#2 still real: no save / promote affordance on the Records Surface

Unchanged from the original framing. The user has 98 URLs in 45 rows captured today on official_updates_index_urls. They are in the v5 row-store. The backup confirms it. But:

• No “promote to v6” button exists on the Records Surface.
• No way to checkpoint today’s curation as a named milestone.
• No visible confirmation in the UI that today’s picks are persisted.

The backend capability record_set.promote exists and (per audit above) works correctly. Wiring it into the Records Surface is small.

#3 (NEW): the store multiplies rows across versions

The architectural concern from [[../reminders/Record-Count-Stays-Stable-Across-Versions]] is real and is the reason the 231 number exists. Each promote creates 96 new row records, even though the user conceptually has one set of 96 records with multiple versions of field values.

Implications:

• Backups balloon: 3 versions × 96 rows = 288 in storage instead of 96 rows × 3 version layers.
• Promote is slow at scale (every value gets copied row-by-row).
• The user has no way to reason about “this row across versions” because the row_id is different per version.
• Confidence in the data model erodes when totals don’t match expectations (the moment that triggered this whole investigation).

This is a row-store architecture change, not a Records Surface UI change. Lives in services/row-store/src/store.ts. Separate follow-up scope.

What the user actually saw — surface what’s there

Even though the data isn’t lost, the user’s experience IS the bug: they shouldn’t have to wonder where their work is. Surfaces that display rows should make the URL field visible enough that “is this populated?” is answerable at a glance. The Records Surface does this already (URL shown inline per row); Pack Runner / Augment does not.

Backup status — what’s safe right now

Before any further changes:

.backups/2026-06-02_records-surface-acceptances/
├── rows.json          544 KB — 45 rows with accepted URLs, 98 URLs total
├── responses.json     1.6 MB
├── prompts.json       650 B
└── sessions.json      1.4 KB

Verified rows.json parses + the official_updates_index_urls arrays are populated on the expected rows (Arthur Blank, Ballmer, Benwood, Gates, Bridgespan, Britebound, Carnegie, Schusterman, …). If the docker volume is lost or corrupted, this file is the source of truth for the session’s hand-curation work.

Issue 2 — Immediate blocker: no save / promote affordance on the Records Surface

The user spent ~45+ minutes hand-curating 98 OfficialUpdate index URLs across 45 records on the new Records Surface (apps/records-surface/). Each pick wrote to the row via workspace.invoke('row.update', { row_id, fields: { official_updates_index_urls: [...] } }), which the row-store auto-persists to disk inside its docker volume.

There is no UI affordance to:

• Save / commit / freeze this work as a milestone.
• Promote the current record set to a new canonical version (the v3→v4→v5 pattern the user previously relied on).
• Export the curated state to a CSV / XLSX for safekeeping.
• See a per-row indicator that “your acceptance is persisted.”

The data IS persisted on disk inside the row-store volume — but the user has no in-app confidence of that, no way to checkpoint the work as a named milestone, and no way to “lock” the work against further edits or accidental clobbering by a later run.

What previously existed (per user recall)

The user described a prior flow:

“We had an original version, where we set about to find the correct homepage url for each organization and it also allowed me to edit the name. I did that, transformed 15-20 records as I recall. Then we saved that as the next version, v4.”

“With v4, we created a new column and that column was filled with a JSON object which contained all the valid ‘socials’ or social links across Facebook, Linkedin, Wikipedia, X, BlueSky, Youtube … I validated every entry. Then I saved it to a v5.”

The capability exists in code today:

• Workspace capability: record_set.promote → NATS subject record_set.promote.requested (services/workspace/src/capabilities.ts).
• Row-store handler: promoteRecordSet({ source_record_set_id, name }) in services/row-store/src/store.ts (~line 447). Walks derivations, unions schemas (“ALL DATA CONTINUES” comment), produces a new canonical record set.

So the BACKEND for promote-to-next-version is built. The Records Surface UI does not surface it. When the user reaches the end of a curation pass, there is no button to click.

Done-when for issue 2

1. The Records Surface shows the user a count of accepted URLs across all rows (“45 rows accepted, 98 URLs total”).
2. A “Promote to next version” button exists. Clicking it invokes record_set.promote with the current active record set as source.
3. The new record set inherits ALL row data — including the official_updates_index_urls arrays and any other field edits the user made during this pass.
4. After promote, the Records Surface auto-switches to the new record set (the user doesn’t have to navigate manually).
5. The promote action is undoable for a session (or at least restorable) — if it goes wrong, the user can recover the prior record set.
6. The pre-promote state is captured to a host-filesystem backup automatically (the same pattern the user just had to do manually).

Issue 1 — Systemic: hand-curated field edits from v3→v4 are not present in v5

This is the deeper problem and the reason issue 2 became urgent.

The user describes a previous augment-it session where they:

1. Loaded an initial record set (let’s call it v3 for clarity).
2. Hand-curated the homepage URL on ~15-20 organizations directly in the Record Collector / by-record card surface.
3. Hand-curated entity-name corrections on the same rows.
4. Saved that work as a “next version” — call it v4.
5. In v4, fan-out fired the socials-bundle (Profile Builder), produced N candidate socials per row, the user triaged each one, accepted into row.socials.
6. Saved THAT as v5.

The user’s current observation:

“As of now, I’ve now been working from v5, which for no reason I can understand, has none of the homepage url fetching I did.”

If true, this is a data integrity failure at a layer that’s load- bearing for the whole product. The Augment-It thesis is:

Augment a record set → promote → your augmentations carry forward.

If field-level edits made in v3 don’t appear in v5, the thesis is broken. The user wastes work; the user can’t trust the product; the user has to redo curation passes they already finished.

Possible root causes — hypotheses to investigate

A. The promote merge is column-selective, not field-comprehensive. promoteRecordSet does a schema union, but the actual VALUES carried forward may be limited to particular field name patterns (e.g. row.socials because the socials-bundle’s accept handler explicitly writes there, but NOT free-form fields like “Homepage” or “Website” the user typed by hand). Worth checking what the merge logic actually copies vs drops per row.

B. The hand-curated edits never landed in the source set in the first place. If the by-record card’s inline URL edit fires a capability that writes only to responses (for the post-flight triage) and NOT to row.fields, then the v3 row data was never mutated — only the response record was. Promote would naturally lose that data because it was never on the row to begin with.

C. The promote was done from the wrong source. If v5 was actually promoted directly from v3 (skipping v4), the v4 edits wouldn’t appear. This is a UX trap rather than a data bug.

D. Field-name drift. The user’s hand-edits may have landed in a field whose name differs from the field the promote-merge logic reads. E.g. user edits “Website URL” but the merge only knows about “Website” or “url” or “homepage”. Schema union catches the field’s existence but the column might be empty for rows that came from a different prior generation.

E. Promote is non-idempotent and a re-promote overwrote. If the user fired promote on v4 a second time after later edits, and the re-promote happened against a stale base, the result could land with intermediate work missing.

What to investigate

1. Audit the row-store contents — list all record sets, their promoted_from, their schemas, and sample rows. For the user’s specific case: find v3 (where the homepage edits should be), v4 (where the socials work happened), v5 (the current active). Diff the row data for one specific organization the user remembers editing (e.g. one of the first 15-20). Where IS the curated homepage URL — v3 only? v4 only? Nowhere?
2. Trace the by-record card’s URL-edit path — what capability does the inline URL edit fire? Does it call row.update (mutates the row) or response.set_structured (mutates the response)? If only the latter, the field-level edit never lands on the row.
3. Read promoteRecordSet end-to-end — what fields make the trip? The “ALL DATA CONTINUES” comment is aspirational; verify against the actual code.
4. Identity-with-cursor logic — promote walks derivations and matches rows by schema.fields[0] (the identity column). If the identity column NAME differs between v3 and v4 (e.g. v3 has “Prospect / Organization”, v4 has the same but normalized), the matching could fail silently.

Done-when for issue 1

1. We can name, for a specific record the user remembers editing, exactly which record set version holds the edit and which does not. (Audit the backup data first; we have the snapshots.)
2. The root cause of the loss is identified among A-E above (or something not yet enumerated).
3. A code or UX fix is landed that GUARANTEES augmentations survive promotion — either by changing the merge logic, by changing where the by-record edits write, or by changing the promote UX so the user can’t accidentally promote from a stale source.
4. A regression test exists: load v3 → make an edit → promote → assert the edit appears in v4. Run on every promote-flow change.
5. The 15-20 records the user edited in the lost v3→v4 pass are recovered. If they exist in an earlier record set in the row-store, reconstruct them. If they’re truly gone, restore from .backups/... or accept the loss and re-do.

What I am NOT doing right now (deliberately)

Per the user’s instruction “if we have to restitch it all together by hand that’s fine, it’s only one record set of 96 records” — I am NOT making any destructive changes to the row-store. The backup is in place. The current official_updates_index_urls data is safe.

Until the user signals to proceed:

• No promote attempted.
• No row-store volume modification.
• No record set archive / delete.
• No schema migrations.

The investigation in §“What to investigate” above is a read-only exercise — it can proceed without risk to the user’s curation work.

What needs to happen next

In priority order:

1. Confirm backup integrity. The user verifies the backup file contains their work by spot-checking a few records’ URL arrays. Done above; this is a checkpoint, not a TODO.
2. Audit the row-store backup to determine which record set holds what. Specifically, look for the 15-20 records the user remembers curating in the v3→v4 homepage pass — find which set they’re in, and what fields are populated.
3. Reproduce one promote in isolation — pick a small test record set (or scratch one off the user’s), make a known edit, promote, verify the edit makes the trip. If it doesn’t, the merge logic is the bug. If it does, the user’s specific v3→v4 promote went wrong for a different reason (e.g. source confusion).
4. Decide on the immediate save affordance for the Records Surface. Two options: a. Add a manual “Snapshot to backup” button that hits the same docker exec ... cat extraction this session did, but from the UI side. Quick. Belt-and-suspenders against any future data loss while we investigate the promote bug. b. Wire the existing record_set.promote capability into the Records Surface as a “Save as next version” button. Bigger scope; can’t ship until promote is proven to actually carry forward the new column. Lean: (a) first, so the user has zero-risk save while we investigate (b)‘s reliability.
5. Restitch v3 → v5 by hand if needed. The user has explicitly said this is acceptable for the 96-row set. We can build a small merge script that reads v3’s hand-curated homepage edits and applies them to v5’s rows by identity match. One-time fixup.

Related

• [[../specs/Flow-for-Bundles-Packs]] — the new Records Surface spec; needs an addendum for save/promote UX once we know the right answer
• services/row-store/src/store.ts §promoteRecordSet — the existing promote implementation; primary code reference for issue 1
• services/workspace/src/capabilities.ts record_set.promote — the capability mapping that wires browser → NATS for promote
• services/row-store/src/handlers.ts ~line 249 — the NATS handler that triggers promote on incoming requests
• .backups/2026-06-02_records-surface-acceptances/ — the snapshot of the user’s current session work; do not delete

Status

Open. Backup made. No destructive actions taken. Awaiting user direction on whether to investigate first or build the manual snapshot button first.