← Corpus / content-farm / plan
Cite-Wide — dedupe inline repetition of identical hex citations
- Path
- plans/Cite-Wide-Dedupe-Inline-Repetition.md
Cite-Wide — dedupe inline repetition of identical hex citations
Symptom
After running Convert All Citations to Hex Format in cite-wide, the body sometimes ends up with consecutive identical hex markers separated only by whitespace:
… driving adoption of agentic workflows.[^jtas3k] [^jtas3k] [^jtas3k]
The repetition is upstream noise — usually from LLM output that emitted the same numeric citation 2–3 times in a row ([1] [1] [1] or [1][1][1]), or from the LLM-paste pipeline expanding a comma group like [1, 1, 1]. Hex conversion preserves the repetition because each token is a syntactically valid citation reference — but readers see it as a typo.
Existing dedupe is the wrong tool
Cite-wide already ships Dedupe Citations by URL (src/services/dedupeByUrlService.ts, modal at src/modals/DedupeByUrlModal.ts, command id dedupe-citations-by-url). That service operates on the reference-definition section: it finds two distinct hex IDs whose [^hex]: … lines point to the same URL, picks a canonical, and rewrites every occurrence of the duplicate hex to the canonical hex throughout the document.
That’s a different problem. URL-dedupe consolidates different hex IDs that happen to point to the same source. Inline-repetition dedupe collapses runs of the same hex ID that appear consecutively in body prose. Two separate operations; URL-dedupe should not be extended to handle this.
Proposed behavior
Add a small inline-repetition collapse pass that runs as the last step of convertAllCitations (in main.ts:499) and inside the LLM-paste conversion pipeline (src/services/llmCitationParserService.ts). The pass:
- Scans the document for runs of two-or-more occurrences of the same hex citation marker
[^xxxxxx]separated only by whitespace (spaces, tabs) — never across newlines. - Collapses each run to a single occurrence.
- Reports total collapsed runs in the existing post-conversion
Notice.
The regex shape (one match per run):
// /(\[\^([a-z0-9]+)\])(\s+\[\^\2\])+/g
const INLINE_REPEAT_RE = /(\[\^([a-z0-9]+)\])(\s+\[\^\2\])+/g;
const collapsed = content.replace(INLINE_REPEAT_RE, '$1');
The backreference \2 is what makes this safe: it only collapses when the same hex repeats. Two different hex IDs side-by-side ([^abc123] [^def456]) are legitimately distinct citations and must be preserved.
Where it fires
| Surface | File / function | When |
|---|---|---|
| Convert All Citations to Hex | main.ts:499 convertAllCitations | After citationService.convertCitation finishes the loop, before editor.setValue |
| LLM-paste pipeline | src/services/llmCitationParserService.ts | After numeric→hex token expansion, before the inserted text is returned |
| Selected-citation conversion | main.ts:209 convert-selected-citation-to-hex | NOT here — selection-scoped converts only one ref; out of scope |
| Dedupe by URL | src/services/dedupeByUrlService.ts | NOT here — keep dedupe-by-URL focused on its single concern |
The right home for the helper is a tiny export in src/services/citationService.ts (it already owns convertCitation), e.g. collapseInlineRepeats(content: string): { content: string; collapsed: number }. Both surfaces import it.
Edge cases — what NOT to dedupe
- Different hex IDs adjacent:
[^abc123] [^def456]→ unchanged. The backreference protects this. - Same hex separated by newline:
[^abc123]\n[^abc123]→ unchanged. Newlines mean the second usage is in a different paragraph or list item; dedupe could destroy meaningful repetition (e.g. each bullet citing the same source). - Same hex in reference section:
[^abc123]: …\n[^abc123]: …→ out of scope. The regex requires the markers themselves to be inline (no trailing:); reference-definition lines won’t match because the colon makes them syntactically distinct from body markers. - Same hex on either side of punctuation:
[^abc123], [^abc123]→ unchanged. The whitespace-only separator requirement keeps us conservative. If users want this collapsed too, treat it as a v2 enhancement after the conservative version proves safe. - Leading/trailing whitespace preserved: collapsing a run leaves the surrounding spaces of the first marker intact; we replace the run with the first marker only, so neighbors keep their spacing.
Out of scope (explicit)
- Cross-paragraph dedupe (newline separator).
- Punctuation-separator dedupe (
[^x], [^x]). - Reference-definition dedupe — that’s already what
dedupeByUrlServicecovers (for different hex IDs sharing a URL); identical-hex duplicate definitions are a separate pathology that should be flagged for the user, not silently collapsed. - Adding a standalone command. The user explicitly wants this built into Convert-All-To-Hex, not a separate command. If a one-off post-hoc cleanup is needed later, expose the helper through the existing dedupe-by-URL modal as an optional checkbox — don’t add a new command surface.
Acceptance criteria
- Running Convert All Citations to Hex Format on a document containing
[^jtas3k] [^jtas3k] [^jtas3k]produces[^jtas3k](single marker) at that position. [^abc123] [^def456](two distinct IDs) remains exactly[^abc123] [^def456].[^abc123]followed by a newline and[^abc123]on the next line remains unchanged.- Reference-definition lines (
[^abc123]: https://…) are never touched by this pass, even when adjacent. - Post-conversion
Noticeis updated to mention collapsed runs when any occurred, e.g.Converted 12 citations to hex format. Collapsed 3 inline repetitions. - The helper has unit-test coverage in the existing test surface (or in
scripts/parse-llm-citations.mjsif that’s where parser-style logic is exercised). pnpm buildpasses with no new TypeScript or ESLint warnings; the helper signature uses noanyper the type-safety contract.
Implementation order
- Add
collapseInlineRepeatstosrc/services/citationService.ts. - Wire it into
main.ts:499convertAllCitations(between the loop andeditor.setValue). - Wire it into
llmCitationParserService.tsat the appropriate post-tokenization step. - Update the
Noticestrings in both surfaces to surface the collapsed-runs count. - Add a changelog entry under
cite-wide/context-v/changelogs/YYYY-MM-DD_NN.mdper repo convention.
Why this is small enough to ship without a spike
The regex is one line. The integration touches two callsites. There’s no new modal, no new command, no new service. The whole change is on the order of 30–50 lines including the helper, the wiring, and a handful of unit tests.