LFM Inline Link Substitutions: OG fetch foundation + `:::link-preview` / `:::link-rollup` directives end-to-end

Overview

Built the first end-to-end vertical slice of the inline link substitution family from the LFM spec (context-v/specs/Codifying-a-Comprehensive-Extended-Markdown-Flavor-and-Shared-Package.md §4.23.6) — from URL classifier through OG fetcher, through directive annotation, through Astro renderer, all the way to a visible card on the page at /promote/_demo/memo/version-1.

Where this leaves us: the _demo/memo/v1.md file now exercises every format the spec defines (row, card, thumb, livesite, column, gallery, carousel, thumb-row--horizontal-scroll) using both the ::: directive form and the Obsidian-portability code-fence form. Cards populate with real OG metadata via OpenGraph.io and degrade to favicon-only when fetch fails or is disabled. The infrastructure is in place for the popover work and the canonical-sources promotion flow described in the spec.

What still needs eyeballs: visual styling polish across the eight formats (the user paused us at “we can troubleshoot these later”), rate-limit tuning against the OpenGraph.io plan, and Vercel env-var wiring for production.

Changes by Area

1. `@lossless-group/lfm` — OG fetch foundation (Phase 0)

The spec required the OG fetcher to land before any component work. This is that foundation, built as small composable modules so swapping the network layer (e.g. for a future scraping proxy) is one new file rather than a refactor.

New types in packages/lfm/src/types/index.ts:

LinkPreviewData — render-surface metadata, field names aligned with the canonical Sources schema in cite-wide so future “promote to canonical” enrichment is additive (full mapping table in spec §4.23.6).
OGFetchResult, OGBackend, OGBackendOptions, OGBackendName, OGFetchOptions — the backend contract and per-site config shape.

New utility modules in packages/lfm/src/utils/:

og-cache.ts — JSON cache file at src/data/og-cache.json (gitignored), keyed by 16-char SHA-256 prefix of the URL, separate TTLs for hits (default 7 days) and failures (default 1 day) so transient upstream errors don’t poison for a week. Atomic write via .tmp + rename.
og-backends/direct.ts — naive fetch() with regex-based meta extraction. Free, fails ~10-20% of real-world URLs (Cloudflare, JS-rendering).
og-backends/opengraph-io.ts — the production backend. Maps internal apiKey field → OpenGraph.io’s wire-level app_id query param at the boundary so the rest of the codebase consistently says “key” (it authenticates, it’s billable). Reads from hybridGraph first, falls back through openGraph → htmlInferred.
og-backends/frontmatter-only.ts — no-network stub for highly-curated content.
og-backends/index.ts — registry; proxy falls through to direct until the future Browserless / self-hosted scraper backend lands.
og-dispatcher.ts — wraps a backend with cache hits, retries (exponential backoff, retryable-status detection), in-process semaphore for concurrency, and a sliding-window per-minute rate limiter + per-month soft-cap accounting (warns at 80% of plan ceiling, doesn’t hard-stop because OpenGraph.io’s monthly counter is the authoritative source).

New plugin in packages/lfm/src/plugins/:

og-fetcher.ts (remarkOgFetcher) — walks the MDAST for external link nodes, deduplicates by URL, batches through the dispatcher, attaches LinkPreviewData to each link.data.linkPreview. No-op fast path when enabled !== true. The MDAST LinkData interface is augmented in-place via TypeScript module augmentation so renderers get the typed field.

2. `@lossless-group/lfm` — Inline link classifier + `:::link-preview` directive

Lifted classifier to packages/lfm/src/utils/classify-link.ts:

Source: sites/mpstaton-site/src/lib/markdown/classify-bare-link.ts (which the README already documented as the temporary site-side stand-in for the package).
Added classifyLink(url) — pure URL classifier (no paragraph-shape requirement) for use by the directive plugin and any future inline-popover work.
Added collectLinkNodes(root) — recursive walker returning every external link node in a subtree.
Added previewType field on LinkClassification — maps catalog kind (‘video’, ‘short’, ‘playlist’, etc.) to LinkPreviewData.type (‘video’, ‘article’, etc.) so the directive renderer can pick the right component family.
Vimeo matcher now captures the optional unlisted-hash suffix into extra.hash (the prior site-local version threw it away).

New plugin in packages/lfm/src/plugins/remark-link-preview.ts:

Walks containerDirective nodes named link-preview or link-rollup.
Reads attributes (type, format, columns, aside, width, kind, trusted).
Classifies every URL in the subtree, stamps each link.data.linkClassification.
Pulls per-URL linkPreview data the og-fetcher attached and copies it into spec.items (URL → partial OG data) so the renderer doesn’t have to re-walk children.
For :::link-rollup without an explicit type=, infers a single shared type when every URL classifies the same way.
Stamps the resolved LinkPreviewSpec on node.data.linkPreviewSpec.

Pipeline order matters (preset.ts): gfm → directives → callouts → citations → og-fetcher → link-preview. The link-preview plugin runs after the og-fetcher so the spec carries fully-hydrated per-URL items into the renderer in a single pass. Reversing this order made spec.items empty.

3. `mpstaton-site` — Renderer + Astro components

New components in sites/mpstaton-site/src/components/markdown/:

LinkPreviewCard.astro — single component that handles all four single-URL densities (row, card, thumb, livesite, fullplayer) via a format prop. Reads OG data when available, degrades to URL + favicon when not. Favicons sourced from Google’s service (https://www.google.com/s2/favicons?domain={host}&sz=64) — no API key, instant, browser-cacheable.
LinkRollup.astro — multi-URL container. Iterates urls, picks the matching child format per the spec mapping (column→row, gallery→card, carousel→card, thumb-row→thumb), passes per-URL itemData through.

Modified renderer in sites/mpstaton-site/src/components/markdown/AstroMarkdown.astro:

New dispatch arm in the containerDirective branch reads data.linkPreviewSpec and routes to LinkPreviewCard (kind=link-preview) or LinkRollup (kind=link-rollup). Falls through to the existing children-as-div fallback when no spec is present, so any non-LFM containerDirectives keep their current behavior.

Modified loader in sites/mpstaton-site/src/lib/promote/memos.ts:

loadMemo now passes ogFetch: { enabled: true, backend: 'opengraph-io' | 'direct' (auto), apiKey: import.meta.env.OPENGRAPH_IO_API_KEY, cachePath: 'src/data/og-cache.json', maxConcurrent: 4, rateLimit: { perMinute: 60, perMonth: 100 } } to parseMarkdown. Backend auto-selects opengraph-io when the API key is present in env, falls back to direct for offline dev or unauthenticated runs.
.gitignore updated to exclude src/data/og-cache.json.

4. Demo content additions

sites/mpstaton-site/src/content/promote/_demo/memo/v1.md — extended with samples covering:

Inline external links (Stripe, NYT, Wikipedia) for the popover/og-enrichment path.
:::link-preview directive in default form, with explicit type + format, with margin-track positioning (aside=right-escape), with video-as-substitution (vs. bare-URL auto-unfurl), with Layer-1 escape-hatch attributes (class=, data-track=).
:::link-rollup directive in column / gallery (3-column video) / horizontal-scroll-thumb forms.
Obsidian code-fence equivalents for both directive families (the spec’s remark-code-fence-as-directive portability path — not yet implemented but content is staged).

URLs picked deliberately to span clean-OG (Stripe, Wikipedia), Cloudflare-protected (NYT), and provider-classified (YouTube, Vimeo, YouTube Shorts) so first-build feedback tells us exactly which surfaces work cleanly through OpenGraph.io.

Architectural decisions captured with rationale

These are the close-call calls worth preserving for the next session.

OpenGraph.io as production default, direct as the always-available fallback. Direct fetch() loses 10-20% of real URLs to Cloudflare/JS-rendering. With popovers that’s a missing hover; with substitutions it’s a visible degraded card in the prose — different visibility, same code path, much higher quality bar. The dispatcher is a small router over backend modules so adding a self-hosted scraping proxy later is one new file, no plugin-API churn.
apiKey (config) vs app_id (wire). OpenGraph.io’s API parameter is app_id. Our internal naming is apiKey — it authenticates, it’s billable, “id” implies non-secret. The boundary translation happens in exactly one place: og-backends/opengraph-io.ts. Env var: OPENGRAPH_IO_API_KEY.
Cache layer first, then backend. Built og-cache.ts and the backend interface as a coherent unit before wiring OpenGraph.io specifically. Two payoffs: (a) the cache is provably independent of which backend filled it; (b) we can dev/test against fixtures without spending API quota.
Separate TTLs for hit and fail. A failed fetch shouldn’t be re-tried for 7 days (the success TTL) — that means a transient OpenGraph.io 502 poisons a card for a week. Default failCacheTtl is 1 day. Not configurable per-failure-type yet; revisit if a particular host’s transient failures need shorter retry.
Per-month rate limit is soft. The dispatcher counts outbound calls and warns at 80% of the configured perMonth cap, but doesn’t hard-stop. Reason: OpenGraph.io’s monthly counter is the authoritative source; a build-side estimate could falsely block legitimate work after a partial-month config change. The per-minute limit IS hard (sliding window) — that’s a client-side throttle the dispatcher controls fully.
og-fetcher runs before link-preview, not after. First version had link-preview annotating directives first, then og-fetcher enriching links. Result: spec.items was always empty because the data didn’t exist when link-preview walked. Reversed: the og-fetcher attaches linkPreview to every external link node, then link-preview walks containerDirectives and copies the now-present per-URL data into the spec. Renderer reads from one place (spec.items) regardless of whether the link is inside a directive or not.
One LinkPreviewCard.astro over four files. Spec describes LinkPreview__Article--Row.astro, --Card.astro, --Thumb.astro, --LiveSite.astro etc. as separate files. We collapsed to one component with a format prop that switches CSS classes. Reason: 90% of the markup is shared, the variants differ only in layout. If a single format grows enough custom logic to warrant its own file, splitting later is cheap.
Favicon as the always-on brand mark. Even when OG image is present, the Card shows a 16px favicon next to the host label — recognition cue that doesn’t depend on OG fetch having succeeded. When the OG image is absent the card collapses to body-only (no giant favicon-as-hero — that was the first attempt and looked bad; the user flagged it immediately).
Site components first, package extraction later. Per the astro-knots philosophy (CLAUDE.md): build in the site, extract to packages/lfm-astro/components/ once the pattern is validated. Card + Rollup live in sites/mpstaton-site/src/components/markdown/ for now. Once OG enrichment is visually polished and the formats prove themselves, we’ll lift them into the canonical pattern source.

What’s intentionally NOT in this work

Hover popovers (spec §4.23.1–5) — the OG cache feeds them directly, but the global popover element + event-delegation infra is a separate UI layer. Not built.
Wikilinks — flagged in the spec as wish-list, no implementation.
livesite format — has a stub (sandboxed iframe with trusted=true opt-in) but no real testing. Author opt-in only by design; off by default for security.
proxy backend — registered in the dispatcher, falls through to direct. Will become a real module when we hit a URL OpenGraph.io can’t reach (intranet, IP-banned hosts, paywall).
remark-code-fence-as-directive — the Obsidian-portability path that converts ```link-preview ... ``` fences into directive nodes. Demo content includes the syntax but the transform isn’t built. Without it those fences render as syntax-highlighted code blocks.
Catalog YAML→JSON build-step extractor — the bare-link catalog (Bare-Link-Provider-Catalog.md) is documented as having a tsup-time JSON emit step. Still doesn’t. The classify-link.ts matchers are hand-mirrored from the YAML; drift is possible. Worth fixing before adding more providers.
“Promote to Canonical Source” pipeline — LinkPreviewData.canonicalSource field exists for forward-compatibility, but the agent flow that fills it (cite-wide blueprint) is not in scope.
Per-site variant registry (src/config/lfm-variants.yaml) — Layer 2 escape hatch from the spec. Schema is documented, runtime not built.
Aside positioning CSS — aside attribute is parsed and stored on the spec but the renderer doesn’t use it yet (no margin-track grid in the demo layout). Cards render inline regardless of aside=.

Files

New (LFM package — `packages/lfm/`)

src/utils/og-cache.ts
src/utils/og-dispatcher.ts
src/utils/classify-link.ts
src/utils/og-backends/index.ts
src/utils/og-backends/direct.ts
src/utils/og-backends/opengraph-io.ts
src/utils/og-backends/frontmatter-only.ts
src/plugins/og-fetcher.ts
src/plugins/remark-link-preview.ts

New (mpstaton-site — `sites/mpstaton-site/`)

src/components/markdown/LinkPreviewCard.astro
src/components/markdown/LinkRollup.astro

Modified (LFM package)

src/types/index.ts — added 6 new exported types, extended RemarkLfmOptions with ogFetch?.
src/index.ts — exports for new types, plugins, utilities.
src/preset.ts — wired remarkOgFetcher + remarkLinkPreview into the chain in the right order.
tsup.config.ts — added 6 new entry points.
package.json — added @types/node devDep (needed for node:crypto/fs/path imports in og-cache).

Modified (mpstaton-site)

src/components/markdown/AstroMarkdown.astro — new containerDirective dispatch arm for link-preview / link-rollup.
src/lib/promote/memos.ts — enabled ogFetch in parseMarkdown options.
src/content/promote/_demo/memo/v1.md — added five sections of samples covering the full directive grammar.
.gitignore — excluded src/data/og-cache.json.

Spec status

context-v/specs/Codifying-a-Comprehensive-Extended-Markdown-Flavor-and-Shared-Package.md lines 47-77 (the link-rendering task list):

Phase 1 substantially complete from a wiring perspective. Visual polish + LiveSite + per-format edge cases are the open items.

Next session

Natural openings, in priority order:

Visual polish across all eight formats at /promote/_demo/memo/version-1 — the user paused us with “we can troubleshoot these later.” First eyeball pass should focus on: card grid alignment when OG images vary in aspect ratio, carousel controls (currently no prev/next), thumb-row card-width tuning, mobile collapse behavior.
Vercel env-var wiring — OPENGRAPH_IO_API_KEY needs to be set in Vercel’s env-var UI for production. Build will fall back to direct backend without it (and produce mostly-favicon cards).
Promote LinkPreviewCard + LinkRollup to packages/lfm-astro/components/ once the formats stabilize — they’re currently site-local. Per CLAUDE.md, this is the canonical pattern source other sites copy from.
remark-code-fence-as-directive — converts Obsidian-style code fences to the directive nodes the existing renderer already handles. Demo content includes both forms; only one renders correctly.
OG popover infrastructure (spec §4.23.1–5) — the cache is ready; what’s missing is the global popover element + event-delegation script in the site layout. Different work surface (UI, not parser).
Catalog YAML→JSON build extractor — keeps classify-link.ts matchers and Bare-Link-Provider-Catalog.md from drifting. Worth doing before adding Loom / Spotify / SoundCloud providers.