Pickup notes — 2026-09-12: the watchdog got fixed, and so did the bug it was never going to catch

Read the "Out-of-band changes" section first. A meaningful share of this session's work is not in git — Railway service settings, a new Vercel project, and two secrets. A fresh clone will not tell you any of it exists.

TL;DR

1. The opening report — "Railway deployment issues on our Twenty instances" — was a false lead. All four live Twenty instances (lossless, palmer-ai, the-water-foundation, reach-edu) were and are healthy. Two dormant projects (laudable-art, perfect-dedication) hold Twenty stacks with zero deployments; probably abandoned, never confirmed.

2. The real thread was augment.didi.sh, and it split into an app bug and a tooling bug. Both are now fixed, but they consumed the session in the wrong order — roughly two days on the observability tooling before the app bug that prompted the report.

3. gh #95 is shipped and deployed — the Corpora Curator can no longer render one corpus's sources under another corpus's name.

4. gh #94 is shipped — deploy-watch.yml has gone from never having run green in its entire history to green on both its dispatch and schedule paths.

5. gh #96 and #97 remain open, both with plans written and evidence inline.

6. Railway watch paths do not support ! negation. Two measured attempts; current config deploys everything, which is the safe failure mode and matches the behaviour before this session. Doc commits still rebuild ten services — see "Watch paths" for the explicit-includes fix if that noise is worth removing.

What shipped this session

The app bug (gh #95) — the original complaint

Three corpora in a row showed the same three sources, each under its own name with its own confident "3 sources" badge.

Ruled out first, both innocent:

• The data. Production SurrealDB is clean and correctly scoped — seven corpora with counts 3/2/4/1/5/1/1 and distinct source_uuids.

• The query. assembleDomain filters on (client_slug, domain_type, domain_slug), and so does every write path.

The cause was the socket. workspace-service logged ws close at 20:30:39 and did not reconnect until 20:58:43; all three clicks landed inside that window. select() sets activeSlug synchronously and assigns sources after an await that never resolved. Identity moved, content did not, and nothing could detect the disagreement because a list of sources carried no trace of the corpus it came from.

Fix: domain.assemble now echoes the (type, slug) it answered for, and the curator refuses replies that do not match the current selection. That closes the class rather than the symptom — an out-of-order reply can no longer win either.

Deliberate softness: the match is enforced only when the echo is present, because the resolver and the curator deploy independently. A resolver on the old build sends no echo, and rejecting that would blank the surface instead of merely misfilling it. Tighten to a required echo once both sides are deployed everywhere — that is a real follow-up, not a nicety.

The watchdog (gh #94)

deploy-watch.yml had never once run green. All 200 retained runs failed, back to 2026-08-23; the secret was last set 2026-08-15. It had been blind since roughly the day it was wired.

Three separate defects, each hidden behind the previous one:

1. The token was malformed — raw=70 trimmed=63, not the 36-char UUID a Railway project token is, and rejected as an account token too.

2. gh had no repo to infer. The job never checks out, so every gh issue call died on fatal: not a git repository. Unreachable while the token was broken. Fixed with GH_REPO on both gh-using steps.

3. The cron was fiction. Measured: 38 scheduled runs delivered against 496 requested — 8%, median gap 185 minutes against a requested 15. GitHub drops high-frequency schedules under load. The old comment worried about Railway's rate limit, which was never the binding constraint.

So detection moved to push (Railway webhook → Vercel relay → repository_dispatch) and the cron was demoted to an hourly reconciler at :23.

The design rule is in Webhook As Wake Up Not As Truth: the relay forwards a wake-up, not a payload. The woken job re-queries Railway and reaches its own conclusion. Railway payloads are unsigned, so this is what makes a forged dispatch harmless — verified by sending a lie, which produced a green run that opened nothing.

Out-of-band changes — NOT IN GIT

This is the part a fresh clone cannot tell you.

nats deliberately has no watch paths — it runs from the image nats:2.10-alpine and a git push never rebuilds it.

Rotating RELAY_SECRET: change it in Vercel, redeploy, then update the Railway webhook URL. Order matters — the old URL 404s the moment the env changes.

Watch paths: resolved, and NOT the way it was attempted

Current state is safe: every push deploys everything, exactly as before this session touched anything. Nothing is silently undeployed. Read on only if you want the doc-commit noise gone.

Two attempts, both measured:

So ** matches correctly and Railway's watch paths do not honour gitignore-style ! negation, whatever the docs' "gitignore-style patterns" phrasing implies. The current config is therefore equivalent to having no watch paths at all.

If you want the exclusion, use explicit includes rather than negation — list what should trigger a build and let omission do the excluding:

apps/**
packages/**
services/**
shell/**
package.json
pnpm-lock.yaml
pnpm-workspace.yaml
Dockerfile

That leaves context-v/, changelog/, and root *.md out by omission. A .md inside apps/** would still trigger a build; acceptable.

Two traps, both paid for already:

1. get_service_config accepts watch_patterns on write and does not return them on read. There is no way to confirm the setting from the API — only by pushing and observing SKIPPED vs BUILDING. Budget a real commit per attempt.

2. The failure modes are asymmetric. Over-matching costs wasted rebuilds. Under-matching silently stops shipping your code and looks identical to a quiet day. If an attempt skips a code commit, clear the patterns immediately rather than iterating on syntax.

Shipping past a skipped deploy, if it happens again:

railway deployment redeploy --from-source -y -s <service> -e production \
  -p a45c72fa-8d40-464a-ba0d-a0940522de8e

A pattern worth naming

Three instances of one shape turned up in this codebase in three days:

• A curator rendering an unanswered fetch as content

• A vitest suite reporting a throwing import as "no tests", not a failure — which is why apps/corpora-curator/test/curation.test.ts sat dead since gh #93 and nobody noticed eight tests vanish

• A watchdog that would have reported healthy on an empty read, had its own author not guarded against exactly that

Absence of signal rendered as positive signal. The guard already written into deploy-watch.yml says it best: an empty service list must NEVER read as healthy. Worth treating as a review question rather than three incidents.

Open work

Branch + git state

• Branch rebuild/turbo-rsbuild (this is the default branch — matters, because schedule and repository_dispatch only fire workflows on the default branch)

• Clean and fully pushed, except clients/reach-edu — a submodule pointer that was already dirty before this session and is not mine. Left alone deliberately.

• This session: 52ba60c → 802d45c, eight commits.

Verify state on next session

# app is up
curl -s -o /dev/null -w '%{http_code}\n' https://augment.didi.sh/
curl -s https://ws.augment.didi.sh/config          # {"didi_auth_mode":"required"}

# watchdog is green on BOTH paths
gh run list --workflow deploy-watch.yml --limit 5 \
  --json event,conclusion -q '.[] | "\(.event)\t\(.conclusion)"'

# relay is alive (404 on a bad key is correct)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  'https://deploy-relay.vercel.app/api/railway?key=wrong' \
  -H 'content-type: application/json' -d '{}'

# the fix is actually deployed
curl -s https://corpora-curator-production.up.railway.app/static/js/5.*.js \
  | grep -c sourcesStatus

As of writing: app 200, watchdog green on schedule and repository_dispatch, relay 404s bad keys, sourcesStatus present in the deployed bundle.

Related

• Stale Corpus Sources When The Workspace Socket Is Down — Partially-Shipped

• Re Mint The Deploy Watch Railway Token — Shipped

• Back Up The Content Ingest Corpus Volume — Open

• Fix Env Aliasing That Drops Public Vars In Developers Menu — Open

• Webhook As Wake Up Not As Truth

• changelog/2026-09-11_01_A-Corpus-Now-Refuses-To-Show-Another-Corpus-Sources.md

• services/deploy-relay/README.md — relay setup, rotation, curl recipes