How code reaches
a student.
Every deploy path, every gate, every trap. One ARM box in Helsinki runs both products, the database, the queue, the job runner and the monitoring — so the blast radius of a mistake is larger here than the architecture diagram suggests.
/internal/decks.deploy-school and deploy-campus skills, and docs/ in both repos. Where this deck and a runbook disagree, check reality before trusting either; slide 14 lists the places they already do.Everything is on one box
the production host — Hetzner CAX41 ARM, Helsinki
Renewable-powered, and the single point of failure for the entire organisation. Coolify is the PaaS layer at port 8000; Traefik is the reverse proxy with automatic Let's Encrypt; Cloudflare fronts it in Full Strict mode.
| Service | Container | Notes |
|---|---|---|
| School prod | school-prod | Flask, gunicorn 2×2, port 8080 |
| School staging | school-staging | Auto-deploys from production |
| PostgreSQL 16 | postgres | Shared by school + campus |
| Redis 7 | redis | Sessions, shared for SSO |
| Job runner | job-runner | Separate deploy path — see slide 6 |
The consequence, stated plainly
On 2026-08-02 the disk hit 100%. Postgres crashed, Redis went into MISCONF and rejected every write. Nobody noticed until Aug 4. On 2026-07-23, concurrent PR-preview builds starved the production host during a live class.
Neither was a code defect. Both were one box doing too many jobs with no resource-threshold alerting. Campus #1073 asks for exactly that and is still open.
The shared database is a coupling nobody drew
Merging to school's production auto-deploys staging, which auto-migrates the shared production database. Schema lands before the production app does. This is intentional and it is also the sharpest edge in the whole system.
Where code goes
| Repo | Branch | Deploys to | Trigger |
|---|---|---|---|
| school | feature/* fix/* claude/* | nothing | CI only |
main | nothing | Integration only — far behind, do not target PRs here | |
production | staging → then prod | Push runs CI; staging auto-deploys on pass. Production is manual. | |
| campus | main | nothing directly | PRs target here |
production | prod | Promoted from main — and the two have diverged in both directions |
Campus has a fork it hasn't healed
171 commits exist only on production; 209 exist only on main. That is not drift, it is two products. It is why campus cannot be given a version number — a version is a claim about what's deployed, and campus's deployed state isn't legible enough to make the claim honestly.
The human-led reconcile (campus #650, step 1) is a prerequisite for release gates, not a parallel track.
The naming collision waiting to bite
Campus has a real branch named 0.0.x. It is an integration branch, not a version — but it reads as one, and it will read as a stale one the moment campus is stamped 1.0.0. Rename it to integration or next during the reconcile.
Deploying the school
productionThe status check
One GitHub commit status: ci/hetzner-lint. GitHub Actions was removed entirely — if you are looking at a green Actions badge, you are looking at a ghost. The --test phase builds Dockerfile --target test and runs the unit suite, so school's gate genuinely blocks.
Arm the deploy guard first
POST /internal/monitoring/arm-deploy-guard before any production restart. Without it, the restart looks like an outage to Sentinel and escalates a false incident. The guard writes a Redis key that Sentinel checks. This step is required, not advisory.
Deploy ≠ restart
A restart reuses the same image. If you changed code and restarted, you deployed nothing and it will look like your fix didn't work. Use the deploy endpoint.
Rolling deploys are briefly two-faced
Old and new containers both serve for roughly two minutes after a deploy. Verify with docker exec … grep <new-symbol>, not with a request to /healthz — the health check will pass on the old container and tell you nothing.
Deploying the campus
The shape
Node/React/Vite/Socket.IO, promoted main → production, deployed through the same Coolify instance. LiveKit provides the SFU for calls.
- PRs target
main, notproduction— the opposite of school - Promotion is a human decision, currently done by hand
- Cherry-picks to
productionare common, which is why the fork exists
The gate bites — but it only checks one thing
ci/hetzner-lint is a required, strict status check on main and production, plus one review. It genuinely blocks. The problem is coverage: ci-check.sh takes MODE="${1:---typecheck}" and the webhook runner passes no arguments.
So on campus the gate runs typecheck only. The migration content guard, lint, build and the entire server suite exist and pass — but only under flags nothing calls. Per the stabilization doc: “4,115 server tests existed and zero ran on any PR.”
LiveKit is in the wrong hemisphere
Campus #301: the SFU sits in Hetzner DE, which degrades video for US students on typical home connections. This is an infrastructure placement decision masquerading as a stream of call-quality bug reports.
What campus does better
Preview deployments. PR + the deploy-preview label gives you your branch running in the live world on an isolated database branch, at pr-{N}.preview.campus.themultiverse.school. Behavioural verification in one click. Use it every time.
The job runner deploys separately
Deploying the app does not deploy the jobs
jobs/ does not ship with the school application deploy. It lives at /opt/job-runner/ on Hetzner and is updated by its own deploy.sh, which hard-resets /opt/school-repo and rsyncs jobs/ across.
If you fixed a job, deployed the app, and the job still misbehaves — this is why. Run the job-runner deploy too.
Schedules live in Windmill, not the repo
The /run schedules are driven by Windmill (workspace multiverse), not by the n8n JSON files in the repo and not by host cron. The repo's n8n files are decoration. Six-field cron syntax.
Cross-service dispatch
School calls campus over RPC; campus enqueues on pg-boss; pg-boss calls back to job-runner:5050/run/<job>. Campus #1116 notes there is no retry — a job-runner restart of a few seconds fails the whole pg-boss job.
The critical loop
full_enrollment_process.py runs every ten minutes and does 17 phases: Stripe sync, enrolment, membership, calendar, emails, receipts, weekly digest, standup.
It is the safety net under the two real-time webhook paths. Several operations — membership updates, calendar invites, receipts, the weekly digest, standup invites — happen only here. If the batch stops, students get enrolled and silently miss everything else.
It has stopped before
The orchestrator was killed at its 1800-second timeout on roughly 96% of runs, from an unbounded receipt fetch plus a poison record. Fixed in July. Separately, backfill_unmatched crash-looped for over ten days on a case-sensitivity mismatch between an email column and a lower(email) index.
What actually stops a bad merge
| Check | School | Campus | Blocking? |
|---|---|---|---|
| Syntax / build | yes | yes | Both |
| Typecheck | n/a | gates | Campus: the only thing the gate runs |
| Lint | gates | exists | Campus: behind --lint, which nothing calls |
| Unit tests | gates | never run | 4,115 campus server tests, zero executed on any PR |
| Tests against a real DB | no | no | Campus #893 — flagged highest-yield |
| Migration guard | partial | exists | Campus: behind --check-migrations, uncalled. School's ledger has false positives (#708) |
| Branch protection | yes | yes | Strict + 1 review on every deploying branch — but enforce_admins: false on all three |
| Smoke test after deploy | manual | manual | Auto-rollback is R4, unbuilt |
| Secret-hygiene scan | no | no | R13, unbuilt |
ci/hetzner-lint, and both genuinely block. But school's runs lint + tests + build — and because the deploy builds the same Dockerfile, a red test blocks the deploy, not just the PR. Campus's runs typecheck. A slide reading “both repos gate on ci/hetzner-lint” would be true and badly misleading.Sort by cost of being wrong, not by confidence
The opening move of AUTOMATION_FRAMEWORK.md, and the reason our process looks different from a normal shop's:
The axis instead is reversibility × blast radius × machine-verifiability. “A per-run confidence of ‘only’ 90% is fine for full autonomy when a deterministic check catches the other 10% and rollback is free. Conversely, 99% is not enough to automate an irreversible, wide-blast action.”
Full autonomy
Deterministic success check or cheap rollback, and a regression is visible in minutes.
- PR gating, lint autofix, lockfiles
- Preview spin-up and teardown
- Read-only audits
- Auto-rollback on failed smoke test — automated harder than deploy, because rollback is the safe direction
Confidence to run: 95–99%.
AI proposes, human merges
Alters product behaviour, or can't be fully machine-verified.
- Must open a draft PR
- Must run the verify harness — drive the real flow, attach a recording
- A human clicks merge
Confidence to run: 70–85%. Any diff touching a Tier-3 surface is force-demoted to Tier 3 no matter how clean it looks.
Human decides, AI assists
Irreversible, wide blast, or values-laden. Confidence is capped by policy.
- Moderation against a real student
- Anything fanning out to all users
- Non-additive migration on the shared DB
- prod↔main promotion
- Secret rotation, Stripe changes
AI prepares the diff, the rollback plan and the evidence — and pulls no trigger.
automation_events) is specified and not built.Automate loops aggressively. Automate decisions almost never.
Three lists across three docs, under three different titles. They compose rather than compete — and where school's versioning doc and campus #650 disagree, #650 wins on promotion authority.
School — “What never automates”
Every version bump is Tier 3: the critical-bugfix patch, the Thursday release, the cohort-end bump, the break-week promote.
“What the automation does instead is assemble the evidence and propose the bump… A human merges and promotes. The one exception is rollback (R4), which fires on its own.”
Campus — “What stays human”
- The prod/main reconciliation decision and every promotion
- Any non-additive migration on the shared DB
- Flipping an enforcing flag for all users
- Secret rotation and Stripe/billing
Stabilization — “permanently”
Adds the product half: promotion, moderation against real students, pricing — and the bit.
“Whether a joke is funny, whether a feature is delightful, whether the goose should also steal hats — taste is the one verification loop we never automate, because the moment whimsy is machine-approved it stops being whimsy.”
The structural control
Not a policy, an architecture: “a routine here should not be given deploy credentials at all — a loop that cannot reach the deploy API cannot quietly become the thing that promotes.”
School's enforcement plan records that an earlier draft had routines deploying production on a schedule, and corrects it.
The whole point, in one line
“The art is making the loop deposit enough evidence at the human's feet that the irreversible decision takes 30 seconds instead of 30 minutes.”
Or: “This is the difference between ‘the release process runs itself’ and ‘the release process does your paperwork.’ We're building the second one.”
Flying on instruments that always read level
Written the night of the 1.14.0 release. 1.14.1 switched on error capture, and in the following two hours ten pre-existing production bugs surfaced. Not one was a regression from the release — whole subsystems, silently non-functional, for weeks.
The checks that reported success
deploy/smoke-test.shpassed 5/5, twice, while an entire integration was deadmeetupTools.test.tspassed 26/26 while the function it covers was provably broken in prod- 4,115 server tests existed and zero ran on any PR
“A green check is consumed as evidence. We were not flying blind; we were flying on instruments that always read level.”
The rule that came out of it
“Every verification layer must be able to demonstrate a red result on a known-broken system. If it cannot, it is not a check — it is decoration, and it should be deleted or fixed, never trusted.”
Eight of the ten bugs were JS-value ↔ Postgres-column contract violations. Not one was a logic error. 297 of 431 server test files mock the database — and a mock accepts any parameter, so a type mismatch is unrepresentable in the test.
Five checks named. One runs.
What the process asks for
A change is done when:
- R1 green — build, typecheck, lint, test, migrations
- For behaviour changes: R7 verify evidence, a recording of the flow working, attached to the PR
- Migrations are additive-only (R12)
- No new committed secrets (R13)
- The PR template's “How to test” filled with real steps
Where we're going
“The unit of agent work today is the diff. The unit of agent work after this strategy is the diff plus machine-generated evidence that it works.”
Which inverts the human role: from tester of last resort to auditor of evidence.
What is actually enforced
On campus: typecheck
R7 does not exist. R12 exists but runs behind an uncalled flag. R13 is unbuilt. Evidence coverage today is 0%; the target is >80%.
The diagnosis behind all of it
“52 of those 105 commits — essentially half — are fix: commits. We are not slow. We are fast in a loop where a large share of capacity goes to repairing what the previous lap broke.”
“Code-writing capacity is effectively unlimited now; verification is the scarce resource, we spend almost none of it before merge, and so the deficit is paid later.”
The scoreboard
Fix ratio ~50% → <25%. Human minutes per merged PR → <5. Auto-remediation rate → >60%. Detection latency, unbounded today → <1 hour.
“Fix ratio falling because auto-remediation rate is rising is the shape of success.”
Preview deployments
What you get
Add the deploy-preview label to a PR and your branch runs in the live world at pr-{N}.preview.campus.themultiverse.school, against an isolated database branch. This is the single best verification tool either repo has.
The cap, and why it exists
preview_reaper.py holds six previews per repo, evicting the oldest non-release preview and removing its label. Coolify deletes are asynchronous — allow around 90 seconds before the slot frees.
The cap is not tidiness. Concurrent preview builds starved the production host during a live class on 2026-07-23.
It is currently unreliable
Campus #842: previews fail with "no available server" and the label requirements are undocumented. A verification tool people cannot trust is a verification tool people stop using.
What's missing
Campus #843 — previews have no jobs runner, so anything driven by pg-boss or cron cannot be verified in a preview at all. That's a blind spot covering most of the enrollment and notification surface.
The versioning scheme
MAJOR . MINOR . PATCH → 4 . 2 . 304
- MAJOR — the thing you can see. A visual or structural overhaul big enough that people say "new campus" versus "old campus".
- MINOR — the release counter, tied to cohorts. Even is stable. Odd is unstable.
- PATCH —
WD, week and day within the cohort.304is week 3, day 04.
The rhythm
The cohort is the release cadence, because our cohorts are real rather than invented from signup dates. Thursday is release day. During a cohort, production runs an even/stable minor. At cohort end everything built ships as the next odd/unstable minor, with the previous stable container still on deck. Break week is when the unstable release gets tested — the only window where production knowingly runs unproven code, and the only week nobody is in class.
"Unstable" is about confidence, not standards
An odd release clears exactly the same gates as an even one and a human still pulls the trigger. What differs is that we have declared a window where we expect to find problems, and staffed it accordingly.
Campus ships two incompatible schemes right now
docs/VERSIONING.md is live on both campus branches and defines something else entirely: cohort.week.staging-deploys, where minor is calendar weeks since the Apr 20 kickoff and patch resets weekly.
Live state: main:VERSION is 1.16.2, production:VERSION is 0.14.7. 1.16.2 is legal under both schemes and means different things under each. That is a live footgun, not a stale doc.
Deliberately excluded
The 2026-03 GCP→Hetzner migration is not a MAJOR bump. It was the largest engineering event in the repo's history and users saw nothing. That exclusion is the definition working correctly.
And our milestone names encode the losing rule
Two WD rules are live. cohorts.yml (merged, but confirmed: false) would make Aug 6 104. The rule actually in use makes it 204 — confirmed by tag v4.2.203 stamped Wed Aug 5.
The plan says “#713's rule is the better one” and declines to adopt it while the file is unconfirmed. So every August milestone title is off by one week. Fixing it is one gh api --method PATCH per row — but confirm cohorts.yml first.
Where secrets live, and where they leak
The intended path
Self-hosted the secret manager on Hetzner, Bitwarden-compatible, driven by the bw CLI. Production values are injected as Coolify environment variables. The repo's .env is for local development only.
Rules: never commit, never log in full, mask as sk-…last4, and investigation agents are read-only.
Where the gaps are
- School #494 — a P0 credential-hygiene item, tracked privately.
- Campus #680 — a credential rotation and history purge, tracked privately.
- Campus #1076 — build-time secret handling, tracked privately.
- A hardcoded-credential item flagged in June, still pending. Tracked privately.
Why rotation is stuck
We will not rotate a live Stripe key without restart-free rotation (school #287), because rotating it today means a restart, and a restart during a class is an outage. #287 therefore blocks #288 and #290. That ordering is correct and should be stated out loud so nobody re-litigates it.
The missing control
Campus #665 — a secret-hygiene scan in CI. Every leak above would have been caught at PR time by a check that takes seconds. There is also #846: redact secrets from Claude transcripts before they are stored.
Backups: three layers, one lesson
on-host
60-day retention
off-site
fails closed
Layer 3 was silently dead for three and a half months
The R2 credentials were simply missing. The backup job reported success. Nobody had off-site backups and nobody knew. This is the organisation's signature failure mode in its purest form: the thing reported success and did nothing.
The fix was not "add backups" — backups existed. The fix was a check that verifies a real object's freshness in the destination and fails closed when it can't.
The stale-path trap
/mnt/db-backups/ is stale. The real backups are at /mnt/storage/db-backups/. Someone restoring under pressure will find the first path, and it will look plausible, and it will be old.
The backup volume was resized 10GB → 500GB live during the disk incident. Hetzner volumes never shrink; that is a permanent cost line now.
Restores have been exercised
Seven hard-deleted /x/ pages were fully restored from backup in July. The restore path works — which is worth far more than knowing the backup job runs.
Unbounded growth is the next one
Campus #1074 — retention policies for unbounded tables plus two runaway producers. #1075 — Redis has no maxmemory and a noeviction policy, on the host that just filled its disk.
What is watching, and what isn't
In place
GlitchTip → GitHub issues
Errors become deduplicated tickets automatically. Live since July, validated end to end.
Grafana + Loki
Log aggregation. Traefik 5xx flows in; an http_error_alert cron reports to the infra channel.
Gatus + Dozzle
Uptime checks and live log tailing.
Prometheus + Pushgateway
App metrics at /metrics, IP-restricted. Job metrics persist between runs via the Pushgateway.
Deploy guard
Suppresses false Sentinel escalations during a planned restart.
Missing — and each one has an incident behind it
Resource thresholds
Campus #1073. We can only detect a full disk after it breaks something. That is precisely how the Aug 2 incident went unnoticed for two days.
Synthetic funnel monitoring
School #735. No robot buys a class on a schedule. Enrollment webhooks were silently broken for four months and the batch job masked it.
Stripe reconciliation
School #665. Nothing compares successful charges against rows in purchases. Campus took $19 and fulfilled none of it.
Environment separation
Campus #1114. SENTRY_ENVIRONMENT is unset, so GlitchTip cannot tell staging from production. Every alert is ambiguous.
Usable error payloads
School #770 — auto-filed issues carry no stack trace. Campus #1055 — client errors arrive as <unknown>. The pipeline works; the contents don't.
What actually went wrong
| When | Incident | Root cause | Class |
|---|---|---|---|
| Aug 2–4 | Disk 100%, Postgres crashed, Redis rejecting writes | No resource-threshold alerting; Docker deploy history filling root | capacity |
| Jul 23 | Production host starved during a live class | Concurrent PR-preview builds, uncapped | capacity |
| Jul 24 | School prod 503s — worker starvation | Timeout-less synchronous NATS publish wedged gunicorn threads | code |
| Jul 24 | Enrollment orchestrator killed on ~96% of runs | Unbounded receipt fetch + a poison record, against a 1800s timeout | code |
| ~Apr–Jul | Off-site backups silently absent, 3.5 months | R2 credentials missing; job reported success | silent |
| Feb–Jun | Welcome emails never sent, ~5 months | App imported job_tools/, which isn't on the app path | silent |
| Nov–Mar | Enrollment webhooks broken, ~4 months | A two-argument TypeError; the batch job carried the load and hid it | silent |
| Jul | 797 Matrix invites stuck ~8 days | Drainer had no LIMIT against a 300s kill | code |
| Jul | Seven /x/ pages hard-deleted | Human action, no soft-delete. Fully restored from backup. | data |
Four of nine were silent. They ran for months, reported success, and were found by someone going to look — not by an alert. Every gauntlet on the team-update deck exists to convert that category into the "found in ten minutes" category.
Gotchas that have cost us real hours
The deploy endpoint in the docs 404s
The runbook says POST /applications/{uuid}/deploy. Reality is GET /api/v1/deploy?uuid=, run from the box. Docs and reality disagree — check reality.
Restart is not deploy
A restart reuses the same image. Code changes need a deploy.
Rolling deploy split-brain
Both containers serve for ~2 minutes. Verify by grepping for a new symbol inside the container.
The homepage is Cloudflare-cached
Bare / is served from the edge and needs a purge. /classes and detail pages are dynamic.
jobs/ doesn't ship with the app
Separate deploy path. Run /opt/job-runner/deploy.sh too.
The git remote is github, not origin
Commands using origin fail in confusing ways, and the local production branch lags the remote.
Timestamps in classtimes lie
Pacific wall-clock stored as +00. A raw > now() comparison is seven to eight hours early.
Calendar updates fail silently
You must fetch the existing event before calling update(), or nothing happens and nothing complains.
A migration ledger row is not proof
A repair baselined 155 rows in one second. Three recorded-as-applied columns don't exist in production.
Partial unique indexes need their predicate
Repeat it in ON CONFLICT or the upsert silently doesn't match. Campus has 563 upsert sites and one predicate.
flask-limiter is per-worker
Storage is memory://, so rate limits aren't global. Use Redis INCR + TTL for anything real.
SendGrid ignores our unsubscribes
Confirm suppression handling before any campaign. This is a compliance problem, not a nuisance.
The R-series: process gaps, already ticketed
Campus #652 enumerates fourteen routines plus four infrastructure prerequisites. Read this as a design, not a status board. The tiers, the outcome ledger and the R-numbers are adopted vocabulary in active use in planning and review — but automation_events exists in no code, and school's VERSION, release/version.py, /version and docs/fires/ do not exist either. Layers 0, 1 and 3 are unbuilt.
R1 — the PR gate
#653. Build, typecheck, lint, test, migrations as a required status check. Everything else depends on this.
R2 — drift report
#654. prod ↔ main divergence, reported continuously so a fork can never grow to 380 commits again.
R3 — env-drift audit
#655. Armed flags and environment variables that differ between what's deployed and what's expected.
R4 — auto-rollback
#656. A failed smoke test reverts the deploy without waiting for a human to notice.
R5 / R6 — the monitors
#657 activation funnel, #658 call-join success rate. The two numbers that describe whether the product works.
R7 — verify harness
#659. Evidence attached to every PR, so review reads evidence rather than re-deriving claims.
R8 / R9 — autonomy
#660 bug-motel triage stays autonomous; #661 repair PRs propose, never merge.
R10 — moderation assistant
#662. Recommend and queue, never execute. The autonomy boundary is deliberate.
R13 — secret hygiene
#665. This one is overdue.
Infra — branch protection
#667 and school #608. Required checks, no direct push. Cheap, and currently absent from both repos.
Infra — outcome ledger
#669. An automation_events table, so we can tell which automations earn their keep.
Infra — scoped identity
#670 and school #607. A dedicated automation environment with credentials that are deliberately not Coolify access.
The card to keep open
Ship something on school
- Branch off
production— nevermain - Push; wait for
ci/hetzner-lint - PR targets
production - Merge → staging deploys and auto-migrates the shared prod DB
- Verify on
staging.themultiverse.school - Arm the deploy guard
- Deploy production from the box
- Grep the container for a new symbol — not
/healthz - Purge Cloudflare if you touched
/
Ship something on campus
- Branch off
main - PR to
main, adddeploy-preview - Verify behaviour in the live-world preview
- Merge; promote
main→productiondeliberately - Remember the gate did not block you — you are the gate
Where to look when it breaks
- Live logs — Dozzle, or
docker logsby container filter - Aggregated — Grafana/Loki
- Errors — GlitchTip, and the auto-filed GitHub issue
- Uptime — Gatus
- Prod data — the DB query MCP; no SSH needed
- Jobs — Windmill for schedules, Pushgateway for metrics
Two standing rules
Never plan an outage. Restructure it as a live cutover with a rollback plan. A 36GB volume move was done as a cutover with nine seconds of downtime and no compose changes — that is the bar.
Evidence beats assertion. A PR that says "fixed the login bug" is worth much less than one showing the bug reproducing before and not after. Nearly every gate in this deck exists to produce evidence a reviewer can check in under a minute.