Application Releases Runbook

This runbook tells how a promoted Candidate becomes a live app. It also tells how to check a release.

Promotion advances the Canonical Generation. A release then serves the promoted commit. ReleaseWorkflow (workers/integration/src/release-workflow.ts) checks each release with executable Evidence. No model decides whether a release is authorized.

A release keeps all history. A failed release keeps the Canonical Generation. To recover, promote a fix Change or a rollback as the next generation.

1. Live app release

Each Project has a live app at https://<slug>.rhumbatron.localhost. The apps service runs it on workerd with the Worker Loader (ADR 0020, live-apps.md). A deploy is runtime data: a bundle in the object store and a pointer row in app_deployments. A deploy changes no container and no configuration.

sequenceDiagram
  participant I as integration (promotion)
  participant Q as release queue
  participant R as ReleaseWorkflow
  participant S as object store bundles
  participant D as app_deployments
  participant A as apps router
  I->>Q: release.start rel_project_generation
  Q->>R: start the instance
  R->>R: still canonical? build.started
  R->>S: bundle of the promoted commit (reuse or build)
  R->>D: move production pointer (compare-and-swap)
  R->>A: probe / and up to 3 GET routes
  alt smoke passes
    R->>R: Evidence release-sN, status deployed
  else smoke fails
    R->>D: move pointer back to the earlier version
    R->>R: Evidence, status smoke_failed
  end

1. Start: the promotion fast-forwards canonical main in the Project repository on the git service. It records a queued release with the ID rel_<project>_<generation>. It sends release.start on the release command queue. The integration module starts ReleaseWorkflow with the release ID as the instance ID.
2. Check: the workflow confirms that the generation is still canonical. A newer promotion during the release sets the release to superseded.
3. Bundle: the workflow uses the stored bundle of the promoted commit, the bundle of the same tree, or the bundle of the Candidate preview. When none of these exists, it builds the bundle from the Project source on the apps service. The cap is 60 builds per UTC day (APP_BUILDS_PER_DAY).
4. Pointer: the workflow moves the production pointer by compare-and-swap on the row that it read. The workflow keeps the earlier version.
5. Smoke check: the workflow probes the app through the apps router. It probes / (a real page, not a router status page) and up to 3 static GET routes from the route table of the app (workers/integration/src/release-model.ts). After a failed check, it waits 10 seconds and runs the check once more.
6. Evidence: the workflow stores release Evidence with the check ID release-s<generation> in the object store and in evidence_index.
7. Status: a passed smoke check sets deployed. A failed smoke check moves the pointer back to the earlier version and sets smoke_failed. A missing bundle or a failed build sets failed.

Hosts

Host	Serves
<slug>.rhumbatron.localhost	The production version.
<previewId>--<slug>.rhumbatron.localhost	The preview of one Candidate. A Change keeps at most 3 previews. The daily sweep deletes preview rows and their data stores after 7 days (cleanup.md).
v0000001--<slug>.rhumbatron.localhost	One kept version. The number is the Canonical Generation with 7 digits.

A Project without a shipped Change shows “Nothing built yet”. After “Remove app”, the host answers 410.

Rollback

A rollback to generation Sk is one more canonical transition, from Sn to Sn+1. The tree of Sn+1 is equal to the tree of Sk. It takes the same path as every promotion, with the approval policy and the release.

Stuck releases

Every 5 minutes, the event router sweep queues release.recover for each Change that is releasing for more than 45 minutes (RELEASE_STUCK_AFTER_MS). It queues at most 10 per sweep. The integration module then does one of these actions.

• A finished release settles the Change to completed or failed.
• A release that is still queued or building is marked failed, and the Change fails.
• A Project without a live app completes the Change.
• A Project with a live app but no release gets a failed release record.

The sweep runs only while the stack runs. After bun run stack:up, the next sweep does the recovery.

2. Check a release

1. Open the Candidate page. The release panel shows the status, the version, and the smoke Evidence.
2. Run bun run e2e local (local-stack.md). It checks the API routes for apps, versions, code reads, and the rollback refusals.