Shipyard regulates itself.
Here is the wiring.

An autonomous software pipeline is only as trustworthy as its feedback loops. Shipyard is built as six control-loop families — each with a measured signal, an actuator, a hard cap, and a proof. No loop runs without a lid; no lid without a human behind it.

Projectmarcmantei/shipyard

SheetControl loops — 1 of 1

Scalems → weekly

Rev2026-07-23

01 · The loop stack

Six families, nested by timescale

Fast loops sit inside slow ones. Each family regulates one thing, caps itself, and escalates upward — the author is the outermost ring, not a bystander.

per issueminutes–hours

Build loop

Issue → PRD → implement worker → independent review worker → fix rounds → merge. Creator ≠ reviewer ≠ merger; workers run in isolated worktrees with a 30-minute wall clock.

cap: round budget 3–8

per outputseconds

Eval loop

LLM-as-judge scores every outgoing artifact (PRD, review, comment) against a versioned YAML rubric before it ships. Below 0.7: regenerate with the judge's feedback.

cap: 2 retries

10 minheartbeat

Cascade loop

Wave orchestration: the next wave unblocks only when the current one is 100% merged and the system is healthy. Epics close only against a passing acceptance command.

gate: health + acceptance

continuous5–10 min sweeps

Self-healing loop

Metrics, logs, and Docker events feed a supervisor that classifies failures into 9 prioritized root causes and heals cause-specifically — WAL checkpoint, worker cancel, CPU throttle.

act ≥ 2 timeouts, cooldowns

6 hhill-climbing

Engine loop

Clusters production trace failures into patterns and files improvement proposals. It observes everything — its only write action is a GitHub issue labeled needs-approval. It never self-modifies.

gate: ≥ 3 traces + human

outermosthuman ring

Author loop

Every escalation converges on one label — needs-human — triaged first by a meta-observer, then decided by the author with full context. Kill switches work without a redeploy.

backstop for every cap

02 · The inner loop

Review ↔ fix, as a closed control circuit

How does the system know an AI-written fix is actually good? It doesn't count iterations — it measures error, verifies every claim adversarially, and regulates on the trajectory. Confidence is earned from evidence, never self-reported.

The measured loop (blue ring): findings become a verified error signal; the controller regulates on error and confidence and closes the ring through a fresh fix worker. Green exit: convergence. Red rail: five named stop conditions into the human lane.

How the error is measured

error = Σ severity × verification
severity: P1 = 3 · P2 = 1 · P3 = 0
verification: actionable 1.0 · unverified 0.5 · disproved 0

Every finding is attacked before it counts: structural claims get grepped, world-knowledge claims get checked against the deployed repo, behavioral claims get tests executed. Disproved phantoms cost nothing; unproven claims count half. The fixer can dispute a finding — but only with an executable proof.

How confidence is earned

C = 0.40·(1 − error/E₀) + 0.25·judge + 0.20·CI_green + 0.15·test-evidence

red CI → cap 0.40 gate FAIL → cap 0.30 gate unproven → cap 0.60 gate PASS → +0.15

Missing evidence counts zero, not neutral — a round without CI can never score like a round with a green pipeline. Hard disproof dominates everything else. Tier high ≥ 0.75 is the only tier that may auto-merge.

A real trajectory — one PR, four rounds

Round	Error	Confidence	decide()
0	6.0	0.24	continue — first round
1	3.0	0.51	continue — progress
2	1.0	0.68	continue — progress
3	0	0.82	converged → auto-merge

Budget for this PR: ceil(6/2) = 3 fix rounds — earned by the size of the initial problem, not a constant. A one-line fix gets the floor (3); a heavy review gets up to 8. Both bounds are live-tunable Airflow Variables.

The whole state — error per round, finding fingerprints, evidence — is persisted as a hidden comment on the PR itself. The loop survives restarts, and a human can read every decision.

After the merge the loop keeps score on itself: a daily job checks whether merged PRs got reverted or hotfixed, and a weekly calibration holds claimed confidence against reality.

03 · In production

What a loop looks like while it runs

Every review↔fix loop renders as a live card in Mission Control: the verdict with its judge score, the round against its budget, the error and judge trajectories as sparklines, agent count, and the token cost of the whole loop. These are unedited captures from the running system.

Mission Control loop card for mycelium PR 43: verdict APPROVE 0.87, round 2 of 3, 3 agents, 737K tokens for $0.70, error trajectory 2 to 1 to 0 to 0, judge trajectory steady around 0.87, state converged, waiting for merge gates — **Convergence, measured round by round** The review found two weighted findings; two fix rounds drove the error 2 → 1 → 0 while the judge score held at ~0.87. The card shows the loop stayed inside its budget (round 2/3) and what the whole loop cost: 737K tokens, $0.70. State: converged — the auto-merge gates run next.

Mission Control loop card for shipyard PR 446: fixing the merge_reached_production_flow DAG failure, verdict APPROVE 0.91, round 1 of 3, 4 agents, 2.8M tokens for $13.49, error 2 to 0, judge 0.88 to 0.91 — **The platform repairing its own watchdog** The DAG-failure observer flagged a broken production-reconciliation DAG and filed issue #444; this PR is the resulting fix, built and reviewed by the normal loop: error 2 → 0 in a single round, judge rising to 0.91. The system maintains its own control loops through the same pipeline it uses for feature work.

Mission Control loop card for shipyard PR 460, merged: verdict APPROVE 0.92, error 0 in the first round, judge 0.92, one agent — **First-pass clean** Not every loop needs rounds: here the independent review found nothing blocking (error 0, judge 0.92), so the PR converged immediately and merged 16 minutes after dispatch. The trajectory that *didn't* happen is evidence too.

Mission Control loop card for mycelium PR 46, merged: verdict APPROVE 0.86, round 1 of 3, 5 agents, 8.5M tokens for $41.40, error 1 to 0 to 0, judge 0.86 to 0.90 to 0.86 — **Merged, with the full story attached** After the merge the card keeps the evidence: the error trajectory (1 → 0 → 0), the judge series, five agents, and the full loop cost of 8.5M tokens / $41.40 — with a Langfuse link to every underlying LLM call. Cost and quality stay auditable per PR, not just per month.

Mission Control issue row for mycelium issue 47, labels auto-ticket and ai-task: lifecycle strip queued, working, review, merged; escalated, waiting for you for 59 minutes; badge waiting for human decision; button clear needs-human; 93K tokens for $0.08 — **An escalation waiting at the front door** This ticket was auto-filed from review findings, worked, reviewed — and then the loop hit a stop condition and escalated. The lifecycle strip shows exactly where it stands: *escalated — waiting for you*. The human decision is one click (*clear needs-human*), and until it happens, nothing auto-merges.

Mission Control issue row for mycelium issue 51, label auto-ticket: strict-embeddings silently ignored in session bootstrap path; lifecycle queued, not yet dispatched; buttons dispatch agent and escalate — **The system files its own work** A review finding that didn't block the PR became a standalone auto-ticket. It sits queued — visible, costed at zero, untouched — until the dispatcher or a human sends an agent. Nothing the system discovers gets lost in a chat log; everything becomes a tracked, addressable unit of work.

04 · Stop conditions

Five named ways to stop — checked in this order

"Loop four times, then give up" is not control engineering. The controller distinguishes why it stops, and every escalation names the specific finding that is stuck.

Gate failed

verify.yml check exit ≠ 0

The last fix measurably broke the repo's own reference check. Overrides everything — you don't iterate on top of a regression. The human's job is to revert, not to judge prose.

Oscillation

fingerprint returns after "fixed"

A finding disappears and comes back — ping-pong between reviewer and fixer. That's a structural disagreement no further round will resolve.

Budget exhausted

rounds > ceil(E₀/2) ∈ [3,8]

The parametric lid. Escalates with the full error trajectory so the human sees whether it was converging slowly or going nowhere.

No progress

Δerror ≤ 0 ∧ same finding survived

The error didn't fall and the identical finding is still open — the fix channel is producing motion, not progress.

Confidence plateau

ΔC < 0.02 despite a new round

The error may still inch down while the wider evidence — judge, CI, tests — has stopped improving. More rounds now add cost, not confidence. Can also file targeted branch-scoped tickets instead of dead-ending.

✓

Otherwise: continue

error falling, evidence growing

Progress is measured, not assumed. And convergence — error → 0 → APPROVE — exits the loop entirely, through the four auto-merge locks.

05 · The operational inventory

Every loop: signal, actuator, cap, proof

The full fleet of closed loops in production. Each row is a running DAG; each has a scenario test that replays its decision logic — several replay real incidents.

Loop	Cadence	Signal	Actuator	Cap / escalation
The inner circuit
Review↔fix controller	per push	verified weighted error + confidence trajectory	fix worker re-dispatch	budget 3–8 → named-finding escalation
Review-loop watchdog	10 min	push/review timestamps in fix cycles	nudge dead channels, re-trigger lost reviews (SHA-dedup)	2 nudges → needs-human
Eval gate	per output	judge rubric score	regenerate with feedback	2 retries → escalate with transcript
Workers & dispatch
Worker-timeout healer	10 min	timeouts via Loki, 9 root-cause classes	WAL checkpoint · cancel · CPU throttle	acts ≥ 2 timeouts, per-cause cooldown
Progress liveness	per sweep	rising tool calls, not wall clock	alive / stalled (5 min) / dead (30 min) verdicts	feeds healer & preemption
Dispatch reconciliation	5 min	ack without PR/branch	reconcile or re-dispatch	unreconcilable → escalate
Implementation resume	15 min	'implementing' with no PR, no worker	re-dispatch until a PR exists	then human
Stuck-issue observer	15 min	dispatched ticket silent ≥ 90 min	exactly one re-dispatch	1 re-dispatch → needs-human + page
Merge & cascade
Project cascade	10 min	wave completion + system health	unblock next wave	health red → block + page
Acceptance gate	on cascade	exit code of the epic's acceptance command	label + gate epic closure	infra failure → blocked, terminal until human
Closure gate	on close	acceptance criteria of bot-closed issues	bounded reopen	1 reopen → needs-human
Merge-readiness shepherd	6 h	approved-but-blocked PRs	branch update, re-trigger CI	true conflict → named files + needs-human
Transport & deploy
Webhook reconciler	10 min	GitHub delivery status	redelivery	4 attempts → single page
Infrastructure watchdog	5 min	HTTP/TCP health, OAuth expiry, inodes	alert → triage → self-heal	unresolved → page
Deploy drift monitor	hourly	live DAGs vs deploy manifest	drift alert	0 out-of-band edits tolerated
Post-deploy canary	after deploy	live canary checks	FAIL verdict	page — a deploy proves itself in prod
Merge-reached-production	hourly	main HEAD vs deployed commit + smoke checks	reconciliation alert	silent deploy block → loud failure
Meta & grounding
DAG-failure observer	30 min	FAILED runs + diagnosis classifier	retry infra flakes · ticket code errors · page auth	3 tickets, 5 retries per sweep; never itself
Coherence auditor	4 h	contradictions across the dashboard's own payloads	propose only	human or meta-observer disposes
Meta-observer	event + 4 h	every needs-human escalation	resolve resolvable classes, brief the rest	1 re-dispatch → human; modes off/dry_run/active
Post-merge outcomes	daily	reverts, CI failures, hotfixes after merge	outcome log + benchmark proposal	revert → page
Confidence calibration	weekly	Brier score, reliability per decile	empirical recalibration mapping	overconfidence gap > 0.25 → page

06 · The observer hierarchy

Loops that watch loops

Three tiers of observation, with the human on top. The design rule at every tier: pure, unit-tested decision cores; reversible actuators; a hard cap; and a kill switch per observer.

Object observersheal primitives

Watch workers, issues, and events directly: the timeout healer (9 root causes), progress liveness, dispatch reconciliation, the stuck-issue observer, the webhook reconciler. They fix things — restart, re-dispatch, redeliver — under strict caps.

Loop observersguard the loops themselves

Watch the control loops from outside: the review-loop watchdog (the inner loop's blind spots), the merge-readiness shepherd (approved-but-blocked), the DAG-failure observer (the whole fleet, including other observers — never itself), the coherence auditor (the reporting layer's own contradictions).

Meta-observerwatches the escalations

needs-human is not a terminal state — it's this observer's front door. It gathers context, resolves resolvable classes itself (exactly one re-dispatch), and hands everything else to the author as a synthesized brief. Three operating modes, switchable live: off · dry_run · active.

Authorthe terminal backstop

Behind every cap: re-dispatch 1 · nudges 2 · tickets 3 · retries 5 · redeliveries 4. No inner loop can run unbounded without the author hearing about it — and autonomy is granted per issue, never globally.

any loop hits its cap → label: needs-human → meta-observer triage → resolvable? re-dispatch (cap 1) → author: brief + options

07 · Grounding

The loop that audits the loops

A confidence score that is never checked against reality is just a number. Shipyard closes the outermost circle: real merge outcomes feed back into the scoring.

daily

Post-merge outcomes

Every merged PR is re-checked: reverted? CI failed on the merge commit? hotfixed? The verdict is logged next to the loop's final confidence. Reverts page the author and auto-generate a benchmark proposal.

weekly

Confidence calibration

Claimed confidence vs. empirical accuracy, in ten deciles, plus a Brier score. An overconfidence gap > 0.25 pages. Corrections apply only once a decile has ≥ 10 resolved cases — until then, the mapping honestly stays identity.

continuous

Loop KPIs & scenario proofs

Zero-touch completion rate, phantom-finding rate, fix-rounds-to-convergence — plus scenario tests that replay whole loop trajectories (convergence, oscillation, plateau, crash-resume) as invariants. Several replay real incidents.

A control loop without proof is a guess with a cron schedule. — the design rule every loop in this system is held to