# case-study
Case study: The Marketplace Rebuild
A real, anonymized commercial rebuild. The product and the people behind it are deliberately unnamed.
The app
A niche two-sided marketplace connecting independent creators with venues and audiences: screenings, ticketing, payouts, and media delivery. Non-public, commercial, in active development.
Why the rebuild happened
The original stack — Node/Express with a relational database and an NFS file store, deployed on EC2 with Docker Compose — could not host the product's core payload: full-length media masters, generated cinema packages, and streaming renditions. Storage was the trigger, and the documented reasons were:
- Large-object storage — masters and renders don't belong in a relational database; they need a dedicated object store with storage classes and lifecycle rules.
- Resumable uploads — big files must stream straight from the browser to storage, not through an app server.
- One shared backend — the whole product family converges on a single platform with one identity layer (MFA), rather than a bespoke per-product stack.
- Nothing to migrate — the relational database held demo/test data, so the decision was "no data survives," not a migration.
- Prototype → production — the original build was a working prototype; this is the step up to real production requirements: scale, reliability, and real money moving through the platform.
Before → after
The agent setup — three models, one human boundary
The process
- -A phased plan (P0–P12), one PR per phase with explicit exit criteria.
- -Decisions of record at the top — settled once, not re-litigated per phase.
- -Every phase ships with tests and a tech-debt quota; a phase that only adds code is incomplete.
- -Pilot runs drive real test-mode payments through a headless browser against the live site, with screenshots and per-step results.
- -~100 PRs over roughly three months, from first phase to production cutover.
What the pilot harness was
The pilot harness was the independent verification layer — the part the agents did not write. It had three pieces: emulator suites (the functions ran against local copies of the cloud services), headless-browser journeys (real sign-up, invite, and booking flows driven against the live site in test mode, with screenshots and per-step results), and an audited security log (every auth-claims write and security event logged and reviewed, not self-reported).
What the pilot harness caught before production
Each of these is a published teardown — the bug, the fix, and the lesson:
- -TD-047 `undefined !== false` — every non-staff sign-in rewrote auth claims and flooded the security log.
- -TD-048 The invite gate fired on a not-yet-loaded profile, bouncing every deep link to the wrong screen.
- -TD-049 A booking could be completed before the night ended and before the money was paid.
These never reached production because the pilot harness — emulator suites, browser journeys, and an audited security log — surfaced them first. That is Detectable, working as designed.
What stayed human
- -Live payment keys and the payment-provider dashboard setup.
- -Frontend deploys (a human runs the deploy script).
- -Genuine business calls — e.g. true cinema-package mastering is a partner job with licensed tooling, not a code change.
The lesson
The marketplace rebuild was the enterprise mode of Part 2 — gated, reviewed, phased — run by a multi-agent team with a human holding the keys. The prototyping mode described in Part 3 is how you loosen that for exploration without losing the safety net.
Together they bound the whole autonomy spectrum:
- 1.Directing the Agent — the per-task discipline.
- 2.Engineering the Agent Environment — the systems discipline (enterprise: gate and verify).
- 3.Engineering the Prototyping Environment — the exploration discipline (prototype: isolate and restore).
The sandbox harness — the source code
The code is the point here. This is the sandbox harness — the autonomous coding loop that writes, tests, and merges. Don't confuse it with the pilot harness above: that one verified; this one builds. Three models, a path-safety filter, independent tests, and a human kill-switch.
The prose stays short; the code is complete below, collapsible per file.
What's what
- architectureThe loop at a glance — Codex initializes, Opus builds, Fable reviews, pytest runs, auto-merge.
- orchestrate.pyThe brain: a bounded build → review loop (max 4 cycles) with a path-safety filter and a full audit trace.
- agent-sandbox.ymlThe GitHub Actions workflow — a fresh runner VM per run; runs the loop, pytest, then opens and merges the PR.
- CODEOWNERSMarks the harness files human-owned, so the agents cannot self-approve changes to their own guardrails.
- emergency-rollback.ymlThe human kill-switch — force-resets main to a known-good SHA, fully outside the agent loop.
- local_test_runner.pyRuns the same loop locally in Docker — no CI minutes, no risk to the remote repo.
- audit traceThe JSON the loop writes after each run — every prompt and completion, replayable for debugging.
- -Branch protection: a pull request is required, and the orchestrate-and-validate status check must pass before merge.
- -A dedicated GitHub App token (not GITHUB_TOKEN) — traceable, and able to merge under branch protection.
- -The path-safety filter aborts with exit 403 on any write into .github/ or any path traversal.
- -Emergency rollback is a separate, human-triggered workflow — git reset --hard with fetch-depth: 0.
A limitation to watch out for
The review gate is still an LLM. Fable's "approved" is a model's opinion, and the pytest step only proves that Opus's own tests pass — LLM checking LLM is not independent verification. Recall Rule 11 (Demand Tests — and Independent Verification): for anything that matters — payments, auth, destructive migrations — add a human review gate or an independent test suite the agents did not write. In this rebuild, that independent layer was the pilot harness.
Last updated: November 6, 2026