Behind the case studies

The systems I build to work with agents

Agents write all my code now. What’s left is knowing what to build and keeping the agents honest. These are the systems I built for both.

Each system below leads with its single most useful idea, so those six lines are the short version. The detail under each is there if you want to check it.

01

Agent-filed issue intake

Agents file their own issues, check them against the backlog, and flag the overlaps.

The cost isn’t fixing issues faster – it’s rebuilding context. Three issues in the same module should pay that cost once.

How it works

New issues get checked against open ones before they are filed. Near-duplicates fold in. Real overlaps — same module, same root cause — get a note on each issue describing the relationship.

The guardrail

Auto-close requires 50 labeled cases with zero wrong closes, each labeled by a person or by a verified panel whose models come from families other than the assessor’s. Append requires at least 20 labeled cases with at most one wrong in every twenty. With zero errors, 20 cases only pin the true error rate near 14%; 50 bring it near 6%. Filing caps at 15 issues a day.

Why this was the hard part

Deciding what the system may close on its own. A wrong close hides open work. A wrong comment is ignorable. Those are not the same risk – they don’t get the same bar.

02

Roadmap re-sort

The roadmap re-ranks itself against the strategy and tells me what to build next.

The sorter is graded on the decision it reached – not on how well it explained itself. The explanation is what you read, which makes it the easiest thing to be fooled by.

How it works

It reads the strategy, the open issues, and the code each one touches, then ranks with reasoning attached. Nothing is written to the board until I approve the exact ranking it proposed. Since November it also says what has to happen before an issue can start: an investigation, a definition, or a product decision from me.

The guardrail

Thin or ambiguous evidence produces recorded uncertainty, not a confident rank. Two model families, Codex and GLM, grade each call – one family grading its own call is not a second opinion.

Why this was the hard part

Separating a good decision from a good write-up. Evidence resolves against a pinned commit, so a rank cannot rest on code that has since changed.

03

Parallel worker coordination

Each issue spins up its own worker in its own worktree. A coordinator runs them in parallel, caps concurrency, and only surfaces to me when a decision is required.

My answer binds to the exact question that triggered it. If the worker has moved on, the answer is refused – not silently delivered into the wrong context.

How it works

The coordinator launches one worker per issue, limits how many run simultaneously, and follows an event stream. A worker’s question reaches me verbatim. A finished worker reports its pull request – nothing more. It can also adopt a session I started by hand, so that worker’s questions and finished turns appear beside the others without it being sent a new prompt.

The guardrail

Every relay carries a token tied to its question – late answers are refused outright. Events are acknowledged in order; replaying one is safe. A send that can’t be confirmed is reported, never retried automatically.

Why this was the hard part

Cheap execution means you can launch more work than you can inspect. This layer’s job is to subtract attention, not add surface area. The coordinator never does a worker’s task, never answers its product questions, never merges. The moment it starts deciding on content, it becomes a shortcut around the gates it exists to feed.

04

Easy-issue auto-fix

An operator-invoked pipeline that reproduces a reported defect as a failing test, evaluates a candidate fix, and prepares it for review with the evidence attached. Paused since October 8, 2026; the unattended tier was never built.

The check must fail before the fix – for exactly the reason the issue predicted. A test that only passes proves nothing unless you watched it fail first.

How it works

Red run at base. Apply the fix. Survival assertions confirm the check wasn’t weakened – same bytes, no dropped cases. Byte-audit the changed files. Replay the whole sequence in a second fresh worktree.

The guardrail

Three outcome classes – verified, refused, and harness-fault – so a tooling problem never reads as a bad fix. Candidate code runs minimal and never sees my credentials. Results arrive as drafts. Before the final attempt I set a stopping rule: a green draft pull request with no hand repairs to the pipeline and no new gap. It did not meet that bar, so I paused it. Restarting needs five eligible issues with authored acceptance criteria open at once, and ordinary debugging runs passing the reproduction gate without help.

Why this was the hard part

Every expensive failure here has been a verification failure, not an implementation one. A rewrite corrupted encoding while every gate stayed green. A cleanup loop deleted a block of tests and the suite kept running – reporting four fewer, no alarm. No gate may report success without having checked. The harness’s hardest job is watching its own gates.

Where it was wrong

800+ unit tests passed before it ran end to end. The first six real runs found six structural defects anyway – including a ledger whose stage order was impossible in either direction. Same root cause every time: fixtures pre-seeding state the real flow was supposed to create. The pause taught the other half: an issue can carry acceptance criteria and still hide an unresolved product decision, so before automating a fix, establish that the criteria describe a settled decision and a checkable outcome. The red-first reproduction gate moved into ordinary debugging.

05

Unattended worktree cleanup

Parallel work leaves finished worktrees behind. A machine-wide sweep reclaims the dead ones without losing anything: agents mark work done, and the sweep owns removal.

Every safety probe can answer unknown, and unknown stops the deletion. Most of the engineering here is refusing to let uncertainty collapse into yes.

How it works

The sweep finds repositories under configured roots and examines their registered worktrees. A session records who created a tree and what for, and marks it done when the work is finished; a tree whose newest file and latest commit are both at least 14 days old is also eligible. Merged, clean, inactive trees are removed and their branches kept. Unfinished work is preserved first: tracked and untracked changes, plus unreachable commits from the reflog, are committed to a local salvage branch, and ignored files that cannot be rebuilt go to a recovery folder. The report records where everything went. Nothing preserved is deleted automatically.

The guardrail

Recovery has a concrete endpoint: a named branch and saved files the operator can restore. Morning reports separate removals, cache clearing, preserved work, and holds, and I read the exceptions. It is still under development; I do not claim sustained unattended reliability or measured time savings.

Why this was the hard part

The only cases that matter are the ones you cannot prove safe. Activity, an open pull request, a lock, an incoming link, a protected path, or uncertain state keeps a tree in place, and the sweep rechecks Git, process, link, and job evidence immediately before it acts, because a tree can change after inventory.

06

Norm, a curated knowledge library

Everything I’ve collected on product, AI, design, and strategy, picked and annotated. It gives the agents a current, nuanced view of the field instead of whatever their training data remembers.

“Nothing matched – treat this as ungrounded” is a required output. A tool that always returns something is indistinguishable from one that fabricates.

How it works

Notes come from podcasts, newsletters, talks, and research I’ve picked, filed under declared topics. Each carries a summary, usage guidance, caveats, and pointers to whatever corrects it. Retrieval merges keyword and semantic recall. Agents pull at defined triggers – not when asked, but when the work demands it.

The guardrail

Consults from other projects are read-only – retrieval cannot rewrite what it cites. Sponsored sources are flagged in the note and count against independence.

Why this was the hard part

Confident retrieval on a topic the library doesn’t hold is indistinguishable from knowledge. So every answer surfaces sources, caveats, and disagreements. Claims are labeled by whether their sources are genuinely independent. Three notes drawn from one interview cannot pose as corroboration.

Which model does what — and what it may not decide

Routing is not about collecting models. Each one owns work it is suited to, and each one has a boundary that keeps it from quietly becoming the deciding voice.

Claude

Owns

Product intent, plan authorship, orchestration, and final synthesis.

Does not decide

No vote on technical deadlocks – those are settled with code, tests, traces, or a contract, not by polling another model.

Codex

Owns

Repository grounding, implementation, debugging, tests, and most code review.

Does not decide

Its review of its own work counts as verification, never independent corroboration.

Astra

Owns

The senior checkpoint when a result can shift an architectural or release decision.

Does not decide

Capped at one full pass plus one bounded delta pass; not spent on formatting or routine fan-out.

GLM

Owns

Independent premise falsification: what framing or assumption is most likely wrong, asked before the solution locks.

Does not decide

Advisory only – never implements, never authorizes scope expansion, and sees the problem without my proposed answer attached.

Fable

Owns

Bounded design arbitration when two viable options diverge in their consequence for the user.

Does not decide

Same model family as the planner, so it contributes design judgment, not independent corroboration.

Kimi

Owns

Advisory scope audit of UI-bearing plans and final UI diffs: which controls, messages, or states exceed what was asked for, and which required ones are missing.

Does not decide

Owns no gate and earns no independence credit. A finding enters a verdict only after a reviewer verifies it against the plan and the source; kept because at its five-PR checkpoint it caught misses on four of the five.

Code gets reviewed by a different model family than the one that wrote it – same-family review inherits the author’s blind spots. Every commit is mine. I read the diff first.

Where I overruled a model

Routing work to the right model is useful. Letting one set product scope is not. A design model once recommended an inspectable eligibility checklist before a user applies. I kept the simpler single check for the first release and moved the richer version into its own plan.

Beliefs that expired

Rules I worked by until the evidence broke them, oldest first.

  1. By June 2025

    Generating the code is the hard part.

    My first AI builds started in a code generator (V0) and ended in disorganized files. Moving code between a chat window and the editor (Claude and Cursor) meant endless copy-paste and error-correction loops.

    The model works in the repository itself, first through a file-server connection (MCP) and now from the terminal (Claude Code).

  2. By June 2025

    The model remembers what it already built.

    On Mixed Signals, my first large AI-built app, the model lost track of its own earlier work, and building the front end and back end at once created cascading errors.

    I build the front end before the back end, and the model works from a written error log and task list instead of its memory.

  3. By June 2025

    Better prompts are enough to control the model.

    Rules documents, research reports, and project personas improved my control of the model, but I still lacked control over how each project started and was tracked.

    Every project starts from four files the model generates and then works against: the project brief, the rules, the plan, and the task list.

  4. By June 2025

    The strongest general model is the right one for every job.

    Building automation workflows (n8n), an agent tool (Manus) produced a working automation and outperformed the other models I tried.

    I pick the model by the job. That habit became the role table on this page.

  5. October 2025

    An agent can grade its own work.

    An agent pipeline I built had an agent check its own output and approve it once its confidence reached 0.90. On one change it reported 0.985 confidence and a passing build while the deploy failed, and the code it had written took six fixes for render and reload loops the next morning.

    I dropped the pipeline after two changes. Code is now reviewed by a different model family than the one that wrote it, and nothing merges without me.

  6. December 2025

    An optimizer makes a prompt better.

    An automatic optimization pass (DSPy) grew a product-requirements prompt from about 1,000 words to about 3,000 without making it better.

    I switched to an optimizer (GEPA) scored on a metric that penalizes a prompt for growing.

  7. By September 2026

    Passing checks mean the code is right.

    A file written by the implementing model (Codex) passed type checks, tests, and lint with corrupted bytes: a stray byte-order mark and double-encoded characters. A reviewer from a different model family caught what every check had passed.

    Review comes from a different model family than the author, and every diff gets a byte check before it is committed.

  8. 2026-09-08

    Each review finding describes a real risk on my machine.

    A reviewer from a different model family, reading code the implementing model (Codex) wrote, found a window where an interrupt would delete the backup of a file the module existed to protect. Nothing else had caught it. Another finding, about backups under a symlinked settings file, did not apply when measured: the file was not a symlink.

    Every review finding is a hypothesis, checked against the code or a measurement before it changes the plan. A cheap guard for a risk that does not apply yet can stay.

  9. 2026-09-13

    A measured zero means the thing never happened.

    A plan to change how my agents decide when to consult my library rested on a count of zero: they never did it on their own. I asked whether we had already built this. A four-minute check of the raw session records found 4, not 0. The counting tool could not see the event it was counting.

    Before a zero becomes a premise, I check that the instrument could have shown anything else.

  10. 2026-09-28

    Calling the coding model through its plugin integration is the simplest route.

    The integration (Codex over MCP) could stall silently, waiting for approvals nobody was there to give, and the vendor later dropped the command it relied on.

    Work goes to the coding model through its command line, under a supervisor that names the model and effort for every job and records what each job did.

  11. 2026-10-15

    More reviewers means better review.

    I audited 119 decisions by the design model (Fable). On design and product questions it found things the others missed. On CI, evaluation, schema, and test-policy questions it mostly agreed with the model that wrote its brief, which is confirmation, not review.

    It stays on design and product-intent decisions and no longer sits on the other panels.

  12. 2026-10-15

    A question I can measure is one I should measure.

    An audit left open whether the senior coding model wrote better code than the routine one. A fair test meant splitting about eight real units of work between them, and with the cost acceptable I chose not to run it.

    I kept the senior model on architecture-bearing work and wrote down what would reopen the question: rising cost or more escaped defects.

  13. 2026-11-03

    Writing down a lesson can wait until merge.

    Of about 14 lesson-capture attempts deferred to merge, none ran.

    When a change adds something the plan did not ask for, or a debugging guess turns out wrong, the capture attempt runs before the PR opens, and the PR records whether a lesson was written or why it was skipped.

  14. 2026-11-03

    A safeguard that answers a real incident is always justified.

    Many of the safeguards in one cleanup tool traced to real failures, yet together they had grown it to about 7,000 lines of code and 14,000 of tests, for a job whose core is cleaning up while saving any unmerged work.

    I adopted a rule to choose the smallest adequate solution: a fix names the safeguards it makes redundant and removes them, and one tested recovery path is enough where harm cannot be undone.

    The rule in my public workflow notes
  15. 2026-11-12

    A model that looks strong belongs in the workflow.

    I gave a new reviewer model (Kimi) a dated trial with an early-stop check instead of a seat. At its five-PR checkpoint it had caught scope misses on four of the five.

    It stays as an advisory reviewer: it owns no gate, and its findings count only after another reviewer checks them against the plan and the source.

Where the evidence comes from

Research and retrieval sit outside the models that do the work. Each one has a job and a boundary.

Norm

My curated knowledge on product, AI, and design – current thinking and pre-worked decisions, caveats and disagreements included.

Returns ungrounded rather than filling gaps from memory. Every cross-project consult is read-only.

Jev

Relevance rankings, checks of supplied evidence, and rubric scoring when a checkable answer could change the next step.

Advisory only; never a review gate.

Manus

Bounded outside research when a decision turns on current evidence the repository cannot supply.

Research context only. Claims that drive a decision get verified before they are used.

Podscan

Practitioner discussion and the language customers actually use, cited to a specific episode.

One person’s account stays an account – never promoted to market evidence.

Monid

A route to structured data or a real API, checked before anyone writes a scraper that breaks next month.

Consulted to find a path, not used in place of a dedicated integration that already exists.

These are not reviews. They supply evidence; the models still have to judge it. A research tool agreeing with me is not corroboration.

Current as of . Auto-fix is paused; the other systems are under active development. The engineering described here is what they are built on.

Building something complex? Let's talk.

Looking for my next Product Engineer role at an AI-native company, on a small team with real ownership.

Get in Touch

Designed & Built by Drew Miller

© 2026. Version 3.3