Risk one
Unreviewable volume
Generated changes arrive faster than people can read them. Nobody reads what they did not write, and review becomes a rubber stamp on work no human has actually inspected.
Matthew Dresden · DevBench · Technical overview
AI can write the code. Something still has to prove it is safe to merge.
DevBench is an open-source orchestration system that drives a structured backlog from claim to merged pull request. Development agents write code. Judge agents review it. No work unit closes until five independent verdicts pass. Where a human signs off is a setting you control, not a change to the software.
This is the technical overview: what the system enforces, what you configure, and what it deliberately does not do. The engineering view carries the component-level reference, and the architecture diagram is clickable down to individual hooks and edges.
Green: shipping today in the open-source repository Purple: proposed enterprise vision, not built
The problem
The risk is not that AI writes code. It is that nobody reads it.
Ungoverned AI coding produces output faster than any team can review it. The failure mode is quiet: work looks finished, the audit trail is a paragraph of confident prose, and the first real review happens in production.
Risk two
Scope drift
An agent asked to change one file changes five. Without a contract declared before the work starts, there is no way to tell the difference between the fix you asked for and everything else that came with it.
Risk three
Self-reported success
"Tests pass" is a sentence, not evidence. When the same agent that wrote the change also declares the change good, the review step exists on paper only.
DevBench's answer, in one line: nothing an agent says is load-bearing. The loop advances on artifacts it can check for itself: files on disk, process exit codes, and reviewer verdicts recorded through the devbench CLI. Every one of those calls passes a guard before it runs. An agent claiming success moves nothing.
Governance by construction
These are properties of the system, not rules people are asked to follow
Every control below is enforced by the software at the moment an agent tries to act. There is no honour system and no configuration required to switch these on.
The worker cannot grade its own work Enforced
The executor agent implements the change. A PreToolUse guard rejects its log-verdict call outright if it names any of the five canonical reviewer verdicts. The only verdict name the executor may write is its own audit-only executor entry, and that name does not satisfy the done-gate.
Done requires five independent passes Enforced
A work unit cannot be marked done unless the most recent review round carries a passing verdict from all four review judges and from the security reviewer. A rejection resets the round, so every judge must re-approve the fixed work.
Agents cannot exceed their authority Enforced
Seven guard scripts sit in front of tool calls. The review supervisor is read-only and may only spawn the four named review judges. Destructive git commands are refused with a named safe alternative. Work-unit files cannot be edited with the Write or Edit tool by any agent: guard-work-unit-write.sh rejects any such call against a work-unit .md file under backlog/. Status transitions, verdicts and comments are the sanctioned path through the devbench command line, where the format guards inspect them. The guards cover tool-call writes; a shell-level edit such as sed -i is a bypass the hook does not see, documented as deliberate in the architecture decision record that introduced the role-scoped bypass.
Evidence is injected, not asserted Enforced
Nine of the ten agents run with no file read, write, or search tools at all: only the executor can touch the filesystem directly. What they see is the output of devbench read-unit and get-diff (and run-tests for the test judge), executed and inlined before the prompt is evaluated. They act only through the command line.
And no text an agent writes can steer the loop. The orchestrator stops for exactly two reasons: devbench next reports no actionable work, or the stop-hook circuit breaker trips. A reviewer writing "halt orchestration" into a comment halts nothing, because a PreToolUse guard rejects eleven such control phrases before the comment is ever written. The recovery paths branch the same way: on whether an artifact exists on disk, not on what an agent called its own verdict.
"All review decisions come from Claude LLM evaluation; there are no hard-coded pass/fail rules." DevBench README, project self-description
Every verdict is a judgment made by a reviewing model against a written rubric, then recorded as a timestamped comment on the work unit. The gates that count those verdicts are deterministic code.
What must be true before a change merges
-
The change matches what was declared
Every work unit declares a Changes Manifest before implementation starts. At commit time, every staged path must appear in that manifest. The check is skipped, with a logged warning, only when the work-unit file cannot be resolved. The assertion is deterministic, with no model involved:
assert_staged_compares the staged set against the declared set and one out-of-scope path stops the commit.matches_ manifest -
The tests actually pass
Implementation follows a test-first cycle, and each phase is logged as structured evidence. A
PostToolUsehook,assert-tests-pass.sh, reads the test command's own exit code and blocks progress past a non-zero one. The agent's summary of the run is not consulted. -
Four review judges pass, in parallel
Code quality, test discipline, documentation accuracy, and declared-versus-actual changes are each reviewed by a separate judge running concurrently. All four must pass. A failing judge writes its objections to a per-judge JSON file that the executor reads on its next attempt, and the cycle repeats within the retry budget, ten attempts by default.
-
The security reviewer passes
Security runs once, on its own, after the four review judges pass. It reviews against SOC 2, PCI DSS, FINRA, SEC, GDPR, CCPA and SOX expectations, and fails on any critical or high finding. It is never retried automatically: a security failure blocks the work unit for a person.
-
Continuous integration is green
The pull request is opened and checks are watched to completion. If
gh pr checksreports no checks while a workflow file exists in the repository, DevBench retries rather than accepting the silence, and refuses the merge if registration still cannot be confirmed. Then, and only then, the merge happens under the configured strategy.
Closing the work unit re-checks all five verdicts a second time. If any is missing from the current round, the close is refused.
5passing verdicts required before a work unit can be closed: four review judges plus the security reviewer
10specialist agents in the execution plugin, each with a declared tool policy
7guard scripts that inspect agent tool calls before they run
16notification event types, each independently toggled and all off by default
Human control
You decide where the line is, and the line is configuration
DevBench does not have one fixed level of autonomy. It has a set of checkpoints you turn on and off in a single configuration file or through environment variables. Moving a checkpoint never means changing the software.
How work reaches the main branch
Four mutually exclusive delivery modes, from fully manual to fully autonomous.
| Mode | What happens | Who merges |
|---|---|---|
| Multi-pull-request default |
One branch and one pull request per work unit, merged once its checks are green. | DevBench |
| Single-branch batch | Every work unit commits to one shared branch. Nothing is pushed until the batch is finalized, producing one pull request per repository. | Human |
| Pause before merge | The pull request is pushed and checks are watched to green, then the work unit parks in an in-review state and the orchestrator moves on to other work. Set by git_ops.pause_before_merge. |
Human |
| Local only | No remote, no push, no pull request, no continuous integration. Commits stay on the machine. | Nothing leaves |
Two opt-in toggles layer onto batch mode: one opens the batch pull request automatically once every unit is finished, the other merges it once checks are green. Both are off by default, which means a human merges by default in batch mode. In pause-before-merge mode the orchestrator never sits waiting on a person: it reports how many units are awaiting a human merge and exits cleanly, then reconciles the merge whenever it happens via devbench check-merge.
The other places a person can stand
Intake approval
Set backlog.default_ to draft and new work units land unclaimed. Draft units are invisible to the orchestrator until a person promotes them, so an operator can review every generated task before autonomous execution begins.
Human review blocks the merge
Turn on git_ops.pr_review_resolution and a GitHub CHANGES_REQUESTED decision hard-blocks the merge, regardless of anything else. Named review bots can block too. A settle window, 60 seconds by default, waits for asynchronous review comments to land before deciding.
Scope authorization
An operator restricts which parts of the backlog a run may touch, by identifier or by range. A second session whose scope overlaps an active one is refused unless overlap is explicitly allowed.
Proposal approval
When DevBench generates new work units from a rejected change request, task_factory. decides whether they are accepted automatically or wait for an explicit human promote or reject decision. It defaults to automatic acceptance, so this is a gate you opt into.
Bulk-change confirmation
Status changes affecting more than backlog.bulk_ work units, ten by default, prompt for confirmation and append to an audit log. Set the threshold to zero to be asked every time.
Notifications on your terms
Sixteen event types cover completions, every blocked classification, pull requests, check failures and orchestrator stops. Three separate switches must all be on before a single message is sent, and every one is off by default.
Automation is bounded, and the bounds are yours
| Bound | Default | What happens when it is reached |
|---|---|---|
| Executor retry budget, shared across review failures, check failures and review-comment cycles | 10 | The work unit is blocked and handed to an operator. Per-judge budgets can be set individually. |
| Recovery cascade depth, capping recovery work generated from recovery work | 2 | The cascade stops. No further recovery layer is generated and the source task is handed to an operator. |
| Recovery window, how long an in-flight recovery may run before it is treated as stuck | 30 minutes | The work unit is reclassified as needing operator action, converting a silent wait into a visible alert. |
| Loop circuit breaker, consecutive continuation blocks inside a quiet window | 5 in 180 seconds | The breaker trips, the loop is allowed to stop, and an audit comment records that human intervention may be needed. |
| Self-restart cap after a known agent-runtime fault | 3 | The launcher fails fast with an explicit instruction to investigate, rather than restarting forever. |
One caveat on how coarse these bounds can get. An unattended run (make start) executes with Claude Code's permission prompts bypassed by design; there is no per-tool-call approval mode for it. A live observation session (make start-interactive) can be launched without the permission-bypass flag, in which case Claude Code prompts before every sensitive tool use, but the repository documents that mode as observation-oriented rather than operational. For a real first week, the controls that bound the run are the delivery, intake and proposal gates above.
Nothing stalls silently
When work stops, someone owns it
Every blocked work unit is classified into exactly one of seven states by a single deterministic function, classify_blocked_task, which is first-match-wins across a fixed priority order. Three always require a person. Four clear themselves, inside limits you set. Each state has its own alert, its own panel in the status report, and its own resolution instruction written for the operator.
7
defined block-resolution paths. There is no eighth, unclassified, "we are not sure why it stopped" state.
Always a person Three of the seven
Human owns it
On hold
An operator deliberately paused this unit, usually because it is under debate. The orchestrator skips it. Only an operator resumes it.
Human owns it
Blocked on a held unit
This unit is waiting on something a person paused. Because a hold is not a finished state, no automation can clear it. Release the other unit and the dependency cascade re-queues this one automatically.
Human owns it
Operator action required
The catch-all: no automation exists that will move this. Reached by an exhausted retry budget, a security failure, a hard delivery failure, or a pull request closed without merging. The report names which of those four it was.
Clears itself, within limits Four of the seven
Automatic
Awaiting dependency
Another work unit must finish first. When it does, this one is re-queued automatically. No human action is required.
Automatic
Auto-clearing via proposal
Recovery work has been generated and wired in as a dependency. The moment all of it is finished, this unit re-queues itself. Turn off automatic proposal acceptance and this becomes a human approval gate instead.
Automatic, then human
Awaiting amendment recovery
Evidence on disk shows a recovery agent is mid-flight, so the orchestrator's next sweep will advance it. If that has not happened inside the 30-minute recovery window, the unit is reclassified as needing operator action.
Automatic, then human
Runtime degradation
A known agent-runtime fault where a reviewing session loses the ability to dispatch reviewers at all. DevBench detects the empty reviewer list rather than reading it as approval, restarts itself up to three times, and hands over to a person once that cap is spent.
Two safeguards worth knowing. A blocked unit is never reported as passing review: the reviewing supervisor is instructed to confirm it can dispatch reviewers before dispatching any, and the [BLOCKED] agent-tool-unavailable comment it writes is what the classifier reads to bucket the unit as runtime degradation. And the system will not restart itself while any unit is waiting on operator action, so automation never runs ahead of a decision you have not made yet.
Adoption
Adopt it at the speed of your own confidence
The sensible rollout is not a switch. It is a handful of settings, changed over three phases, each one widening automation only after the previous phase has produced evidence.
Review everything
- Set new work units to land as drafts. A person promotes each one before it is eligible.
- Pause before merge (
git_ops.pause_before_merge): every pull request waits for a human. - Notifications on for blocked states and pull-request events.
- Scope limited to a single epic.
DevBench does the work. You do the merging, and you read every diff.
Spot-check
- Move to batch delivery: one pull request per repository, still merged by a person.
- Turn on pull-request review resolution and its
decision_blocksflag, so a requested change hard-blocks the merge. - Widen scope from one epic to several.
- Sample work units for deep review instead of reading all of them.
You stop reading every line and start auditing the evidence trail.
Widen automation
- Enable
git_ops.auto_finalize, thengit_ops.auto_mergeonce green checks have earned it. - Keep the security gate, the manifest assertion and continuous integration exactly where they were.
- Keep the operator alerts on, because blocked work still needs an owner.
A clean batch lands without a hand on it. Nothing about the review gates changed.
The checkpoints are configuration, not code changes. Every step above is a line in one configuration file or an environment variable. The configuration is validated the moment it loads: unknown keys and illegal combinations fail immediately with a named error, so a misconfigured trust level cannot be discovered halfway through a run.
Honest scope
What DevBench is today
DevBench is a working open-source system, not a platform. Being precise about its edges is part of the trust argument.
Today
- Open source, Apache-2.0. The code, the agent prompts, the guard scripts and the configuration schema are all public and readable at github.com/matthew-dresden/devbench.
- Runs on a developer workstation or a virtual machine. Optional infrastructure code exists for provisioning a remote per-developer instance, but the project states plainly that you can run the full loop locally without any AWS infrastructure.
- One workspace, one backlog. A workspace holds the backlog and the target repository clones side by side. More than one orchestrator session can run against that backlog concurrently, and overlapping scopes are refused unless you allow them explicitly.
- State lives on disk. Every judge verdict, test-first phase and status transition is written as a timestamped comment on the work-unit file, and every tool call is appended to a separate hook event log. The orchestrator resumes from any point after a restart, because nothing important is held in memory.
- Observable from the command line. A streaming progress report, a live activity dashboard, a hook-level event stream, and per-run cost attribution.
Not in scope today
- Nothing deploys. The pipeline terminates at merge. There is no release, promotion or deployment stage anywhere in the pipeline. A release-tag helper exists in the code but has no caller outside tests.
- No web interface. Operation and observability are entirely command-line today.
- No hosted service. There is no multi-tenant control plane, no shared queue and no central identity model.
- One notification transport. Slack is the only endpoint implemented.
- Not a replacement for your delivery process. It plugs into the pull request and continuous-integration process you already have: it opens pull requests and merges through the GitHub command line under the merge strategy you configure.
Vision: proposed architecture
What the proven core could become at enterprise scale
Everything above exists today. What follows does not, and is presented as a proposal rather than a product. The same governed core, run as a shared service: the orchestrator as a containerized service instead of a process on a workstation, a backlog store with a web interface, a portal that shows what is blocked and what it is costing, multiple teams and tenants with the identity, access-control and audit expectations that implies, and the same progressive trust model applied across an organization instead of a single workspace.
The distinction matters. Nothing on this page describing today's system depends on any of it.
Read the enterprise vision