The complete technical reference

System overview

DevBench is three phases sharing one filesystem: a human-driven authoring phase that produces a spec and a backlog, an autonomous orchestration phase that executes one work unit at a time under judged review, and a gitops phase that commits, opens the PR, watches CI and merges. Each phase ships as its own artifact.

The orchestrator is not a daemon

State lives on disk, in markdown

Markdown work-unit files are the source of truth. BACKLOG.md is a derived index; validate-backlog reports drift between them as an error rather than auto-correcting, and the parser logs a warning and trusts the file. Work-unit files carry no YAML frontmatter: status, repo, branch, dependencies and acceptance criteria are parsed out of markdown by regex.

There is no message broker, no database server and no external issue tracker in the data path. SQLite is present, but only as a rebuildable report cache (section 9). Because state is on disk and every agent action appends a timestamped audit comment, the loop resumes from any point after a restart.

Per-work-unit pipeline

validate-backlog → next → claim → ensure-branch → executor (TDD)
   → [manifest-amender, conditional] → review-supervisor (4 judges in parallel)
   → security-reviewer → git-ops (commit / push / PR / CI / merge) → mark-done → loop

The 10 agents

Ten agent prompt files ship in the orchestrate plugin: six at the top level and four in agents/review_team/. Each takes a single argument (a work-unit ID) and each opens with an ## Evidence block whose commands are executed and inlined before the prompt is evaluated, so agents reason over CLI output rather than over free filesystem access.

The pipeline

Three hard invariants the skill states

• "Never bypass the done-gate -- review-supervisor must pass before git-ops."
• "Security review runs exactly once per work unit -- after review-supervisor passes."
• "The retry loop (step 6) re-runs only review-supervisor, never security-reviewer."

Verdict names, rubrics and rejection vocabularies

Verdicts must be logged under underscored canonical names, not the agents' hyphenated frontmatter names; the done-gate parses only the underscored forms, so a hyphenated verdict means mark-done can never succeed. Five names satisfy the gate; four more are accepted by log-verdict as audit-only.

Evidence injection

Every agent opens with an ## Evidence section whose backtick-bang lines run before the prompt is evaluated. That is how Bash-only agents see code at all:

read-unit

work-unit content, repo_path and work_unit_path as JSON (injected for all ten agents). The four review-team judges (plus manifest-amender and task-factory) pass --strip-comments so prior verdicts do not bias them; security-reviewer, review-supervisor, blocker-resolver and executor read the unit with its comment history intact for their injected evidence; blocker-resolver additionally issues a stripped read inside its recovery recipe blocker-resolver.md:87

get-diff

the authoritative scope source for every judge, mode-aware per ADR-12, so judges never compute scope with a raw git diff origin/main

run-tests

injected only for test-reviewer

proposal / amendment JSON

cat-ed directly for task-factory and manifest-amender

Hook system: the deterministic floor

Agent prompts are instructions; hooks are enforcement. One file (hooks/hooks.json) registers 10 event types, 14 matcher groups and 21 command registrations, all pointing at ${CLAUDE_PLUGIN_ROOT}/scripts/*.sh. The contract is blunt: exit 0 allows the tool call, exit 2 blocks it and the guard's stderr becomes Claude's feedback. The Stop hook is the exception: it emits a JSON decision envelope instead.

Complete registration map

These ten event names are registered handlers. Whether a given Claude Code build emits all ten was not verified; the safe phrasing is "the plugin registers handlers for ten hook event types".

What each guard enforces

The Stop hook: what keeps a prompt-driven loop running

continue-orchestration.sh is the only Stop handler and the only script that emits a JSON decision envelope: {"decision": "block", "reason": "…", "hookSpecificOutput": {…}}. It always exits 0: the block lives in the JSON, not the exit code. Its purpose, per its own header, is to "prevent the orchestrator from stopping mid-loop" and to inject a continuation instruction "so Claude re-enters the loop without human intervention".

Circuit breaker and forensics

The counter resets when the window expires. At max_blocks the breaker allows the stop, writes [CIRCUIT_BREAKER] Allowed stop after N blocks in Ns. Human intervention may be needed. as an audit comment and clears its state file (/tmp/devbench-stop-hook-state-<session>.json, per-session so concurrent orchestrators keep independent counters). Every invocation also writes a forensic JSON file to <workspace>/.devbench/stop-hook-diag/<timestamp>-<task-id>.json capturing the task, last action, block count, window, elapsed seconds, stale warning, next step and the exact stdout emitted. The stated purpose: "so a future hang can be post-mortemed from evidence rather than speculation."

Why the guards are written in jq and pure bash

Three separate real-bug fixes are encoded in the scripts and worth citing when people ask why hook code looks defensive: python3 resolution "fails silently under asdf-shim PATHs", which made guards bow out and let dangerous calls through, so jq is tried first everywhere; local -n namerefs broke every PreToolUse hook on stock macOS bash 3.2 and were replaced with indirect reads; and a shlex.split tokeniser was silently returning zero tokens, causing legitimate verdicts to be blocked as "missing arguments".

Skills: five, across two plugins

A skill is a SKILL.md prompt Claude Code loads by name. DevBench ships five: four authoring skills that a human drives interactively, and one execution skill that is the orchestration loop. Amendment handling, block resolution, reporting and tailing are not skills: they are agents and CLI commands.

The chain

Correct invocation names post-0.4.0: devbench-authoring:create-spec, devbench-authoring:spec-to-backlog, devbench-authoring:configure-devbench, devbench-authoring:bootstrap-environment, devbench-orchestrate:orchestrate. Only the last one is automatic: devbench start issues its prompt for you.

What the spec must contain

Eighteen canonical sections under sixteen top-level numbers (0 through 15, with 3.5 and 3.6 as separate sections), every one present or explicitly marked "N/A -- reason" (the skill's own heading says "these 16 top-level sections" and then enumerates 18 bullets): behavior-change items · context · goals · existing primitives to reuse · standards audit · trust model · new command surface · data format · version and interoperability semantics · error handling, logging and configuration · documentation updates · parallel or multi-repo scope · testing requirements · completions/integrations matrix · out of scope · resolved decisions · CLI --help reference · future work.

Authoring is an interview (17 numbered questions in 7 blocks), then section-by-section drafting, then a bounded self-critique loop against an 8-item rubric, then operator sign-off before the file is written. The skill emits a [QUALITY_REFERENCE] audit line naming the exemplar it used, or the literal token <embedded-canonical-sections> when none was configured, so the audit trail proves no external exemplar was consulted.

What the backlog must contain

Fifteen canonical sections per task file, in required order: title · ## Status: · Target Repository · Description · Definition of Ready (5 task-tailored items) · Depends On This · Approach (task-specific RED/GREEN/REFACTOR steps) · Code Standards (6 subsections) · Related Specifications · Dependencies · Acceptance Criteria · Changes Manifest (exactly 2 columns) · Definition of Done (~9 items) · TDD Cycle Log (header only) · Comments (header only).

Bounded loops, resumable runs

Both authoring skills run an iterate-until-perfect loop bounded by two constants: SKILL_MAX_ITERATIONS = 5 (exhaustion emits [SKILL_MAX_ITERATIONS_REACHED] and exits non-zero) and SKILL_QUALITY_THRESHOLD = 0 (reaching it emits [SKILL_QUALITY_THRESHOLD_REACHED] and exits success). Checkpoints live under .devbench/skill-state/; spec-to-backlog keeps a second per-task checkpoint so a crash resumes mid-backlog instead of regenerating every file. Above skills.fan_out_threshold (default 10) leaf tasks, it spawns one sub-agent per Feature to author that Feature's tasks in parallel.

The orchestrate loop's own discipline

Three CRITICAL directives sit at the top of the execution skill, and all three exist because of observed failure modes: never stop to ask the user (the only valid exits are ALL_DONE or NO_ACTIONABLE); after every Agent tool result the next tool use must be another Agent call or a devbench Bash call ("ending your turn before emitting that next tool call is a loop-exit bug"); and ending a turn with recap prose is forbidden, because Claude Code reads a turn-end-without-tool-call as "agent done" and fires Stop. That last rule is regression-pinned by a test.

Correspondingly, the loop halts on exactly two signals: devbench next returning ALL_DONE/NO_ACTIONABLE, and the stop-hook circuit breaker. Everything a subagent says is diagnostic: "Subagent text is diagnostic, not control flow." That prompt rule is backed by the deterministic guard-comment-format.sh floor.

Work-unit lifecycle

Nine statuses, two of them terminal, two of them claimable. Four exist specifically as human-decision parking states.

Happy path and side branches

draft → in-queue → in-progress → in-review → done

in-queue | in-progress | in-review  → blocked   (non-terminal)
in-queue | in-progress              → hold      (non-terminal)
in-queue | in-progress | blocked    → declined  (terminal)

Claimability and the scheduling truth

Only in-queue and in-progress Task work units are actionable; proposed, draft, hold, blocked, declined and in-review are all inert to the scan. But "the next item" is not simply the earliest queued one:

A dependency is satisfied when its status is done or declined. A dep on an Epic/Feature/Story is satisfied when every descendant task is terminal; a parent with no task descendants is vacuously satisfied. An unknown dep ID is treated as satisfied so a typo cannot deadlock the loop; validate-backlog reports it as an integrity error instead.

Claim is race-safe

devbench claim <id> refuses a work unit whose Changes Manifest still contains a TBD placeholder row (the same rule as validator 19, enforced a second time at claim time), acquires an exclusive flock(BACKLOG.lock), re-reads the on-disk status under the lock, and raises ClaimRaceError without writing if the status changed. Every transition into in-progress appends a [WU_CLAIMED] audit comment stamped with the session name.

The done-gate

mark-done refuses unless the most recent review round carries a [REVIEW_PASS] from all five canonical judges: code_review, test_review, doc_review, changes_manifest and security_review. The round boundary is the most recent [REVIEW_REJECTED] line: the reader walks the Comments section in reverse and stops there. In practice it never stops early, because no code path writes that token. log-verdict emits only REVIEW_PASS or REVIEW_FAIL cli.py:3607, and [REVIEW_REJECTED] appears in src/ only as a read pattern manager.py:1092 · proposal.py:247, so the window degenerates to the whole comment history. Verdicts from executor, blocker_resolver, manifest_amender and task_factory are accepted by log-verdict but do not count.

On success, _set_status writes the status line in the work-unit file, updates the BACKLOG.md row, rewrites the Status Summary table, ticks every checkbox in Acceptance Criteria and Definition of Done, writes the per-task window-stats aggregate, fires the auto-requeue cascades for anything blocked on this unit, and rolls the parent up. Rollup cascades Task → Story → Feature → Epic and deliberately bypasses the done-gate: parent units are structurally done when their children are, and need no judge review.

Retry budgets

Global budget max_executor_retries (default 10, env DEVBENCH_MAX_RETRIES), optionally overridden per judge via max_executor_retries_per_judge; the block trips the moment any single judge's budget is exhausted. CI-failure retries (git-ops rc=2) and PR-bot-feedback retries (rc=3) share the same budget, so total per-task work stays bounded. Security review is never retried.

Block resolution: seven paths

Every work unit whose status is blocked is classified into exactly one of seven states by a single pure function, classify_blocked_task, evaluated first-match-wins in priority order. The classification decides what the operator sees, which notification fires, and whether anything automatic will move the task.

The seven states are mirrored consistently across the operator surfaces: seven sub-buckets in devbench status, seven panels in devbench report (each with a resolution hint), and seven of the sixteen notification events map to them 1:1. The report's panel router raises RuntimeError on an unhandled enum member rather than silently defaulting; the codebase forbids silent fallbacks.

Configuration surface

One file (backlog/config/devbench.yaml), validated against a draft-07 JSON Schema with additionalProperties: false at the root and on every nested object except report.models, which is an intentionally open map of model id → rate table so operators can price new models without a code change. Any other unknown key is a hard failure at load. 24 top-level keys, 112 leaf keys (109 live plus 3 retired-but-schema-accepted so the runtime can fail fast with a message naming the replacement).

Two mandatory environment variables

Both are checked at devbench.config import time and exit 2 with a one-line diagnostic if unset:

DEVBENCH_WORKSPACE_ROOT

Absolute path to the workspace root: the parent directory holding BACKLOG.md, backlog/ and the target repo checkouts as siblings. It is not the backlog repo itself.

DEVBENCH_CLAUDE_MODEL

Model identifier for the SDK caller. It governs the orchestrate skill's coordination calls only; it does not route the work agents, each of which loads its model from plugin frontmatter or an agents: override.

In total, 75 environment variables are read by the Python source, the two above included (64 DEVBENCH_*, 6 legacy JUDGE_*, 5 third-party: AWS_REGION, GH_TOKEN, NO_COLOR, USER, USERNAME), plus 16 read only by shell: the Makefile, the plugin hooks and the EC2 provisioning targets.

The git_ops mode matrix

Four mutually exclusive modes decide PR granularity and who merges. This is the primary human-checkpoint surface in the whole system.

Eight named mutual-exclusion errors, enforced at config load

The human-gate knobs

These are the settings that decide how much rope the system gets. Defaults are the code defaults from config_loader.py. They are worth stating precisely, because two of them are documented backwards in the repo (see section 10).

Scope: authorizing a slice of the backlog

Scope is a printer-pages selector persisted to scope.json. E2 matches epic E2 and every descendant; E1-E3 expands inclusively across epics; E5-F1-S1-T2-T5 expands across tasks; comma-separated tokens are unioned. Reverse ranges and malformed tokens raise immediately; out-of-range tokens warn without aborting. Written either by devbench start --include … or by devbench scope set; the skill honours the file, not the command that wrote it. Before registering a session, start refuses a scope that overlaps an active session unless --allow-overlap is passed.

Per-agent model overrides

Every field in the agents: block defaults to null, meaning "use the plugin frontmatter". Setting one materialises a workspace-local shadow plugin tree and rewrites that agent's model: line. Values containing haiku (in any form, short name, Anthropic ID or Bedrock ARN) raise at config load with no override path.

CLI: 56 commands

A single registry (_COMMANDS: dict[str, tuple[Callable, int, str]]) maps each name to its handler, minimum positional arity and description. 28 of the 56 are "variadic": they own their flag parsing and receive the raw trailing argv; the rest get a fixed-arity slice and warn on extras. Across all of them there are 52 distinct DevBench long flags.

The CLI is deliberately the only way agents touch state. Agents cannot open files or run git freely; they call devbench, and the guards police those calls.

Plus one runtime-support command: prepare-plugin-shadow, which materialises the per-agent model-override shadow plugin and prints its path. 10 + 14 + 8 + 9 + 6 + 8 + 1 = 56.

The git-ops exit-code contract

One command, four outcomes, and the orchestrate skill branches on the number, never on prose.

Deterministic merge rails

Two assertions apply to every commit path and involve no LLM: assert_on_branch rejects orphan-branch commits, and assert_staged_matches_manifest rejects any staged path that is not declared in the Changes Manifest. On top of those, a merge can be stopped by the orphan-pattern gate, by CI (check-registration retry exhaustion refuses the merge, an explicit no-fallback decision), by unresolved PR review feedback, by pause_before_merge, or by batch mode leaving the PR open for a human.

Observability

Three read-only surfaces, each answering a different question, plus an append-only audit trail that lives in the work-unit files themselves.

report

Renders 19 metric rows per window across two windows by default: all-time, and the current run (bounded by a session gap). Four rows are window-agnostic and span the table: recent pace, estimated time to complete, estimated completion date, and estimated total cost at completion. The first line is always a liveness banner ([ORCHESTRATOR ALIVE] / [STOPPED] / [STARTING]), derived from log recency against stop_hook.window_seconds.

Beyond the metrics: in-progress rows carry a humanised attempt duration, plus a Declined panel, a pending-proposals panel, and seven blocked panels, one per BlockedTaskState, each with a resolution hint written for the operator:

hook-tail: the JSON firehose, made readable

hook-tail pretty-tails $DEVBENCH_WORKSPACE_ROOT/hook-logs.jsonl, the raw stream every hook registration writes. Columns: timestamp (HH:MM:SS), a two-character event glyph, agent (12 chars), tool (8), description (120), and a stdout preview (80, PostToolUse rows only). All four widths are operator-tunable via hook_tail.*. Timezone resolves --tz > display_timezone > OS local; the log itself always stores UTC. Rotation is detected by inode change. There are deliberately no filter flags: pipe to grep or jq.

Because the catch-all PreToolUse/PostToolUse matchers are .*, every tool call by every agent is recorded, plus SubagentStart/SubagentStop at each agent boundary and PreCompact before every context compaction. Each record carries an orchestrator_session field so hook-tail can scope its output to the orchestrator's own session: a mid-run investigation in a side pane no longer pollutes the audit stream.

Notifications: 16 events, all off by default

Slack is the only transport. A POST requires three independent switches to all be true: notifications.enabled, notifications.slack.enabled, and the specific notifications.events.<name> toggle. All default to false, and the entire block is commented out in the sample config. Webhook URLs are validated at config load (string or null, https:// only) and are expected to arrive via DEVBENCH_NOTIFICATIONS_SLACK_WEBHOOK_URL, never in tracked YAML.

Firing is transition-gated, not level-gated: a ping fires when the classification actually changes (initial entry counts), and repeated observations are no-ops. The state cache at .devbench/notification-state.json is written on every call regardless of the toggle, so enabling a toggle later does not retro-fire a backlog of alerts. Notifications are dispatched only from write sites, never from the renderers, so re-rendering a report cannot spam a channel. devbench notify-test --event <name> smoke-tests one event.

What is written where

Docs gaps this site closes

This page was generated by reading the repository (source files, plugin prompts, JSON schema, shell guards and the CLI registry) rather than by summarizing the repo's own documentation. That turned up a set of places where the docs lag the code. Every row below was verified against both sides. None of them are defects in the running system; all of them are reasons to trust the code over a prose page.

Two things this site deliberately does not claim