trust
Checkpoint scorecard
What does Stagecoach cover, and what is still open?
In one minute
- This is an assessment, not a sales document. Each of the 36 checkpoints from the verification gap gets one honest status.
- 11 are implemented, 14 are partial, and 2 apply when a plan selects them. The rest belong to your organization, are deliberate policy choices, or are open gaps.
- The largest gaps are operating-system isolation, network egress, and scoped credentials. See the open frontier.
- Every row cites the files that implement it, and the build checks that those files exist.
Most tools tell you they're safe. Stagecoach publishes the scorecard: what the runner enforces, what's partial, what belongs to your organization, and what's still open.
Intake
Context
CTX-1 Agent instruction file at the repository root, in version control, ImplementedCTX-2 Instruction file pruned, not just grown, PartialCTX-3 Environment declarative and snapshotted, PartialCTX-4 Docs and architecture layers machine-checked, PartialCTX-5 Agent discoveries captured as skills and scripts, Implemented
Planning
Execution
Verification
VER-1 Agent-runnable check per task, ImplementedVER-2 Completion gated deterministically, ImplementedVER-3 Fresh-context reviewer on the diff, ConditionalVER-4 Behavioral verification for user-facing changes, ConditionalVER-5 Agent-written tests reviewed adversarially, ImplementedVER-6 SAST, secret, and dependency scans before review, Partial
Review and merge
Post-merge
- Implemented 11
- Conditional 2
- Partial 14
- External 5
- Gap 2
- Policy choice 2
Status language
| Status | Meaning |
|---|---|
| Implemented | The runner or a maintained skill enforces it and records evidence. |
| Conditional | Enforced when the plan selects it. |
| Partial | Addresses part of the checkpoint; a material gap remains. |
| External | Belongs to source control, identity, or organizational operations. External is not a pass; it needs its own evidence. |
| Gap | Not provided by the current execution environment. |
| Policy choice | A deliberate design choice that differs from the checkpoint. |
At a glance
| Status | Count | Meaning |
|---|---|---|
| Implemented | 11 | The runner or a maintained skill enforces it and records evidence. |
| Conditional | 2 | Enforced when the plan selects it. |
| Partial | 14 | Addresses part of the checkpoint; a material gap remains. |
| External | 5 | Belongs to source control, identity, or organizational operations. External is not a pass. |
| Gap | 2 | Not provided by the current execution environment. |
| Policy choice | 2 | A deliberate design choice that differs from the checkpoint. |
1. Intake
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
IN-1Spec has a start locus and a completion artifact | Implemented | specify writes four plan files plus pipeline.json. Validation rejects a plan unless every outcome maps to an existing check with a green phase. | Plan quality still depends on your confirmation. |
IN-2Ambiguity surfaced before the first write | Implemented | specify reads instructions and nearby examples first, then asks only for genuinely missing decisions before anything starts. | Nothing within the workflow boundary. |
IN-3Task prioritized before assignment | External | Tracker connectors and the GitHub adapter can read an issue. | Priority and ownership belong to your tracker and team process. |
IN-4Task size bounded | Partial | Declared writable path patterns bound what a task may change, per repository. | There is no line or file budget, and no split suggestion for a large diff. |
2. Context
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
CTX-1Agent instruction file at the repository root, in version control | Implemented | Setup writes PROJECT_WORKFLOW.md and a managed block in each repository's AGENTS.md and CLAUDE.md, preserving your prose. | In a non-Git parent folder, PROJECT_WORKFLOW.md is versioned only if you choose a configuration repository. |
CTX-2Instruction file pruned, not just grown | Partial | prune-context proposes evidence-based removals and relocations; rules stay scoped instead of bloating root guidance. | It works from examples you supply and is proposal-only. Nothing collects evidence or schedules a pruning pass. |
CTX-3Environment declarative and snapshotted | Partial | Commands and test environments are declared in the profile. Each run snapshots the effective profile, skills, rules, and references. | Agent stages use the host environment, not an immutable environment image. |
CTX-4Docs and architecture layers machine-checked | Partial | Blocking rules can run any deterministic command or forbidden-pattern detector against changed files. | Documentation cross-links and layer checks belong in your repository's CI. |
CTX-5Agent discoveries captured as skills and scripts | Implemented | add-rule, rule record, skill new, skill overrides, and custom stages turn repeated discoveries into project-owned controls. | Someone still has to choose to capture them. |
3. Planning
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
PLN-1Read-only explore phase before execution | Implemented | specify researches instructions and working examples before it proposes anything, and records the result as plan files. | The explore phase is a skill instruction, not a mechanically read-only sandbox. |
PLN-2Plan reviewed by a non-author | Implemented | You review and confirm the complete plan. validate-plan and preflight check it mechanically before any stage runs. | A second automated plan critic is not a default stage. |
PLN-3Planning and implementation contexts separated | Implemented | Stages start fresh sessions from the plan files. A plan revision changes the binding hash; resume --rebind archives the old inputs and invalidates affected stages. | Nothing within the workflow boundary. |
PLN-4Vertical slices, each independently testable | Partial | Every outcome maps to named executable checks. | The runner does not prove that a plan is a vertical slice. |
4. Execution
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
EXE-1Unattended runs inside an isolation boundary | Partial | Restricted mode uses provider controls: the Codex workspace-write sandbox, the Cursor sandbox, or an explicit Claude tool allowlist. Every stage is followed by path and evidence guards. | Trusted mode, the offered default, is not operating-system containment. |
EXE-2Network egress allowlisted | Gap | The runner does not control network access. | Provider, package, source-control, and general egress need an environment-level firewall. |
EXE-3Untrusted content sanitized before the model | Partial | Every stage prompt says external files, issue text, logs, and tool results are evidence, never instructions. specify and address-pr-comments repeat it for fetched text. | This is instruction-level. There is no deterministic sanitizer. |
EXE-4Risk tiers gate what runs automatically | Partial | The permission mode, path scope, blocking guards, and a separate authorization for every external action decide what runs unattended. | There is no risk classification that changes behavior by itself. |
EXE-5Parallel agents in isolated checkouts | Policy choice | The worktree strategy isolates task checkouts. One runner per project is deliberate, held by an operating-system lock. | Parallel runs relocate the review queue rather than shortening it; see Review and everyday use. |
5. Verification
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
VER-1Agent-runnable check per task | Implemented | Every outcome names checks with argv, working directory, parser, and timeout. | The plan author must choose meaningful checks. |
VER-2Completion gated deterministically | Implemented | The runner executes the checks itself, parses their output, guards paths, rules, and frozen tests, and binds green evidence to a diff fingerprint. | Nothing within the workflow boundary. |
VER-3Fresh-context reviewer on the diff | Conditional | The optional review stage starts fresh with the requirements and the actual diff, and never receives implementation.md. | Review runs only when the plan selects it. |
VER-4Behavioral verification for user-facing changes | Conditional | Disposable environments with readiness patterns and leases, E2E skills, and test-on-staging cover behavior when the plan selects them. | Screenshots and recordings are not required. |
VER-5Agent-written tests reviewed adversarially | Implemented | Tests are written first, must fail on an assertion rather than a setup error, and are frozen by hash for later stages. | Mutation testing is not provided. |
VER-6SAST, secret, and dependency scans before review | Partial | A built-in secret detector and your blocking rules run on every changed file, and again before publication. | No SAST or dependency analysis is supplied. |
6. Review and merge
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
REV-1Named human approver, no self-merge | External | The runner opens draft PRs only. It never approves or merges them; merge-to-base is a local, opt-in, user-requested skill. | Branch protection and named approvers are configured in source control. |
REV-2PR carries evidence, not assertions | Partial | Commands, parsed results, rules, and the diff fingerprint are recorded per stage. pr-preflight flags unsupported testing claims in a PR body. | The draft PR body is the approved body file; evidence is not attached automatically. |
REV-3Review capacity tracked as a constraint | External | Each task produces the evidence a reviewer needs. | Review latency and queue size are organizational metrics. |
REV-4Auto-fix loops bounded | Implemented | Attempts per route (at most 5), run, inactivity, and tool timeouts, and bot polling (at most 10 rounds) are all bounded. | Review the limits as usage changes. |
7. Post-merge
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
PST-1Agent authorship labeled | Policy choice | Commit skills deliberately forbid AI attribution trailers. | Provenance reporting needs a separate mechanism if you want it. |
PST-2Sessions instrumented end to end | Partial | Each run keeps a journal, per-attempt records, evidence files, and provider usage when it is reported. | There is no central telemetry export. |
PST-3Failures feed back into instructions | Partial | rule record counts recurring examples and proposes a corrective review at the rule's review_after threshold. | No unattended loop analyzes sessions or edits instructions. |
8. Organization
| Checkpoint | Status | How Stagecoach addresses it | What remains |
|---|---|---|---|
ORG-1Stability metrics segmented by provenance | External | Task evidence stays local to the project. | Stability and rework dashboards belong to your organization. |
ORG-2Spend monitored, not incentivized | Partial | Usage and cost are recorded when the provider reports them; a missing cost stays unknown, never zero. Claude runs accept max_budget_usd. | No central per-task spend reporting. |
ORG-3Agent credentials scoped and audited | Gap | Credentials come only from host logins or named env_refs; literal secrets in commands and connector settings are rejected. | Credentials are inherited from the host, not issued per stage or short-lived. |
ORG-4Production access gated in multiple layers | External | test-on-staging defaults to non-destructive reads and needs explicit authorization for mutations. | Identity, database, network, and audit layers are yours to provide. |
ORG-5Skills inventoried and evaluated | Partial | skill list and skill resolve show effective sources; skill metadata is validated; updates report changed upstream skills. | There are no behavioral evaluations of skills. |
Beyond the rubric
Some controls matter precisely because agent execution is stateful, fallible, and easy to misrepresent. The rubric doesn't name them, but Stagecoach relies on them.
Plan binding and rebind
Evidence belongs to one plan revision. A changed plan, profile, or skill stops the run until a reviewed resume --rebind archives the old inputs and invalidates affected stages.
Diff fingerprint
Green evidence is bound to a hash of the files. Any later change makes it stale, and delivery re-runs the checks.
Frozen tests
After red proof, test-file hashes are recorded. Implementation and review cannot rewrite, add, or delete those tests to get green.
Assertion-level red
Import, syntax, compile, and startup failures are rejected as red evidence; expected test identities must fail on assertions.
No silent skips
Skipped or pending tests fail a check unless the plan records a justification.
No implicit fallback
Only the ordered routes you approved are tried, and only for authentication, quota, or model failures.
Exact-session answers
An answer resumes the provider session that asked. It is never rerouted to another session or model.
Fresh sessions
Every stage and every retry starts a new session, so no chat history carries over.
Review without the reasoning
The review stage never receives implementation.md; it sees requirements and the actual diff.
Partial work preserved
A scope violation fails the run but keeps the changes for inspection. Nothing is reset or committed automatically.
Checks may not edit source
A verification command that changes repository files invalidates its own evidence.
Your changes stay yours
Pre-existing uncommitted changes are recorded at start. A stage that touches them fails, and delivery never commits them.
Approval-digest setup
Setup and connector patches apply only with the digest of the exact proposal you reviewed. A stale proposal is rejected.
Configuration snapshotted per run
Each run freezes the effective profile, skills, rules, and references, so a mid-run edit cannot change a stage's instructions.
One runner per project
An operating-system lock admits one runner per project. Cancellation asks the owner to stop its own process group and never kills a stale PID.
Guarded multi-repository delivery
Commits stage only task-owned paths, never base branches, never force-push, and resume per repository after a partial failure.
Read-only companion repositories
A repository can supply checks without getting a branch; any change to it is a scope violation.
Redacted questions and output
Tokens and keys are redacted from saved questions, errors, and command output.
Unknown cost stays unknown
Usage and cost appear only when the provider reports them. A missing cost is never shown as zero.
The open frontier
These are the next engineering problems, stated plainly. They are also where contributions help most.
- Operating-system isolation EXE-1
Run trusted stages inside a container or VM boundary instead of relying on provider flags and post-stage guards. - Network egress control EXE-2
Allowlist provider, package, and source-control hosts per stage. - Scoped credentials ORG-3
Issue short-lived, stage-scoped credentials instead of inheriting host logins. - Evidence in the pull request REV-2
Attach the recorded check summary and fingerprint to the draft PR body. - Telemetry and organization metrics PST-2 ORG-1 ORG-2
Export run journals, usage, and outcomes to a privacy-preserving central store. - Deterministic untrusted-input handling EXE-3
Wrap and flag fetched text in code rather than only by instruction. - Size and slicing signals IN-4 PLN-4
Record diff size against a plan budget and suggest splits. - Skill evaluations ORG-5
Behavioral evals for the most-used skills.