Gate
The instrumentation spec and the gate itself — explicit scale / hold / kill thresholds set in advance, plus one worked kill example. No live telemetry, and no fake numbers presented as outcomes.
Event schema
Every run, decision, and resolution emits one structured event. Weekly active users are derived from distinct analyst actors per week after the champion hands off. Actors are opaque hashes — no identity is stored.
{
"event": "run | accept | override | escalate | resolve",
"exception_id": "E-…",
"skill_version": "1.1.0",
"model": "claude-sonnet-5",
"ts": "<ISO-8601>",
"actor": "skill | analyst:<opaque-hash>",
// triage payload (run / escalate)
"classification": "<enum>",
"confidence": "high | medium | low",
"escalated": true,
// decision telemetry (accept / override)
"analyst_accepted": true,
"override_from": "<enum | null>", // what the human changed it to
// resolution telemetry (resolve)
"time_to_resolution_ms": 0,
"baseline_ttr_ms": 0, // matched manual baseline
// adoption telemetry
"champion_handed_off": false // drives WAU-after-handoff
}Scale / hold / kill criteria
Thresholds set in advance — not measured results| Metric | Week 2 | Week 6 | Week 12 |
|---|---|---|---|
Override rate Share of triages an analyst overrides. Ceiling — lower is better. | ≤ 40% | ≤ 25% | ≤ 15% |
Escalation precision Of items the skill escalated, share a human agreed needed it. Floor. | ≥ 60% | ≥ 75% | ≥ 85% |
Cycle-time delta Median time-to-resolution vs. the matched manual baseline. Improvement. | — | ≥ 20% faster | ≥ 30% faster |
WAU after champion handoff Weekly active analysts once the champion steps back, vs. the pre-handoff active set. | n/a | n/a | ≥ 70% retained |
All thresholds met at the checkpoint, and WAU holds after handoff. Fill the next seam (procurement RFP intake).
Any single threshold missed. Diagnose (prompt, rubric, data), re-record, and re-check at the next window before expanding.
Override rate rising across windows, or WAU-after-handoff below 40%. The team has voted with its behavior — retire it.
Worked kill example
A plausible adjacent variant: draft the vendor dispute email once triage confirms an overcharge. On paper it saves the most time of any candidate. In the hypothetical pilot it fails at the week-6 gate: because every draft goes to an external party, analysts read each one end-to-end and rewrite for tone and account specifics. The override/edit rate stays near 60% and cycle time does not improve — the review burden exceeds the drafting time it was meant to remove. The gate says kill, and it is retired rather than defended. The triage skill it sat on top of keeps running; only the drafting layer is cut.
Instrumentation wireframe
Every value on this wireframe is a placeholder. Bars show shape, not data. The dashboard populates only from the event stream above — there is no live telemetry on this site, and none is implied.