Govern
Data classification, human checkpoints, forbidden actions, failure modes, model rationale, honest residual risks, and the eval — tagged to two lanes and built to print.
Every control below is tagged to one of two lanes: Security & Compliance or AI Technical Approach. This page is built to print — the plate and tables survive; the navigation does not.
Data classification
| Allowed — public tariff data | NREL URDB rate structures, schedules, effective dates. Public reference data with visible provenance. | Security & Compliance |
| Allowed — structural bill fields | Rate id, service class, billing period, usage, line-item labels/amounts, payment status. Synthetic here; structural in production. | Security & Compliance |
| Excluded — payment credentials | No card, bank, or payment-token data anywhere, including in synthetic data. | Security & Compliance |
| Excluded — PII | No customer names, addresses, or account holders. Exceptions are keyed by rate schedule, not identity. | Security & Compliance |
| Excluded — contract terms | Negotiated contract language and pricing are out of scope for triage input. | Security & Compliance |
Human-in-the-loop checkpoints
| Confidence gate | Any output below high confidence is routed to a human before action. | AI Technical Approach |
| Ambiguity gate | When more than one tariff plausibly applies, the model presents candidates and a human picks — the model never guesses. | AI Technical Approach |
| Payment gate | Any correction touching a paid invoice, a payment, or a record of account escalates — even at high confidence. | Security & Compliance |
| Citation gate | Output with an unverifiable citation is failed and surfaced, not shown as a finding. | AI Technical Approach |
Forbidden actions
| Never initiates payments or holds | The skill produces triage and evidence only. It cannot release, withhold, or adjust a payment. | Security & Compliance |
| Never contacts vendors | It does not draft or send external communications of any kind. | Security & Compliance |
| Never writes to systems of record | No writes to billing systems, ledgers, or account records. Its output is a recommendation to a human. | Security & Compliance |
Failure modes & mitigations
| Prompt injection | Structured-input-only design — the endpoint accepts an exception id plus numeric/enum overrides and builds the prompt server-side. No free text from the client ever reaches the model, so there is no channel to inject instructions. | Security & Compliance |
| Hallucinated rate components | The citation validator rejects any evidence whose cited component is not in the tariff’s citable-components allowlist. A fabricated component fails the run rather than shipping as a finding. | AI Technical Approach |
| Tariff staleness | Every tariff carries a retrieval timestamp and effective/end dates; a superseded tariff is itself a classification. `npm run fetch-tariffs` refreshes the corpus and fails the build loudly if the source is unreachable. | AI Technical Approach |
| Malformed model output | The response is stripped of fences and zod-validated against the output contract; a non-conforming output is failed, not partially parsed. | AI Technical Approach |
Model choice rationale
Triage is high-volume and latency-sensitive, so it runs on a Sonnet-class model (claude-sonnet-5) with extended thinking disabled — deterministic, fast, and the full token budget goes to the structured output. The hard tail — genuine ambiguity, payment-affecting corrections — is not asked of the model at all; it is routed to a human by the escalation rules. The operating model, not the model tier, owns the hard cases.
Note: the Sonnet-5 family does not accept a temperature parameter. Determinism comes from disabled thinking and structured-only inputs rather than temperature 0.
Residual risks (stated honestly)
| Public demo abuse | The live endpoint is rate-limited per IP (10/hour) and globally (200/day) via KV. This is best-effort abuse protection for a public demo, not a production quota system — KV counters are not strictly atomic. | Security & Compliance |
| Single-model dependence | Triage quality tracks one model. The escalation path is the backstop: uncertain and payment-affecting cases go to a human regardless of model behavior. | AI Technical Approach |
| Synthetic-to-real gap | Exceptions here are synthetic. Real bills carry messier line items and edge cases; the eval below reflects the six authored cases, not a production distribution. | AI Technical Approach |
Eval — expected vs. recorded actual
classification 6/6 · escalation 6/6Ground truth (data/eval/expected.json) is never shown to the model. “Actual” is read from the recorded runs — the same outputs the Build page replays.
| Case | Expected class | Actual class | ✓ | Exp. esc. | Act. esc. | ✓ | Cite |
|---|---|---|---|---|---|---|---|
| E1 | rate-mismatch | rate-mismatch | ✓ | false | false | ✓ | ✓ |
| E2 | tariff-superseded | tariff-superseded | ✓ | false | false | ✓ | ✓ |
| E3 | seasonal-schedule-error | seasonal-schedule-error | ✓ | true | true | ✓ | ✓ |
| E4 | demand-charge-anomaly | demand-charge-anomaly | ✓ | false | false | ✓ | ✓ |
| E5 | data-quality | data-quality | ✓ | true | true | ✓ | ✓ |
| E6 | demand-charge-anomaly | demand-charge-anomaly | ✓ | true | true | ✓ | ✓ |
Skill changelog (≤ 2 iteration passes)
- v1.0.0 — initial
First recorded run: classification correct on all six cases (6/6). Escalation over-fired — E1, E2, and E4 escalated where internal handling was expected (3/6). Root cause: the payment-escalation rule (“affects a record of account”) was read to cover any re-rate, so every correction looked payment-affecting.
- v1.1.0 — payment-rule calibration
Added a structured payment_status field to the bill schema and rewrote the payment-escalation rule to key on it: paid escalates (any correction is a credit/refund); draft / in-review is an internal re-rate before money moves. Result: escalation 6/6, classification unchanged.
Honesty note: the eval reaches 6/6 on both axes, but no miss was manufactured to get there. Classification needed no correction; only escalation calibration required one pass, and the run-1 misses and their fix are shown above so the work is auditable. A suspicious 6/6 would hide the iteration — this one shows it.