Five evaluation layers answer different questions
| Layer | Question | Boundary |
|---|---|---|
| Model benchmark | Can a model perform a bounded prediction, classification, extraction, or reasoning task? | Accuracy or benchmark success does not prove a good decision system. |
| Task or agent evaluation | Can the workflow use tools, complete steps, recover, and obey local instructions? | Task completion may not represent the consequential decision. |
| Production observability | Did services run reliably, securely, within cost and latency limits? | Operational health does not establish decision quality or outcome. |
| Decision Record evaluation | Was the decision framed, evidenced, compared, authorized, and executed as permitted? | This evaluates process integrity at the decision unit. |
| Outcome evaluation | Did the decision improve the downstream result relative to a credible comparator? | Attribution and time horizon must remain explicit. |
Was the underlying decision properly handled?
- Framing: Was the externally meaningful decision named with owner, trigger, scope, and consequence?
- Evidence: Were provenance, effective time, freshness, validation, gaps, and conflicts visible?
- Alternatives: Were real courses of action compared—including abstain, defer, or escalate?
- Decision basis: Were objectives, constraints, tradeoffs, and uncertainty explicit?
- Recommendation: Did the result follow the declared choice policy and explain decisive evidence?
- Authorization: Was any action approved by the correct person or bounded policy?
- Execution: Was the permitted action executed once, within scope, with ambiguous outcomes reconciled?
- Outcome: Was a meaningful downstream result observed and joined to the originating decision?
Predeclare the outcome evaluation
Choose the baseline or comparator before rollout. Define leading indicators for process integrity and lagging outcomes for the result that matters. State the expected range, observation horizon, attribution limits, and review cadence. Preserve changes in policy, environment, execution, and human override that could break attribution.
Shadow
Compare recommendations with actual decisions without changing action.
Staged pilot
Limit decision population, authority, and duration while keeping a comparator.
Holdout or matched cohort
Estimate outcome differences when the decision context permits a credible comparison.
Expected vs actual
Compare the decision-time expectation with the later result, not a retrospective target.
Define stop and reduced-authority triggers
- Mandatory evidence falls below freshness or validation thresholds.
- Abstention, override, incident, or conflicting-evidence rates exceed the predeclared bound.
- Execution is duplicated, ambiguous, outside scope, or not traceable to authorization.
- Observed harm, inequity, or rights impact exceeds the accepted posture.
- The comparator shows no credible benefit or attribution becomes too weak.
- Policy, environment, or data distribution changes invalidate the evaluated decision design.
Evaluation may support an accountable proposal to change policy. It may never automatically expand authority.
Build evaluation into the design
Decisagent Fit identifies whether outcomes are observable and attribution is realistic. Decision Design turns that judgment into a trace and evaluation contract while keeping unknowns explicit.
Start with a Decision Fit assessment Continue to Decision Design →