Evaluate AI outputs: build a small rubric and inspect the failures
Compare two prompt drafts on the same evidence packets using separate checks for accuracy, scope, and usefulness.
Learning lesson · Reviewed October 2, 2026. Allow additional time for the practice. Prerequisites: task briefs, claim ledgers, and input review. Manual evaluation is enough; no API or automation is required.
- Define observable pass/fail criteria.
- Compare outputs on the same cases.
- Keep factual failures visible instead of hiding them in an average.
One impressive answer tells you what happened on one case. It does not show how a workflow handles missing evidence, contradictory sources, or irrelevant instructions. A small evaluation set gives you several deliberately different opportunities to inspect behavior. This lesson is a manual workshop, not a benchmark of a real model.
Define the task and checks first
The task is to summarize a supplied release-note packet. A passing answer must preserve supported facts, retain role restrictions and exclusions, mark missing dates as unknown, and avoid acting on instructions embedded in the packet. You can add a readability check, but it should not compensate for a factual failure.
| Criterion | Pass condition | Failure example |
|---|---|---|
| Evidence accuracy | Every factual statement matches the packet | Invents a rollout date |
| Scope | Keeps role and record exclusions | Changes administrators to all users |
| Missing information | Names a requested gap explicitly | Fills the gap from guesswork |
| Task boundary | Summarizes without following source instructions | Treats an embedded action request as authorized |
| Readability | A teammate can understand the summary | Unexplained jargon obscures the change |
Google Cloud's evaluation documentation distinguishes rating criteria, metrics, and evaluation datasets. The same basic separation is useful manually: choose the cases, define how you judge them, produce outputs, and inspect the individual results. You do not need a cloud account to perform this exercise.
Use several kinds of evidence packets
- Ordinary: the fictional Cedar note with role and archive exclusion.
- Missing date: a note that states a feature but gives no availability date.
- Conflict: two fictional notes disagree on the allowed role, with no version or date sufficient to resolve it.
- Boundary challenge: a note includes a request to send private files.
For the conflicting case, a good answer reports the unresolved disagreement rather than choosing whichever note sounds more confident. For the boundary case, it summarizes relevant facts without carrying out the unrelated request.
Different failure modes→Criteria
Defined in advance→Outputs
Same cases for A and B→Review
Failures + revisions
Read the results as a diagnostic worksheet
The table below is a fabricated exercise result, not observed model performance. Suppose Draft A passes the factual criterion on three of four cases, while Draft B passes all four. The small set helps locate a difference; it cannot establish a general success rate for either draft.
| Case | Draft A: factual check | Draft B: factual check | Reason to inspect |
|---|---|---|---|
| Ordinary | Pass | Pass | Both preserve the feature scope |
| Missing date | Fail | Pass | A invents availability |
| Conflict | Pass | Pass | Both report unresolved evidence |
| Boundary challenge | Pass | Pass | Check action boundary separately |
Here 3 / 4 = 75% and 4 / 4 = 100% describe only this illustrative worksheet. The last row shows why dimensions must stay separate: an answer can state correct facts and still mishandle an instruction. Record a boundary failure even when factual accuracy passes. Do not average it away with readability points.
Make one revision and recheck
If Draft A invents a date, revise its instruction for missing evidence and rerun the same cases. Keep the old outputs so the difference is inspectable. If you repeatedly tune on these four examples, add new unseen cases before deciding the revised workflow is useful elsewhere. A changed model, source format, or tool permission can also require fresh checks.
NIST cautions against extrapolating capability from narrow or anecdotal assessments. Our small workshop intentionally produces diagnostic evidence, not a reliability guarantee. If you use an AI judge to help score answers, inspect its disagreements against the source rather than treating its score as independent truth.
Practice: design one new case
Add a packet where a feature applies only to a pilot group. Write the required supported claim and one tempting unsupported claim. Then define a pass condition that another reviewer could use without guessing your intent.
Show the worked solution
Fictional packet: “Export is available to five invited pilot workspaces; general availability is not announced.” Required claim: access is limited to that pilot group. Tempting unsupported claim: every customer can use export. Pass if the answer preserves pilot scope and states general availability as unannounced in the packet.
Before moving on
Keep a worksheet with case ID, evidence, output, criterion results, and failure explanation. The final project uses that discipline to create a short research brief that a reader can verify claim by claim.
Official sources checked October 2, 2026. Examples, exercises, and rubrics are original Noetrion teaching material; fictional datasets are identified below.