Independent AI Evaluation: Read the Access Terms Before the Verdict

A product page says its AI has been independently evaluated. The phrase sounds reassuring, but it leaves several practical questions unanswered. Which version was examined? Could the evaluator choose the tests? Did they see operational evidence or only a prepared report? Could an unfavorable conclusion be published?

These are questions about how evidence was produced. They matter whether you are buying a customer-support tool, reviewing an internal assistant, or reading a frontier-model risk assessment. An external organization can contribute valuable analysis while working within important limits.

October 2026 offers a particularly useful reading exercise. Anthropic announced expanded evaluator access on October 15, with a clarification on October 16. METR then published a review on October 20 that explicitly says it was conducted under earlier, more limited arrangements. A later publication date did not mean the newer access terms applied to that review.

Begin with the exact claim you need to evaluate

“The model is safe” is too broad for a procurement decision. “The assistant must not submit a refund without approval” identifies an action, a boundary, and an observable outcome. A report about a different capability may provide context without answering that question.

Write a short claim before reading the headline score. Name the system version, tools, user permissions, task, relevant failure, and setting. Then look for corresponding evidence. If the report tests text responses without the tools enabled, it cannot directly establish the safety of a tool-enabled deployment.

This framing also makes missing evidence easier to discuss. Instead of dismissing a study as irrelevant, specify the transfer you would have to assume: from one model version to another, a laboratory setup to a live queue, or a constrained task to a longer workflow.

Keep publication time separate from assessment time

The October 20 METR review concerns a section of Anthropic's August 2026 risk report. Its scope statement says the review relied on that internal report and did not receive the access described in the later announcement. Its disclosure also says Anthropic had publication-veto and redaction rights for this review; no redactions were made to the review, and its conclusions were unchanged during publication review.

Those details are not interchangeable. A right to intervene describes the process. Whether that right was exercised describes what happened in this instance. The reader needs both. The disclosure does not establish that a negative finding was suppressed, and the absence of redactions does not erase the contractual restriction.

For the expanded arrangement, Anthropic describes employee-like access for a small METR team and publication without a company veto, subject to predefined redaction categories whose application Anthropic ultimately decides. That is the company's description of an arrangement, not a result proving that a model is safe. Read the announcement and its updated footnote together.

Use a report-specific evidence worksheet

METR's October 14 explanation of its approach emphasizes making processes, restrictions, evidence limits, and relevant conflicts visible. You can apply the same reading discipline to ordinary commercial evaluations without assuming they all need identical institutional arrangements.

The worksheet below is an original editorial aid. It is not an accreditation scheme or a numerical independence rating. Keep an unresolved answer visible rather than converting a blank cell into an implied pass.

Attach a page or section reference to each answer. A general partnership announcement should not fill a report-specific access cell unless the report confirms that the arrangement applied to its assessment.

Do not turn access into a guarantee

More access can allow stronger questions. It does not guarantee that every important risk was investigated or that the evaluator's method would detect it. Expertise, time, task design, and the ability to investigate unexpected findings still matter.

Consider a fictional invoice assistant. A reviewer sees a useful demonstration and a summary of successful runs. Another reviewer can examine failed runs, reproduce the configuration, vary invoice layouts, and inspect the approval log. The second arrangement supports more direct investigation of failure pathways, but its conclusions still depend on the cases and methods used.

That difference should affect the wording of a decision memo. “We reproduced the authorization boundary on the listed cases” is assessable. “An external reviewer attended, therefore the workflow is safe” substitutes a relationship for a tested claim.

For one concrete technical boundary, our RAG prompt-injection guide separates retrieved evidence from authority to act. An evaluation should observe the relevant action, not merely a reassuring explanation from the assistant.

Read gains and regressions in the same cases

Independence and test design are separate dimensions. Even a transparent report can hide a useful distinction inside an aggregate score. Use this original fictional comparison of an incumbent and a candidate on the same 100 cases. Each result is a binary pass or fail against a predeclared rubric.

The incumbent passes 70 + 8 = 78 cases. The candidate passes 70 + 12 = 82. The improvement is four percentage points. Yet eight previously passing cases now fail, and ten remain failures. “Four more passes” does not communicate the entire change.

Suppose the eight regressions all involve refunds above an approval threshold. That would be operationally different from eight minor formatting failures. The aggregate score cannot decide whether those consequences are acceptable. Inspect failures by predeclared severity and workflow slice before recommending a release.

Match the denominator to the conclusion

The fictional 82% result is a statement about these 100 evaluated cases. It is not a measured probability that every future request will succeed. The test set may underrepresent unusual documents, new languages, long sessions, or adversarial inputs. Repeating a stochastic system can also change outcomes on the same case.

Record whether cases were sampled from real work, deliberately constructed, or selected after observing failures. Each choice can be useful, but it answers a different question. A stress suite diagnoses difficult situations; its failure rate need not estimate ordinary production frequency.

If you use repeated trials, report the case count, runs per case, aggregation rule, and treatment of invalid or missing runs. Do not silently discard a tool timeout and calculate success only among completed runs. That changes the operational question from “Did the workflow finish correctly?” to “Was it correct when it finished?”

Our fictional arithmetic is not a statistical significance test. A real claim about improvement needs an appropriate sampling and uncertainty analysis, alongside consequence-based acceptance criteria. Avoid choosing the release rule after seeing which system wins.

Turn the reading exercise into a purchase decision

A small organization can build a one-page evidence request without commissioning a frontier-risk assessment. Start with the workflow you intend to deploy. Ask for the evaluated configuration, test methods, important failure categories, and what has changed since the assessment.

Then run a bounded local check using approved, non-sensitive cases. Keep production permissions disabled until the relevant controls are verified. If the vendor cannot share detailed logs, ask what alternative evidence or supervised demonstration could answer the same question. Missing access is a constraint to manage, not automatic proof of misconduct.

Record the decision and its expiry conditions. A new tool integration, permission change, or model update can invalidate assumptions even when the original report remains available. For broader operational planning, see our AI provider exit exercise.

Ask three questions when the next headline arrives

First, what exactly was evaluated, and when? Second, what could the evaluators observe, test, and publish? Third, which decision does the evidence actually support?

Those questions avoid two unhelpful extremes: accepting every external review as certification, or dismissing constrained reviews as worthless. A clearly limited report can reveal important failures and improve decisions. Its value grows when readers can see how its conclusions were produced.

Sources and review limits

Reviewed October 21, 2026. Primary sources: Anthropic's October 15 announcement, clarified October 16; METR's October 14 approach statement; and METR's October 20 review of the August risk report. These documents describe particular voluntary arrangements and assessments. They do not establish a universal auditing standard. All business scenarios, paired outcomes, and the worksheet in this article are original illustrative material, not reported evaluation results.