Skip to main content

Model evaluation and accuracy theater are not a defensible practice: why "the AI told me so" fails a clinical decision

OpenEvidence is scaling clinician clinical decision support (CDS) with Anthropic and Penn Medicine, and it leads with rigorous evaluation. Eval proves harness performance. It does not by itself defend the answer that changed care, and "the AI told me so" does not explain a clinical decision.

Statistical accuracy cannot answer why the physician decided. Without reconstructable lineage informing that human decision, the claim that changed care cannot be defended.

What moved

As Fierce Healthcare reported (Heather Landi, 28 September 2026), OpenEvidence is partnering with Anthropic and Penn Medicine to expand AI clinical decision support globally.

The plan includes a specialized free version for lower-resourced regions across Africa and Asia, with a rollout in 100 countries (a Reuters-cited figure, as reported there), local-guideline tailoring, and Penn's Botswana-UPenn design partnership. OpenEvidence will also be available to 10,000 clinicians across Penn's seven-hospital U.S. system. The same report cites 1.12M medically licensed-verified U.S. clinicians and 42M U.S. clinician consultations in August.

OpenEvidence CMO Travis Zack described the work as building and "rigorously evaluating" tools for varied practice conditions worldwide.

Reach is real. The open question is whether evaluation packages travel with the answer.

Honest cuts

Accuracy has become the default assurance in clinical AI sales. Here is what that sounds like, and why it does not hold when care is examined.

"Our model is 99% accurate; you're covered." No. Aggregate accuracy does not answer why the physician made this decision.

"Low hallucination rates and rigorous eval prove our Trust Layer works." No. Harness performance is not what the physician stands on when care is examined.

"If the answer is right, how it got there is secondary." No. The human is still the decision-maker; "the AI told me" fails.

"Accuracy monitoring means you're audit-ready." No. Live metrics do not reconstruct what informed the clinician months later.

"ScientixAI will replace OpenEvidence or validate clinical AI." No. We build evidence infrastructure. We do not replace clinician judgment or speak for regulators.

The problem with this move

We call it structural amnesia.

Eval and accuracy theater answer the question: is the model usually right? When care is challenged, the examiner asks a different one: why did the physician make this decision?

Structural amnesia hurts the physician most. They pick up the pieces and reconstruct what informed that decision. "My 99% accurate AI told me so" is not a defense.

What they need is a reconstructable record: what was retrieved, which guideline version, who attested, and how a wrong answer is recovered.

Examiners hold the human accountable; an AI scoreboard does not transfer that accountability. Regulators have been plain about this mindset in other settings. An FDA warning letter to Purolea Cosmetics Lab (2 April 2026) addressed human clearance of AI aids. It is a different context, not a CDS precedent, but the posture is familiar: the human who relied on the tool still answers for the outcome.

Local-guideline tailoring, model builds, and evaluation packages must be reconstructable objects informing that human decision. Reach without lineage is a governance gap, not a solved evidence network.

What we are

ScientixAI builds evidence infrastructure for healthcare and life sciences: a record you can reconstruct so the claim can be defended when someone must stand behind it, across systems you already run.

We do not sell another CDS chatbot. We make defensibility quantitatively demonstrable, not marketing. That is what we mean by defensible AI.

Why now

Global CDS scale plus local-guideline tailoring multiplies orphan answers unless lineage travels with every answer that changes care. Each new region, each local guideline, each model build adds versions someone may later need to reconstruct.

Architecture chosen this year decides whether expanded reach arrives attested, or expanded with answers no one can reconstruct.

Trust is never the input, evidence is. Let's build evidence infrastructure together.

Become a design partner

We are choosing a small number of design partners to build defensible AI together. Design partners set the order of the roadmap and get evidence infrastructure inside their own boundary while it is being built.

Start the conversation

Subscribe via RSS