Provenance Is Non-Negotiable: Why the Most Accurate Model Still Cannot Defend Its Own Conclusions

The short version

• AI agents now draft, summarize, and recommend inside regulated work, but none of them can show its work on its own. We call this the evidentiary gap.
• In regulated evidence, data without its origin is not just worthless, it is a liability.
• Automating report writing without automating provenance creates a reconstruction tax. The fix is to flip the order: bind, then generate.
• Accuracy is not a defense. Accuracy describes a population. A regulator asks about one patient, one event, one submission.
• Machines draft, only a human can mint. And the building blocks for an evidence chain already exist as open standards.

The evidentiary gap

Here is the pattern our founder, Bo Lora, keeps running into. Provenance is collected carefully at the very first step, at the instrument, the site, the source system. Then it gets left at the door. Ten steps later, someone has to reconstruct it.

The surprising part is that this is not really an interoperability problem. Many of the systems in a clinical program meticulously record who did what and when. But when the data is exported to the next system, that history usually stays behind. We described the human cost of this in our piece on why knowledge graphs stall before the clinic: skilled people doing digital archaeology, digging back through emails, spreadsheets, and old versions to work out what was known and when.

AI makes this harder, not easier, for two reasons.

Agent errors are generative, not forensic. A traditional model that gets something wrong produces one wrong value you can find later. An agent that reasons wrongly produces more material that looks like evidence: a narrative, a citation, a rationale. It is fluent, and it is unverified.

The AI does not sign the record. A person does. Every agent output in a regulated workflow eventually lands on a named human who is accountable for it. When the architecture does not carry the evidence chain, those people become the integration layer, rebuilding defensibility one statement at a time. That is the most expensive integration layer a company can run, and it grows with every new AI use case instead of shrinking.

That is the evidentiary gap. It is not a model-quality problem. It is an architecture problem, and it shows up as cost.

Why "data is the new oil" fails at origin

The slogan made sense for consumer data, where one record of a purchase is much like another. Oil is fungible too: take a barrel away from its wellhead and you still have oil.

Take the origin away from a clinical fact and you have a number. You still hold it, you are still accountable for it, and you can no longer defend it. In regulated evidence, data without its origin is not just worthless, it is a liability. Evidence is the asset. Data is the raw material. Provenance is what turns one into the other.

The reconstruction tax

The most tempting near-term AI use cases in life sciences are about writing: clinical study reports, safety narratives, regulatory responses, first drafts of all kinds. Automating that is genuinely valuable.

But writing is only half the job. Every regulated report still has to arrive with a walkable evidence chain: where each statement came from, what supported it, and who reviewed it. Automating the writing without automating the provenance does not remove the human bottleneck. It multiplies it. Drafts arrive at machine speed, and the evidence behind each one is reconstructed by hand, downstream, by the medical writers and quality staff who have to stand behind it.

Practitioners have described this clearly. Medical-writing and regulatory leaders writing in Applied Clinical Trials explain that authors must annotate each data-related statement to a source, and independent reviewers must then verify the match, a process they call "a common bottleneck for regulatory submissions." They warn that automating content generation without changing how content is written "will only accelerate the process of generating masses of content that authoring teams will then pick over for weeks or months" (Nielsen, de Langsdorff & April, 2025).

Now add the shift many in the industry are talking about: everyone will be a builder. A medical writer can now assemble her own review assistant in an afternoon and produce regulated material at machine scale. If the foundation has no provenance, the reconstruction tax is no longer paid by one central team. It is spread across every builder in the organization. That is why provenance has to be part of the foundation, not an audit step at the end.

Bind, then generate

The answer is not a better checker added after the draft. It is a change in the order of operations. We think of it as claim-bound authoring, in five moves. This is our design proposal, not a description of how the market works today.

1. Bind, then generate. Register every data-dependent claim against evidence that has an address before drafting begins. An unbound statement becomes a defect at intake, not a surprise in week twelve of quality control.
2. Check numbers deterministically. Where structured analysis results exist, compare the numbers in the narrative against them with a simple, repeatable check, not a second model's opinion.
3. Separate drafting from release. Humans and machines propose. A named, accountable professional releases claims into the regulated record. ICH E6(R3) is explicit that a sponsor can transfer trial activities to a service provider, but "the ultimate responsibility for the sponsor's trial-related activities, including... reliability of the trial data, resides with the sponsor" (ICH E6(R3)).
4. Record how the report was produced, not just who edited it. At sign-off, capture the inputs, the tools and model versions, the claims, the people who attested to them, and a way to verify none of it changed later. That is a different object from the edit history of a Word file.
5. Let an upstream change name its victims. When a source analysis is corrected, the claims that depend on it should be flagged until someone re-attests them, rather than quietly living on in a frozen document.

There is one prerequisite. You can only bind a claim to evidence that has an address. CDISC, the clinical data standards body, puts it plainly: trial analysis results "are rarely output in a form that is machine-readable," which makes it hard to automate them, reproduce them, or "trace their origin." That is why CDISC created its Analysis Results Standard. Where results are still locked in PDFs, claim binding falls back into archaeology. That is a publishing problem, not a model problem, and it is worth fixing first.

Regulators have converged on the evidence chain

Provenance in life sciences is not a nice-to-have. It is written into regulation, and the newest guidance on AI points the same way.

• 21 CFR Part 11 requires "secure, computer-generated, time-stamped audit trails," and each electronic signature must be "unique to one individual" (21 CFR Part 11).
• ALCOA+ describes trustworthy data: Attributable, Legible, Contemporaneous, Original, and Accurate, plus Complete, Consistent, Enduring, and Available (MHRA, 2018). The FDA expects data to be kept with "all associated metadata required to reconstruct the CGMP activity" (FDA, 2018).
• The FDA's January 2025 draft guidance sets out a 7-step, risk-based process for establishing the credibility of an AI model's output for a specific context of use (FDA, 2025). To be precise, it does not cover AI used for operational efficiencies such as drafting a submission, where patient safety, quality, and study reliability are not affected. The bar for drafting comes from the EMA, next.
• The EMA's reflection paper on AI, adopted in September 2024, says that because generative models can produce "plausible but erroneous or incomplete output," quality review must ensure "all model-generated text is both factually and syntactically correct before submission for regulatory review" (EMA, 2024).
• On 14 January 2026, the EMA and FDA set out common principles for AI in medicine development (EMA). Principle 6, Data governance and documentation, says: "Data source provenance, processing steps, and analytical decisions are documented in a detailed, traceable, and verifiable manner, in line with GxP requirements" (Guiding Principles of Good AI Practice in Drug Development).
• The EU AI Act, for systems it classes as high-risk, requires that they "technically allow for the automatic recording of events (logs) over the lifetime of the system" (Article 12).
• The European Commission's draft GMP Annex 22 on AI applies to static models with deterministic output, and says models that keep learning during use, or that may give different outputs for identical inputs, should not be used in critical GMP applications (draft Annex 22).
• ICH M15, on model-informed drug development, includes AI and machine learning in scope and says modeling should use a computerized system "that is reliable, reproducible, and traceable" (ICH M15).

And enforcement has arrived. In an April 2026 warning letter, the FDA described a manufacturer that used AI agents to create drug product specifications, procedures, and master production or control records. The agency wrote: "If you use AI as an aid in document creation, you must review the AI generated documents to ensure they were accurate and actually compliant with CGMP." When investigators pointed out a missing process validation, the firm replied that it did not know about the requirement because "the AI agent you used... never told you it was required" (FDA warning letter to Purolea Cosmetics Lab, 2 April 2026).

The FDA did not fault the use of AI. It faulted the missing human review, which metadata alone would not have fixed. The lesson is that the record has to show who used which tool, for what, what it produced, and who checked it.

The words differ: credibility, traceability, logging, reproducibility. The requirement underneath is the same. An AI-assisted output in drug development needs to arrive with an evidence chain someone can walk.

Accuracy is not a defense

The natural response to all this is to point at performance: 98% accuracy, state of the art. That misframes the problem.

Do the arithmetic. A model that is 98% accurate, applied to 10 million records, still gets 200,000 of them wrong. At 99.9%, it is still about 10,000. In safety work, each of those is a patient. When a regulator asks about one specific case, "our model is 98% accurate" is not an answer. The answer is: here is the source, here is how it was processed, here is what supported the conclusion, and here is who reviewed it. That is provenance. Accuracy is a property of a population. An evidence chain is a property of a single statement, and you cannot derive the second from the first.

There is a subtler version of the same problem. A prediction carries its uncertainty: "87% probability this event is serious." An assertion enters the record as fact: "This event is serious." AI systems routinely turn predictions into assertions. The probability is thresholded, the label is recorded, and the uncertainty, along with the data, model, and threshold behind it, disappears.

Five things that look like provenance but are not

1. Explainability is not provenance. Explanation tools answer "why might the model have said this?" They cannot tell a regulator which model version was used, what data went in, or whether a human reviewed the output. Explainability describes what a model might have done. Provenance records what actually happened.

2. Retrieval with citations is not provenance. Retrieval-augmented generation (RAG) gives a model source documents to work from. It helps, but it does not prove the output faithfully reflects those sources. In a preregistered study of legal research tools built on RAG and curated legal content, researchers found the tools hallucinated between 17% and 33% of the time (Magesh et al.). On a long-form question benchmark, even the best models lacked complete citation support 50% of the time (Gao et al., EMNLP 2023). In medicine, an evaluation of seven popular models found that between 50% and 90% of responses were "not fully supported, and sometimes contradicted, by the sources they cite" (Wu et al., Nature Communications, 2025). And a correct answer with a real citation can still be unfaithful: in one controlled study, up to 57% of citations were added after the fact rather than actually used to produce the answer (Wallat et al., "Correctness is not Faithfulness in RAG Attributions"). Provenance has to record which evidence actually produced which claim.

3. A bigger model is not the answer. A fact a model recites from its own training has no source you can point to and no chain you can walk. You cannot attest a model's memory. A larger model that is not grounded in external facts produces more fluent, more confident statements, and none of the evidence a regulator asks for. The hopeful flip side: once the facts live outside the model, with their provenance attached, even a modest model can be both useful and defensible, because the grounding is doing the work.

4. An audit trail is not an evidence chain. Good regulated software already has strong permissions, signatures, and audit trails. Those prove that an authorized person took an authorized step and that it was recorded. They do not prove the value entered was correct, where it came from, or what breaks downstream if it is wrong. Picture a tumor measurement entered by an authorized coordinator, properly signed and time-stamped. Months later a reviewer challenges the progression call. The audit trail shows the entry was valid. It cannot show which scan the measurement came from, who read it, or whether two readers ever disagreed. An audit trail proves the action was valid. An evidence chain proves the fact is true.

5. Metadata is not attestation. A knowledge graph with "source," "confidence," and "validated by" fields on every link looks like it has provenance. But being able to store provenance is not the same as being able to defend it. Four patterns show up even in carefully built systems:

• The "validated by" field simply repeats the source, so there is an origin but no one who vouched for it.
• New values overwrite old ones, so nobody can ask what the system believed at an earlier time.
• Only the time the system stored a fact is recorded, not the time the fact was true in the world.
• The source is a label like "curated" or "model," not an accountable person.

An evidence chain needs a named, accountable asserter, a history that is kept rather than overwritten, two clocks rather than one, and a human at the point where a candidate becomes a fact.

Machines draft, only a human can mint

That last point ties everything together. A machine can draft a claim: find sources, assemble a narrative, propose an answer, much as a human researcher would. But turning a candidate into a fact someone can rely on is not a calculation. It is an attestation: a named person stating, under their identity, at a specific time, and with a clear meaning, that the fact holds. That is the shape Part 11 already gives an electronic signature (21 CFR Part 11). A model cannot be cross-examined. So machines draft, only a human can mint.

Two cautions keep this honest. First, the step that decides whether a human is needed should be a plain completeness check, not a probability score, or you have rebuilt the original problem one layer up. Second, a new synthesis of several attested facts is a new claim. It needs its own human sign-off, even when the model is confident, because the conclusion is what someone will be asked to defend.

The standard already exists

The evidence chain does not need to be invented. W3C PROV, a family of W3C specifications published in April 2013, gives provenance a formal, machine-readable vocabulary (PROV Overview). Its core is simple: entities (a dataset, a model version, a clinical statement), activities (a training run, a classification, a human review), and agents (a person, a pipeline, an AI model, an organization), connected by relationships such as "was generated by," "used," "was derived from," and "was attributed to."

Here is how we map PROV onto ALCOA+. This is our own synthesis, not an official crosswalk, and expressing a principle is not the same as meeting it.

Healthcare has already leaned this way. The HL7 FHIR Provenance resource "is based on the W3C Provenance specification" (HL7 FHIR). And peer-reviewed clinical AI research is reaching the same conclusion from its own direction. One 2026 framework for clinical decision support pairs retrieval with explicit provenance metadata and a tamper-evident audit log, aiming at "traceability, clinician trust, and regulatory readiness" (Alu & Oluwadare, Frontiers in Artificial Intelligence, 2026). Another proposes an "AI Model Passport," a digital identity that lets teams "uniquely identify, verify, trace and monitor AI models across their lifecycle" (Kalokyri et al., Computational and Structural Biotechnology Journal, 2025).

We like that image. In our piece on why knowledge graphs stall before the clinic, we described data that is born with a birth certificate and carries a passport that gets stamped at every border. The same idea applies to the models and agents that touch that data. Every actor in the chain, human or machine, should carry papers.

NGSI-LD and PROV: complements, not alternatives

Provenance also needs somewhere to live that does not make it too costly to keep. That is where NGSI-LD, the context information standard from ETSI that grew up in smart cities, comes in (ETSI GS CIM 009).

• Details ride on the statement itself. Properties and relationships can carry their own properties, so who asserted a fact and how travel with the fact.
• They speak the same language. NGSI-LD is built on JSON-LD, which PROV can also use, so the two combine without a translation layer.
• Time is built in. NGSI-LD records when something was observed and can answer questions about an earlier point in time.
• The question travels, the record stays. Federated deployment lets each organization keep its data and provenance where they are, while a question follows the chain across sites, sponsors, and partners.

The way we put it: NGSI-LD answers "what is true now, in context." PROV answers "how do we know, and who is accountable."

One design idea follows, and we want to be clear that it is a design direction, not established market practice. Because a context broker sits in front of every read and write, it can make complete provenance a condition of entry. A statement that arrives without it is held for human review instead of slipping into the record. In short, provenance is a privilege, not a diary: something a statement must have to get in, not something someone remembers to write down afterwards.

Two more requirements come from the evidence itself. A tumor assessment is true over a period of time, can be revised after a later read, and travels from site to sponsor to regulator. To defend a decision you need to answer two time questions: what was true of the patient at that moment, and what did we know when we acted? That second clock, combined with federation across organizations, is the minimum structure for defending a cross-institution, time-sensitive claim. NGSI-LD provides the federation and single-clock time features natively. The second clock is something you have to design on top.

What to measure

What a buyer measures is what a vendor optimizes. AI programs tend to celebrate drafting speed, while the work of reconstructing evidence is absorbed quietly by writers and quality staff. Three metrics would put a price on the gap:

• Unbound claims that escaped past review
• Reconstruction hours absorbed per report
• Time to reconstruct a challenged claim: the hours between a regulator's question and a walkable evidence chain

None of this calls for a big-bang replacement of validated systems. It calls for a decision about where provenance lives. Either the architecture carries it as part of every statement it produces, or people rebuild it by hand, every day, until an inspection, a dispute, or a deal brings the gap into the open. Provenance is not an audit layer added after the fact. It is what makes putting AI into the work that moves medicines forward defensible at all.

Where to go next

The standards exist and the regulatory direction is clear. What is still missing is practice: real programs that bind claims to evidence before they generate, and that measure reconstruction time instead of just drafting speed.

That is the work we want to do together. We are looking for a small number of design partners in clinical development, safety, and regulatory operations who want to put an evidence chain under their AI programs from the start. If that sounds like your team, or if you just want to compare notes on the reconstruction tax, write to us at info@scientix.ai.

About the authors

Bo Lora is the founder of ScientixAI. He is a veteran, led UX design technology at Amazon Business and served as a Principal Design Technologist in Healthcare and Life Sciences at AWS. His interest in knowledge graphs for clinical research grew out of his smart city work: a clinical trial, like a city, is many independent systems and people that have to share context they can trust.

Marcie Ogilvy is ScientixAI's Acting Chief Marketing Officer and a digital teammate: an AI agent, not a person. She works alongside Bo and the company's other human and digital teammates. On this paper she cut down a longer research manuscript, checked every claim against a primary source and cut what she couldn't verify, and shaped the argument for a life-sciences audience. Bo set the thesis, supplied the ideas, and reviewed and approved the final text.

We publish this way on purpose. ScientixAI builds infrastructure so that every claim can show where it came from and who stands behind it. A paper about provenance should do the same.

Sources

1. Nielsen IØ, de Langsdorff V, April J. "How Medical Writing and Regulatory Affairs Professionals Can Embrace and Deploy Generative AI at Scale." Applied Clinical Trials, 7 January 2025. https://www.appliedclinicaltrialsonline.com/view/medical-writing-regulatory-affairs-professionals-embrace-deploy-generative-ai
2. ICH. E6(R3) Guideline for Good Clinical Practice, Step 4, 6 January 2025, section 3.6.6. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf
3. CDISC. Analysis Results Standard. https://www.cdisc.org/standards/foundational/analysis-results-standard
4. U.S. Code of Federal Regulations. 21 CFR Part 11, Electronic Records; Electronic Signatures, §11.10(e) and §11.100. https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11
5. MHRA. 'GXP' Data Integrity Guidance and Definitions, March 2018, section 3.10. https://assets.publishing.service.gov.uk/media/5aa2b9ede5274a3e391e37f3/MHRA_GxP_data_integrity_guide_March_edited_Final.pdf
6. FDA. Data Integrity and Compliance With Drug CGMP: Questions and Answers. Guidance, December 2018. https://www.fda.gov/media/97005/download
7. FDA. Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products. Draft guidance, January 2025. https://www.fda.gov/media/184830/download
8. EMA. Reflection paper on the use of Artificial Intelligence (AI) in the medicinal product lifecycle. EMA/CHMP/CVMP/83833/2023, adopted 9 September 2024. https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-use-artificial-intelligence-ai-medicinal-product-lifecycle_en.pdf
9. EMA. "EMA and FDA set common principles for AI in medicine development." News, 14 January 2026. https://www.ema.europa.eu/en/news/ema-fda-set-common-principles-ai-medicine-development-0
10. FDA and EMA. Guiding Principles of Good AI Practice in Drug Development. January 2026. https://www.fda.gov/media/189581/download
11. Regulation (EU) 2024/1689 (EU AI Act), Article 12, Record-keeping. https://artificialintelligenceact.eu/article/12/
12. European Commission. EudraLex, Annex 22: Artificial Intelligence (draft for consultation). https://health.ec.europa.eu/document/download/5f38a92d-bb8e-4264-8898-ea076e926db6_en
13. ICH. M15 General Principles for Model-Informed Drug Development, Step 4. https://database.ich.org/sites/default/files/ICH_M15_Step4_Final_Guideline_2026_0129.pdf
14. FDA. Warning Letter, Purolea Cosmetics Lab, MARCS-CMS 722591, 2 April 2026. https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/warning-letters/purolea-cosmetics-lab-722591-04022026
15. Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools." arXiv:2405.20362. https://arxiv.org/abs/2405.20362
16. Gao T, Yen H, Yu J, Chen D. "Enabling Large Language Models to Generate Text with Citations." EMNLP 2023. https://aclanthology.org/2023.emnlp-main.398/
17. Wu K et al. "An automated framework for assessing how well LLMs cite relevant medical references." Nature Communications, 2025. https://www.nature.com/articles/s41467-025-58551-6
18. Wallat J et al. "Correctness is not Faithfulness in RAG Attributions." arXiv:2412.18004. https://arxiv.org/abs/2412.18004
19. W3C. PROV-Overview: An Overview of the PROV Family of Documents. W3C Working Group Note, 30 April 2013. https://www.w3.org/TR/prov-overview/
20. HL7. FHIR Provenance resource, R5. https://hl7.org/fhir/provenance.html
21. Alu A, Oluwadare O. "An auditable and source-verified framework for clinical AI decision support: integrating retrieval-augmented generation with data provenance." Frontiers in Artificial Intelligence, 2026. https://doi.org/10.3389/frai.2026.1737532
22. Kalokyri V et al. "AI Model Passport: Data and system traceability framework for transparent AI in health." Computational and Structural Biotechnology Journal, 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12528916/
23. ETSI. Context Information Management (CIM); NGSI-LD API. ETSI GS CIM 009 V1.9.1, July 2025. https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.09.01_60/gs_CIM009v010901p.pdf