Whether you can reconstruct why the system did what it did.
43documents on this topic
26organizations represented
4issues named
10sourced citations
2sourced statistics
The state of it
One of 7 topics within Governance & oversight.
43 documents from 26 organizations address whether you can reconstruct why a system did what it did. Two findings stand out, and they point in opposite directions.
The first is that explainability protocols largely exist and largely have not been tested. Infosys reports that while a majority of boards have protocols, most have not rigorously tested them - and that not enough boards take responsibility for explainability at all.
The second is a genuine technical regression that the governance literature has barely absorbed. McKinsey's data-readiness work points out that when a response is assembled from fragments of many unstructured documents, traceability is not straightforward - and that this shift toward unstructured data moving in and out of AI systems with little traceability is a major challenge. Retrieval-based systems are harder to explain than the models governance frameworks were written for, and agentic systems are harder again.
There is also a live disagreement about how much this matters, set out below.
The issues, by agreement
How many independent organizations name each issue as a problem. An issue is only as real as the number of separate publishers that identify it, so the count is the ranking. Bars are organizations, not documents. Where the count reads ours, no publisher here states the issue and the analysis is our own.
The chart above counts positions; this shows whose they are. Read down a column for what one organization holds across the whole topic, and across a row for who lines up on one issue. Where a cell carries more than one position, the strongest is shown and the rest are in the tooltip.
Ddisputes itQqualifies itNnames it as a problemPproposes a fix
Audit trails & explainability: 4 issues against the 7 organizations cited on them. The number under each name is how many of these issues it is cited on.
A dot means this organization is not cited on that issue. It does not mean they are silent on it: an organization is cited where its document takes a position we could locate, and the absence of a citation is the absence of a finding, not a finding of absence. Who is represented lists everyone working on this topic, including those not cited above.
Where they disagree
No contradictions recorded on this topic yet.
The issues in full
Each issue carries the organizations that name it, the numbers behind it, and the remedies proposed - with the concrete steps under each. Every citation points at a section of a named document, so any count here can be checked.
Issue 012 organizations name it2026 evidence
Agents take actions, and the chain leading to them cannot be reconstructed
Explainability frameworks were written for a system that produces an output a human then acts on. An agent that calls tools, holds state and acts across applications leaves a trail that most organizations are not capturing.
Record every tool call, parameter and state change an agent makes, in a store the agent cannot write to.
Done when Every production agent writes to an append-only action log it cannot itself modify, recording identity, tool called, parameters and authorisation basis, and one past action has been reconstructed as a rehearsal.
Require an append-only action log for every agent in production, separate from application logging.30-90 daysCTO
Include the identity, the tool called, the parameters and the authorisation basis.30-90 daysCISO
Rehearse an investigation: pick one past agent action and reconstruct why it happened.90-180 daysRisk
The evidence — 4 documents
Organization
Document
Position
AnthropicFrontier lab · July 2026
Anthropic’s CISO guide to agentic AIOur reading Poses the test as two concrete questions most organizations answer no to: can you tell agent actions from user actions, and does that record reach your SIEM.Four questions: what observability do I have
names it
IBMHyperscaler
Agentic AI risk and opportunityOur reading Notes agentic systems act independently in ways whose inherent complexity leads to unpredictable behaviour, complicating efforts to assure safety and reliability.Risks and key mitigations of autonomous action
names it
Cloud Security AllianceInstitution · July 2026
Defining non-human identityOur reading Proposes recording delegator, intent and action together in the audit log so that an action taken on a person's behalf can still be attributed to them afterwards.Agent identities; delegated access and audit
proposes a fix
McKinsey & CompanyConsultancy
McKinsey: Rethinking AI decision-makingOur reading Proposes agents operate under guardrails including transparency, auditability and fail-safes, on the same footing as human policy.Governance and oversight
proposes a fix
Issue 021 organization name it2026 evidence
Retrieval from unstructured sources breaks the traceability model governance assumes
A single answer assembled from fragments of many documents has no clean lineage. Governance frameworks written for a model with a training set and a feature vector do not describe this system.
This is a technical change that the governance literature has largely not caught up with. McKinsey notes that one answer can be assembled from fragments of many documents at once, which is what makes the trail hard to follow, and identifies the movement of unstructured data in and out of AI systems with little traceability as a major challenge. The practical consequence is that "which data produced this answer" stops being a query and becomes an engineering problem that must be designed for in advance.
Lineage cannot be reconstructed after the fact. Record which sources contributed to each response at the moment the response is produced.
Done when Retrieval systems log the source fragments used for each answer with document id and version, those logs are retained on the same schedule as the decisions they support, and a sample past answer has had its sources named on demand.
Require retrieval systems to log the source fragments used for each answer, with document id and version.30-90 daysCTO
Retain those logs on the same schedule as the decisions they support.30-90 daysRisk
Test reconstruction on a sample: pick a past answer, name its sources.90-180 daysRisk
The evidence — 2 documents
Organization
Document
Position
McKinsey & CompanyConsultancy · June 2026
Data readiness for scaling AI impactOur reading States that one answer can be assembled from fragments of many documents at once, so the trail is not straightforward, and identifies unstructured data entering and exiting AI systems with little traceability as a major challenge.Unstructured data and traceability
names it
McKinsey & CompanyConsultancy · June 2026
Data readiness for scaling AI impactOur reading Proposes connecting structured and unstructured data into a governed, traceable and reusable foundation as the precondition for scale.Governed, traceable, reusable foundation
proposes a fix
Issue 03Our analysis2 qualifies itnewest evidence May 2025
Explainability protocols exist on paper and have not been tested
Most organizations can produce an explainability policy. Far fewer have run it against a real decision and checked that the explanation it produces would satisfy a regulator, a customer or a court.
25%Named AI risks attributed to impersonationInfosys · Oct 2025
16%Named AI risks attributed to lack of explainabilityInfosys · Oct 2025
How to fix it — 1 approach, 3 steps
Run the protocol against a real adverse decision
Pick a decision that went against someone and produce the explanation the policy promises. The gap between the policy and the output is the finding.
Done when One adverse customer-affecting decision from the last quarter has been explained end to end using only what the protocol provides, and the elapsed time and what was missing have gone to the audit committee.
Select one adverse, customer-affecting decision from the last quarter.0-30 daysRisk
Produce the explanation end to end, timed, using only what the protocol provides.0-30 daysRisk
Report the elapsed time and what was missing to the audit committee.30-90 daysAudit Committee
The evidence — 3 documents
Organization
Document
Position
CapgeminiConsultancy
How AI’s reshaping CXO decisionsOur reading Reports explainability among the leading executive concerns in AI-supported decisions, and separately that systems to validate and explain AI output are wanted far more often than they are in place.Executive summary: what CXOs worry about
qualifies it
Dario Amodei (Anthropic)Other · April 2025
The Urgency of InterpretabilityOur reading Qualifies every protocol on this page from the position of someone building the models: the frontier labs do not yet understand how their own systems reach an output, and the interpretability that would explain one is described as a research goal rather than an available capability. A signed position by an interested party, not a measurement - but it is the builder conceding the limit an enterprise protocol assumes away.The Dangers of Ignorance; the MRI analogy
qualifies it
PwCConsultancy · May 2025
AI leadership playbook overviewOur reading Places interpretability and explainability as a named pillar alongside robustness, security and bias, embedded into design rather than added afterwards.Interpretability & explainability pillar
proposes a fix
Issue 04Our analysis1 qualifies it2026 evidence
Audit trails are built to satisfy a regulator, not to investigate an incident
Logging is scoped to what a compliance requirement names, which is rarely what an investigator needs at 2am - the inputs, the version, the configuration and the reasoning path.
Start from the question an investigator will ask, then check the log answers it. Compliance scope is a floor, not a design.
Done when The three questions an investigator would ask after a bad output are written down, current logs have been tested against them, and model version, configuration and input capture have been added wherever the answer was no.
Write the three questions an investigator would ask after a bad output, then check whether current logs answer them.0-30 daysRisk
Add model version, configuration and input capture wherever the answer was no.30-90 daysCTO
The evidence — 1 document
Organization
Document
Position
McKinsey & CompanyConsultancy · June 2026
Data readiness for scaling AI impactOur reading Frames the requirement as reliability and reuse for the business as well as an investigative capability, naming audit and discovery as where the gap becomes material.Traceable and reusable foundation
qualifies it
Who is represented
This dossier is drawn from 27 organizations working on the subject, 7 of which are cited directly in the issues above.
Cloud Security Alliance 3NIST 3World Economic Forum 2Association of Corporate Counsel 1Australian Institute of Company Directors 1Institute of Directors 1Moody’s 1