Whether anyone is watching the system after it goes live.
38documents on this topic
21organizations represented
4issues named
9sourced citations
0sourced statistics
The state of it
One of 6 topics within Evaluation & observability.
38 documents from 21 organizations address whether anyone is watching after go-live. It is the largest topic here and still the one where the gap between what is recommended and what is described being done is widest.
The recommendations are consistent and unremarkable: continuous monitoring, tested fallbacks, outcome-based metrics. What is missing almost everywhere is the operating detail - who looks, how often, against what threshold, and what happens automatically when it is crossed. Accenture is the exception worth reading, because it names the specific decay: unchecked model drift alongside shadow GPU spend, duplicated vector stores and manual approval workarounds, compounding every quarter.
Read across the set, monitoring is treated as a platform feature to be procured rather than an operating discipline to be staffed.
The issues, by agreement
How many independent organizations name each issue as a problem. An issue is only as real as the number of separate publishers that identify it, so the count is the ranking. Bars are organizations, not documents. Where the count reads ours, no publisher here states the issue and the analysis is our own.
The chart above counts positions; this shows whose they are. Read down a column for what one organization holds across the whole topic, and across a row for who lines up on one issue. Where a cell carries more than one position, the strongest is shown and the rest are in the tooltip.
Ddisputes itQqualifies itNnames it as a problemPproposes a fix
Monitoring in production: 4 issues against the 8 organizations cited on them. The number under each name is how many of these issues it is cited on.
A dot means this organization is not cited on that issue. It does not mean they are silent on it: an organization is cited where its document takes a position we could locate, and the absence of a citation is the absence of a finding, not a finding of absence. Who is represented lists everyone working on this topic, including those not cited above.
Where they disagree
No contradictions recorded on this topic yet.
The issues in full
Each issue carries the organizations that name it, the numbers behind it, and the remedies proposed - with the concrete steps under each. Every citation points at a section of a named document, so any count here can be checked.
Issue 012 organizations name it2026 evidence
Every team chose its own stack, so there is nothing to observe across them
Teams are encouraged to move fast and each picks its own tools. The result is a set of systems with no shared way to instrument, compare or shut down any of them - so oversight has to be negotiated separately with every team.
McKinsey's engineering unit describes this as the predictable cost of getting people hands-on: fragmentation across vendor, in-house and mixed stacks, little reuse between teams, diverging standards, and technical debt accumulating in both version support and engineering shortcuts. That is a reasonable price for learning, and it becomes a governance problem the moment anything goes to production, because there is no layer at which a question like "how many agents are running and what did they touch" can be answered at all. The organization ends up able to supervise each system and unable to supervise the estate.
Let teams choose their frameworks; do not let them choose whether they are observable. Make the shared layer the condition of going to production.
Done when The minimum every agent must emit is written down, production access is refused to anything that does not emit it, and the estate inventory is generated from that emission rather than from a survey.
Define the minimum every agent must emit - identity, action, tools called, outcome.0-30 daysCTO
Make emitting it a condition of production access, not a recommendation.30-90 daysCIO
Publish an estate-wide inventory that is generated, not surveyed.90-180 daysCIO
The evidence — 2 documents
Organization
Document
Position
AnthropicFrontier lab · July 2026
Anthropic’s CISO guide to agentic AIOur reading Names the precondition for any supervision at all: every action runs under some identity, and an agent acting has to be distinguishable from the person it acts for - which is why the fix offered is issuing that identity centrally and streaming the record into the systems the organization already watches.Identity and the widening aperture
names it
QuantumBlack, AI by McKinseyConsultancy · June 2025
‘Agentic AI Mesh’Our reading Names fragmentation as the leading obstacle in moving from proof of concept to production: competing tool choices, limited reuse, diverging standards and mounting technical debt across siloed teams.Operational challenges of distributed agentic systems
names it
Issue 021 organization name it2026 evidence
Monitoring watches the infrastructure and not the behaviour
Uptime, latency and error rates are instrumented because the existing platform already does it. Whether the output is still correct is usually measured by nobody.
This is the specific way AI monitoring fails: the dashboards are green because they are measuring the things a conventional service exposes. A model that has drifted answers just as fast, with the same status code, and the same cost per call. Nothing in an infrastructure monitoring stack notices that the answers got worse, which is why the first signal is usually a customer or a regulator rather than an alert.
Add a quality signal to the same dashboard that already carries latency and error rate, so behaviour and infrastructure are watched by the same people.
Done when Each production system shows a quality signal on the same operational dashboard as its latency and error rate, owned by whoever is on call, and that signal raises an alert at the same severity as an outage.
Pick one measurable quality signal per production system - acceptance rate, escalation rate, correction rate.0-30 daysCTO
Put it on the operational dashboard next to latency, owned by whoever is on call.30-90 daysCIO
Alert on the quality signal with the same seriousness as an outage.30-90 daysCTO
Name who looks and how often
Continuous monitoring with no named owner is a dashboard nobody opens. Assign the review, and record that it happened.
Done when Each production system names a review owner and a cadence, and the log of completed reviews makes a missed one visible rather than silent.
Assign a named owner and a review cadence per production system.0-30 daysCIO
Require the review to be logged, so its absence is visible.30-90 daysRisk
The evidence — 3 documents
Organization
Document
Position
AccentureConsultancy · April 2026
AI-ready cloud strategiesOur reading Names unchecked model drift alongside shadow GPU spend, duplicated vector stores, one-off data pipelines and manual approval workarounds as costs that compound every quarter, and grades cloud estates on observability maturity.Cloud estate maturity; accumulated decay
names it
Google CloudHyperscaler
Google’s agentic AI transformation frameworkOur reading Proposes continuous agent auditing, automated evaluation, human-in-the-loop verification and robust monitoring together, specifically to manage emergent behaviours after deployment.Continuous auditing and monitoring
proposes a fix
UberEnterprise
Uber - Journey to Generative AIOur reading Describes codifying safe deployment, retraining and feature monitoring into the platform so the discipline applies uniformly rather than per project.Feature monitoring in the platform
proposes a fix
Issue 031 organization name it2026 evidence
There is a dashboard, and no threshold that triggers anything
Metrics are collected and displayed. Almost nowhere is there a stated level at which the system must stop, retrain, or route to a human automatically.
A metric with no bound consequence is reporting, not control. State the level and what happens when it is crossed, without a meeting.
Done when Each monitored signal states the level at which the system must not continue, that level is wired to an action rather than an alert, and a scheduled production test shows the fallback firing.
For each monitored signal, define the level at which the system must not continue operating.0-30 daysRisk
Wire the threshold to an action - retrain, route to human, disable - not to an alert.30-90 daysCTO
Test the fallback actually works, in production, on a schedule.90-180 daysCTO
The evidence — 2 documents
Organization
Document
Position
AnthropicFrontier lab · July 2026
Anthropic’s CISO guide to agentic AIOur reading Argues that what a system reports has to arrive where a team already investigates and acts, rather than in a screen somebody has to remember to open, and that a response measured in days is the wrong unit once agents are running.Case study: Claude Cowork - telemetry into the SIEM
names it
McKinsey & CompanyConsultancy
McKinsey: Rethinking AI decision-makingOur reading Proposes that efficiency, accuracy and user satisfaction be measured and monitored over time with underperforming agents retrained or retired, which requires a defined level at which that happens.Measure, monitor, retrain or retire
proposes a fix
Issue 04Our analysis2026 evidence
Evaluation is budgeted as overhead and turns out to be most of the system
Forecasts are built from the model that does the work. In a deployed agentic flow the model that checks the work can account for the large majority of calls, which makes the original number wrong by an order of magnitude rather than by a margin.
This is the most concrete operating fact in the theme and it comes from an engineering account rather than a survey. LinkedIn reports that nearly 90% of the language model calls in its hiring assistant flow come from the evaluation agent - the step that assesses a candidate against extracted requirements - not from the step a reader would call the product. Evaluation at that share stops being a quality practice and becomes the capacity plan: it decides which model you can afford, how fast the system can run, and whether continuous checking is viable at all. A budget that treats it as a line item under testing is not wrong at the edges. It is wrong about what the system is.
Forecast the evaluation calls as a first-class part of the workload. If they are the majority of traffic, they are the system you are buying.
Done when Calls are instrumented by purpose so the production-versus-evaluation split is reported, the unit cost includes every check the flow performs, and the chosen check rate is recorded with the quality it buys.
Instrument calls by purpose - production versus evaluation - and report the split.0-30 daysCTO
Rebuild the unit cost so it includes every check the flow performs.30-90 daysCFO
Decide the check rate deliberately, and record what quality it buys.90-180 daysCTO
The evidence — 2 documents
Organization
Document
Position
OpenAIFrontier lab · July 2026
How to control AI usage and spendOur reading Proposes treating usage as a managed budget with visibility per workload, which is the mechanism that would make an evaluation share of this size visible before it appears on an invoice.Controlling usage and spend
proposes a fix
USTConsultancy · November 2025
CIOs guide to agentic AIOur reading Proposes cost per task as a headline measure, which folds checking into the unit economics rather than leaving it outside them.Cost per Task as a governing metric
proposes a fix
Who is represented
This dossier is drawn from 22 organizations working on the subject, 8 of which are cited directly in the issues above.
Consultancy — 7
McKinsey & Company 5Accenture 2QuantumBlack, AI by McKinsey 1UST 1Deloitte 1Infosys 1KPMG 1
Institution — 3
Cloud Security Alliance 3NIST 2World Economic Forum 1