Issue 011 organization name itnewest evidence Aug 2026
Adoption is measured because the vendor console reports it, not because it answers anything
Seats, licences and weekly active users are counted and reported upward. None of them distinguishes a tool that has absorbed a process from one that is opened daily and completes nothing.
This is the default rather than a decision: usage counts arrive free with the subscription while work-completed figures have to be defined, instrumented and argued over. The alternative that the material converges on is a unit of successfully completed work and what it cost to produce - which reframes model choice too, since cheaper tokens that need three attempts and a human review are not cheaper per finished outcome. Note who is making that argument: a model vendor benefits from a measure that rewards work done over seats held. The reasoning is sound and the interest is real, and both belong on the page.
How to fix it — 1 approach, 3 steps
Define one unit of completed work per use case, and cost it
For each deployed use case, name the thing that counts as one finished piece of work, and track how many are completed and what each costs end to end - including retries and human review.
Done when Each live use case defines one completed unit of work, completion counts and full cost per unit including retries and review are instrumented, and seat and active-user counts have been removed from management reporting.
- Write down, for each live use case, what one completed unit of work is.
- Instrument completion counts and full cost per completed unit, retries and review included.
- Retire seat and active-user counts from management reporting once the completion measure is running.
The evidence — 3 documents
| Organization | Document | Position |
|---|---|---|
| DeloitteConsultancy · October 2025 | AI is capturing the digital dollar: what is left for the rest of the tech estateOur reading Names the measurement gap outright: indicator usage across 46 measures has flattened rather than shifted toward outcomes, and use of process-effectiveness measurement fell from 81% to 75% in a year in which AI attention rose, which it calls a missed opportunity rather than a caveat.Return on key performance indicators | names it |
| OpenAIFrontier lab · August 2026 | A scorecard for the AI ageOur reading Argues the software industry habit of counting seats, active users and renewals answers the wrong question, and proposes cost per successfully completed unit of work as the replacement. Written by a model vendor, whose commercial interest points the same way as the argument.Measuring work accomplished rather than seats purchased | proposes a fix |
| USTConsultancy · November 2025 | CIOs guide to agentic AIOur reading Sets out a specific task-level set - completion rate, work reduction, cost and cycle time per task, safety violations - explicitly in place of technical model scores, which are equally unrelated to whether work got done.Task-level performance indicators | proposes a fix |