The reckoning has arrived. Across UK boardrooms and technology leadership teams, a quiet but uncomfortable conversation is taking place: the chatbot pilot that absorbed six months of budget and goodwill has delivered little beyond a slightly faster FAQ page. According to multiple industry surveys, a significant proportion of enterprise AI investments have failed to meet their original business cases — and the pressure to explain why, and what comes next, is intensifying. But the more urgent question is not what went wrong with the last wave of AI projects. It is whether organisations are measuring the right things as they consider the next one.
Agentic AI — systems capable of reasoning, planning, and taking multi-step actions autonomously rather than simply responding to prompts — represents a fundamentally different proposition from the conversational tools that dominated recent pilots. And if the technology is fundamentally different, it follows that the metrics used to evaluate it should be too. The businesses that get this right will greenlight the right projects. Those that apply yesterday's measurement frameworks to tomorrow's technology will continue to be disappointed.
Why Traditional ROI Metrics Fail Agentic AI
Classic AI ROI calculations tend to anchor on cost reduction: headcount avoided, time saved per query, support tickets deflected. These are legitimate measures for tools that augment individual tasks. They are poorly suited to agentic systems, which operate across workflows rather than within single interactions. When an agentic system autonomously processes a supplier onboarding request — pulling data from multiple sources, cross-referencing compliance requirements, drafting communications, and escalating only genuine exceptions — the value is not captured by counting how many minutes a human would have spent on the equivalent task.
The deeper problem is that cost-saving frameworks are inherently backward-looking. They ask: what did we used to spend, and how much less are we spending now? Agentic AI, at its best, enables organisations to do things they were not doing before — to process volumes of decisions that were previously impossible, to operate at hours that were previously uneconomical, to maintain consistency at a scale that human teams cannot match. A metric that only measures substitution will never capture this expansion of organisational capability. It will systematically undervalue the most transformative deployments and make conservative, low-ambition use cases look relatively attractive.
The Case for Decisions Automated Per Hour
The emerging alternative is to measure agentic AI performance in terms of decisions automated per hour — a throughput-oriented framing that shifts attention from what the system replaces to what it produces. A decision, in this context, is any bounded judgement with a defined outcome: approving or flagging a transaction, categorising an inbound query and routing it appropriately, assessing a document against a compliance checklist, generating a personalised recommendation within defined parameters. These are the currency of organisational value, and agentic systems are fundamentally decision engines.
This framing has several practical advantages. It is outcome-oriented rather than process-oriented, which means it aligns better with how senior stakeholders think about business performance. It scales naturally — doubling decisions automated per hour is a meaningful improvement regardless of the absolute baseline. It also surfaces capacity constraints that cost metrics obscure. If an agentic system can process two hundred supplier risk assessments per hour against a previous human capacity of twelve, the business case becomes visible in a way that a percentage cost reduction simply does not convey. Crucially, it creates a common language across technical and commercial teams, reducing the translation friction that has undermined so many AI business cases.
What Changes When You Adopt This Framing
Shifting to a decisions-per-hour framework does not just change how you measure AI projects — it changes which ones you choose to pursue. Under cost-reduction logic, the most attractive targets are high-volume, low-complexity tasks performed by relatively expensive staff. Under throughput logic, the most attractive targets are high-volume processes where decision quality is currently limited by human bandwidth, regardless of the seniority of the people involved. This opens up a different set of use cases: regulatory monitoring, dynamic pricing decisions, contract clause analysis, clinical triage support, infrastructure incident classification. These are not low-status tasks. They are decisions that matter, and that currently happen more slowly and less consistently than they should.
It also changes how you structure pilots. A cost-reduction pilot is typically designed to demonstrate that the AI matches human output at lower expense, often by running in parallel with existing processes. A throughput pilot is designed to demonstrate what volume of decisions the system can handle, at what quality threshold, under what conditions. The questions are different, the success criteria are different, and critically, the failure modes are different and more instructive. Teams that have been frustrated by inconclusive parallel-run pilots often find that reframing around throughput gives them clearer signals within shorter timeframes.
Addressing the Quality Dimension
The obvious challenge with any throughput metric is that volume without quality is not value. Automating ten thousand poor decisions per hour is worse than making a hundred good ones. Any robust measurement framework for agentic AI must therefore treat decision quality as a constraint, not an afterthought. The practical approach is to define a minimum acceptable accuracy or consistency threshold — informed by the error rates of the existing human process, not by a theoretical ideal of perfection — and measure throughput only within that quality envelope.
This is where many organisations benefit from expert guidance in designing evaluation frameworks before deployment begins. The specific quality thresholds, the mechanisms for detecting drift, the escalation criteria for edge cases — these design decisions shape whether a deployment is measurably successful or indefinitely ambiguous. Getting them right is not primarily a data science problem. It is a business design problem that requires clear thinking about how decisions are made today, what constitutes an acceptable outcome, and who is accountable when the system is wrong. These conversations are often more valuable than the technology selection itself.
If your organisation is approaching its next AI investment decision, the most useful thing you can do before scoping the technology is to define your unit of value. What decisions are currently throttling your organisation's capacity? Where does human bandwidth create bottlenecks in processes that are otherwise well understood? Which judgements are being made inconsistently, slowly, or not at all because the volume is simply too high? These questions point you towards the use cases where agentic AI creates genuine, measurable advantage — and away from the low-ambition deployments that have already disappointed.
The shift from cost savings to decisions automated per hour is not a rhetorical exercise. It is a practical change in how business cases are constructed, how pilots are designed, and how success is recognised when it arrives. Organisations that make this shift are better positioned to invest with confidence, to demonstrate value to boards and finance functions, and to build on early deployments rather than write them off. The technology has matured. The measurement frameworks now need to catch up.
What exactly is agentic AI, and how does it differ from the chatbots we have already deployed?
Agentic AI refers to systems that can autonomously plan, reason, and execute multi-step tasks without human intervention at each stage. Unlike conversational chatbots, which respond to individual prompts, agentic systems can pursue a goal across multiple tools, data sources, and actions — such as processing an application end-to-end rather than simply answering a question about it.
Is decisions-per-hour a recognised industry standard metric, or is this an emerging concept?
It is an emerging framing rather than a formally standardised metric, and definitions vary across vendors and practitioners. However, throughput-oriented measurement is increasingly being adopted by enterprise AI teams as a more meaningful alternative to cost-reduction metrics, particularly in the UK financial services and professional services sectors. The specific unit can be adapted to context — some organisations use decisions per day or per processing cycle.
How do we determine what counts as a 'decision' for measurement purposes in our specific industry?
A decision, in this context, is any bounded judgement with a defined, observable outcome — such as approve or reject, categorise as X, escalate or resolve. The best starting point is to map your existing processes and identify points where a human currently makes a discrete call based on available information. If that call can be defined with clear inputs, rules, and acceptable outcomes, it is a candidate for agentic automation and a candidate unit of measurement.
What quality threshold should we set when measuring decisions automated per hour?
The threshold should be anchored to your current human process performance, not to a theoretical ideal. If human staff make accurate decisions 94% of the time on a given task, that is your baseline — not 100%. Setting unrealistic quality bars is a common reason AI deployments appear to fail when they are, in practice, performing comparably to or better than the existing process. Stakeholder alignment on this baseline before deployment begins is essential.
How should we handle regulatory and compliance risk when automating decisions in a heavily regulated UK sector?
Regulated environments require that automated decision-making is explainable, auditable, and subject to defined human oversight thresholds. This typically means building escalation logic for edge cases, maintaining detailed decision logs for regulatory review, and ensuring that the system's decision criteria can be articulated to the FCA, ICO, or relevant body. Human-in-the-loop design is not a limitation — it is a feature that makes deployment viable in regulated contexts.
Can smaller UK organisations benefit from agentic AI, or is this primarily relevant to large enterprises?
Agentic AI is increasingly accessible to mid-market organisations through cloud-based platforms and specialist implementation partners, rather than requiring large in-house data science teams. For smaller organisations, the throughput argument is often even more compelling — a team of twenty processing decisions at the rate of a team of two hundred is a proportionally greater advantage than it would be for a large enterprise with existing scale.
How long does a realistic agentic AI pilot take before it produces meaningful data?
A well-designed throughput-oriented pilot can produce meaningful signals within four to eight weeks, provided the decision unit, quality threshold, and success criteria are defined before the pilot begins. This is significantly faster than many parallel-run chatbot pilots, which have often run for six months or more without producing clear evidence of value or failure. Tight scoping and pre-agreed metrics are the primary drivers of pilot speed.
What are the most common reasons agentic AI deployments underdeliver, beyond measurement problems?
Beyond measurement, the most frequent causes of underdelivery are poor data quality feeding the agent's decisions, insufficient definition of escalation and exception-handling logic, and misalignment between technical teams and business process owners on what the system is actually supposed to do. Many deployments also underestimate the change management required to integrate automated decisions into existing human workflows without creating friction or distrust.
How do we present a decisions-per-hour business case to a finance director or CFO who expects cost-reduction framing?
The most effective approach is to translate throughput into capacity equivalence: express the decisions-per-hour figure as the number of additional full-time employees that would be required to match that throughput, then apply a fully-loaded cost to that headcount. This connects throughput to familiar financial language without requiring the CFO to adopt a new mental model. You can then layer in the secondary benefits — consistency, operating hours, speed — as additional value rather than the primary case.
Should we build agentic AI capability in-house or work with a specialist implementation partner?
For most UK organisations outside major technology businesses, working with an experienced implementation partner is lower risk and faster to value than building in-house. The core competency required is not just technical — it is the ability to design decision frameworks, define quality thresholds, and integrate agents into existing business processes. Partners who have done this across multiple deployments bring pattern recognition that in-house teams typically take years to develop. In-house capability becomes more valuable once you have a clear understanding of your own use cases and data landscape.
Get in touch today
Book a call at a time to suit you, or fill out our enquiry form or get in touch using the contact details below