How to Tell Whether an AI System Is Actually Working

You can only tell whether an AI system is working if you measured the process before it arrived. The simplest approach is to track four numbers: time per item, error rate, correction rate and cost per item.

This sounds obvious, but it is where many small projects go wrong. The system goes live, everyone agrees it feels quicker, and six months later nobody can say what it achieved or whether it is still doing it.

Why is "it feels faster" not enough?

New systems feel impressive for the first fortnight. Then small problems appear, and people quietly go back to the old way for the awkward cases. Soon the system handles only the easy half of the work, the staff still do the hard half, and the saving is much smaller than anyone believes.

Without numbers you cannot tell the difference between a system that works and one that is being worked around. With numbers you can see the drift early, while it is still easy to fix.

What are the four numbers?

1. Time per item. How long it takes to handle one invoice, enquiry or document from start to finish, including waiting time. Measure it before the project, even roughly, by timing twenty real items. Use the median rather than the average, because a few awful cases can distort the average badly.
2. Error rate. How many outputs were wrong. Define "wrong" in advance. For invoices it might be a wrong supplier, total or tax figure. For an inbox it might be a misrouted email. Write the definition down so it cannot shift later.
3. Correction rate. How often a person had to change what the system produced. This is different from the error rate. A system can be accurate on paper and still need constant tidying, and tidying costs time. A high correction rate is the clearest sign that a system is not yet earning its keep.
4. Cost per item. Staff time plus software and usage costs, divided by the number of items handled. Include the review time, because that is where savings often leak away. A system that saves ten minutes of typing and adds eight minutes of checking has saved two.

What does a before-and-after look like?

Here is an invented example, for illustration. Before the project, handling one supplier invoice takes eight minutes in total, and about one in twenty contains a keying error. After a month with the system, the median time per invoice is three minutes, including a quick human check. Four in ten invoices need some correction, mostly from two suppliers with unusual layouts. The keying error rate has dropped sharply.

That is a real improvement, and it also tells you where to look next. The correction rate is high because of those two suppliers. Fixing that is worth more than any change to the model.

How should you collect them?

Keep it simple. A spreadsheet with columns for the date, the item, whether it was corrected, what was corrected and how long the check took is enough. The person doing the checking fills it in as they go, which takes seconds. Review it weekly for the first month, then monthly.

Do not build a dashboard early. A spreadsheet that people actually fill in is more reliable.

What should you watch for?

• A correction rate that falls and then creeps back up. This often means the inputs have changed, such as a new supplier layout or a new kind of enquiry.
• Staff skipping the review step. If the checks stop, the error rate stops being visible, and an unseen problem looks like a good result.
• Savings that exist only on paper. If the freed-up time is never reallocated, the business sees no benefit even though the numbers improved.
• Selective measurement. If only the easy items are logged, the figures will flatter the system.

When do you stop or change course?

Decide in advance, ideally before launch. For example: if the correction rate is still above a figure you agree after six weeks, you change the setup or stop. Setting that line early turns the decision into a factual one instead of an argument, and it protects you from the sunk-cost feeling that keeps weak projects alive.

What is a reasonable expectation?

Not perfection. A reasonable early result is a clear reduction in time per item, a stable error rate no worse than the manual process, and a correction rate that falls month by month as the setup is tuned. If you can show those three things on one page, you have a system that is working.