Measuring automation honestly: baselines, sample sizes and what to report
'Saved 70% of time' sounds great until someone asks: compared with what, measured how, over how long? Here's a simple standard for numbers you can stand behind.
By the Po team · 17 September 2026 · 6 min read
Automation marketing is full of round, impressive numbers. Most of them can't be checked. For a small business deciding where to spend limited money, that's a problem, because the only number that matters is the one measured in your own operation.
Four things every claim needs
- 01A baseline: how the task performed before the change.
- 02A measurement period: the dates the numbers cover.
- 03A sample size: how many enquiries, tickets or documents were counted.
- 04A source: where the numbers came from, such as CRM timestamps or a time log.
If any one of these is missing, treat the result as an impression, not evidence.
Pick metrics people already understand
- Median time to first reply on enquiries.
- Hours per week spent on a specific task.
- Errors found per hundred records.
- Share of enquiries that receive a second follow-up.
- Share of AI drafts sent without edits.
Medians are usually more honest than averages, because one very slow day can distort an average.
Compare like with like
Seasonality fools everyone. Comparing a quiet month before automation with a busy month after it will flatter or punish the change unfairly. Where you can, compare the same weeks of the year or run the old and new process side by side for a short period.
Report what didn't work
Every honest evaluation finds limits: message types the AI handles badly, a language it struggles with, edge cases that still need a person. Writing them down makes the next improvement obvious and makes the positive results more believable.
When there's no data yet
A new business won't have client results on day one, and that's fine. Benchmark tests on realistic inputs, clearly labelled as prototype findings, are a perfectly respectable substitute for invented case studies.