Your team uses AI tools. Everyone seems busier — or at least, everyone seems to be doing more things in less time, which is what you hoped for. But when you look at the actual output — the decisions made, the errors caught, the work your clients see — can you point to a number that tells you this is better than before?
Most businesses can’t. And that’s the problem.
BDC research finds that 97% of Canadian SMEs that have adopted AI report tangible benefits. I don’t doubt it — AI tools can genuinely make people more effective. But “reporting tangible benefits” is not the same as being able to demonstrate them. Most of the business owners I talk to know, viscerally, that something has improved. Very few of them have built a way to show it — to their teams, to their clients, or to themselves when it’s time to decide whether to renew a $500/month subscription or cut it.
This post is about building that.
Why “I feel more productive” isn’t a measurement
Productivity is notoriously hard to measure even without AI in the mix — and AI makes it harder, not easier, because the outputs are less tangible. Your team generates more content, processes more information, moves through tasks faster. But faster at what? More of what? Better in what direction?
There’s a particular trap here that I want to name before we go further: the activity trap. AI tools are very good at generating activity — drafts, summaries, proposals, responses — and humans are very good at mistaking activity for progress. If your team is producing twice as many things, but the quality of what you’re delivering to clients hasn’t changed, or the decisions being made haven’t improved, you haven’t gained efficiency; you’ve just added noise.
Measurement is what lets you tell the difference. Without it, you’re flying on instinct — and instinct, as anyone who has ever built a reporting system to check their own assumptions can tell you, is frequently wrong in consistent and invisible ways.
NOVIPRO’s 2026 IT Trends Report (p.15) found that 91% of Canadian organisations are using data to inform business decisions. That’s a genuinely high number, and it suggests most businesses have at least some comfort with data-driven thinking. But using data to run your business and using data to evaluate whether a specific tool is earning its place are two very different things — and the second one is rarer than the first by a wide margin.
If NOVIPRO’s 58% AI adoption rate (p.15) and Statistics Canada’s 19.2% active production figure tell us one thing clearly, it’s that a lot of Canadian businesses are using AI tools before they’ve figured out how to measure whether those tools work. The gap between adoption and evaluated adoption is where most of the wasted spend — and the missed opportunity — lives.
Four metrics that actually tell you something
The goal here is not a complex measurement framework. The goal is to be able to answer the question “is this working?” with something more than a feeling. Here are four metrics that give you that, without requiring a data team to implement.
1. Task completion time, before and after
Pick a specific, repeatable task — drafting a client proposal, summarizing a meeting, generating a first-cut report — and measure how long it takes with the AI tool versus without it. Not once; at least five times, with different people, to account for variability. The number you’re looking for is the average time saved per task, multiplied by how often that task happens. That’s your efficiency case. If the number is small or zero, that’s worth knowing.
The reason most teams skip this step: they didn’t measure before they adopted the tool, so they have nothing to compare to. I’ll come back to that.
2. Revision rate and error frequency
How often does the output from an AI tool need significant revision before it can be used? And is that rate going up, down, or staying flat over time? A tool that produces a first draft that’s usable 30% of the time is a different value proposition than one that’s usable 80% of the time — but if you’re not tracking it, you probably don’t know which you have.
Error frequency matters particularly for outputs that go to clients or influence decisions. If you’re using AI-generated content and you’re not auditing it before it goes out, you’re not using AI to improve quality — you’re using it to move faster, and “faster with unchecked errors” is a step backward.
3. Output quality, as assessed by the person receiving it
This one is less structured, but it’s important: ask the people on the other end of the AI-assisted output whether the quality has changed. Your clients, your collaborators, your direct reports. Not “did you notice we’re using AI?” but “is the work landing better, worse, or the same?”
External quality perception is the measure that actually matters, and it’s the one most teams forget to ask about. Faster and cheaper internally is irrelevant if the end result isn’t as good as it was.
4. Cost per outcome compared to the alternative
Add up the full cost of the tool: subscription, plus any time spent learning it, setting it up, and maintaining it. Compare it to the cost of the alternative — staff time, freelancer costs, previous software. Then look at whether the outcomes are actually equivalent.
Sometimes AI tools cost less and deliver the same result; that’s a genuine win. Sometimes they cost less but require more oversight, making the real cost higher than it looks. Sometimes they cost more and deliver more. All of those are fine — but you need to know which one you have.
How to set a baseline when you’ve already adopted
Here’s the problem I mentioned earlier: most SMBs adopted AI tools without measuring first. The tools got added, people started using them, and now you’re in the awkward position of wanting to know if they’re working without a “before” state to compare to.
This is fixable — it just requires some creative thinking.
For some metrics, you can reconstruct a baseline. Look at output volumes, revision logs, and client feedback from before the tool was in place; if you have records, use them. If you don’t — and many small businesses don’t — you can start measuring now and establish a baseline over the next quarter, then measure again at the end of the following quarter. You lose the comparison to before, but you gain the ability to track forward, which is what actually helps you make decisions.
For task time specifically, you can do a controlled test right now: have team members complete a task without the AI tool, time it, then use the tool and time that. Do it for three to five tasks across different roles. That gives you a rough before/after even without the historical data.
The People Nerds Guide to AI Evaluations (2026) makes a useful distinction here between summative evaluation (did it work?) and formative evaluation (how is it working, and where does it break down?). Most businesses jump straight to summative — “is this tool worth keeping?” — without having done the formative work that makes that question answerable. Slowing down to ask the formative questions first gives you a much better picture of what you’re actually dealing with.
Red flags: signs your team is using AI to look busy, not do better work
These signals are worth watching for, because they’re common and because the people exhibiting them are almost never doing it intentionally:
Nobody can explain what a tool does for them. If you ask a team member what they use their AI tool for and the answer is vague — “just general stuff,” “I use it for writing” — that’s worth a follow-up. Not a punitive one; a curious one. What specifically? When did you last use it? What would you have done instead?
Output volume has increased but outcome quality hasn’t. If your team is producing more content, more responses, more proposals — but winning rates, client satisfaction, or decision quality haven’t moved — you may have traded depth for speed in a way that isn’t serving you. CFIB’s AI adoption research notes this as a pattern among small businesses in early adoption phases: productivity metrics look good before business outcome metrics catch up.
The tool gets used when it shouldn’t. Nuanced client conversations, sensitive communications, anything that requires genuine judgment and institutional knowledge — these should not be AI-first. If your team is defaulting to AI for things that require a human brain and relationship context, the problem isn’t the tool; it’s the absence of guidance about when to use it.
Vendor promises and actual usage don’t match. If you’re paying for an enterprise AI platform that promised to transform your operations and three people check in occasionally to generate summaries — that’s a signal. Not a failure, necessarily; sometimes the right response is to expand adoption and training, sometimes it’s to cut the subscription.
A 30-minute monthly check-in you can run yourself
You don’t need a consultant to build an AI measurement practice. You need 30 minutes a month, a shared document, and the discipline to actually do it.
What we’re using — a current list of active AI tools and who is using them.
What we measured — this month’s data on whichever two or three metrics are most relevant to your business. Task time, revision rate, cost comparison, client feedback — pick what matters for your context and track it consistently.
What changed — any changes in tool use, adoption, or output quality since last month.
What’s next — one decision or action based on what the data shows. Renew a subscription, cancel one, expand use to a new team member, add a policy guardrail. One thing.
Thirty minutes, once a month, written down somewhere the team can see it. Over six months, you’ll have a picture of what’s working, what isn’t, and what you should do next. That picture is worth far more than a vague sense that AI is probably helping.
Not sure where to start? An AI assessment is a structured look at what tools you’re using, whether they’re delivering, and what a measurement approach looks like for your specific team. Book a 30-minute coffee meeting and we’ll build the picture together.