We're measuring the wrong thing, and AI is about to make it worse

There's a metric that has haunted L&D for decades, sitting quietly at the centre of almost every learning report ever produced, dressed up in dashboard colours and presented to leadership as evidence that something is working.

The completion rate.

Seventy-eight percent of employees completed the module. Ninety-one percent finished the course. A hundred percent of the team has been through the induction. Tick. Done. Great job everyone. On to the next one. Here's what completion rates don't tell you: whether anyone did anything differently last Thursday than they did the Thursday before.

I've spent the better part of a decade trying to understand why so much training doesn't work. Not why it's badly designed- though we all know plenty of it is- but why even well-designed, well-intentioned, genuinely expensive learning programmes so often produce... not a fat lot. People leave the session, return to their desks, and within a fortnight the content has dissolved into the general fog of work life, leaving behind nothing more than a faint memory of a slide with too many bullet points, and the bang-average catering spread.

The answer, I've come to believe, is that we built the entire infrastructure of L&D around the wrong question.

We asked: did they consume the content?

We should have asked: did anything change?

These are not the same question- not even close.

Consumption is easy to measure. A learning management system can tell you, to the minute, how long someone spent on a module. It can tell you their quiz score, their completion timestamp, their click-through rate on supplementary materials. It generates data with the enthusiasm of my cockapoo, Dylan, fetching a ball- endlessly, happily, without ever stopping to consider if the ball even has a point (Note: this is no criticism of my dog, he more than makes up in lovability and loyalty what he lacks in critical thinking skills). Behaviour change is harder to measure. It requires you to decide, before you build a single piece of content, what you actually want people to do differently. It requires you to observe whether they do it. It requires patience, and follow-through, and a willingness to look at the results even when they're uncomfortable. It requires, in short, that you treat learning as something with consequences in the real world rather than as an event that takes place in a platform and concludes with a certificate.

Most organisations have chosen the cockapoo approach. And for a long time, the costs of that choice have been hidden inside the general murkiness of how hard it is to attribute business outcomes to learning interventions. Who can really say whether that leadership programme caused the improvement in team performance, or whether it was the new hire, or the restructure, or the fact that the office coffee got better? Measurement is hard, so we measure what's easy, and we call it good enough. It isn't good enough, and AI is about to make that very clear.

Here's what's coming. Within the next two to three years, the barrier to producing learning content will approach zero. What currently takes a team of instructional designers, subject matter experts, and learning technologists several months will take an afternoon. Courses, modules, microlearning, scenario-based practice, facilitator guides, assessment banks- all of it generatable, at scale, at speed, for almost nothing. This sounds like good news. And I think that, in many ways, it is. The democratisation of content production means that small, scrappy teams can punch above their weight, that learning can be updated in hours rather than quarters, that the gap between "this thing changed and our people need to know about it" and "our people know about it" can close dramatically.

But here's the problem: if you're optimising for completions, AI makes that optimisation catastrophically efficient. You can produce more content for more people to consume more quickly and measure more completion of, while behaviour change remains exactly where it's always been- stubbornly, frustratingly unmoved.

More content isn't the answer to a measurement problem. It's an acceleration of it.

The research on this is not new, but it bears repeating in the context of what's coming.

Bjork's work on desirable difficulties showed that the conditions that feel like learning- easy recall, smooth consumption, the warm glow of familiarity- are often the conditions least likely to produce durable change. Retrieval practice, spaced repetition, interleaving: all of them feel harder than passive consumption, produce more anxiety in the learner, and can even worsen immediate performance. They also produce dramatically better retention and transfer. The things that look good in a completion dashboard are frequently the things that work least well.

Ebbinghaus mapped the forgetting curve in the late 1800s and, much like Coca-Cola and Levi's blue jeans, nobody has really improved on it since: without reinforcement, we forget roughly 70% of new information within 24 hours. A module completed on Monday is, in cognitive terms, largely gone by Wednesday. If the measure of success is whether someone finished the module, you will never be forced to confront this. If the measure of success is whether someone is doing something differently six weeks later, you cannot avoid it. Self-determination theory- Deci and Ryan's decades of research on human motivation- tells us that durable learning requires autonomy, competence, and relatedness. People need to feel that they have agency in what they're learning, that they're developing genuine mastery rather than performing it, and that the learning is happening in the context of real relationships. A completed e-learning module, experienced alone, on a laptop, because it appeared in a mandatory training queue, fails all three criteria simultaneously. None of this is a secret. It's taught on every instructional design course. It sits in the behavioural science literature, clearly signposted, waiting to be applied, and yet the completion rate reigns.

I want to be fair to the people who built these systems. Behaviour change is genuinely difficult to measure. It requires you to define, with real precision, what you want someone to be able to do that they couldn't do before- and that definition has to happen before you build anything, which means it has to survive the political reality of learning commissioning, where stakeholders often want a programme that makes people "better at communication" or "more commercially minded" or "more innovative," without being especially willing to say what better, more commercially minded, or more innovative would look like if you observed it in a Tuesday morning meeting.That definitional work is hard. It requires conviction, and sometimes confrontation, and a willingness to say: "I won't build this until we've agreed what success looks like." Most learning functions don't have the organisational standing to make that demand. So they build the programme, measure the completions, and move on. AI won't solve this problem, but it will make the choice visible in a way it hasn't been before.

When content is expensive and slow to produce, a learning function's output is constrained by its capacity. The question "is this working?" gets deferred partly because it's hard, but also because the machine is already running at full speed and there's no room to stop and interrogate it. When content is cheap and fast to produce, that constraint disappears. You can produce more content than any organisation could ever consume. The bottleneck moves- from production to something else entirely.

If you haven't built the measurement infrastructure to know whether your learning is changing behaviour, that bottleneck will be visible as a pile of content that nobody can demonstrate the impact of. More modules, more completions, more data that tells you nothing worth knowing. If you have- if you've done the hard work of defining what change looks like, building observation frameworks, creating feedback loops between learning and performance, designing for behaviour rather than consumption- then AI becomes something genuinely transformative. Not because it produces content faster, but because it allows you to iterate on that content in response to real signal. You can update, refine, personalise, and improve in near real time, closing the loop between "what we're teaching" and "what's changing" in ways that were previously impossible.

The question AI forces L&D to answer is the one we've been avoiding for decades.

Not "how do we produce more content?" but "how do we know if any of it is working?"

It's an uncomfortable question. It requires admitting that a lot of what we've measured- neatly, reliably, at scale- has been measuring the wrong thing. It requires building new frameworks, having harder conversations, and accepting that "seventy-eight percent completion" is not, on its own, a success story. But it's also the most interesting question in the field right now. Because if you get the measurement right- if you design learning systems around the actual question of behaviour change rather than the proxy question of consumption- then AI is extraordinary. It gives you the capacity to produce content that responds to real learner need, at the moment it arises, with the specificity that one-size-fits-all content has never been able to achieve.

Alice Veitch is a Learning & Development Lead with a background in behavioural science. She has led learning strategy across EMEIA and is a respected disruptor in the field.