Stop Evaluating Legal AI by the Demo

I should be upfront about where I sit. I co-founded a legal technology company, which means I sit on the vendor side of the table, usually one of several names on a GC’s shortlist. That’s not a neutral vantage point, and I won’t pretend otherwise.

But what it does mean is that I spend a lot of time with legal leaders during and after their evaluations, hearing how they weigh their decisions. What I’ve come to believe is that their decisions tend to reward whoever’s demo is best, rather than the platform that performs best.

The demo is a 15-minute format in which you’re shown a question being typed into an AI assistant with a clean interface. Then a confident response appears within seconds, and the room nods.

That’s because the interface is the part every vendor in this category has, by now, figured out how to make look effortless: a clean chat window, a fast response, and a fluent and confident-sounding answer.

But what the demo can’t show you is whether the system can reason over your organization’s own playbooks, precedents, and decision history, or simply generate a plausible answer.

Infrastructure vs. interface

What I’m observing is that most legal AI evaluations happen backwards. A team comes away impressed from a demo, only to discover months later, usually the first time the tool gets asked a more complex question, that it has no idea how the organization or legal function works.

There’s a simple tell: ask the tool a question that depends on your organisation’s own history. Not “explain a limitation of liability clause,” but “why is this clause different from the position we normally take?” A capable model may retrieve previous contracts, policies, or correspondence. But retrieval alone doesn’t guarantee understanding. The real test is whether it can connect those fragments into the reasoning, trade-offs, approvals, and risk decisions that shaped the position in the first place.

That’s the real limit of horizontal AI tools, whether it’s ChatGPT, Claude or Copilot. Modern enterprise versions can retain conversational context and connect to document repositories. They are often very good at helping a user find information. But legal teams are rarely being asked to find information. They’re being asked to make defensible decisions.

A document may tell you what position was taken. It rarely tells you why it was taken, what alternatives were considered, which risks were accepted, who approved the approach, or whether those assumptions still hold today. Those forms of organizational judgment are often spread across emails, meetings, negotiations and conversations, if they’re captured at all. That’s the gap.

The tool may retrieve the artefacts of a decision, but not the decision-making record itself. It can help someone reconstruct the past, yet it doesn’t inherently maintain the governed institutional memory that lets a legal function consistently explain and defend its reasoning over time. Nor does it automatically create the audit trail a GC needs when asked who decided, what factors were considered, and on what basis the final position was approved.

For legal teams, that’s often the difference between information and accountability.

This isn’t only an efficiency question, either. An IDC study put a number on the gap: 83% of legal teams say administrative work keeps them from focusing on strategic priorities, and only 1 in 5 business leaders rate their legal function as highly effective. In my experience, part of that comes from teams adopting tools without ever really testing what’s underneath them.

A single connected record

The industry’s answer has mostly been to talk about connecting AI to more data. I’d push back on that framing, though not on the idea of connection—connection is exactly right. More accurately, it comes down to how deep the connection goes. Because connecting AI to more documents is not the same as connecting it to the decisions those documents record.

An AI tool plugged into a single contract repository or a shared drive can now see more documents, but it can’t see the negotiation history behind a non-standard clause, or the reasoning that led a team to accept one fallback position over another, the last time a similar deal came up.

That’s why the real differentiator isn’t how many sources a system connects to—it’s whether it can reason across matters, contracts, and past decisions as a single connected record and take the next operational step, rather than simply answering a question.

None of this makes the interface unimportant. A confusing tool with low adoption is its own kind of failure. But a good interface is now a baseline expectation, not a differentiator. What separates one tool from another is what sits beneath the surface: whether the system understands your organization once the questions get specific.

If I could leave in-house legal leaders with one question to bring into their next round of vendor evaluations, it wouldn’t be about the interface at all. It would be this: what happens when the tool is asked something only your legal team knows the answer to?

That single question tends to separate the tools genuinely reasoning over an organization’s context from the tools that are “pretty”. And the teams that get this right won’t be persuaded by the quality of the demo; they’ll evaluate the quality of the context behind it.