Is your legal AI working? Four numbers to track from month one

By Michael Nadalin, Founder, Market Lead · 23 September 2026 · 6 min read

Three months after signing up, most firms can tell you exactly what the licence costs and almost nothing about what it returns. The partner who pushed for it says it is saving time. The senior associate who never opened it says it is a toy. Neither of them has a number, and the renewal invoice arrives regardless.

That is fixable, but only if you decide what to count before the tool goes live. Once people have started using it, you have lost your baseline, and every later claim about time saved is a guess dressed up as a finding.

You do not need a dashboard or a consultant. You need four numbers, collected from the first month: how many people use it on real matters, how long a defined task takes with and without it, how often its output fails verification, and whether the time recorded against that work survives billing. Each one is cheap to collect. Together they tell you whether to renew, expand or walk away.

Why a gut feel will not survive the partners' meeting

Impressions about AI tools are unreliable in both directions. Enthusiasts remember the research memo it drafted in ten minutes and forget the forty minutes they then spent checking every authority. Sceptics remember the one confidently wrong answer and ignore the dozen routine questions it handled well.

There is also a professional reason to measure. Australian courts have issued guidance on generative AI, including the NSW Supreme Court's Practice Note SC Gen 23, and practitioners have already been referred to regulators over citations nobody checked. If your firm uses these tools, you should be able to show how you supervise them. Numbers you collect for commercial reasons double as evidence of that supervision.

Number one: real use on real matters

Logins tell you almost nothing. A graduate who opens the tool to try it on a hypothetical is counted the same as a senior associate who used it to scope the authorities on a live injunction. What you want is the number of people who used it on a billable or pro bono matter in a given fortnight, as a share of the people who have access.

The simplest way to get this is a matter-level tag. Ask people to add a short code to their time entry whenever AI contributed to the work. It is imperfect and some entries will be missed, but the trend over the first twelve weeks is what matters. A tool that is used on real matters by a steady or growing share of the team is doing something. A tool whose use peaks in week two and then falls away has told you its answer.

Number two: time on a task you already understand

Do not try to measure time saved across the whole practice. Pick one or two tasks that recur often, that your team does in a reasonably consistent way, and that you can time. Good candidates are a first-pass research note on a limitation question, a summary of the authorities on a discrete point of statutory interpretation, or a chronology from a bundle of correspondence.

Before rollout, have a few people record how long that task takes them, start to finish, including checking. After rollout, record the same thing, and include the verification time. This last part is where most internal pilots cheat themselves. A draft that arrives in five minutes and takes an hour to check is not faster than a careful research note written in fifty minutes.

Number three: how often output fails verification

This is the number that protects you. Every time someone checks an AI-assisted answer and finds a problem, they should log it in a shared register with the category of failure. It takes thirty seconds and it gives you two things: a failure rate you can watch over time, and a profile of the kinds of mistakes the tool makes.

The profile matters more than the rate. A tool that occasionally gets a pinpoint reference wrong is a very different risk from one that produces authorities that do not exist, or states the law of the wrong jurisdiction without saying so. In Australia the jurisdiction problem is real: a proposition that holds in Victoria may not hold in Queensland, and legislation is amended often enough that a correct answer from two years ago can be wrong today. Rank your failures by how much harm they could do if they reached a court or a client.

Number four: whether the time survives billing

The last number is commercial. Time spent on AI-assisted work only helps if it ends up on an invoice the client pays, or if it frees someone for work that does. Track write-downs and write-offs on matter-tagged research against your usual rate for comparable work. If the AI-assisted entries are written off more often, find out why before you draw conclusions.

Sometimes the reason is benign: a supervising partner trims the entry because the research felt too quick to charge at full value. That is a pricing conversation, and it may push you towards fixed fees for certain tasks. Sometimes the reason is worse: the work had to be redone. Your verification register should tell you which. Run the four numbers together on a simple timetable so you are not making the renewal decision on anecdote.

What to do with the numbers at day 90

The decision is usually clearer than people expect. If usage has settled at a meaningful share of the team, the defined task is genuinely faster including checking, serious verification failures are rare and caught, and the work is not being written off, renew and consider expanding to a second task. If any serious failure category keeps recurring, restrict use to tasks where that failure cannot cause harm, or stop. If nobody uses it after week three, no feature list will change that.

Be wary of a vendor's own figures about time saved or accuracy. They were measured on someone else's tasks, in someone else's practice, often outside Australian law. Your own numbers, however rough, are worth more. Whichever tool you trial, whether that is Legal Brain or anything else, hold it to the same four measures and write down what you found. When a partner or a court asks how you supervise AI in your practice, a short file note with twelve weeks of data is a far better answer than a feeling.

Frequently asked questions

How do you measure ROI on legal AI in a law firm?

Measure real use on matters, time on a defined recurring task including verification, the rate and type of verification failures, and write-offs on AI-assisted work. Take a baseline before rollout, because without one any claim of time saved is a guess. Twelve weeks of data is usually enough to decide whether to renew.

Do Australian courts allow lawyers to use AI for legal research?

Generally yes, but practitioners stay responsible for everything they file. Several Australian courts have issued practice notes or guidelines on generative AI, including the NSW Supreme Court's Practice Note SC Gen 23, which places specific limits on its use in some court documents. Check the current guidance of the court you are appearing in and verify every authority yourself.

How long should a legal AI pilot run before deciding?

About 90 days is a sensible minimum. The first couple of weeks are inflated by curiosity use, and you need enough real matters to see whether usage holds and whether serious verification failures recur. Shorter pilots tend to measure novelty rather than value.

What is the biggest risk of using AI for legal research?

Citing an authority that does not exist, or one that does not support the proposition claimed. Both can reach a court if output is not checked line by line against the source. A simple register of verification failures helps you see how often this happens with the tool you are using.