What to ask a legal AI vendor before you sign

By Michael Nadalin, Founder, Market Lead · 12 August 2026 · 9 min read

The demo always works. The salesperson types a question about a well settled area of law, the tool returns four tidy paragraphs with citations underneath, and everyone in the room nods. Nobody in that room asks the tool a question about a 2019 amendment to a state Act, or a point where the Federal Court and a state Supreme Court have pulled in different directions, because the demo is not designed to go there.

That gap is where the money gets lost. Not in the licence fee, which most firms can absorb, but in the six months a practice spends half trusting a tool, checking everything it says anyway, and getting no leverage from it at all. Worse is the firm that stops checking.

Below are the questions worth putting to any legal AI vendor before signature, and more usefully, what a real answer sounds like versus a rehearsed one. None of these require a technical background. They require you to keep asking until the answer is specific.

Where does the law actually come from, and how old is it

Every legal AI product sits on top of something. Some are wired into a licensed publisher's database. Some read public sources such as AustLII, JADE, the Federal Register of Legislation and state legislation sites. Some are a general purpose model with a legal prompt bolted on and no source corpus at all. The third kind will still produce a confident answer about section 51 of a state Act it has never read.

Ask three things. Which specific sources are indexed. How often each one is refreshed. And what the tool does about legislation that has been amended but where the compilation has not yet been published. If the answer to the second question is a vague 'continuously', ask for the date of the most recent update to a specific Act you work with daily. A vendor who knows their pipeline can tell you within a minute.

The follow up that separates the serious products: ask whether the tool distinguishes between the version of a provision in force today and the version in force at the date of the conduct. Practitioners live in that distinction. A lot of tools do not know it exists.

What does it do when it does not know

This is the single most important question and almost nobody asks it, because it sounds like a technicality. It is not. A tool that fabricates a plausible case name has not made a small error, it has produced something that reads exactly like real work product and will pass a skim read by a busy supervisor.

The question to ask: show me a query where your tool refuses to answer. Then watch what happens. A vendor with a well built product has examples ready, because they have spent months tuning the abstention behaviour and they are proud of it. A vendor who has not thought about it will tell you their model is very accurate, which is a non answer, or quote you a benchmark score you cannot verify.

Push further. Does every proposition in the output carry a link back to the source passage it came from, or does the tool generate prose first and attach citations afterwards. Those are two different architectures and they fail in different ways. The second one produces citations that exist but do not say what the paragraph claims they say, which is harder to catch than an invented case, not easier.

Where does your client's material go

Rule 9 of the Australian Solicitors' Conduct Rules does not carve out an exception for software you found useful. If you paste a client's affidavit into a tool, you need to be able to say where that text went, who could read it, how long it was kept and whether it was used to train anything.

Ask for the answer in writing, and ask it as a chain rather than a single question. Most legal AI vendors are not running their own models. They are calling someone else's, which means your client's material is passing through at least two organisations and possibly a third if there is a hosting layer in between. The contract you sign is with the vendor. The subprocessors are where the actual risk sits.

Specific things to nail down: whether prompts and documents are used for model training or evaluation, whether a human reviewer can access your inputs and under what circumstances, what the retention period is and whether you can set it to zero, and where the servers are. Data residency in Australia is not always achievable and it is not always necessary, but you should know the answer rather than assume it. If you act for government or health clients, check your own obligations before the demo, not after.

The commercial questions that bite in month eleven

Legal software pricing has a pattern. The first year is priced to win the deal. The renewal is priced against how much work you have moved into the platform. That is not dishonest, it is just how the category works, and you can plan for it if you ask the right questions before signing rather than after.

Ask what the price is at renewal, in writing, with a cap. Ask what happens if your headcount drops, since most per seat agreements are happy to go up mid term and refuse to come down. Ask whether the seat count is named users or concurrent. Ask what is metered: some tools charge per query or per document beyond a threshold, and the threshold is always comfortable until the firm actually adopts the tool.

Then ask the export question. If you cancel, what do you take with you. Saved research, matter notes, prompt libraries and anything your team has built inside the product should come out in a usable format. A vendor confident in their product will answer this without flinching. A vendor whose retention strategy is friction will get vague.

Test it yourself, on work you already know the answer to

The only assessment that means anything is one you run on a question you have already researched properly. You know the answer, you know which authority governs it, and you know where the ambiguity sits. That is the control. A vendor demo cannot give you this because the vendor chooses the question.

Pick five questions across the range of what your practice actually does. One settled and simple. One where a recent amendment changed the position. One that turns on a state versus Commonwealth distinction. One where the authorities genuinely conflict. And one that sits just outside your practice area, where a wrong answer is plausible enough that you would not catch it if you were tired.

Score them on two axes only: was it correct, and how long did it take you to satisfy yourself that it was correct. The second number is the whole business case. A tool that is right most of the time but requires full verification of every proposition has not reduced your workload, it has moved it. Several Australian superior courts have now issued practice notes or guidance on the use of generative AI in litigation, and the direction of travel is that the practitioner wears responsibility for what is filed. Your verification time is not optional, so it belongs in the assessment.

What a good vendor sounds like

The tell is not enthusiasm, it is specificity. A vendor who has built something real will tell you what their tool is bad at without being asked twice. They will name the model provider. They will give you a refresh date rather than an adverb. They will show you a query the product declines to answer and treat that as a feature rather than an embarrassment. They will let you run your own questions during the trial without a sales engineer sitting on the call steering you away from the hard ones.

The opposite pattern is equally clear. Answers that redirect to accuracy claims you cannot verify. Reluctance to put data terms in writing. A trial too short to include a real matter. Pricing that requires a call. And the most common one: a demo where every question comes from their list, not yours.

None of this is unique to legal AI. It is ordinary software procurement discipline applied to a category where the failure mode is unusually expensive, because the output looks like competent legal work right up until someone checks. We built Legal Brain to survive exactly this line of questioning, and the honest suggestion is that you put every product you assess, ours included, through the two week test above rather than taking anyone's word for it. A vendor who is uncomfortable with that test has told you what you needed to know.

Frequently asked questions

What is the most important question to ask a legal AI vendor?

Ask them to show you a question their tool refuses to answer. A product built for legal work has deliberate abstention behaviour and the vendor will have examples ready. If the answer is a general accuracy claim rather than a live demonstration, treat it as a tool that will guess rather than decline.

Can I use legal AI without breaching client confidentiality?

Yes, but only if you know where the material goes. Get written answers on whether inputs are used for training, who can access them, how long they are retained and which subprocessors are involved, since most vendors are calling another company's model. Your obligations under the Australian Solicitors' Conduct Rules and the Privacy Act 1988 (Cth) sit with you, not the vendor.

How long should a legal AI trial run before deciding?

Long enough to cover at least one real matter from start to finish, which usually means two weeks minimum rather than the standard seven day trial. Short trials only ever test the demo scenario. Ask for an extension if the default period does not reach a genuine piece of work.

How do I check whether a legal AI tool is using current Australian law?

Pick a provision in your practice area that was amended recently and ask the tool about it directly. Then ask the vendor for the last refresh date of that specific source. A vendor who cannot give you a date for a named Act does not have a controlled pipeline behind the product.