Chat with your data: what it takes to trust the answers

Typing a question and getting a confident paragraph back is the easy part of "chatting with your data". The hard part is knowing whether the paragraph is right. A fluent answer can be wrong in ways that read exactly like a correct one. This guide is what to look for, whatever tool you use.

1. Who did the arithmetic?

A language model is good at reading a question and writing a sentence. It is not a calculator. The safest tools run the sums with ordinary code or a database query, then let the model describe the result. Ask any tool: was this figure computed, or was it written? If you cannot tell, treat it as written.

2. Can you see the working?

A trustworthy answer comes with the query or calculation behind it, the number of rows it used and the columns it read. You should be able to take the same figure to a spreadsheet and reproduce it. If the tool offers only a paragraph, you are being asked to trust it on faith.

3. Does it say how sure it is?

• A pattern across a handful of rows is a hint, not a finding. Good tools say how many records sit behind a claim.
• A search over many segments will turn up odd-looking ones by luck. Good tools say how many things they checked.
• When the data cannot answer the question, the right reply is "it cannot", with the reason.

4. Does it know what it did not read?

Large files are often sampled. A tool that reads the first few thousand rows of a million-row file and answers as if it saw everything is misleading you. Ask what fraction of the data the answer rests on.

5. Spot-check three numbers

Pick a total you already know, such as last month’s revenue. Ask for it. Then ask for a figure you can check against another system, and one you can work out by hand from a small filter. If all three match, you can lean on the tool for the questions you cannot check. If one is off, find out why before trusting the rest.

How AnalyzeIt handles this

AnalyzeIt runs the statistics first and uses the model to explain them. Numbers come from the data or from a named tool, a "Show the working" panel gives the figures and the query, and a check flags figures in the written answer that do not match what was computed. Findings are labelled by how sure the evidence is, and when a question cannot be answered from the data, it says so.

Read more: How answers are produced, or try the live demo with no account.