Putting AI to work on your data works. But not like this.

Right now, there are probably a handful of people in your organization seriously considering an AI agent that can answer questions about your data. “Just let Claude query our warehouse.” It sounds appealing: less ad hoc work for analysts, everyone can find answers for themselves, and your data team can finally do what they were hired to do.

It really can work. Anthropic recently published an article explaining how they built this internally: 95% of all internal analytics questions automated, with approximately 95% accuracy. Interestingly, a controlled experiment at a medical clinic reached almost exactly the same conclusions. But what went wrong along the way is at least as instructive.

So the question is not just whether your AI tool is good enough, but more importantly: is your data ready for AI?

The real problem is not the technology

The core problem with analytics agents is not writing queries. It is correctly matching a question to the right table, the right definition, and the right filter.

“How many active users did we have last month?” sounds simple. But an agent needs to know which table you mean, what “active” means in your context, which time window you use, and whether certain user groups should be excluded. If any of that is unclear, the agent will provide an answer that is technically correct but substantively wrong. And no one notices, because it looks perfectly normal.

A concrete example: a field named slot_status with the values Open and Filled. The agent is asked about available slots, searches for slot_status = 'Available'—a value that does not exist—and neatly returns zero results. No error message. No indication that anything went wrong. An experienced analyst would probably have checked which values actually occur first.

What doesn’t work, but is still widely attempted

Automatically deriving definitions

Have an LLM automatically generate your data catalog from your tables and query logs. Many tooling vendors are actively building this into their solutions right now.

Anthropic tried it. It didn’t work. Automatically generated definitions look complete, but they encode exactly the ambiguity you are trying to eliminate. The difference is in the details. An automatically generated description says:

“The status of the order.”

A useful description says:

“The lifecycle status of the order. Use this field for revenue analysis. Do not use this field to determine whether inventory has been reserved; use allocation_status for that.”

The first description identifies the field. The second teaches the agent how the domain works. Use AI to draft documentation, but have a person establish the definition.

Providing thousands of reference queries

Anthropic gave the agent access to thousands of historical SQL files. Accuracy barely improved. In 80% of the cases where the agent was wrong, the correct answer was already in those queries. It just couldn’t find it. The bottleneck was not access to data, but structure.

Choosing a better or more expensive model

In the experiment at the medical clinic, the model was never changed. Accuracy rose from 0% to 92% purely through better context. The model is rarely the bottleneck.

All three cases revealed the same problem: the agent did not need more information, but better information and structure.

What does work

A solid data foundation

Not fifty tables that all have something to do with “revenue,” but one place that everyone knows is the source. Clear ownership, documented definitions, and explicit decisions about what is and is not included. This is not a new lesson: it is exactly what you also need for reliable dashboards. An AI agent simply makes it more obvious when something is wrong.

Is AI part of a broader effort to develop your data environment? A data strategy helps you address architecture, ownership, definitions, and priorities as a whole.

The instruction layer that makes the difference

A semantic layer helps. In Power BI, it is explicitly part of the semantic model. On other platforms, the same principle exists under different names, such as curated datasets, business logic, and centralized definitions. That resolves ambiguity at the metric level. But an agent needs more: it must know which data sources to consult, what the pitfalls are, and how to clarify a vague question. That context is not part of your semantic model.

Anthropic builds that context into manually created instruction documents that it calls “skills.” ChatGPT has the same concept under the name “GPTs,” while Gemini calls them “gems.” Without that layer, its agent achieved only 21% correct answers. With well-designed skills: above 95%.

The knowledge these documents need to contain is domain expertise that cannot simply be written down. It resides in the minds of the people who have worked with the data for years: why a particular table exists, which filters always apply, and which exceptions were once built in after a painful mistake. These people are not the data engineers, but the analysts and domain experts. They are also the only ones who can assess whether an answer is substantively correct; domain expertise is the real differentiator.

The agent is never finished

Anthropic saw accuracy fall from 95% to 65% within one month after it stopped maintaining the agent. There are two reasons for this. Tables are renamed, definitions shift, new sources are added, and the context an analytics agent relies on changes constantly, so the instruction documents gradually become inaccurate. But it also works the other way around: you fix a bug in your data model or add a better filter, while the workaround you documented earlier remains in place. The agent then applies it unnecessarily or does something that is now actually wrong.

Anthropic sees the same pattern found in every data project: the build gets attention and budget, but subsequent maintenance does not. With a dashboard, you notice when someone complains that the figures are incorrect. With an AI agent, nobody notices because it simply keeps providing answers.

Where do you start?

A handful of well-documented datasets, a few dozen test cases, and a simple instruction layer already deliver most of the gains. But first, answer three questions honestly: is your data model documented well enough for an agent to navigate it without guidance? Is someone responsible for maintaining the instruction documents when something changes? And do you have domain experts who are willing to write down their knowledge and can assess whether the answers are correct?

If the answer to any of those questions is “not really,” that is no reason to stop. But it is a reason to improve your data foundation first.

Want to know whether your data is ready for AI?

In our AI readiness assessment, we review your data models, definitions, documentation, and ownership. Afterward, you will know where you stand, what is already usable, and what needs to happen first.

View the AI readiness assessment