An analyst asks an AI assistant a simple question: summarise sales for the third quarter. The answer arrives quickly, in clean sentences, with numbers that look right. Only later does someone notice the numbers are from the second quarter. Nothing crashed. No error appeared. The answer was simply wrong, in a confident voice.

LLM tracing is the step-by-step record that lets you find out why. Instead of only seeing the final answer, you see every step the system took to reach it: what it was asked, what data it fetched, what tools it called, and what the model did with the results. When an answer is wrong, the trace shows which step went wrong, so you fix that step instead of guessing.

This matters most once an AI feature leaves a demo. In a demo, you test a handful of neat questions. In real use, questions are messy, data changes, and a rare mistake repeats at scale. A trace turns a mystery into a checklist. It is the same instinct data scientists already use: do not trust a result you cannot reproduce and inspect.

What LLM tracing actually is

Think of a trace as one complete run, from question to answer. Inside it are spans. A span is a timed record of one step: a model call, a data lookup, or a tool call. Spans sit inside each other like a tree. The whole run is the trunk, and each step is a branch.

That tree shape is the key difference from a log. A log is a list of lines: a chat happened at 3 p.m., it took two seconds, it succeeded. A trace shows order and cause. You can see that the model called a search tool first, that the tool returned a second-quarter file, and that the model then summarised that file correctly. The model was not the problem. The data it was given was.

Traces can also be grouped into sessions. A session is a whole conversation or a whole agent run, made of several traces. Instead of reading disconnected requests, you read the full story: what the user asked first, what the assistant tried, and where it went off track.

What a trace records

A useful trace records the same facts for every step, so traces can be compared. The table below is the minimum worth asking for from any tracing tool.

Step	What the trace should show
Prompt and version	The exact instructions and prompt version behind the call
Model call	Model used, tokens in and out, latency, and cost
Data and tool calls	What was fetched or called, what came back, and how long it took
Final answer	The exact answer sent to the user, linked to the steps above

Tokens and cost matter because they turn quality problems into measurable ones. If one kind of question suddenly costs twice as much, or takes twice as long, the trace shows whether the cause was a longer prompt, a repeated tool call, or a bigger model than needed.

Reading a trace to find the bad step

Go back to the sales example. The trace for that answer would read, in order: user question, model decides it needs sales data, tool call fetches a file, model summarises the file, final answer. The file name in the tool step is the giveaway: it says Q2, not Q3.

Each possible failure points to a different fix. If the tool fetched the wrong file, fix the search or the file naming. If the tool fetched the right file but the model ignored it, fix the prompt. If the model never called the tool at all, fix the instructions that tell it when a tool is required. Without the trace, all three look identical from the outside: a wrong number in a nice sentence.

This is also where LLM monitoring starts. Monitoring is watching traces over time, not just reading one. If the same tool step starts failing, or answers for one kind of question start costing more, the pattern shows up in the traces before users start complaining about it.

Figure: one trace as a tree of steps. Illustrative example, not a real trace.

What to track first

You do not need to track everything on day one. Start with these, for every call that matters:

• A trace id and a session id, so one answer can always be found again.
• The prompt version, so you know exactly which instructions produced the answer.
• The model, tokens, latency, and cost for each step, not just for the whole run.
• Tool inputs and outputs, so a wrong tool result is visible and not hidden inside a summary.
• Feedback when a human marks an answer right or wrong, so bad traces become test cases later.

Keep the failed examples. A small set of questions that went wrong before is the best test set for a fix. If a new prompt or a new data source does not pass those examples, it is not ready, however good it looks on easy questions.

How to choose a tracing tool

There are two common ways to get traces. One is to route calls through a gateway, so tracing is already on for every call that passes through it, with nothing extra to instrument. The other is to keep your existing model calls and send spans over OpenTelemetry, which suits teams that do not want to move their calls behind a new service.

Whichever route you take, check four things. Can you filter traces by model, cost, latency, prompt version, and session, so you can find the one bad trace among thousands? Can you walk the span tree and read the input and output of each step? Can you control what is stored, so sensitive request and response text is only kept when your team decides it should be? And can a bad span be rated and linked back to the prompt version behind it, so the fix starts in the right place?

A simple start this week

First, turn tracing on for one feature, not for everything at once. Second, read one bad trace and one slow trace end to end, and write down which step failed in each. Third, fix only that step, then re-test against your failed examples. Fourth, repeat weekly, and watch for patterns: the same tool failing, the same question type costing more, or the same prompt version behind the bad answers.

That routine is the whole point. An AI answer is not a single event. It is a chain of steps, and a chain can be inspected, measured, and fixed one link at a time. Once you can follow an answer step by step, a confident wrong answer stops being a mystery. It becomes a trace, a failed step, and a fix you can prove.