The thing that defines an agent is that it decides its own steps. Imagine a coffee chain hands its customer support to an AI agent, and a customer writes in: my order never showed up, I want my money back. Nobody scripted what happens next. The agent looks up the order, checks the courier status, reads the refund policy, weighs the customer’s history, and decides: refund, redelivery, or a voucher and an apology. We give it some tools and instructions, and on every turn it looks at what has happened so far and works out the next move for itself.
And that is exactly where it stops behaving like the software we are used to. Traditional software could fail too, but it failed along branches we had written ourselves, and given the same input it failed the same way every time. An agent’s decisions are unique to the situation in front of it. That is why observing these systems has to be different from how we observe traditional software, on two fronts: what we log, and how we evaluate it.
Start by finding every point in the system where an intelligent decision gets made. Each one has to be captured. For each, we want two things: the reasoning behind the decision, and the action it led to. The action is usually a tool call, a subagent, or a skill being read. The reasoning is the model’s own thinking before it acts. Just the few lines where it works out what it has understood, and why the next step makes sense.
The action tells us the agent refunded $6. Only the reasoning tells us why. Maybe the courier marked the order delivered while the photo shows someone else’s door. Maybe the agent was just being generous with our money. The log looks the same either way.
Miss it, and we are where engineers have always been at 2 am. Staring at an action, wondering what the system was thinking. This time it really was thinking. We just did not write it down.
“Courier marked the order delivered, but the delivery photo shows someone else’s door. The customer is right. A refund is the fair call here.”
issue_refund(order_8127, $6.00) ✓ EXECUTED
The hard part is scale. Settling that one missing latte took the agent four or five decisions. String them together with everything else in the conversation and you have a trace: the full record of one run, every decision point in order. A longer conversation makes a longer trace, and a chain doing this across every store produces thousands of them a day. Traditional software gave us a way out here. Correctness was clearly defined, so we could write deterministic checks, flag the mistakes, and have a human look at them. Here there is no single correct answer to check against. Refund, redelivery, or voucher can each be the right call depending on the situation, and there are far too many traces for a human to go through. Nobody reads thousands of transcripts a day. The ones who say they do are skimming.
So we need an LLM to do the evaluation, but not a bare one. Ask an LLM cold whether a trace looks fine and it will usually say yes, in the same agreeable way it says yes to most things. The judgment is only worth something when the LLM knows how to judge. It needs the heuristics for checking these decisions, and a rule book to validate against. For our coffee agent, most of that rule book already exists. It is the refund policy, the escalation rules, the tone the brand wants. The work is handing it to the judge.
Running an LLM over every trace, every day, is also not the answer. The cost is the obvious problem. The subtler one is that it only moves the pile. Judge thousands of traces and you have thousands of judgments, and nobody reads those either. You have swapped an unreadable stack of transcripts for an unreadable stack of verdicts, and paid for the privilege.
So we need a hybrid. Through smart deterministic checks we can give a very rough score to each trace first. If the agent should always look up the order before promising anything, we penalize the trace where it did not. If it issued a refund, did it check the delivery status first? Simple, almost dumb, and that is fine. These checks do not tell us whether the refund was the right call. They tell us whether the agent did what it was supposed to do.
The goal here is not a system that catches every wrong decision. That system does not exist. Chasing it is how we overfit. A new check for every mistake we have ever seen. A rule book so long nobody trusts it. You can feel the tipping point when the checks stop describing the job and start describing incidents. Look up the order before issuing a refund is a rule about the work. Do not offer a voucher over $15 on a Sunday is a scar from one bad ticket. The goal is to go from zero visibility to a meaningfully better place, with checks simple enough that we still believe what they tell us.
Once we have those rough scores, we take a daily sample: the low-scoring traces, plus some drawn at random. That goes to the LLM, which already has the heuristics and the rule book. The random slice matters more than it looks. The low scores tell us where our rules were broken. The random draw is the only thing that tells us our rules were incomplete.
Judging all 4,000 instead would cost around $1,100 a day — and produce 4,000 verdicts nobody has time to read.
This last distinction is worth being careful about. The deterministic score checks whether the agent followed the process. The LLM checks whether it actually got the answer right. These are two different things. Our agent can look up the order, check the courier, follow every step, and still refund a coffee that was sitting at the customer’s door the whole time. And it can break the path we expected and still be right. Skip the standard escalation. Send a fresh coffee with an apology. Turn an angry customer into a regular. A good agent finds routes we did not think to script, and punishing that is punishing the exact quality we hired it for. So we keep the two separate: we score the process one way, judge correctness another, and never average them into a single number that means neither.
This is not theory for us. At Orcana we run exactly this setup on our own analytics agents. Every run is scored the moment it finishes. We keep the axes separate rather than blending them into one number. Did the answer stay grounded in the data? Was the run reliable? Did it trip anything we treat as critical? That score narrows thousands of runs down to the roughly 30% worth a second look. From there the daily sample is genuinely small: the worst ten runs plus five drawn at random. That is what a person actually sits down with.
In the first week, that sample surfaced seven blind spots we had simply never noticed. None of them stayed a finding. Two became new deterministic checks. Three turned into fixes in the tools and libraries the agent calls. Two were prompt changes. By the end of the week the average deterministic score had risen 15%. On the questions that had exposed those blind spots, the judge came back clean.
Both of those numbers deserve an asterisk, and they are the same asterisks we would put on anyone else’s. Coming back clean on the cases that prompted your fixes is not evidence the system is fixed. It is evidence you fixed those cases. Only the random slice can tell us whether the problem was really the shape we thought it was. And the 15% should shrink. It should keep shrinking every week, because a large jump means there was a lot lying around to find, which is flattering exactly once.
That feedback step is the whole point, and it is the one most teams skip. Findings that do not change the system are just a nicer form of not knowing. Every blind spot should end up somewhere. A process failure becomes a new cheap check. Something the judge should have caught becomes a line in the rule book. An agent that was set up to fail becomes a prompt or tool change. The pipeline is not the product. The loop is.
And what the loop buys is not really a rising number. It is that we can say what our system did, and why, on a given day, for a given customer. That is what lets us ship a change rather than hope about it. We are all adding features to these systems quickly. Being able to ship them with confidence is the rarer thing, and right now it is the expensive one.
Most teams are still logging far less than they will wish they had, and scoring almost none of it. But the direction is clear enough: capture the decisions as they happen, use cheap checks to triage at scale, spend real judgment only where it is needed, and feed what you learn back in. Agents are taking on more of the actual decision making. Refunds for cold lattes today. Things that matter far more soon. Being able to see and question those decisions stops being a nice to have. It becomes the main way we keep the system honest.
Aryan builds the agent and evaluation systems behind Orcana’s analytics platform. Reehan is co-founder and CTO. The observability setup described here runs against our own agents every day.