Blog

Why Your Analyst Only Answers Part of the Question

At 3pm someone asks your analyst: why did new patient starts drop? They pull the trend. Down nine percent. They cut by region. Nothing. Then by specialty, where oncology looks soft. Deeper into oncology by site of care, then by start month. And there it is: a cluster of community sites that all stalled after February.

Three hours and eleven queries later, they write one memo: oncology starts at community sites fell by half after the February formulary change. The memo is probably true. It is also one branch: the one they had time to walk.

That is the whole problem. A question like this opens too many places that need investigation, and an afternoon covers less than a tenth of them.

by region by payer mix by referral source dermatology rheumatology HCP turnover adherence gaps by specialty oncology Feb formulary change ← the memo new starts −9% walked: 3 hours, 11 queries, 1 of 125 paths unexplored: everything the deadline ate
Fig. 1. One walk through a branching question. The finding is real. The coverage is an accident of which cut got tried first.

Today’s tools automate the query, not the decision

This has not gone unnoticed. Most analysts already have a mitigation: a text-to-SQL product, a BI copilot, a visualization agent. These collapse the cost of asking one question and getting a decent answer back.
But that is not a sufficient fix. They automate query writing. The analyst still has to read the result, validate it, and decide what to ask next. The bottleneck is the decision-making, and automating that brings its own problems.

Problem 1

The plan gets written before anyone looks at the data

Agentic systems mostly solve this by planning up front: break the question into five or six smaller ones, run them, and stitch the answers together. For retrieval that works, because those smaller questions do not depend on each other.
However, analysis is not retrieval. Question four only exists because of what questions one through three found, so a plan written up front is a plan written blind.

Problem 2

Acting on a summary alone misses the context that matters

The obvious fix is a supervisor that reads the agent’s outputs and decides what to ask next. That solves the blindness, but only partly. The supervisor sees only results or summaries, and the nuances of the investigation (the assumptions, the underlying distributions, the discarded segments) get heavily distilled, or lost. So the supervisor confidently steers an analytical agent that has the full picture, using a distilled view of it. The same failure as before, one level up.

Problem 3

Going deeper is not the same as looking wider

Even if systems were able to go deeper in an analysis, the breadth would still not be covered. The memo on the drop in new patient starts only focused on one branch because the investigation anchored on specialty, and everything after was constrained by that choice. This is the core of the problem: the highest-leverage decision gets made first, with the least information. No amount of depth can rescue a poor decision at the top, and the unexplored branch never speaks up.

Problem 4

Organizational context matters most, and it is not in your data

This is the most direct blocker to solving problem 3. In practice, good starting points or anchors are rarely in the data. Nothing in a table says that payer mix was ruled out in the last business review, that the brand is near loss of exclusivity, or that shipments slipped because of a vendor issue. That lives in the organization: team meetings, who owns a territory this quarter, assumptions that shift, and what has driven a pattern before.

Four things a fix has to get right

The first two problems share a root: a decision got made somewhere other than where the evidence lives. The other two are also a pair: both are about where an investigation starts. Hence, the following four principles are created.

  • The plan is a whiteboard, not a checklist Lines of investigation get added when the data surfaces them, closed when they are covered, dropped when they stop mattering.
  • Stop when it’s answered, not when the steps run out A thread ends when it has enough to answer the question, not at a preset number of steps. There still has to be a ceiling, so a thread going nowhere stops before it burns time and cost.
  • Start in several places, not one Coverage should not be a side effect of whichever cut got tried first. Open several starting points at once, and let each go as deep as its own thread deserves.
  • Good starting points come from context, not from the data What a team owns, which cuts are usually load-bearing, what external events are hitting the numbers. None of that is in a table. It is what turns a large pool of possible cuts into a few worth opening.

The first two principles settle the architecture. The machinery around an investigation has to be systematic. It carries out a decision already made in front of the data, runs long enough to finish the thread, and stops before it runs away with itself.

So how do we fix the planner? Remove it. Let the analytics agent drive the direction itself. That provides a two-fold advantage. You cover the cost of the answers and nothing else, with no planning overhead on top. And the investigation is exactly as smart as the agent, so every improvement to the agent lands inside it for free, skipping the cost of training a supervisor separately from an agent.

01020304 Answer in full Say what it showed Revise the plan Pick one next same rigor as any turn one line, on the record add / resolve / drop question, and why repeat until the thread is answered Only step 01 calls a model. Steps 02 to 04 just record what it decided. There is no separate planner.
Fig. 2. The cycle. Four stages, three of which are a state machine. The return arrow only carries out a decision that was already made in front of the data.

The fix for breadth is not to pick a single vertical at all. The investigation is not one deep dive. It is several, fanned across breadth points, each digging on its own. Anchor one on specialty, one on referral timing, one on payer mix, one on patient support, one on the competitive picture. Let each go as deep as its thread deserves.

Agreement and disagreement both become evidence. When two investigations reach the same conclusion through unrelated data, that is far stronger evidence than one reaching it through five careful steps. And when the timing branch says formulary while the support branch says prior-auth delays, that tension is the most useful sentence in the report.

Branches can be abandoned without ego. Sunk cost is a property of the analyst, not the analysis. A branch that dies in two steps costs a parallel system nothing to drop. That makes the marginal branch cheap enough to actually try. That is the unlock: not that the machine is smarter, but that it can afford to be wrong four times out of five and still find the one branch that matters.

same question · same clock
referral timing  · never opened payer mix  · never opened patient support  · never opened competitive launch  · never opened specialty → oncology → site of care → start month … out of time new starts −9%
1 of 5anchors opened
11hops, all on one thread
0independent corroboration
specialty referral timing payer mix patient support competitive launch stops: answered stops: answered stops: thread dead stops: ceiling stops: answered one read of five walks new starts −9%
5 of 5anchors opened
11hops, spread by need
4independent cross-checks
Fig. 3. The same question, two strategies. Switch between them. Serial spends every hop proving one branch; fanned spends them establishing which branch was worth proving. Each thread stops on its own terms: answered, dead, or at its ceiling.

How we built it

Analytical Loops

In Orcana this is an Analytical Loop. Point one at a question and it takes the depth off your hands. It decides what to ask next in front of the result, runs each step as a full analysis, and keeps going until the thread is answered rather than until time runs out.

And you do not fire just one. Loops run concurrently, anchored on different breadth points. A single question comes back as one multi-level analysis across every angle worth opening, instead of you slicing the work into pieces and driving each piece by hand.

You also do not have to scroll through every question-and-answer pair. Each loop ends with an artifact written for three readers: the bottom line for a VP, the method for an analytics lead, and the full list of technical decisions (assumptions, definitions, derivations) for whoever has to defend the number in the room.

✦ Bottom linefor the VP

Oncology starts fell nine percent, concentrated in community sites after the February formulary change.

Recommendation: Treat the formulary change as the primary driver and the prior-auth backlog as a compounding factor. Brief the field on Q3 access before the next start wave.

What we found

for the analytics lead
  1. Raw trend. Starts fell from 1,240 to 1,128 a month. No regional variation.
  2. Specialty split. Oncology only. Dermatology held flat, rheumatology rose 0.4%.
  3. Site cohorts. Community starts fell 8.1% before February, 17.2% after. The break lands on the formulary change.
  4. Access control. Prior-auth denials rose 34%, but controlling for them moves the gap only from 9.1pp to 7.3pp.

Technical notes

for whoever defends it
  • Start. First dispense within 30 days of start_date, not a sample or bridge fill.
  • Cohort split. On start_date, not first_fill_date, which lags by up to three weeks and would misassign 214 patients.
  • Exclusions. 41 switch patients counted separately, 3 sites with fewer than 10 starts.
Fig. 4. One artifact, three readers. The patient-starts question from the top of this piece, as a loop actually writes it up. The conclusion on top, the reasoning under it, the derivation underneath that. Nobody reads all three, and nobody has to go looking for their part.

We argue hardest for that last layer. A loop, like every question in Orcana, is deterministic and inspectable end to end. Every assumption, every filter, every condition, definition and cohort rule behind a number stays on the record and can be shown on demand.

Nobody has to trust an investigation they cannot audit. Open any step and see exactly what it did and why. That is the difference between a result you can defend in front of a committee and one you can only quote.

Giving the system the context that isn’t in the data

Orcana captures the context that decides where to look. Which team you are on. What they are accountable for this cycle. The lines of questioning this group always ends up wanting. The nuances your data operates under: the six-week restatement, the cut settled last quarter.

That is what makes anchors proposable instead of generic. A market-access lead asking why volume slipped gets a fan pointed at formulary tier movement, prior-authorization rejection, and abandonment at the pharmacy counter. A field-operations lead typing the same five words gets call-plan attainment, territory vacancy, and targeting fit. Same question, different user, different openings.

And it compounds. Orcana learns from the investigations that came before: which anchors paid, which have moved recently, what was asked last month and what came of it. Every fan-out starts closer than the last, which leaves our users doing the only part that was ever theirs: glance, decide, ship.

A customer who didn’t know what to ask

One of our early customers arrived with a problem we had not anticipated. Not I don’t trust the answer. Something more basic.

“I don’t know what I can ask, or how this helps me find things in my data I don’t know are there.”

That turns out to be common. Plenty of business users badly want insight out of a tool like this and have no idea where to start. And once they start, they do not know how to go deeper, or wider.

A fan answers this in a way a question box cannot. They did not need a question list. They needed a topic. Pick something you care about, ask one open-ended question about it, and let the loop take it from there.

We had already ingested a good deal of their context: quarterly business reviews, past and upcoming drug releases, the things that only exist inside the organization. So an open-ended question did not come back generic. It came back as a deep dive that had already covered several bases, chosen from what we knew about their business rather than from the shape of their tables.

The depth of it surprised them. That was the part that mattered. Reading it, they could see which threads deserved more, and they fired four more loops in parallel. Between them, five cuts of the data settled a purchasing decision they had been unsure about.

The only thing they supplied was direction: the business judgment that was theirs and nobody else’s. Everything after that was coverage they did not have to plan, run, or stitch together.