The Agent KPI Is Not Tokens

Every AI product eventually discovers the dashboard.

The easy dashboard counts activity. Prompts sent. Tokens burned. Peak usage. Favorite topics. Anthropic is doing the consumer version with Claude Reflect, a new feature that shows people how they use Claude and nudges them to think about their habits. The Verge notes that it can summarize topics, delegated tasks, usage patterns, and quiet hours.

That is useful. It is also dangerously close to “look how much product you used.”

Enterprise agents need a more annoying question: did the thing pay for itself?

That is why ROI for agents in Microsoft Foundry is worth watching. The private preview connects agent traces, business-value evaluators, and operating costs so teams can see value, cost, net value, and current ROI in the same place. RedmondMag’s coverage frames it well: technical metrics like token usage, latency, and errors do not prove an agent saves time or justifies its cost.

This is the grown-up version of AI observability. Not “was the model fast?” Not “did users click the shiny button?” More like: which workflow created value, which agent version got worse, which tool call is expensive nonsense, and where is cost growing faster than the outcome?

For partners, this changes the demo.

Do not lead with a chatbot answering a happy-path question. Lead with the value model. Here is the task. Here is what completion is worth. Here is the eval. Here are the traces. Here is the cost. Here is what changed after version three.

If the agent cannot survive that dashboard, it probably should not survive production.