AI Observability: How to Tell If Your AI Features Are Actually Working

You shipped an AI feature. Maybe it is a support chatbot, a smart search bar, a contract summarizer, or a copilot inside your SaaS dashboard. The demo went well, the launch went out, and now you have a quiet dashboard, a Stripe charge from OpenAI that grew 40% last month, and absolutely no idea whether users are getting good answers.

Welcome to the part of the AI lifecycle nobody talks about. Building the feature is the easy 20%. Knowing if it works in production is the other 80%, and most teams skip it entirely.

Why Traditional Observability Misses AI

Datadog, New Relic, Sentry, and Grafana are excellent at what they do. They catch 500 errors, slow queries, memory leaks, and degraded uptime. They were designed for a world where the system either works or throws an exception.

LLMs do not throw exceptions when they are wrong. They return a perfectly formatted, confident, JSON-shaped answer that is also completely fabricated. The HTTP status code is 200. The response time is 1.4 seconds. Datadog says everything is green. The user got told that your refund policy is 90 days when it is actually 14, and they are now writing a one-star review.

That is the gap AI observability fills. You need a layer that watches the content of the model output, not just the envelope around it.

The Five Metrics That Actually Matter

You can drown in possible metrics. Start with these five and add more only when you have a specific question to answer.

1. Latency, broken into p50 and p95. Average latency lies. A feature that responds in 800ms for most users but 12 seconds for 5% of users will quietly bleed those 5% out the door. Track the long tail separately. For chat interfaces, anything above 3 seconds at p95 is a problem.

2. Token cost per request, segmented by feature. Not total spend. Per request, per feature. The day someone adds a "summarize this whole document" button without truncation guards is the day your cost per request quintuples. You want to see that on a graph within the hour, not on next month's invoice.

3. Fallback rate. If your system has guardrails (retry logic, model fallbacks from GPT-4 to GPT-3.5, default canned responses when a tool call fails), you need to know how often each one fires. A fallback rate climbing from 2% to 18% over a week usually means a prompt regression, an API change, or a new edge case in user input.

4. User satisfaction signals. Thumbs up and thumbs down buttons are the obvious one, but only about 1-3% of users click them. Better signals: did the user immediately rephrase their question (bad), did they copy the response (good), did they abandon the session within 10 seconds of the answer (very bad), did they keep the chat open for follow-ups (good).

5. Eval pass rate against a frozen test set. This is the one most teams skip and the one that catches the most regressions. More on this in a minute.

The Tools Worth Comparing

The LLM observability space went from zero serious tools to about a dozen credible ones in 18 months. Here are the five worth your shortlist.

• LangSmith from the LangChain team. Best-in-class tracing for LangChain and LangGraph apps, also works with raw OpenAI calls. Strong eval framework. Hosted, with a usage-based price that gets pricey at scale.
• Langfuse. Open source, self-hostable, framework-agnostic. The pragmatic pick if you want full control of your data. Their hosted plan is reasonable.
• Helicone. Simplest to drop in, you change one line (the base URL) and get traces, cost tracking, and caching. Less powerful on evals than LangSmith or Langfuse, but the time-to-first-graph is under 10 minutes.
• Arize Phoenix. Open source, strong on embeddings drift visualization, popular with teams running RAG pipelines who need to debug retrieval quality.
• Braintrust. Eval-first platform. If your bottleneck is "we cannot measure quality," this is the one. Excellent prompt playground for comparing model versions side by side.

Pick one. The worst choice is to spend three weeks evaluating tools and ship nothing. The second worst choice is to roll your own logging table in Postgres because it "seems simple." It is simple for the first week and a maintenance burden by month three.

Why Vibe Checking Breaks at 10 Users

In the early demo phase, you and two coworkers test the AI feature, eyeball the responses, and call it good. This works fine when there are 50 total interactions a week and you can read all of them at lunch.

It stops working the moment you have 100 users generating 5,000 interactions a week. Nobody is reading 5,000 conversations. Quality regressions slip in unnoticed: a model upgrade changes tone, a prompt edit breaks a niche use case, a new user segment asks questions in Spanish and gets garbage. You will hear about it from a customer complaint, three weeks late, when the damage is already done.

The fix is sampling plus evals. Sample 5-10% of production traffic for human review (your observability tool will do this for you). Build a regression eval suite of 30-50 frozen examples that represent your most important use cases. Run that suite on every prompt change and every model upgrade before you deploy.

How to Actually Set Up Evals

Evals sound academic. They are not. Here is the minimum viable version.

Open a Google Sheet or a markdown file. List 30 to 50 inputs that represent the things users actually ask your AI. Pull these from real production logs. For each input, write down the ideal output, or at minimum the criteria a good output must meet (must mention the refund window, must not invent a product name, must respond in under 200 words).

Now run those 50 inputs through your current prompt and current model. Mark each output pass or fail. That is your baseline. From now on, before any prompt change ships, you re-run the suite and check that the pass rate did not drop.

Once that habit exists, graduate to LLM-as-judge evals (using a stronger model to grade responses against criteria), embedding similarity scores for retrieval evaluation, and structured eval frameworks inside whichever tool you picked above. But start with the spreadsheet. Most teams never even get to the spreadsheet.

The $3,000 Silent Bill

Here is a real pattern we see at least once a quarter when we audit AI features for clients running AI automation in production.

A team ships a chatbot. It works. Three months later, monthly OpenAI spend is $3,200 and nobody knows why. We instrument it with Helicone in 20 minutes and discover three things: 22% of requests are silent retries because of a malformed JSON parser, the system prompt grew from 400 tokens to 3,100 tokens because every PM added their pet instruction, and the context window includes the full conversation history (sometimes 40 messages) on every turn.

Fix the JSON parser, trim the system prompt, summarize old conversation turns. Bill drops to $700. None of those problems would have been caught by Datadog. All of them were obvious within an hour of having traces.

Your Week One Setup

If you have an AI feature in production right now and zero observability, here is what to do this week.

• Day 1: Pick one tool from the list above (Helicone if you want speed, Langfuse if you want self-hosting, LangSmith if you are already on LangChain). Sign up, add the SDK, instrument every LLM call with a trace ID that ties back to a user session.
• Day 2: Log full inputs and outputs, including system prompts and retrieved context. Yes, even with PII concerns, log them with appropriate redaction. You cannot debug what you cannot see.
• Day 3: Set a daily cost alert. Slack webhook into a #ai-cost channel. Threshold at 1.5x your current daily average so you catch runaway costs the same day they start.
• Day 4: Pull 50 real user inputs from your logs and write down the ideal answer. That is your eval suite. Commit it to git.
• Day 5: Add thumbs up/down to the UI. Add a "Was this helpful?" Slack digest that posts the previous day's worst-rated 10 conversations every morning.

That is it. Five days, no platform team required, and you go from flying blind to having more visibility than 80% of companies running AI in production. The rest is iteration.

The teams winning with AI in 2026 are not the ones with the cleverest prompts. They are the ones who treat their AI features like any other production system: instrumented, measured, alerted, and continuously improved based on what real users actually do.