How can you detect AI feature degradation?

An AI feature has degraded when users get less value per attempt, even if every model request returns HTTP 200. That definition matters. A feature can stay technically available while answers become less useful, responses slow down, fallback text replaces the intended result, token consumption doubles, and users quietly leave.

I have seen teams declare a repair successful because the error chart went green while the completion rate kept falling. They had measured the provider call, not the user outcome. Production verification needs one request-level record that connects the entire path: what the user tried, which application and model versions handled it, how long each stage took, what the model produced, whether a fallback appeared, what it cost, and what the user did next.

The six signal families in this article answer different questions. Latency shows whether the feature responds within its useful time window. Error rate finds explicit and disguised failures. Quality checks determine whether a response can do the job. Fallback use exposes hidden loss of capability. Token spend catches inefficient prompts and loops. Abandonment tells you when users have stopped waiting or stopped trusting the result. A remediation is verified only when the affected cohort improves across the relevant signals without making another one worse.

Start with one event for each user attempt

A request-level event is the smallest useful unit of AI production monitoring because averages cannot reconnect a slow call, a poor answer, and the user who abandoned it. Emit one record when the attempt reaches a terminal state. Give it an opaque request ID, then attach that ID to traces, model evaluations, cost records, and product events. Do not log raw prompts or outputs by default; store redacted content, hashes, classifications, or a reference governed by your retention controls.

The event needs enough dimensions to compare like with like. At minimum, record the feature and task type, application release, prompt version, model and provider, tenant or plan class, region, input size band, streaming status, status, retry count, fallback reason, input and output tokens, first-token and total latency, quality result, and subsequent user action. Use bounded labels in metrics. Keep request IDs and other high-cardinality values in logs or traces, not metric labels.

A terminal event can look like this:

{"request_id":"req_7f2c","feature":"support_reply","task":"draft","app_version":"2026.07.4","prompt_version":"p18","model":"model-a","region":"eu-west","input_band":"1k-4k","status":"success","retry_count":1,"fallback":"none","latency_ms":4820,"first_token_ms":910,"input_tokens":1824,"output_tokens":436,"quality":{"policy_pass":true,"schema_pass":true,"grounded":"unknown"},"user_action":"accepted","action_delay_ms":6400}

This record is an instrumentation contract, not a universal schema. A retrieval feature also needs retrieval count, empty-result status, source IDs or source classes, and citation validity. An agent needs tool-call count, tool failure category, loop termination reason, and side-effect status. A classifier needs predicted class, confidence band, and later ground truth when it arrives.

OpenTelemetry's semantic conventions give teams a shared vocabulary for operation duration, model attributes, and token usage. Use those names where your instrumentation supports them, but keep product outcome fields beside them. A standard model span cannot know that a customer accepted a draft or closed the dialog. That last join belongs to your application.

Before changing code, capture a baseline window that represents the same traffic mix you will use for verification. Save the query, cohort filters, sample counts, and dashboard time zone. If you compare Monday's enterprise traffic with Sunday's free traffic, the chart may move more than the repair did.

Tail latency reveals the traffic failure

Latency degradation usually appears in the tail before it changes the average. Track median, p90, p95, and p99 total latency, plus time to first token for streaming responses. The median describes the ordinary request. The upper percentiles expose queueing, rate limits, cold starts, long contexts, retries, and a small tenant or region that the average hides.

Measure at least four boundaries: browser click to visible response, application server time, provider call time, and post-processing time. The provider can stay fast while retrieval or a database lookup stalls. Server timing can look healthy while the browser waits on a blocked stream. A single timer invites the team to fix whichever dependency already has a chart.

Define a service objective from user tolerance, not from a convenient round number. For a drafting feature, you might require 95% of eligible attempts to show the first useful token within the product's measured patience window. For a background extraction job, total completion time matters more. Establish those limits from baseline behavior and product research; do not copy thresholds from another feature.

Prometheus histograms make percentile checks aggregatable when bucket boundaries match the decision you need to make. The Prometheus documentation warns that client summaries and server-side histograms have different aggregation properties. For a classic histogram, a useful query by version is:

histogram_quantile(0.95,
  sum by (le, app_version) (
    rate(ai_feature_duration_seconds_bucket{feature="support_reply",status="success"}[10m])
  )
)

Exclude failed attempts only when an error chart covers them beside the latency chart. Otherwise, dropping timeouts makes latency appear to improve during an outage. Keep separate views for successful calls, all terminal attempts, and timeouts. Also graph request volume because a falling p95 during a traffic collapse is not a recovery.

For streaming responses, first-token latency and total latency can move in opposite directions. A prompt change may produce an early token quickly but ramble for twice as long. Users may perceive the feature as responsive while compute cost and time to a usable answer deteriorate. Treat both timers as independent release checks.

After remediation, compare latency distributions for the affected segment and an unchanged control segment. Require enough observations to populate the tail; a p99 based on a handful of attempts is theatre. Look for retry growth and queue depth at the same time. A lower p95 bought by abandoning more requests at a shorter timeout is a trade, not a fix.

Count soft failures alongside exceptions

Error rate must include any attempt that cannot deliver the promised outcome, not only thrown exceptions and non-2xx responses. Provider timeouts, rate limits, malformed structured output, empty completions, content policy refusals, tool-call failures, retrieval with no usable source, validation rejection, and exhausted retries all deserve explicit terminal categories. If the application replaces them with friendly text and returns 200, the user still experienced a failure.

Use mutually exclusive terminal statuses so the denominator remains honest. Start with success, user_error, system_error, policy_block, and abandoned, then store a specific reason underneath. Decide in writing whether a partial result counts as success for this feature. A generated report missing one required section may be partial success in an editor and a system error in an automated export.

The basic production check is failed eligible attempts / all eligible attempts. Keep user errors, such as an unsupported file type explained before inference, out of the system reliability numerator, but show them separately. A sudden increase can still reveal a confusing interface or a changed input mix. Report rate and count together. One failure out of two attempts and five hundred failures out of a million have the same kind of percentage problem in reverse.

Retries need their own rate and attempt distribution. The Amazon Builders' Library explains that retries add load to a dependency that may already be overloaded, and that retries at several layers can multiply work. That warning applies directly to AI calls, which are slow and expensive enough to make amplification painful. Record the layer that retried, cap attempts, use backoff with jitter, and emit the original failure even when a later attempt succeeds.

Watch recovered errors as a leading signal. If success stays flat while the share of requests needing a second or third provider call rises, capacity or dependency health has already degraded. The user sees extra latency and the finance team sees extra spend before the availability chart changes.

Alert on fast burn and slow burn. A sharp error spike needs a short window, while a prompt parser that fails an extra fraction of traffic for days needs a longer comparison. Slice both by release, model, prompt, region, tenant class, input band, and task. Do not alert on every slice independently; route a top-level alert, then use slices to find where it burns.

To verify a repair, require the targeted reason code to return to or beat baseline while total failures also decline. Otherwise the code may have relabeled the same failure. Inspect a sample of terminal records and confirm that each status matches what the user received.

Output quality needs delayed truth and live proxies

Model output quality is not availability. A fluent answer can be wrong, unsupported, incomplete, unsafe, off-format, or irrelevant while infrastructure metrics remain perfect. Measure quality against a task-specific rubric, and separate hard constraints from graded judgments. Mixing them into one score hides what failed.

Hard checks run on every eligible response when possible. Validate JSON against its schema, required fields against business rules, citations against retrieved sources, tool arguments against allowlists, and language against the requested locale. Record each check as a boolean or named reason. Never let a mean score cancel a schema violation that breaks the next application step.

Graded evaluation needs a stable sample and a rubric with observable criteria. For a support draft, score whether it addresses the customer's request, uses facts present in the approved context, avoids inventing policy, and gives an actionable resolution. Keep the rubric version, evaluator version, and sampling rule on every result. When the evaluator changes, overlap old and new versions on the same sample before joining their time series.

Automatic evaluators are useful for coverage, not as unquestioned truth. Calibrate them against human review, measure disagreement by task and language, and send uncertain or high-impact cases to people. A model grading a model may drift when either model changes. If the evaluation prompt shares the same blind spot as the production prompt, the chart can confidently approve bad work.

Google's guidance on production ML makes an important distinction: live ground truth often arrives late or never, so teams need proxy metrics and should pay more attention to changes than to an isolated raw value. For generative features, useful proxies include schema pass rate, citation support rate, edit distance before acceptance, regeneration rate, copy or accept rate, complaint rate, and downstream task completion. None is universal. A high copy rate can mean a great draft, or it can mean users copied text elsewhere to repair it.

Build a small continuously labeled set from real, redacted production tasks. Sample by important slices rather than pure randomness, or the busiest easy task will dominate. Include failures, long inputs, uncommon languages, new customers, and requests near policy boundaries. Review the same fixed sentinel set on every release for comparability, then add a rotating production sample to catch changing traffic.

Declare degradation when a hard-pass rate breaches its release threshold, a graded criterion moves materially below its baseline with adequate sample size, or a user proxy changes in the same direction as reviewer findings. Do not demand one magical quality number. The evidence is stronger when constraint checks, reviewers, and behavior agree, and more interesting when they disagree.

A remediation passes when the failing criterion improves on the original traffic slice and does not regress safety or another high-impact criterion. Read examples from before and after. Aggregate scores tell you that something moved; paired examples tell you whether the change fixed the actual defect.

Fallback use is hidden loss of capability

A fallback can preserve the page while the AI feature has already failed. Track fallback invocation rate as attempts that returned a fallback / eligible attempts, split by fallback type and reason. Also track fallback success from the user's point of view. Static advice, cached output, a smaller model, and a manual workflow do not provide equivalent service.

Name each path explicitly: provider failover, model downgrade, cached response, deterministic template, feature disabled, or handoff to a person. Record whether the switch happened before inference, after a failed call, after validation, or after the user waited past a deadline. That timing explains whether the cause is capacity, quality, policy, or latency.

Teams often include successful fallback responses in the main success rate. That makes availability look stable while the intended capability disappears. Publish two rates: primary-path success and useful-outcome success. The gap is fallback dependency. If primary success falls from its baseline but the outcome rate stays level, the fallback is doing its job and the primary path is still degraded.

Set a budget for fallback use based on product promises and capacity planning. A failover model may be acceptable for brief incidents but costly or lower quality over a week. A template may prevent a blank screen but cannot count as a completed research answer. Add a maximum continuous duration as well as a rate threshold so low traffic does not hide a fallback stuck on overnight.

Test fallback paths with controlled failure injection. Simulate provider timeout, invalid output, exhausted quota, missing retrieval context, and a validation rejection. Confirm that the terminal event names the original cause, the selected fallback, the added latency, and the user outcome. A fallback that nobody exercises tends to fail exactly when the primary path does.

After a repair, primary-path success should recover and fallback invocation should return to baseline. Check that operators did not simply disable the fallback counter or move the branch. Compare provider call counts with primary successes; unexplained differences often reveal an uninstrumented retry or alternate path.

Token spend must follow a completed outcome

Token spend becomes a degradation signal when the feature consumes more input or output tokens for the same successful task. Total daily tokens mostly measure traffic. Track input tokens, output tokens, cached tokens where applicable, and monetary cost per attempt, then normalize them by accepted or completed outcome.

Useful measures include tokens per eligible attempt, tokens per primary success, cost per accepted output, and cost per completed downstream task. The last two prevent a cheap but useless model from looking efficient. Split spend by model, prompt version, feature, task, input band, retry count, and fallback path. Keep prices in a versioned lookup rather than baking current provider prices into historical events.

OpenTelemetry's GenAI conventions define token usage attributes and duration metrics, with input and output token distinctions. They provide a sound transport vocabulary. Your application still has to connect usage to accepted, edited, regenerated, abandoned, or whatever outcome carries value.

Prompt growth deserves direct alarms. Record static instruction tokens separately from user content and retrieved context if your stack exposes those counts. A templating bug can duplicate the conversation, retrieval can attach the same passage repeatedly, or an agent can loop through tools. Output caps may hide the cost symptom while truncation damages quality, so monitor finish reason and truncation rate too.

Use a reconciliation job between application events and provider billing exports. Telemetry can drop, retries may bypass the main wrapper, and provider accounting may treat cached or reasoning tokens differently. Reconciliation does not need to match each request when the provider lacks IDs, but totals by model and time window should stay within an agreed tolerance. Investigate the residual instead of quietly multiplying by a correction factor.

A practical alert compares cost per successful outcome with both a fixed budget and a recent comparable baseline. Require minimum volume, and exclude known experiments with their own budget. Then pair the alert with token distributions. A higher mean can come from a legitimate increase in long inputs, while a higher p50 within every input band points toward prompt or control-flow growth.

A repair that reduces tokens but lowers acceptance or raises fallback use has moved cost to users. Verify token and monetary deltas on the same cohort, then check quality, completion, and latency guardrails. Efficiency is the amount of useful work per unit of spend, not the smallest bill.

Abandonment connects system health to patience

User abandonment is the clearest evidence that the feature missed its useful window, but it needs an exact event definition. Count an attempt as abandoned when the user closes or leaves the feature before a usable result appears, cancels generation, starts a replacement attempt without using the first result, or remains inactive past a feature-specific timeout. Keep these reasons separate.

The denominator should be eligible attempts that actually started. Do not divide by page views, and do not treat browser disconnects as abandonment without checking whether the job continued and the user later returned. For background work, abandonment may mean deleting the job or never opening the result within a defined period. For an inline assistant, cancel and immediate retry are stronger signals.

Instrument the browser and server with the same request ID. Emit timestamps for submit, first visible token, usable result, cancel, navigation, retry, accept, edit, and downstream completion. Client events can disappear when a tab closes, so use a heartbeat or sendBeacon style delivery where appropriate and label uncertain outcomes. Do not invent certainty by classifying every missing event as a lost user.

Plot abandonment against latency buckets. This reveals the product's patience curve: the share of attempts abandoned before 2 seconds, 5 seconds, 10 seconds, and later, using boundaries that suit the feature. Compare time to first useful output rather than time to any token. A quick preamble that says nothing can improve first-token latency without keeping anyone.

Also measure regeneration and correction behavior. Users may wait for a bad answer, then retry instead of abandoning. Combine abandoned, cancelled, and regenerated_before_use into a broader failed-attempt view, while preserving each component. Acceptance without edits can help, but forced workflows and accidental clicks make it imperfect.

Seasonality and intent complicate behavioral metrics. Slice new and returning users, task type, device class, and entry point. Compare the same hour and weekday when traffic patterns matter. Use an unchanged feature or holdout as a control when a site-wide performance issue, marketing campaign, or interface change could affect behavior.

After remediation, the affected cohort should show lower abandonment at comparable traffic and intent. Confirm that attempt starts did not fall because the button became harder to find. Then check downstream completion; keeping users in a spinner longer can reduce recorded navigation while making the experience worse.

Segment first, aggregate second

Most AI feature failures affect a slice before they affect the fleet. A new prompt may break Spanish inputs, a model route may fail in one region, a retrieval change may hurt long documents, or one tenant's data shape may trigger a tool loop. Global averages dilute each case.

Attach dimensions that correspond to something you can change: application version, prompt version, model, provider, region, feature, task, locale, input band, retrieval index version, experiment arm, and fallback path. For agentic flows, add tool name and termination reason. Use privacy-safe tenant classes rather than exposing customer identity on broad dashboards.

Do not create a metric label for every possible value. High cardinality makes monitoring expensive and queries unreliable. Predefine bounded bands for input size, output size, latency, and tenant class. Keep exact request details in trace or log storage, then jump from a suspicious aggregate to representative attempts.

Every release comparison needs a cohort table. Show request count and traffic share beside latency, failures, hard quality passes, fallback, cost per outcome, and abandonment. Compare candidate versus control and current versus baseline. If traffic composition changed, stratify or reweight before claiming improvement.

Simpson's paradox is common here. Every task can get slower while the global average improves because more traffic shifted to a fast task. The reverse can condemn a good release that received harder inputs. Always inspect within-task changes before the combined number.

Canaries help separate release effects from time effects. Route a stable, deterministic share of eligible traffic to the candidate, keep assignment sticky for a user or workflow, and log eligibility as well as assignment. If only successful assignments reach analytics, the experiment has survivor bias before analysis begins.

Small slices need restraint. Show confidence intervals or uncertainty, enforce minimum counts, and combine adjacent windows when the incident permits. Do not ignore a high-impact failure because its sample is small; inspect the examples and use a hard safety rule. Statistical confidence and operational severity answer different questions.

This segmentation work often exposes AI-generated application code that scattered model calls across routes with no shared wrapper. FixMyMess uses codebase diagnosis and expert verification when repairing that kind of production path, but the durable result still depends on one instrumentation contract that future code must follow.

A repair passes only with a written evidence gate

A remediation is complete when a predefined comparison shows recovery on the broken signal, protects the other signals, and survives real traffic long enough to cover the failure mode. "The graph looks better" is not an acceptance criterion. Write the gate before deploying so the team cannot choose a flattering window afterward.

Use this sequence for an incident or a planned repair:

1. Freeze the incident cohort. Record the affected feature, task, versions, model route, regions, input bands, and time window. Save representative request IDs with sensitive content removed.
2. State the failure in measurable terms. Name the primary metric, baseline, observed value, threshold, and minimum sample. Add guardrails for error rate, hard quality checks, fallback use, cost per outcome, and abandonment.
3. Deploy to a deterministic canary. Confirm telemetry completeness before judging the code. The candidate and control must receive comparable eligible traffic, and every attempt must reach one terminal status.
4. Compare distributions and examples. Check candidate against control, then compare both with the historical baseline. Review paired failing tasks or a fixed sentinel set so an aggregate shift cannot hide the original defect.
5. Promote, hold, or roll back from the written gate. Continue watching through the next relevant traffic cycle, then record the query results and release identifiers with the decision.

A gate might say: p95 total latency for long-document drafting must return below the pre-incident threshold over at least the agreed request count; timeout and fallback rates must not exceed baseline; schema pass and reviewer scores must not decline; cost per accepted draft must stay within budget; abandonment must improve relative to the control. Your actual values must come from your product's baseline and risk tolerance.

Telemetry completeness belongs in the gate. Compare attempt starts with terminal events, provider calls with usage records, and accepted actions with known client delivery rates. If a release loses 15% of terminal events, every downstream rate can look better. Block promotion when missingness changes enough to alter the conclusion.

Keep rollback criteria distinct from alert thresholds. An alert asks someone to investigate. A rollback rule says the candidate has caused enough harm to stop exposure. Quality or safety violations may require immediate rollback from a single confirmed case, while a modest latency change needs a stable sample.

Run a shadow or replay evaluation when real exposure carries high risk, but do not call that production proof. Replays miss live queueing, provider limits, browser behavior, changing retrieval data, and user reactions. They reduce uncertainty before the canary; they do not replace it.

The final evidence packet should let a skeptical engineer reproduce the decision: cohort definition, query text, metric definitions, baseline and candidate windows, sample counts, uncertainty, reviewed examples, telemetry coverage, and the promotion result. If the packet cannot explain why users are better off and what it cost to get there, the remediation is still an opinion.