Skip to main content
When an AI-assisted change reaches production, Terma marks the deploy and aligns outcome signals to the originating session. This gives you a before-and-after view at every deploy boundary — not just whether the change shipped, but what happened to cost, throughput, and resolution rate after it landed. The thread from a specific prompt in a developer’s session to a measurable production outcome becomes traceable.

​
What Terma Tracks in Production

For each AI-assisted deploy, Terma surfaces the following signals:
  • Cost per unit outcome — for example, cost per resolved support ticket or cost per auto-cleared case; calculated from the production AI workload’s token spend divided by the outcome count in the window
  • Loop depth p95 — the 95th-percentile number of tool-call loops the production workload uses to complete a task; an increase after a deploy suggests the change made the agent’s path longer or less efficient
  • Resolution and auto-clear rates — the percentage of tasks the production agent completed without human escalation
  • Latency — end-to-end response time for the production workload, tracked before and after the deploy boundary

​
Canary Analysis

For each AI-assisted deploy, Terma runs a canary comparison over a 7-day window before and a 7-day window after the deploy timestamp. The comparison is shown as a side-by-side diff of each signal, with a direction indicator (improvement, regression, or flat). This window is long enough to smooth out day-of-week variation while staying close enough to the deploy to avoid confounding from unrelated changes.

​
Example Finding

The following is an example of a production finding Terma might surface after a deploy:
CH-812 — support-agent: tighten tool-use instructions → cost regression Origin: prompts/support-agent/system.md, session S-1053 Suggested action: Scope the compliance re-check to policy-relevant tool results only, rather than running it on every tool output. This should reduce loop depth without affecting resolution rate.
In this example, the change tightened tool-use instructions in the system prompt. Loop depth nearly tripled — likely because the compliance re-check now runs on every tool result — and cost per resolved ticket rose by 22%. Resolution rate was flat, so the extra loops did not improve outcomes. The finding links directly to the session and the prompt file that introduced the change.
Production correlation shows signals aligned at a clear boundary — deploy time. It is correlation, not causation. Terma never claims an AI change caused a production outcome. Other deploys, external load changes, and data shifts can all affect the signals in the same window. Use these findings as starting points for investigation, not conclusions.

​
Run-Time Workloads

Every production AI run belongs to a workload and is attributed to a beneficiary. Beneficiary types include:
  • Tenant — a specific customer or account the workload is serving
  • Feature — a product feature that initiated the AI call
  • Shared asset — infrastructure used across multiple features or tenants
  • Platform — internal platform tooling
Terma shows cost and outcome metrics per workload and per beneficiary type, so you can see whether cost regressions are uniform across all tenants or concentrated in a specific customer segment or feature.

​
Beneficiary Mix

The beneficiary mix view breaks down your total production AI spend across tenant, feature, shared asset, and platform categories. This is useful for:
  • Chargeback — understanding how much of your AI infrastructure cost is attributable to each tenant
  • Feature prioritization — seeing which features consume the most AI spend relative to the outcomes they produce
  • Platform efficiency — identifying shared assets with disproportionate spend

​
Retry Storm Protection

Production AI workloads can be exposed to retry storms — for example, a webhook that re-fires hundreds of times due to a downstream error. Terma enforces per-tenant cost envelopes that prevent runaway spend when this happens. When a single tenant’s spend rate spikes beyond the envelope threshold — for example, a webhook re-firing 240 times in a short window — Terma flags the event and the envelope absorbs the excess before it reaches your provider bill. The event appears in the production findings panel with the tenant, the trigger, and the amount intercepted.

​
Linking to Dev-Time

Every production finding links back to its origin in the dev-time record. Click any finding to navigate to:
  • The originating session — the developer session where the change was authored
  • The prompts and files that were edited during that session
  • The commit and PR that carried the change into production
This closes the loop from a production cost regression back to the specific session, the engineer who ran it, and the prompt that drove the change — giving you the full context you need to decide what to change next.