Two questions, answered independently.

What changed?

Measurement plan

Before go-live, the plan fixes who gets the AI when and who waits, simulates the chance of detecting the expected effect with your own history, and records the guardrails, the diagnostics and decision rules that map each outcome to renew, expand, fix or stop. A lottery-based wave order registered before the first wave earns grade A.

Comparison designs

Staggered rollouts are estimated with the imputation estimator (Borusyak, Jaravel and Spiess, 2024). Each AI team-week is compared only with data that had no AI, which avoids the known bias of ordinary regressions when effects grow over time.

Mix adjustment

Every ticket is typed by category. Effects are estimated within type and reported beside the headline number. When AI absorbs password resets, the average remaining ticket is harder. A headline average punishes the AI for its own success; the mix-adjusted effect does not.

Uncertainty and diagnostics

• Team-level ranges. Tickets in a team are not independent, so uncertainty comes from resampling whole teams, with a small-sample correction.
• Pre-trend test. Were AI and comparison teams moving together before rollout?
• Placebo dates. A fake, earlier rollout should show no effect.
• Placebo outcomes. A metric the AI cannot touch should not move.
• Leave-one-team-out. No single team may drive the result.
• Category drift. A taxonomy change must not masquerade as an effect.
• Power. The smallest effect the data could detect, so "no effect found" is never confused with "no effect".

A failed diagnostic moves the result to grade D: shown, never added to totals.

Quality guardrails

Reopen rate, reassignments, escalations, satisfaction and false deflection are estimated with the same design. If any breaches its agreed threshold, the program's value is withheld: faster but worse earns nothing.

Value and ROI

ROI = (value − total cost) ÷ total cost. It is always a range: the measured effect's uncertainty is combined with your finance team's assumptions (loaded hourly cost, realization share) in 10,000 simulations. Capacity ROI counts freed hours. Cash ROI counts only money that provably moved.

Did the AI's claims hold?

Resolution audit

For every resolution an AI claims, the audit looks for a ticket from the same person within 7 days about the same type of issue. People come back after human-resolved tickets too, so the same check runs on your agents' resolutions in the same months, weighted to the same issue mix. Only the excess counts against the AI:

Did not hold = (return rate after the AI − return rate after people) ÷ (1 − return rate after people)

Ranges come from resampling people, widened to a leave-one-month-out jackknife when that is larger. The audit reruns with 3- and 14-day windows, any-ticket matching and a same-people baseline, and reports a high, medium or low confidence with its reasons. It needs no comparison group of teams. It is a floor: failures that end in a phone call leave no ticket.

Bill reconciliation

Every line of a per-resolution bill is matched to your conversation log by the vendor's item ID and to your tickets by person and issue type. Billing errors (billed twice, no matching conversation, not resolved, outside the period) are disputed line by line. If the contract defines a resolution as no repeat ticket within the window, each item that came back is disputed too; otherwise the audit's excess over the human baseline is reported as a negotiation figure.

Regression watch

The latest four weeks are compared with the twelve before: the share of claimed resolutions that did not hold, against the human baseline in the same weeks, and the share of conversations the AI resolved. A regression is a worsening of at least 2 points whose 95% range excludes zero. Known release dates are tested one by one. With no change planted, it raised a false alarm in 1 of 60 simulated desks; a planted jump from 12% to 24% was caught and dated in all 60.

Action audit

For tickets the AI closed by itself: how often they were reopened or followed by a repeat ticket, against your agents' closures of the same types in the same months, with the same estimator and ranges as the resolution audit.

Choosing a vendor

Bake-off

Teams are assigned to vendors and a control group by lottery. Each vendor is measured against the control teams over the same weeks. A vendor leads only if its effect excludes zero with guardrails held, and wins clearly only if its range does not overlap the runner-up's.

Why you can trust it

Validated on known answers

The engine and the audit are tested on simulated service desks with planted truths, including zero, using the same pipeline customers use. With no false resolutions planted, the audit never reported an excess; with 12% and 30% planted, its 95% ranges contained the truth in 98% and 93% of runs. The results are published.