AI agent assist (simulated)
AI agent assist (simulated) changed agent hours per ticket by -0.11 h per ticket (-12%; 95% range -0.13 to -0.09 h, -14% to -11%). Evidence grade B. Guardrails: all pass. Cash ROI 7% to 79% (P10 to P90).
2The question as registered
- Hypothesis: AI summaries and resolution suggestions cut agent hours per ticket by at least 10% without lowering quality; the virtual agent deflects password resets and simple requests.
- Primary metric: effort_hours (expected −10%)
- Registration: Registered 2026-10-17T05:00:27+00:00 after go-live.
3Design
Staggered rollout (imputation difference-in-differences).
- 30 teams adopted in 3 waves, with 10 teams never exposed in the window. Each wave is compared with teams that did not have the AI yet.
- Rollout order was assigned by lottery, so waves are comparable by design.
Evidence grade B: Comparison-group design with diagnostics passed: counts toward measured ROI.
- Randomized rollout, but the question was not registered before go-live, so capped at B.
4Effects
Headline and mix-adjusted results side by side, per ticket. Negative means less time. The mix-adjusted view compares tickets only with tickets of the same type.
| Metric | View | Effect | 95% range | Relative | Total over AI tickets |
|---|---|---|---|---|---|
| Agent hours per ticket | headline | −0.03 h | −0.05 to −0.01 | −3.3% | −429 h |
| Agent hours per ticket | mix-adjusted | −0.11 h | −0.13 to −0.09 | −12.2% | −1,725 h |
| Resolution time | headline | +0.23 h | −0.25 to +0.75 | +1.9% | +3,651 h |
| Resolution time | mix-adjusted | −1.02 h | −1.42 to −0.61 | −7.4% | −16,001 h |
Before and after, AI teams only
| Metric | Before AI | After AI | Change |
|---|---|---|---|
| Headline mean resolution time (h) | 12.6 | 12.7 | +1% |
| Mix-adjusted mean resolution time (h) | 12.6 | 11.6 | −8% |
| Agent hours for the same work | 14,281.9 | 12,460.5 | −13% |
By work type
| Work type | AI tickets | Effect (h) | Range | Relative |
|---|---|---|---|---|
| Software / Application Error | 2,680 | −0.230 | −0.294 to −0.163 | −15% |
| Software / Installation | 2,394 | −0.075 | −0.105 to −0.039 | −10% |
| Hardware / Laptop | 2,190 | −0.126 | −0.219 to −0.043 | −9% |
| Account Access / Permissions | 2,027 | −0.056 | −0.085 to −0.032 | −11% |
| Software / Email | 1,840 | −0.051 | −0.082 to −0.021 | −8% |
| Account Access / Password Reset | 1,708 | −0.045 | −0.059 to −0.031 | −18% |
| Network / Connectivity | 1,465 | −0.117 | −0.201 to −0.040 | −10% |
| Inquiry / Help / General | 1,391 | −0.070 | −0.090 to −0.050 | −17% |
5Effect over time
6Quality guardrails
| Guardrail | Allowed | Status | Detail |
|---|---|---|---|
| Reopen rate | rise of at most 0.01 (share) | Pass | Change +0.00188 (share) (range -0.00261 to +0.00664); within the limit. |
| Reassignments per ticket | rise of at most 0.1 | Pass | Change -0.00464 (range -0.0246 to +0.0162); within the limit. |
| Escalation rate | rise of at most 0.01 (share) | Pass | Change -0.00104 (share) (range -0.00887 to +0.00589); within the limit. |
| Satisfaction score | fall of at most 0.1 pts | Pass | Change +0.000737 pts (range -0.0451 to +0.0528); within the limit. |
| False deflection rate | at most 15% | Pass | 13.6% of resolved sessions came back as a ticket within 7 days (range 12.5% to 14.8%). |
7Diagnostics
| Check | Result | Detail |
|---|---|---|
| Pre-trend test | Pass | AI and comparison teams moved together before rollout (joint p = 0.38 over 8 weeks). |
| Placebo dates | Pass | A fake rollout 8 weeks early shows no effect (-0.0109, range -0.0386 to +0.0147). |
| Placebo outcome: Share of tickets opened outside business hours | Pass | A metric the AI cannot affect did not move (+0.0057). |
| Leave-one-team-out | Pass | Dropping any one AI team keeps the estimate between -0.112 and -0.106. |
| Category drift | Pass | No sign that the category taxonomy changed with the rollout. |
| Power | Pass | With this data the smallest reliably detectable effect is ±0.0218 per unit of work (±2%). |
8Value and ROI
Period: 7.8 months from first rollout. Ranges from 10,000 simulations combining the measured effect's uncertainty with the assumptions in section 10.
| Low (P10) | Middle (P50) | High (P90) | |
|---|---|---|---|
| Agent hours released | 2,343 | 2,590 | 2,849 |
| Capacity value | $138,059 | $156,639 | $177,415 |
| Cash value | $59,413 | $78,103 | $98,716 |
| Total cost | $52,955 | $55,588 | $58,306 |
| Capacity ROI | 146% | 182% | 222% |
| Cash ROI | 7% | 40% | 79% |
| Cost per agent hour released | $19 | $21 | $24 |
Payback, middle case: 3.3 months on a cash basis; 1.1 months on a capacity basis.
Verified deflection
3,543 self-service resolutions: 3,061 verified, 482 false (13.6% came back within 7 days). 1,150 agent hours avoided. Counted: verified resolutions and a measurable fall in ticket volume for AI teams.
Costs for the period
| Component | Description | Basis | Low | Most likely | High |
|---|---|---|---|---|---|
| license | AI add-on licence uplift (illustrative) | annual | $31,146 | $31,146 | $31,146 |
| build_integration | Implementation partner (illustrative) | one_time | $8,000 | $10,000 | $14,000 |
| change_training | Agent training: 300 agents × 20 min × $60 | one_time | $6,000 | $6,000 | $6,000 |
| human_oversight | Knowledge-base upkeep and AI review time | monthly | $3,893 | $7,786 | $11,680 |
9Claimed versus proven
| Source | Claimed hours | Claimed value | Measured hours (P10 to P90) | Note |
|---|---|---|---|---|
| Vendor value dashboard (illustrative) | 9,000 | n/a | 2,343 to 2,849 | Usage × assumed minutes saved per summary |
10Assumptions
| Assumption | Low | Most likely | High | Unit | Owner | Agreed |
|---|---|---|---|---|---|---|
| Loaded cost per agent hour | 52.00 | 60.00 | 70.00 | USD | Finance partner, IT (illustrative) | 2025-11-10 |
| Share of freed hours provably redeployed (realization) | 0.30 | 0.50 | 0.70 | share | Finance partner, IT (illustrative) | 2025-11-10 |
| Extra agent hours per false deflection | 0.25 | 0.50 | 1.00 | h | Finance partner, IT (illustrative) | 2025-11-10 |
11Data lineage
| File | Rows | Modified | SHA-256 |
|---|---|---|---|
| incidents.csv | 74,477 | 2026-10-17T05:00:27+00:00 | a5dd9e825b4798c2… |
| rollout.csv | 40 | 2026-10-17T05:00:27+00:00 | dca626de90094161… |
| virtual_agent.csv | 4,741 | 2026-10-17T05:00:27+00:00 | 45589c60796fe765… |
- Excluded, missing team_id: 74
- Excluded, duplicate ID: 37
- Excluded, resolved before opened: 106
12Recommendation
Renew and expand. Cash ROI is positive even at the low end of the range.
Method appendix
Estimator
Effects are estimated with the imputation estimator for staggered adoption (Borusyak, Jaravel and Spiess, Review of Economic Studies, 2024). Team and week effects are fitted by weighted least squares using only team-weeks without the AI (comparison teams, and AI teams before their rollout). For every team-week with the AI, the fitted model predicts the outcome it would have had without it; the effect is the observed minus the predicted outcome. Effects are averaged with weights equal to the number of tickets in each team-week, so the headline is an effect per ticket.
Because AI team-weeks are only ever compared with untreated data, the estimator avoids the bias of ordinary two-way fixed-effects regressions under staggered rollouts with effects that change over time. A single rollout date with a comparison group reduces to two-group difference-in-differences.
- Design: Staggered rollout (imputation difference-in-differences)
- Teams analysed: 40 (30 with the AI, 10 without it in the window; 3 rollout wave(s))
- Window: 2025-01-06 to 2026-07-05 (78 weeks)
- Rollout weeks that start mid-week are dropped for the team concerned (partly exposed).
Uncertainty
Ranges come from a bootstrap that resamples whole teams (499 draws, seed 20261014), stratified by rollout wave so every draw keeps the design. Intervals are percentile intervals at the 95% level. Tickets within a team are not independent, so resampling teams, not tickets, gives honest uncertainty.
Work typing and mix adjustment
Each ticket is typed from its category fields (and any custom rules). Types below the configured share of baseline volume are folded into 'Other'. Mix-adjusted outcomes subtract each type's baseline mean (computed on untreated team-weeks only) before averaging, so the engine compares tickets only with tickets of the same type. The headline (unadjusted) estimate is always reported beside it.
The descriptive before/after table reweights each period's per-type means to the AI teams' before-rollout mix; it has no comparison group and is not causal.
Diagnostics
- Pre-trend: lead coefficients for the weeks before rollout are estimated on untreated team-weeks with team and week effects; their joint Wald test uses the bootstrap covariance. Fail if p < 0.05.
- Placebo dates: data after each team's real rollout is removed and the rollout is moved earlier; fail if the placebo range excludes zero.
- Placebo outcomes: metrics the AI cannot affect (e.g. share of tickets opened after hours) are estimated with the same design; fail if the range excludes zero.
- Leave-one-team-out: the estimate is recomputed dropping each AI team; fail if one team flips the sign of a significant effect or moves it by more than half.
- Category drift: Jensen-Shannon distance of the category mix before and after rollout, AI teams versus comparison teams; fail if categories appear or vanish.
- Power: minimum detectable effect = 2.80 × standard error (two-sided 5% test, 80% power).
Quality guardrails
Each guardrail is estimated with the same design (type-adjusted). Pass: the upper end of the harm range is below the threshold. Breach: the harm exceeds the threshold and its range excludes zero. Watch: otherwise. Any breach withholds the program's value. The false deflection rate is judged on its level (Wilson interval) against its threshold.
Value and ROI
Capacity value = (agent hours saved + verified deflections × baseline hours for that type − false deflections × extra hours) × loaded hourly cost. Cash value = capacity value × realization share + documented realized savings. ROI = (value − total cost) ÷ total cost.
Hours saved enter through their team-bootstrap draws; each customer assumption enters as a triangular distribution (low, most likely, high). The simulation reports the 10th, 50th and 90th percentiles. Payback = one-time costs ÷ (monthly value − monthly recurring cost).
Deflection counts only when the evidence grade counts, no guardrail breached, and ticket volume per team fell measurably for AI teams.
Evidence grades
- A: randomized holdout or randomized wave order, registered before go-live, diagnostics pass.
- B: staggered rollout or comparison group with pre-trend and placebo tests passed.
- C: valid but weaker (e.g. fewer than 12 teams; synthetic control and interrupted time series from v2): counts, flagged as weaker.
- D: diagnostics failed: indicative only, never added to totals.
- E: vendor estimates, surveys and self-reported time: shown as claimed, never added.
Data handling
- Person and record identifiers are replaced at ingest with keyed HMAC-SHA256 pseudonyms (requester_id, assignee_id, unit_id).
- Free text is used only for work typing and dropped before analysis.
- Teams with fewer people than the minimum group size are left out; results are team-level only.
- Trailing weeks with too many unresolved records are dropped (right-censoring guard).
- Exclusions: missing team_id: 74; duplicate ID: 37; resolved before opened: 106
Reproduce this result
Run `probative analyze -c probative.yaml` with the same data files, configuration and pseudonymization key. Results are deterministic given the seed.
- Probative Labs 0.4.0 (git 699d0675540a42e95acc32585e747b0354230d63-dirty), Python 3.12.15, numpy 2.5.4, pandas 3.0.6
- Configuration SHA-256: 383f5b08a739f949af1e75174d3ddd0ebc1fcd4aada3770385cd2ee4c1329476
- Registration SHA-256: b4210a791a69d2af32f899ee0b17fe354dcc14560d46e1ad7e2d1180c4cdbccd
- incidents.csv: 74,477 rows, SHA-256 a5dd9e825b4798c2…
- rollout.csv: 40 rows, SHA-256 dca626de90094161…
- virtual_agent.csv: 4,741 rows, SHA-256 45589c60796fe765…