AI agent assist (simulated)
Of 3,543 resolutions the AI claimed, an estimated 12% (95% range 11% to 14%) did not hold: the person came back within 7 days, beyond the normal return rate after human-resolved tickets of the same types. At $1.00 per resolution, about $435 ($400 to $479) was paid for resolutions that did not hold. Audit confidence: high.
1What was checked
A claimed resolution holds if the person does not need help again with the same type of issue soon after. For each resolution the AI claimed, the audit looked for a ticket from the same person in the following 7 days about the same type of issue.
Some people come back after any resolution, for reasons that have nothing to do with the AI. So the same check is run on tickets resolved by people, weighted to the same mix of issue types, in the same weeks. Only the excess return rate counts against the AI:
Did not hold = (return rate after the AI − return rate after people) ÷ (1 − return rate after people)
2Result
| Estimate | 95% range | |
|---|---|---|
| Came back within 7 days after an AI-claimed resolution | 13.9% | 12.8% to 15.0% |
| Came back within 7 days after a human-resolved ticket (same issue mix) | 1.8% | 1.5% to 2.0% |
| Claimed resolutions that did not hold | 12.3% | 11.3% to 13.5% |
| Resolutions that held, of 3,543 claimed | 3,108 | 3,064 to 3,143 |
| Billed at $1.00 per claimed resolution | $3,543 | |
| Paid for resolutions that did not hold | $435 | $400 to $479 |
3By type of issue
| Type | Claimed | Came back | After AI | After people | Baseline | Did not hold |
|---|---|---|---|---|---|---|
| Account Access / Password Reset | 1,725 | 236 | 13.7% | 1.9% | same type | 12% |
| Inquiry / Help / General | 705 | 97 | 13.8% | 1.0% | same type | 13% |
| Account Access / Permissions | 675 | 94 | 13.9% | 2.0% | same type | 12% |
| Software / Email | 161 | 25 | 15.5% | 1.9% | same type | 14% |
| Software / Installation | 105 | 15 | 14.3% | 2.7% | same type | 12% |
4Over time
| Month | Claims checked | After AI | After people | Did not hold |
|---|---|---|---|---|
| 2025-11 | 46 | 17.4% | 2.7% | 15% |
| 2025-12 | 142 | 17.6% | 2.7% | 15% |
| 2026-01 | 292 | 13.0% | 2.8% | 11% |
| 2026-02 | 427 | 12.4% | 1.6% | 11% |
| 2026-03 | 507 | 12.6% | 2.0% | 11% |
| 2026-04 | 680 | 13.2% | 1.3% | 12% |
| 2026-05 | 637 | 15.2% | 1.6% | 14% |
| 2026-06 | 640 | 14.4% | 1.7% | 13% |
5Robustness
The same audit with different reasonable choices. If these disagree a lot, read the result with care.
| Check | Claims checked | Did not hold |
|---|---|---|
| Return window 3 days | 3,509 | 4.6% |
| Return window 7 days main result | 3,371 | 12.3% |
| Return window 14 days | 3,217 | 12.2% |
| Any ticket counts as a return | 3,371 | 11.2% |
| Baseline from the same people only | 3,371 | 12.2% |
6Audit confidence: high
- Enough claimed resolutions and human-resolved tickets, a tight range, and the data lines up across systems.
| Coverage | Value |
|---|---|
| Rows in the AI's file | 4,741 |
| Not claimed as resolved (escalated, abandoned…) | 1,198 |
| Claimed resolutions | 3,543 |
| Too recent to check (less than 7 days of follow-up) | 172 |
| Checked | 3,371 |
| People in the AI's records who also appear in the ticket data | 100% |
| Claims whose issue type also appears in the ticket data | 100% |
| Human-resolved tickets in the baseline | 29,151 |
- 172 claims from the last 7 days could not be checked yet; the billing estimate assumes they held at the same rate.
7Data lineage
| File | Rows | Modified | SHA-256 |
|---|---|---|---|
| incidents.csv | 74,477 | 2026-10-17T05:00:27+00:00 | a5dd9e825b4798c2… |
| virtual_agent.csv | 4,741 | 2026-10-17T05:00:27+00:00 | 45589c60796fe765… |
What this audit can and cannot tell you
It can undercount failures
People who give up after a failed AI answer and call, email or ask a colleague instead leave no ticket, so they look like a resolution that held. Treat the result as a floor unless every channel creates a ticket.
Unrelated returns are netted out
Someone who comes back about something else is counted the same way after AI and human resolutions, so it cancels in the comparison; matching on the type of issue removes most of it to begin with.
People must be the same in both systems
The check joins the AI's records to tickets by person. If the two systems use different identifiers, returns are missed; the coverage table shows how many people appear in both.
Human resolutions are the yardstick, not perfection
The question is whether the AI's resolutions hold as well as your agents' resolutions do. Reopened tickets are tracked separately as a quality guardrail in the evidence pack.