Simulated service desk
Of 3,780 resolutions the AI claimed, an estimated 17% (95% range 15% to 19%) did not hold: the person came back within 7 days, beyond the normal return rate after human-resolved tickets. Audit confidence: medium.
1What was checked
A claimed resolution holds if the person does not need help again with the same kind of problem soon after. For each resolution the AI claimed, the audit looked for a ticket from the same person in the following 7 days.
Some people come back after any resolution, for reasons that have nothing to do with the AI. So the same check is run on tickets resolved by people, in the same weeks. Only the excess return rate counts against the AI:
Did not hold = (return rate after the AI − return rate after people) ÷ (1 − return rate after people)
2Result
| Estimate | 95% range | |
|---|---|---|
| Came back within 7 days after an AI-claimed resolution | 28.7% | 27.2% to 30.4% |
| Came back within 7 days after a human-resolved ticket | 13.9% | 13.5% to 14.4% |
| Claimed resolutions that did not hold | 17.2% | 15.4% to 19.0% |
| Resolutions that held, of 3,780 claimed | 3,131 | 3,061 to 3,198 |
How the platform closed the conversations it counted as resolved
| Platform's reason | Claimed | Share | Came back | Did not hold |
|---|---|---|---|---|
| Topic Complete | 1,335 | 37.1% | 19.8% | 7% |
| Auto Closed | 887 | 24.7% | 16.2% | 3% |
| No User Response | 797 | 22.2% | 55.8% | 49% |
| User Ended | 455 | 12.7% | 29.5% | 18% |
| Left for search | 122 | 3.4% | 37.7% | 28% |
3Over time
| Month | Claims checked | After AI | After people | Did not hold |
|---|---|---|---|---|
| 2025-11 | 48 | 33.3% | 15.5% | 21% |
| 2025-12 | 154 | 32.5% | 13.1% | 22% |
| 2026-01 | 318 | 27.7% | 14.6% | 15% |
| 2026-02 | 453 | 30.5% | 14.3% | 19% |
| 2026-03 | 542 | 27.5% | 14.6% | 15% |
| 2026-04 | 729 | 29.4% | 13.7% | 18% |
| 2026-05 | 672 | 28.1% | 13.7% | 17% |
| 2026-06 | 680 | 27.8% | 13.4% | 17% |
4Robustness
The same audit with different reasonable choices. If these disagree a lot, read the result with care.
| Check | Claims checked | Did not hold |
|---|---|---|
| Return window 3 days | 3,744 | 10.5% |
| Return window 7 days main result | 3,596 | 17.2% |
| Return window 14 days | 3,434 | 18.1% |
| Baseline from the same people only | 3,596 | 17.0% |
5Audit confidence: medium
- Returns could not be matched by type of issue, so unrelated tickets count as returns.
| Coverage | Value |
|---|---|
| Rows in the AI's file | 4,741 |
| Not claimed as resolved (escalated, abandoned…) | 961 |
| Claimed resolutions | 3,780 |
| Too recent to check (less than 7 days of follow-up) | 184 |
| Checked | 3,596 |
| People in the AI's records who also appear in the ticket data | 100% |
| Human-resolved tickets in the baseline | 29,169 |
- 237 interactions without the Virtual agent flag were left out.
- The claims carry no category, so a return was any ticket from the same person.
6Data lineage
| File | Rows | Modified | SHA-256 |
|---|---|---|---|
| incidents.csv | 74,477 | 2026-10-17T05:00:27+00:00 | a5dd9e825b4798c2… |
| interactions.csv | 4,978 | 2026-10-17T05:00:27+00:00 | 6ece6678ea983285… |
What this audit can and cannot tell you
It can undercount failures
People who give up after a failed AI answer and call, email or ask a colleague instead leave no ticket, so they look like a resolution that held. Treat the result as a floor unless every channel creates a ticket.
Unrelated returns are netted out
Someone who comes back about something else is counted the same way after AI and human resolutions, so it cancels in the comparison.
People must be the same in both systems
The check joins the AI's records to tickets by person. If the two systems use different identifiers, returns are missed; the coverage table shows how many people appear in both.
Human resolutions are the yardstick, not perfection
The question is whether the AI's resolutions hold as well as your agents' resolutions do. Reopened tickets are tracked separately as a quality guardrail in the evidence pack.