Probative LabsDid your AI's resolutions actually hold?
Virtual agents and AI agents report how many issues they resolved, and that is often what you pay for. The audit checks every claimed resolution against what happened next, independently of the vendor.
A resolution holds only if the person does not need help again.
Find returns
A ticket from the same person, about the same issue, within a week.
Find the normal rate
How often people come back after your agents' resolutions, same months and issue types.
Count the excess
Only returns above that normal rate count against the AI.
State the uncertainty
A range, robustness checks and a confidence level with its reasons.
Put a price on it
If you pay per resolution: what went to ones that did not hold.
Did not hold = (return rate after the AI − return rate after people) ÷ (1 − return rate after people)
Two exports.
- The AI's conversations: who, when, outcome and topic
- Your tickets for the same dates plus a week, from ServiceNow, Jira Service Management, Zendesk, Freshservice, BMC Helix, Ivanti or any CSV
- Optional: the vendor's bill or your price per resolution
People are pseudonymized as the files are read. Results describe the AI, never individuals.
An audit your vendor can check.
- Share of claimed resolutions that did not hold, with a range
- Breakdowns by issue type and by month
- Robustness checks and an audit confidence level
- The method and data fingerprints, so anyone can rerun it
From a finding to money back.
Reconcile the bill
Every billed resolution checked against your records: billed twice, no matching conversation, escalated, outside the period, and, under your contract's definition, did not hold. In the sample month, 110 of 650 billed resolutions were disputable.
Write it into the contract
Define a resolution before you pay for one. The model terms cover what counts as a resolution, the data the vendor provides each month, independent measurement and credits. A starting point for your counsel, not legal advice.
Watch for silent regressions
Vendors update models without notice. Each month, the latest weeks are compared with the weeks before, and known release dates are tested.
Questions
Do we need a staggered rollout?
No. The audit compares the AI's resolutions with your agents' resolutions in the same months, so it works even if every team got the AI on the same day.
What if the AI's resolutions hold up?
Then the audit says so: the range includes zero, and the report states that the AI's resolutions are not shown to fail more often than your agents'.
We use ServiceNow, and Now Assist is billed in assists. Does this apply?
Yes. ServiceNow's reporting counts a Virtual Agent chat as handled when it closes without a live agent, including chats closed because the person stopped responding. The audit reads the Interactions table directly and shows each way ServiceNow closed a conversation with how often the person came back. For assists, a usage check shows where they went and what each resolution that held cost. Every billing model.
What can it miss?
Someone who gives up on the bot and phones or asks a colleague leaves no ticket, so the audit is a floor unless every channel creates a ticket. It also needs the same person identifier in both systems; the report shows how many people it could match.
Can we run it ourselves?
Yes. The audit, the bill reconciliation and the watch are in the open-source engine: probative audit, probative reconcile and probative watch. A plan turns them into results you can act on: a finding from an independent team that the vendor can't dismiss as your own numbers, every bill checked as it arrives, and an alert when the AI slips. See the difference.