Probative LabsCheck what your AI agent actually resolved.
Drop two exports: your AI agent's conversations and your tickets. In about a minute you'll see how many conversations it marked resolved came back as a ticket within a week, next to your agents' own resolutions. Nothing is uploaded.
The steps below are for ServiceNow. On Zendesk, Jira Service Management, Freshservice, BMC Helix, Ivanti or any other tool, export the same fields as CSV. For the conversations, include the column that says how each one ended, and list the values that mean resolved under Options.
1. Virtual Agent conversations
In ServiceNow, open interaction.list, filter to Virtual agent = true for the months you want, and export as CSV with: Number, Opened for, Opened, State, State reason, Agent chat, Type.
2. Incidents for the same months, plus a week
Export incident.list as CSV with: Number, Caller, Opened, Resolved, and Category for the fix list. Use the same kind of person field as the conversations: both names, or both Sys IDs.
What to export, and what to leave out
| File | Columns | Leave out |
|---|---|---|
| Conversations interaction.list, Virtual agent = true | Number, Opened for, Opened, State, State reason, Agent chat, Type | Short description, transcripts, work notes |
| Incidents incident.list | Number, Caller, Opened, Resolved; Category (optional) | Short description, description, comments, work notes, assigned to |
To keep names out of the exports, use the person's Sys ID instead of the name: in the list layout, expand Opened for (or Caller) and choose Sys ID. Do the same in both files so the people match.
Questions
Is this the full audit?
It uses the same definitions as the open engine's resolution audit and gives the same headline figures on the same files. The full audit also matches the type of issue, runs sensitivity checks and widens the range with a month-by-month check. Run it with pip install probative-labs, or book a call and I'll run it with you.
What counts as "came back"?
The same person opened an incident within the window after the conversation started. Some people come back after any help, so the same check runs on incidents your agents resolved in the same months, and only the excess counts as "did not hold".
Can I share the result?
Download the summary: rates and counts only, with no names, IDs or ticket contents. Send that, not the exports, if you'd like a second opinion.
Our virtual agent isn't ServiceNow.
Any export with a person, a start time and an outcome works: pick the columns after you drop the file, and set the outcome values that mean resolved under Options.