Probative LabsTested on real ServiceNow data.
Simulated data shows a method finds what was planted in it. Real data shows it copes with a real export. This page does both: a public log from a real ServiceNow instance, with known failures planted in it.
1. The engine read the real export
The log is the audit trail of an IT company's ServiceNow instance, from 2016-02-29 to 2017-02-16: every update to every incident, from 5,244 callers. The engine kept each incident's last state (141,712 events became 24,918 incidents), mapped it with its ServiceNow preset, and read its day-first dates. Its health check flagged:
- 3 week(s) with no records at all, e.g. 2016-06-13; check the extract is complete
The log has no time worked or survey scores, so the effort and satisfaction measures cannot run on it; the resolution audit needs neither.
2. Real return rates after people's resolutions
How often the same caller opened another incident within 7 days after one was resolved. These are the normal rates an AI's resolutions are compared with, and they vary a lot by category, which is why the audit compares like with like.
| Incidents | Resolved | Came back, any issue | Came back, same type |
|---|---|---|---|
| All | 23,336 | 41.3% | 10.1% |
| Category 42 | 3,124 | 43.2% | 16.5% |
| Category 26 | 3,045 | 37.7% | 7.4% |
| Category 53 | 2,587 | 36.4% | 7.0% |
| Category 46 | 2,356 | 47.3% | 8.8% |
| Category 32 | 1,268 | 43.0% | 9.7% |
| Category 9 | 1,121 | 38.7% | 3.3% |
| Category 37 | 1,098 | 36.1% | 5.7% |
| Category 23 | 1,055 | 41.0% | 6.9% |
3. The audit finds failures planted in real data
In each run, 2,000 real resolved incidents, chosen at random, were relabelled as an AI's claimed resolutions, so they carry real callers, categories and timing. Before planting, they are the same kind of resolution as the rest, so the true share that did not hold is zero. For a planted share, a return incident about the same type of issue was added within 7 days for that share of the claims that did not come back. 50 runs at each level.
| Made to fail | Found, on average | Average error | Range held the truth | Flagged |
|---|---|---|---|---|
| 0% | −0.2% | 0.7 pts | 94% of runs | 0% of runs |
| 10% | 9.8% | 0.6 pts | 100% of runs | 100% of runs |
| 20% | 19.7% | 0.4 pts | 100% of runs | 100% of runs |
With nothing planted, the audit should rarely flag a problem; with failures planted, it should flag them every time, and its 95% range should hold the truth about 95% of the time.
4. What this does not show
- An AI's effect. The log predates AI agents. Results on real AI come from design partners, and are published only with their permission.
- How real failures cluster. The planted returns are spread at random; real ones may bunch by topic or week, which the audit's month-by-month check is there for.
5. Run it yourself
pip install probative-labs
probative validate --real incident_event_log.csv --runs 50 --report real-validation.json