Regression watch

AI agent assist (simulated)

Weeks 2025-11-10 to 2026-06-22 · latest 4 weeks compared with the 12 before · Example vendor
RegressionStatus

Regression: in the last 4 weeks, 23% of the AI's claimed resolutions did not hold, against 13% in the 12 weeks before (+10 pts, 95% range +6 pts to +14 pts). Most likely change: the week of 2026-05-25.

1Latest weeks against the weeks before

MeasureBeforeLatestChange95% rangeStatus
Claimed resolutions that did not hold13.1%23.1%+10.0%+6.0% to +13.5%Regression
Conversations the AI resolved74.1%76.0%+1.9%−1.2% to +5.3%Stable
Conversations per week194210+17For context

A regression needs a worsening of at least 2 points that the range does not include zero for. Shares that did not hold are measured against your agents' resolutions in the same weeks, so a busy or quiet week cannot pose as a change in the AI.

2Known vendor releases

Four weeks before the release against up to four weeks after, with a 95% range.

ReleaseDateBeforeAfterChangeRangeStatus
Model update (illustrative)2026-05-2512.6%22.3%+9.8%+5.6% to +14.4%Regression

3Week by week

Claimed resolutions that did not hold, by week0%5%10%15%20%25%30%11-24: 20%11-2412-08: 16%12-15: 9%12-22: 28%12-2212-29: 10%01-05: 19%01-12: 11%01-1201-19: 8%01-26: 11%02-02: 11%02-0202-09: 10%02-16: 12%02-23: 9%02-2303-02: 13%03-09: 14%03-16: 12%03-1603-23: 6%03-30: 14%04-06: 15%04-0604-13: 12%04-20: 10%04-27: 9%04-2705-04: 11%05-11: 20%05-18: 10%05-1805-25: 24%06-01: 18%06-08: 25%06-0806-15: 22%06-22: 27%Week of the claimed resolution
Share of the AI's claimed resolutions that did not hold, by week of the claim, against your agents' resolutions in the same week. Weeks with fewer than 20 claims are left out.

The largest step up was in the week of 2026-05-25: from 12.2% to 23.3%. This scan is descriptive; the test is section 1.

WeekConversationsResolved by the AIClaims checkedCame back (AI)Came back (people)Did not hold
2026-06-2219678%15328.8%1.9%27%
2026-06-1521474%15823.4%1.7%22%
2026-06-0822077%16925.4%0.7%25%
2026-06-0121275%16020.0%2.4%18%
2026-05-2519376%14625.3%1.3%24%
2026-05-1820476%15512.3%2.6%10%
2026-05-1122171%15820.9%0.8%20%
2026-05-0420475%15211.8%1.3%11%
2026-04-2720172%1459.7%0.8%9%
2026-04-2021672%15610.9%1.5%10%
2026-04-1318979%15013.3%1.9%12%
2026-04-0621276%16115.5%0.4%15%
2026-03-3019877%15215.1%1.4%14%
2026-03-2316974%1259.6%4.0%6%
2026-03-1616172%11614.7%2.5%12%
2026-03-0915668%10615.1%1.8%14%
2026-03-0214171%10014.0%0.8%13%
2026-02-2313170%929.8%1.2%9%
2026-02-1615078%11713.7%1.8%12%
2026-02-0915482%12711.8%2.2%10%
2026-02-0212175%9112.1%0.9%11%
2026-01-2610472%7513.3%2.2%11%
2026-01-1911475%859.4%1.7%8%
2026-01-1210379%8114.8%4.4%11%
2026-01-056269%4320.9%2.7%19%
2025-12-294065%2611.5%1.2%10%
2025-12-224873%3528.6%1.4%28%
2025-12-155571%3912.8%4.7%9%
2025-12-084082%3318.2%2.5%16%
2025-12-012479%19n/an/an/a
2025-11-243269%2222.7%3.8%20%
2025-11-172171%15n/an/an/a
2025-11-101369%9n/an/an/a

Probative Labs 0.4.0 · settings 42017da8a652 · run 2026-10-17T05:00:47+00:00