Report · AI assistant performance

AI Support Analytics

Four questions every team running an AI assistant should be able to answer: what are people asking, does it actually resolve them, are they satisfied, and is the model getting better?

500
conversations in the sample
78%
resolved overall
0.87
F1 score at v1.5, up from 0.67

What people ask about

Conversations by intent

Order Status is the single largest intent (133 of 500), so it is the first place to invest in self-service.

Resolution rate

Share of conversations resolved by intent

“Other” resolves least often (68%). Order Status, the biggest bucket, also sits below average at 75%.

Satisfaction

Average satisfaction by intent

Scores cluster near 3 out of 5 for every intent, with “Other” lowest. There is no intent where the assistant clearly delights.

Model performance

Precision, recall, F1 and misclassification by version

Across six versions F1 rose from 0.67 to 0.87 and misclassification fell from 30% to 12%.

About the data. Values come from the four full Plotly dashboards on the old site (dashboard_*.html), which were generated from the underlying sample conversation log; resolution and satisfaction are averages per intent from that log. The old site also had smaller hand-typed versions of these charts with different numbers; those were not used because they did not match the generated dashboards.

Rebuilt from:

  • dashboard_intent_distribution.html
  • dashboard_resolution_rate.html
  • dashboard_satisfaction.html
  • dashboard_model_performance.html
  • interactive_cases_refined/*_case.html (earlier hand-typed copies, superseded)

More reports