Don't Trust ROUGE: Evaluate AI Products from User Behavior
Offline metrics often lie. This post shares a three-step framework that reverse-engineers model quality from user behavior—helping small teams avoid metric deception and iterate with low-cost signals.
A wake-up call from a real failure
Years ago I was leading an AI writing assistant. Before launch, we ran ROUGE-L and BLEU-4 on our test set—0.65 and 0.42, above average for similar tasks. We were full of confidence. What happened? The retention curve dove within the first week. Users complained: "The output is unusable."
The gap between offline metrics and user perception was much larger than I had imagined.
Why traditional metrics can mislead
ROUGE and BLEU are essentially n-gram overlap. They capture keyword phrases but are nearly blind to semantic coherence, tone consistency, and factual accuracy. A summarization task might hit all key entities yet produce a garbled paragraph. A translation task might score high in BLEU but read poorly.
More critical: these metrics have no idea what users actually need. In an AI writing product, does a user want "fewer edits" or "more inspiration"? The answer varies by scenario. Good metrics don't guarantee business success.
A replacement framework: infer model quality from user behavior
I later developed a three-step method designed for small teams without large-scale human annotation budgets.
Step 1: Define key business behaviors
Start not from model output quality, but from: did the user complete their goal?
- For a writing assistant: Did the user click "insert"? Did they delete and rewrite more than three times after insertion?
- For a language learning app: Did the user finish the day's plan? Did they repeat the same lesson?
- For customer support: Did the user end the conversation after the first reply? Or did they ask follow-ups?
These behaviors are directly tied to business value. Map them to possible model issues:
| User Behavior | Possible Model Issue |
|---|---|
| High insertion + high post-edit rate | Output doesn't match user expectation (style, length, accuracy) |
| Low insertion rate | Suggestions not useful or not visible enough |
| Frequent input changes | Model fails to understand initial intent |
Step 2: Build a mapping from behavior to model output
Once you have behavioral data, diagnose which part of the model went wrong. Simple group comparison works without complex statistics.
Split users into Group A (satisfied—inserted no edits) and Group B (dissatisfied—inserted then heavily edited). Extract model outputs for both groups and compare distributions in sentence length, syntactic complexity, topic coverage, etc. If Group B's sentences average 30% longer, the model might be producing overly long outputs that need truncation.
No human labeling needed—just reverse-label "good" and "bad" outputs from the logs.
Step 3: Design a lightweight online dashboard
Keep offline metrics for regression testing, but the daily dashboard should show behavioral indicators. I usually watch only three numbers:
- Adoption rate: % of generated results that users actively use (copy, insert, save).
- Post-use edit rate: % of used outputs that users then modify. If >50%, output direction is off.
- Task completion: % of users who finish the core action chain (e.g., write one article → click "publish").
These three numbers combined tell you more about model usability than any ROUGE score.
A hypothetical case: AI music recommendation
Suppose I run an AI music recommender. Offline click-through rate is 0.18—not bad. But overall retention is low.
Using the three-step framework:
- Step 1: Key behavior is whether users save a recommended song or skip immediately after listening. High skip rate indicates mismatch.
- Step 2: Compare high-save users vs. low-save users. The low-save group gets recommendations from too narrow a range—model overfits history, creating a filter bubble.
- Step 3: Dashboard monitors "consecutive skips" and "whether the next song has a different style from the previous skip."
This quickly identifies the root cause, rather than blindly tuning model hyperparameters.
Boundaries and caveats
This framework relies on user behavior data. If your product is pre-launch and has zero users, you can simulate with manual testing but at high cost. Also, behavioral signals can be noisy (e.g., network issues causing failed operations). Simple data cleaning is necessary.
Another trap: focusing only on behavioral metrics may encourage short-term click actions at the expense of long-term value (e.g., recommending sensational content). Conduct a quarterly user satisfaction survey as a calibration check.
Finally
AI product evaluation is not a model problem—it's a user problem. Offline metrics are useful during development, but once live, the only valid evaluation is user behavior. Small teams with limited resources should replace expensive human annotation with low-cost behavioral signals, and focus effort where it actually impacts retention and revenue.
PaxLee