Don't Trust Metrics Alone: Evaluate AI Models by User Behavior

Don't Trust Metrics Alone: Evaluate AI Models by User Behavior

We once added a new feature to an AI writing tool: automatic summarization. The model team reported a beautiful ROUGE score, 8% higher than the previous version. But after launch, user engagement didn't go up—it went down. I checked the logs and found many users were generating summaries, then manually deleting and rewriting them.

That's when I realized we had been measuring product success with model metrics, but users don't care about ROUGE.

This isn't to say model metrics are useless. They're essential during training and model selection. But once a model enters a product, the evaluation anchor should shift to user behavior.

Why Model Metrics Can Mislead

ROUGE, BLEU, accuracy—these measure how close a model's output is to a reference. But "good" in a product means users find it useful, use it, and pay for it. These two often diverge.

A few examples I've encountered:

• A translation model had high BLEU, but users complained the translations were too literal and lacked context.
• A Q&A bot had 95% accuracy, but users' questions often fell into that 5% edge, and the edge experience determined whether they kept using it.
• A summarization model had high ROUGE, but summaries frequently missed the specific number users cared about.

Model metrics optimize for "resemblance," while users need "usability."

Evaluating from User Behavior: Three Steps

I recommend small teams avoid building a complex evaluation platform at first. Use three steps to pull evaluation back from model metrics to user behavior.

Step 1: Define Key User Actions

Every AI feature corresponds to a few key user actions. For example:

• Summarization: Do users edit the output? How much? Do they copy it?
• Translation: Do they switch languages? Do they retry?
• Chatbot: Do they ask follow-up questions? Do they end the conversation after a certain response?

These actions don't require model metrics—just event tracking.

Step 2: Set Behavior Thresholds

For each key action, set a "normal range." For instance, an edit rate below 30% for summarization is normal; a retry rate above 20% for translation is a red flag. Thresholds don't need to be precise, but they need to exist. With thresholds, you can quickly judge: is this noise or a signal?

Step 3: Build a Feedback Loop

When behavior data deviates from thresholds, don't just look at numbers—look at specific user sessions. My practice is to pick 10 sessions per week, watch what users are doing, and see where they get stuck. This doesn't require a data analyst; a product manager can do it themselves.

A Decision Framework: Prioritizing When Behavior Deviates

When user behavior goes abnormal, should you fix the model or the interaction first? Here's a simple priority list:

1. If anomalies cluster around a specific input type (e.g., long documents, colloquial phrasing), check the model first.
2. If anomalies are spread across all inputs, check the interaction design—button placement, output format, etc.
3. If anomalies come with high churn, roll back or degrade immediately, then analyze the cause slowly.

This framework isn't complex, but it prevents a common mistake: retraining the model when the metric dips, only to find the real issue was slow page loading.

How Small Teams Can Implement

Small teams have limited resources and can't afford a dedicated evaluation team. My suggestions:

• Don't build an automated dashboard from day one. Use Excel or simple SQL queries to list behavior metrics, and review them weekly.
• Write behavior metrics into the PRD. When developing a new feature, define what behaviors count as "good" and "bad." This is far cheaper than adding tracking afterward.
• Supplement behavior data with user interviews. Behavior data tells you "what happened"; interviews tell you "why." Combine both for correct decisions.

The Boundary: When Model Metrics Still Matter

To be fair, model metrics aren't always useless. They remain primary in these scenarios:

• Model selection: Comparing base models, use offline metrics for quick screening.
• Regression testing: After each update, run the offline set to ensure no degradation.
• Cost control: When deciding whether to use a larger model, offline metrics help estimate the gain.

But once a product is live, let user behavior be the final judge.

A product manager doesn't need to become a machine learning expert, but they do need to know: model metrics are an internal language, user behavior is an external one. Your job is to translate between the two.

PaxLee