Fake It Before You Build: Wizard of Oz Prototyping for AI Products
How do you validate an AI feature before the model is ready? Wizard of Oz testing—simulating backend behavior with humans—lets you test assumptions quickly, cheaply, and honestly. Here’s a practical framework for running these tests in small teams.
Two years ago, I was designing an AI writing assistant: input a paragraph, click a button, and get an expanded version. It sounded useful. The engineering team said a decent generation model would take at least two months. And if users didn't actually need it, those two months would be wasted.
This is the classic AI PM dilemma: models take time, but you want to validate before you commit. My solution was to build a prototype that looked like AI was powering it—except a human was behind the curtain.
Wizard of Oz testing, named after the story, means you simulate the backend manually while users believe it's automated. In software, we use it to test reactions to features that don't exist yet.
Why not just run the model?
Users often don't need perfect AI—they need something plausible. The core assumption for the expand feature was “users want to quickly expand text.” But that breaks into smaller questions: Do they actually want it? What tone and length? How long will they wait? If a human were writing, what would they say?
These can be answered before any model training. Wizard of Oz tests tackle the highest risk: will users even use this? Model performance can be improved later, but the wrong direction means zero value.
Our approach
I built an interactive prototype in Figma. Pressing the button showed a loading spinner for a few seconds, then displayed pre-written expanded text. I prepared templates for several scenarios but also left room to write on the fly. The testers didn't know there was a human on the other side.
We recruited 8 target users and gave them writing tasks: emails, weekly reports, product descriptions. We watched whether they used the expand button and how they reacted. Some edited, some copied, some frowned and went back to typing.
The key wasn't click counts—it was their comments: “This tone is wrong,” “Too formal,” “Longer than I'd write.” Those exposed the gaps in our assumptions.
A simple framework
If you want to run a Wizard of Oz test, here's a four-step model:
1. Identify the core behavior to simulate. Don't simulate the full model—only the interaction you're testing: button, delay, result. Keep the rest crude.
2. Limit the simulation scope. Prepare responses for 3–5 representative inputs. For unexpected inputs, either show “not supported yet” or improvise (and log it).
3. Control quality and latency. Keep a consistent delay (say 2–3 seconds). Quality should be decent—too bad scares users off; too good gives false confidence. Consistency matters; don't let two similar inputs get wildly different responses.
4. Design observation metrics. Beyond task completion, watch for motivation to use the feature, emotional reaction, and whether they'd recommend it. Record sessions and do short debriefs.
Common pitfalls
- Simulator inconsistency: Different people produce different quality. Stick to one simulator and rehearse.
- Lab environment distortion: Users might be more patient in a test. Insert realistic pressure or time limits.
- Fear of exposing the trick: Inform participants afterward. You can get a consent form beforehand that describes the test without revealing the simulation.
- Stopping at the test result: Wizard of Oz tells you if users like the feature, but not how to build the model. Use insights to set model baselines.
When not to use it
Avoid if the core assumption depends on latency (e.g., sub‑second multiple results) or needs live real‑time data (e.g., personalization). Also, if the human‑written output can't be replicated by a model, the test loses value—you might validate a feature the model can't actually deliver.
Outcome
The test led us to drop the one‑click expansion. Instead, we introduced an auto‑suggested short rewrite when text was selected, with an optional detailed version. The validated core behavior was “users are willing to edit from a suggestion,” not “how well the model writes.” We confirmed it with a human simulation, not two months of development.
Wizard of Oz testing isn't deception—it's honest exploration. You pretend the AI is running so you can learn, early, whether it should be.
PaxLee