01Question
Can an AI agent do UX research work without making things up?
On 4 October each agent got work with known mistakes planted in it, and clean work with none. The clean work decided it: every agent invented problems that were not there.
- Accessibility audit · Refine, Buildv1: Not working. Printed 0.307:1 for every colour pair and called 8 of 8 clean elements failures.
- Research-plan check · Researchv1: Not working. Found 2-4 of 13 planted defects and invented 5 on a clean plan.
- Data synthesis · Testv1: Partly. Found the pattern, then invented a trend 26 times too large on flat data.
- Interview guide · Researchv1: Do not use. Copied leading questions into its guide and stamped them 'no leading questions detected'.
- Microcopy test · Refinev1: Not working. Found 0 of 8 planted problems.
Every serious failure had one cause. The model was asked for things it cannot know - arithmetic, sources, counts - and it supplied them anyway.
02Method
- 01
Seal the test first
Five new flawed examples, five clean ones and their answer keys were written and locked with hashes before any new agent code existed. The test cannot move to meet the agents.
- 02
Tune only on the old cases
The 4 October examples were already known, so improving against them is allowed. Their scores are never quoted as the result.
- 03
Freeze, then run once
The agent code was frozen and the sealed test ran 3 times per model. Every run counts, including the bad ones.
- 04
Two models, same test
Local 7B, the same free, offline model that failed in v1, and Cloud 120B, a model about 17 times larger. If only the big model passes, the model did the work, not the design.
- 05
Score by code
A script compares each answer with the key. The bar is the one from 4 October: catch at least half of the planted problems, invent at most one on clean work.
03What changed
Compute, never ask
Contrast comes from axe-core in a real browser; a change in the data comes from a statistical test. The model only explains numbers it was given.
Quote or drop
Every finding must quote the input word for word. Code deletes any finding whose quote is not there - an invented problem has nothing to quote.
One question at a time
The plan check asks about one kind of mistake per call and lists every instance, instead of one long review that skims.
Challenge every finding
Each finding that survives is argued against once, with the whole document in view, before it ships.
Never certify yourself
Code, not the model, checks that a flagged question did not come back into the guide and that the guide fits the time slot.
04Results
3 of 5 pass on the same small model that failed.
| Agent | v1 · 4 Oct | Local 7B | Cloud 120B |
|---|---|---|---|
| Accessibility audit | Not working | Pass3 of 3 runs · caught 7/7, 7/7, 7/7 · invented 0, 1, 1 | Pass3 of 3 runs · caught 7/7, 7/7, 7/7 · invented 0, 0, 0 |
| Research-plan check | Not working | Pass3 of 3 runs · caught 6/11, 6/11, 6/11 · invented 0, 0, 0 | Fail0 of 3 runs · caught 10/11, 6/11, 11/11 · invented 6, 8, 3 |
| Data synthesis | Partly | Pass3 of 3 runs · caught 1/1, 1/1, 1/1 · invented 0, 0, 0 | Fail1 of 3 runs · caught 1/1, 1/1, 1/1 · invented 0, 0, 0 · 2 of 3 after the scorer fix* |
| Interview guide | Do not use | Fail0 of 3 runs · caught 2/4, 2/4, 2/4 · invented 1, 1, 1 | Pass3 of 3 runs · caught 4/4, 4/4, 4/4 · invented 1, 1, 1 |
| Microcopy test | Not working | Fail1 of 3 runs · caught 3/7, 5/7, 2/7 · invented 2, 1, 1 | Fail0 of 3 runs · caught 5/7, 6/7, 7/7 · invented 4, 3, 4 |
* After the run, the scorer was found to split numbers like 6.98e-124 in two and call both made up. The saved answers were re-checked with that one fix; the official result is the one in the tag.
05What still fails
Where a number can be computed, the agents stopped inventing. Where only judgement exists, they still do.
Microcopy fails on both models
Good copy has no answer key the way contrast has a ratio. Both models flag problems in clean, plain sentences.
The bigger model invents more
Cloud 120B catches almost every planted mistake in a study plan, then finds 3 to 8 more in a plan that has none. The small model finds fewer and invents none.
The limits of this test
One flawed and one clean example per agent, written by the same person who built the agents. A pass here is evidence, not a guarantee; the next step is a test set written by someone else.