One Store First

In April 2025, OpenAI shipped an update to GPT-4o, the model behind ChatGPT. It had passed OpenAI's own offline evaluations and its A/B tests. Then real people got it, and it fawned on almost anything, cheering on plainly bad ideas in the same warm, certain voice it used for everything else.

OpenAI rolled the update back within days and, in its own account, called what it had shipped "overly flattering or agreeable." The lesson is not about how a model should behave. It is narrower: the strongest evidence a top AI team had said yes, and yes was wrong. Passing a test is not the same as helping. In a case Kohavi's team later documented, an online store called Doctor FootCare once rebuilt its checkout page certain the new version was better, then lost about 90% of its revenue the moment it ran the old design against the new one head to head. Neither failure was visible to the trained eye. Only a controlled comparison told the truth.

One Store First is the pilot-and-measure playbook for the operator who must decide whether an AI change is safe to roll out everywhere, and who has no data-science team, no experimentation platform, and no patience for a guess dressed up as proof. It takes the discipline the biggest AI-scale companies use to catch their own bad ideas, one number picked in advance, one honest comparison, one rule written down before anyone looks at the result, and shrinks it to what a single store or one queue can actually run.

It is a field manual, not a lecture. Across 18 chapters you build one piece of the method and assemble the whole pilot:

• The Pilot One-Pager. One paste-ready page, built a line at a time, that carries the whole decision: the number, the guardrails, the rule, the ramp.
• The metric definitions library. Pre-committed weights for the one number that decides ship-or-not and the guardrail metrics a win cannot break.
• The sample-size and duration calculator. How many users and days a pilot needs before it can honestly answer the question, and what to do when your traffic can't get there.
• The design chooser. The strongest fair comparison you can run when you can't randomize a clean split.
• Trust-gate checklists. The checks that prove your own measurement is not bent before you trust what it says.
• The AI-change playbook. The traps specific to AI, a model that won't hold still, a vendor deprecation you didn't schedule, that make this harder to judge than an ordinary rollout.

None of it requires a data-science team. It requires the one-pager, the nerve to write the rule down before you look, and honesty about which of two endings you earned: sometimes you can prove the change helped; sometimes your traffic is too thin to prove anything, and the honest win is smaller, that you limited how far a bad change could reach and kept a record of why.

This is a volume in The Operator's AI Library: for operators who must decide whether an AI change is safe to bet the whole chain on, and prove it instead of guessing. Readers of the author's Verifier's Library will know the discipline. That shelf asks whether an AI's answer is trustworthy; this one asks whether the change you made because of it actually helped.