Do not read the p-value until the assignment file is boring

A p-value on a tilted split is a costume. If variant B quietly received fewer new users after a release-night bug, the checkout “win” may be a different mix of people, not a better banner.

We ask for a table with calendar dates in the rows and variant counts in the columns. The table should look almost dull. Dull is the point. A two-point drift by day four is enough to stop the readout and ask engineering what happened to the assignment call.

Teams sometimes average the whole window hoping the drift “washes out.” That only hides which days were contaminated. If the bug lived from Tuesday to Thursday, those days belong in a caveat paragraph, not in a blended success story.

If you are about to close a test, send the daily counts before the metric slide. It is the cheapest way to avoid a planning meeting that has to be unpicked later.