Readout mornings at our Johor Bahru tableJohor Bahru · A/B test analytics
Page Grove Base sits with product teams while an experiment closes. We write the readout, name the caveats, and leave a recommendation you can defend in the next planning meeting.
A closed test
A Johor e-commerce lead sent us logs from a two-week banner test. Conversion lifted on the surface. The assignment ratio had slipped two points by day four. Once we restored the intended split in the analysis, the lift sat inside noise, and the banner stayed off the live template.
That is the work: not a prettier chart, but a decision that survives a sceptical finance question.
How a day runs
We collect the hypothesis, the primary metric, guardrails, the intended split, and the date the assignment actually started — not the date the ticket was opened.
Sample ratio, novelty windows, overlapping tests, and whether the unit of randomisation matches the unit you report. If those fail, the readout stops until they are named.
A short written note: ship, hold, iterate, or stop. Interval estimates sit next to the point estimate. We stay on the call while the team argues with the memo.
What we take on
A written close-out for a finished or nearly finished A/B test: assignment health, primary metric, guardrails, and a ship / hold / stop call.
A critique of the experiment brief before assignment starts: metric choice, minimum effect, duration, and what would count as a failed setup.
A look across several concurrent A/B tests: overlapping assignment, shared metrics, and the habits that create false winners.
“They made us wait two extra days because the assignment log was incomplete. Annoying at the time. The wait stopped us from rolling a false winner into Hari Raya traffic.”
— Farah L., growth lead, Klang Valley retail site
From the journal
Looking at an A/B test every morning through a sale period does not make the result arrive sooner. It makes the stopping rule honest or dishonest.
Why a homepage test in Malaysian retail often needs a metric closer to the hypothesis than a site-wide conversion rate.
Holding the control is a legitimate close-out for an A/B test. Treating ‘not significant’ as a reason to keep the treatment live is how clutter accumulates.
