What teams said after the memo

Farah, Klang Valley retail

“They made us wait two extra days because the assignment log was incomplete. Annoying at the time. The wait stopped us from rolling a false winner into Hari Raya traffic.”

Farah’s team had treated a checkout banner as a closed win. Page Grove Base asked for daily assignment counts. The split had drifted. After the re-analysis the banner stayed off.

Daniel, digital banking onboarding

The onboarding team wanted a shorter KYC screen as the treatment. Primary metric was completed applications. Guardrail was fraud-referral rate. The memo said the completion lift was real on desktop and absent on the phone app the bank actually uses for most new customers. They shipped the shorter screen to desktop only.

Daniel wrote: “I wanted a single nationwide call. The split by device felt fussy. It was also the only version of the story that matched how people actually open an account.”

A longer note: marketplace holdout in Johor

A marketplace kept a 5% holdout “for future tests” and then used it as a convenient comparison whenever a campaign looked weak. Over four months the holdout users were no longer a random leftover; they were people who had been denied every experiment. We spent a health-check week reconstructing who sat in that bucket. The comparison charts were retired. New tests now draw a fresh holdout each quarter instead of recycling the same accounts.

The head of product told us the first debrief felt slow. She also said the recycled holdout had been a quiet habit nobody wanted to put on a slide. Slow was the point.

Mei, content homepage test

“The design review caught that our ‘primary metric’ was pageviews on a page we were about to recache. We would have spent three weeks assigning traffic to a number the cache would flatten.” Mei did not hire us for the later readout; her analyst ran it. That is a fine outcome.