Experiment readout

Who it is for

Product managers, growth leads, and analysts who already ran a two-variant (or simple multi-variant) test and now need a close-out that finance and engineering will not laugh out of the room. Most clients sit in Malaysia’s e-commerce, marketplace, and digital banking teams; we also read experiments for Singapore-facing shops that staff their analysis from Johor.

The result

You leave with a decision memo: ship the treatment, keep the control, iterate on a named element, or stop the line of work. The memo states the interval around the primary metric, lists every guardrail that moved, and records the caveats that would change the call if new logs arrived.

Scope

Included: a scheduled working session, a review of assignment counts by day, a check that the randomisation unit matches the reporting unit, the primary metric with a confidence interval, up to four pre-declared guardrails, and one revision of the memo after the session.

Excluded: building assignment infrastructure, writing production code, running the experiment on your traffic, or a standing weekly reporting cadence. Those belong to your own team or another contractor.

Provider

The readout is led by a Page Grove Base analyst at our Johor Bahru office. A second reader checks the memo before it leaves the desk. You will know the names before the session.

Process

1. You send the brief, the assignment extract, and the metric definitions we list under Preparation.
2. We refuse to quote a p-value until sample-ratio and overlap checks are written down.
3. We meet for ninety minutes to walk the memo, including the awkward paragraphs.
4. You receive a revised PDF or Markdown note the next working day.

Duration and timeline

Simple two-variant tests with clean logs: one working day after intake. Tests with overlapping campaigns, late assignment, or a changed primary metric: two to three working days. We do not compress a messy log into an afternoon because a launch calendar looks tight.

Location and delivery

Sessions run at Level 8, 29 Example Avenue, Johor Bahru 00000, or on a video call if the analysts who own the logs cannot travel. The memo is delivered by email.

Preparation

Before the session we need: the hypothesis in one paragraph, the primary metric formula, the intended split, the actual first-assignment timestamp, a list of overlapping tests, and whether anyone peeked at the live numbers. Missing assignment dates are the most common delay.

Constraints

We will not sign a memo that treats a mid-test metric change as if it were pre-registered. Sequential peeks are named as such. If the sample ratio mismatch cannot be explained, the recommendation is to hold, not to average the variants until the chart looks calmer.

Price basis

From RM 3,200 for a single two-variant close-out with complete logs. Overlapping tests, more than two variants, or a second working session are quoted after we see the brief. See Rates for how estimates are built.

Next step

Write to the desk with the experiment name, the primary metric, and whether the test is still assigning traffic. Prefer a date for the session if your planning meeting is already on the calendar.