Choosing a primary metric when checkout is not the whole story

A team wanted to know whether a denser category grid helped people find festive gift sets. They set the primary metric to site-wide conversion. Half of converting buyers never touched the grid. The test could neither confirm nor deny the hypothesis; it could only move a number that other campaigns also moved.

A better primary was “add-to-cart from the category template among sessions that viewed it.” Guardrails stayed on refunds and bounce from that template. Site-wide conversion remained a curiosity line.

The rule we use: the metric must be able to fail if the idea is wrong. If a metric will rise because of a payment promotion that has nothing to do with the grid, it is a poor primary. Write the hypothesis on paper and underline the verb. The metric should measure that verb, not the entire shop.