THE METHOD · 4 MIN READ
How to run a Shopify 10% OFF vs no-offer holdout test
Define eligibility, randomize before purchase, verify the offer delivered, and wait for enough attributed outcomes before making a discount decision.
1. Define the experiment before the outcome
Start with one clear question: does the normal eligible 10% offer cause additional purchases compared with no experiment offer? Write down eligibility rules, the experimental unit, the evaluation window, and the economic outcomes before looking at a result. Decide how overlapping promotions and ineligible carts will be handled.
All eligible units need a denominator, including those that never check out. Looking only at completed orders would exclude precisely the shoppers whose behavior the discount might have changed.
2. Randomize once, and keep assignment sticky
Assign each eligible journey to CONTROL (normal eligible 10% OFF) or TREATMENT (NO OFFER) before purchase. Refreshing a page, viewing another product, or adding to cart must not switch a shopper to the other arm. Persist the assignment and its experiment version so later outcomes can be interpreted against the rules that existed at the time.
Randomization does not promise that every short-lived snapshot looks identical. It gives the comparison a defensible starting point. Check for unexpected arm imbalance and do not quietly reassign difficult cases.
3. Verify what Shopify actually executed
The experiment label is not the same as the treatment the shopper received. The control arm should retain the normal eligible 10% offer; the treatment arm should receive no experiment discount. Record whether the intended action was authorized and whether the checkout or order shows that it really happened.
If an intervention fails, preserve the merchant-safe fallback and mark the delivery deviation. Do not count a shopper as successfully un-discounted just because a database row says “NO OFFER.”
4. Link checkout, order, and refunds
Connect the Web Pixel journey to cart or checkout identity, then to the Shopify order. Attribute outcomes only when the correlation is defensible. If an order cannot be connected to a pre-outcome assignment, leave it unattributed and exclude it from the causal comparison rather than guessing.
Use actual purchase and refund events for the economic readout. Refund-adjusted measurement explains why the purchase is not the final economic event.
5. Wait for an interpretable readout
Report conversion and revenue per eligible shopper next to discount spend, sample sizes, treatment-delivery quality, and uncertainty. A pre-registered maturity rule helps prevent stopping the test at a favorable-looking moment. A small difference with insufficient data is not evidence of no effect.
Once the result is mature, decide whether the discount’s incremental behavior is worth its cost for the tested population. Keep the conclusion scoped to that population and period; it is not a permanent claim about every shopper or every offer. For the bigger picture, return to the guide to discount incrementality.
CONTINUE THE READING PATH
