THE OUTCOME · 3 MIN READ
Measure Shopify discount results after refunds—not just checkout
A purchase is an important event, but an order that is later refunded changes the economics. Follow outcomes through attribution and a consistent refund window.
Checkout is not the end of the economic story
A completed Shopify order records a purchase, but a later refund can reverse some or all of the revenue. If one experimental arm has more refunded orders, a purchase-only report may overstate its value. Keep purchase, discount, and refund records tied to the same attributed experiment journey.
Web Pixel checkout events are useful behavioral telemetry; the actual Shopify order and refund outcomes are the economic record. Do not treat “checkout started” or an unattributed order as proof that an offer caused revenue.
Use a compact set of honest measures
Start with eligible shoppers, attributed purchases, conversion rate, gross revenue, revenue per eligible shopper, discount spend, and discount per eligible shopper. Add refund amount and a refund rate when enough time and volume have passed for the measure to mean something.
These measures answer different questions. Conversion shows how many eligible shoppers bought. Revenue per eligible shopper shows the top-line effect across the full assigned population. Discount spend shows the observed cost of the offer. A lower cost is not a positive result if revenue or conversion deteriorates more than the cost falls.
Keep attribution and time windows explicit
Persist the assignment before the order and use reliable checkout or order identifiers to connect outcomes back to it. A matching email address or a conveniently nearby timestamp is not a substitute for a deterministic link. Label orders with missing or conflicting links as unattributed and keep them out of the experiment result.
Use the same outcome and refund windows for both arms. Recent orders may still be exposed to refunds that older orders have had time to reveal. A report should say when the refund picture is incomplete instead of silently mixing mature and immature outcomes.
Revenue is not contribution margin
Refund-adjusted revenue is more informative than checkout value alone, but it still is not profit. Reliable product costs, shipping costs, processing fees, and other merchant-specific expenses are needed for a contribution-margin claim. When those inputs are missing, report the measures you actually know.
To understand why these outcomes need a randomized comparison, revisit discount incrementality on Shopify. To plan the two-arm experiment itself, use the 10% OFF vs no-offer holdout guide.
CONTINUE THE READING PATH
