Promotion ROI calculator

Did the bonus-points campaign pay for itself?

“Double points this month” looks like a success when lots of members book, but many would have booked anyway. This calculator compares the members who got the offer with a random holdout, counts only the extra trips, charges every bonus paid, and tests whether the difference is real. The statistics are in Impact Evaluation 101, chapters 04 and 13.

1Compare with a holdoutA random group that didn't get the offer shows what the target group would have done without it. The gap between the two trip rates is the uplift.

2Pay for every tripThe bonus goes to every qualifying trip, including the ones that would have happened anyway. That subsidy is usually most of the cost.

3Check the uplift is realWith a small holdout, a gap can appear by chance. The test and the confidence intervals say how much of the result you can trust.

Preset

Confidence level

Difference-in-differences · optional, for when the holdout isn't a clean random split

Adjust for the trip rates before the campaign Target group, before% booking in an equal window before Holdout, before% booking in the same window

The uplift becomes (target during − target before) − (holdout during − holdout before), so a gap that already existed before the campaign isn't credited to it.

Loading…

Where the money goes

From the bonus paid to the net result

Campaign waterfall

Costs step down from zero, the margin on the extra trips climbs back. The last bar is what the campaign made or lost.

The target group's trips

Every qualifying trip earned the bonus. Only the incremental ones earned margin for the campaign.

ROI against uplift

Same members, same bonus, different uplifts

Is the uplift real?

Reading the test

How big should the holdout be?

Sample size and power, for the next test

A holdout needs to be big enough to tell a real uplift from noise. Choose the smallest uplift worth detecting (the break-even uplift is a natural choice: below it, the campaign doesn't pay) and how sure you want to be of catching it. The target group stays as entered above.

Uplift to detect pp Power

Power by holdout size

How it works

The campaign economics

Each member either books a qualifying trip during the campaign window or doesn't, so a trip rate is the share of members who book. N is the size of the target group, p₁ its trip rate, p₀ the holdout's, m the margin on an extra trip, B the bonus points per trip, c the cost of a point and F the fixed costs.

Uplift u = p₁ − p₀ Incremental trips ΔT = u × N Bonus cost per trip b = B × c Campaign cost C = p₁ × N × b + F (every qualifying trip gets the bonus) of which subsidy p₀ × N × b (trips that would have happened anyway) Incremental margin M = ΔT × m ROI (M − C) / C Break-even uplift u* = (p₀ × N × b + F) / (N × (m − b))

The break-even uplift holds the baseline p₀ fixed and asks how many extra trips the same offer would need to pay for its bonus and fixed costs. It only exists if an extra trip earns more than its own bonus (m > b); otherwise no uplift makes the campaign profitable. The ROI curve uses the same logic: moving along it, the target group's trip rate is p₀ + u and the bonus bill grows with it.

The statistics

Significance uses the two-proportion z-test with a pooled standard error, two-sided. Confidence intervals on the uplift use the unpooled (Wald) standard error.

Pooled rate p̄ = (p₁N + p₀N₀) / (N + N₀) Test statistic z = u / √( p̄(1 − p̄)(1/N + 1/N₀) ), p-value = 2(1 − Φ(|z|)) Standard error of u SE = √( p₁(1 − p₁)/N + p₀(1 − p₀)/N₀ ) Interval on u u ± z₁₋α/₂ × SE Interval on ΔT N × (u ± z₁₋α/₂ × SE) Interval on ROI (N × m × u_low − C) / C to (N × m × u_high − C) / C

For the ROI interval, the cost is held at what the campaign actually paid: the number of qualifying trips in the target group is observed, so only the split between incremental and would-have-happened-anyway trips is uncertain.

Difference-in-differences. When the switch is on, the uplift is (p₁ − p₁,before) − (p₀ − p₀,before) and the counterfactual rate for the target group becomes p₁ − u. Its standard error adds the four variances, and the test is a Wald test z = u / SE. This treats the before and during windows as independent samples; with the same members in both, the true standard error is usually smaller, so the interval is on the cautious side.

An ROI below −100% is possible: it means the target group booked less than the holdout, so the campaign lost margin on top of what it spent.

Sample size and power. For an uplift δ on a baseline p₀ with p₁ = p₀ + δ, a significance level α and a target group of N members, the power of a holdout of N₀ members is

Power(N₀) = Φ( ( δ − z₁₋α/₂ × √( p̄(1 − p̄)(1/N + 1/N₀) ) ) / √( p₁(1 − p₁)/N + p₀(1 − p₀)/N₀ ) )

The calculator solves for the smallest N₀ that reaches the chosen power (bisection), and for the smallest uplift the current holdout can detect. Power rises with the holdout but levels off: if even an infinite holdout isn't enough, the target group itself is too small. The holdout's own cost is the net result its members would have produced if they had received the offer: N₀ × (u × m − (p₀ + u) × b), before fixed costs. When the campaign loses money on each member, that number is negative: the holdout saved money.

Data and sources

All inputs are illustrative assumptions, set September 2026 and stored in assets/data/promoroi.json. The preset rates, margins and costs were chosen to look plausible, not estimated from any programme. The 2,000-point bonus and 0.8 cents per point echo the worked example in Loyalty 101, chapter 09.
NIST/SEMATECH, e-Handbook of Statistical Methods, 7.3.3: comparing two proportions (accessed Sep 2026).
Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020: holdouts, power and the pitfalls of A/B tests.
Difference in differences, including the parallel-trends assumption it rests on (accessed Sep 2026).

Limitations

One trip per member: the trip rate is the share of members who book. If members can book several trips, the counts are larger and the variance is higher than a proportion test assumes; a test on trips per member (for example Poisson or a t-test on member-level counts) would be the right one.
The cost of a point is what it costs the airline to honour it. Points that are never redeemed cost nothing, so the figure should already be net of expected breakage. See the breakage model.
Margin per incremental trip is a single number. On a full flight an extra loyalty booking may displace a paying passenger, so the true margin depends on load factor and fare; on an empty seat it is close to the fare less variable costs.
No pull-forward or halo: trips moved from next month into the campaign window count as incremental, and effects after the window are ignored. Measure over a window long enough to see both, as Loyalty 101 recommends.
One test on one outcome. Looking at many segments or stopping as soon as the result looks significant inflates false positives.
The normal approximation needs enough trips in each group, roughly ten or more on each side. With very small groups or rates, use an exact test.

This is a personal project and isn't affiliated with any airline. For the business side of measuring promotions, read Loyalty 101: Measuring Incrementality.