Airline Impact Evaluation 101

Every airline changes things all the time: a new forecasting model, a bonus-points campaign, a new status tier, a price test. And after every change someone asks: did it work? Answering that honestly is harder than it looks, and most of the numbers that circulate in companies after a launch are wrong in a predictable direction. This page covers the statistics of answering it well, applied to the two areas I know best, revenue management and loyalty. It goes deeper than the short chapter on incrementality in Loyalty 101, and most chapters come with a simulation you can play with: every result is generated in your browser from made-up data, so you can see the methods succeed and fail.

Chapter 00: Why Impact Evaluation?

Three claims I have heard versions of many times:

Three claims that sound convincing

"The campaign worked: members who got the bonus offer booked 40% more than those who didn't." But the offer was sent to active members, who book more anyway.

"The new forecaster added 4% revenue: revenue per seat is up 4% since it went live." But the whole market was up 3% that season.

"Gold members spend three times more than Silver members, so status drives spend." But people become Gold because they already spend a lot.

Each claim compares two numbers and attributes the difference to one cause. Each time, something else could explain most of the gap. Impact evaluation is the discipline of isolating the part of a change that was actually caused by what you did, and of saying how sure you are about it.

In airlines this matters more than in many industries, for three reasons. The baseline moves constantly: seasonality, competitors, fuel prices and economic cycles all shift demand from month to month. The effects we care about are small: a good RM change might add 1 or 2 percent to revenue, a good loyalty campaign a fraction of a percentage point to booking rates. And the costs of being wrong are asymmetric: a campaign that pays points to customers who would have flown anyway can look like a success for years.

The one question

Every method in this page is an answer to the same question: compared to what? The effect of anything is the difference between what happened and what would have happened without it. The second half of that sentence is never observed. It has to be estimated, and the quality of that estimate is the quality of the evaluation.

Chapter 01: The Counterfactual

The cleanest way to think about causal effects is the potential outcomes framework. For each member $i$, imagine two versions of the future: $Y_i(1)$, what they would spend if they receive the offer, and $Y_i(0)$, what they would spend if they don't. The effect of the offer on that member is the difference.

Individual and average effects

\[\tau_i = Y_i(1) - Y_i(0) \qquad\qquad \text{ATE} = \mathbb{E}\big[Y(1) - Y(0)\big] \qquad\qquad \text{ATT} = \mathbb{E}\big[Y(1) - Y(0) \mid T = 1\big]\]

ATE is the average treatment effect over everyone; ATT the average effect on those who were actually treated. For a campaign sent to a chosen segment, the ATT is usually the number you want.

The fundamental problem of causal inference is that we only ever see one of the two outcomes for each member. A member either got the offer or didn't. So any comparison between groups compares different people, and the question is whether those people are comparable.

Where naive comparisons go wrong

The simple comparison, average outcome of the treated minus average outcome of the untreated, can be split exactly into two parts:

Naive difference = effect + selection bias

\[\underbrace{\mathbb{E}[Y \mid T{=}1] - \mathbb{E}[Y \mid T{=}0]}_{\text{what you observe}} \;=\; \underbrace{\mathbb{E}[Y(1) - Y(0) \mid T{=}1]}_{\text{ATT}} \;+\; \underbrace{\mathbb{E}[Y(0) \mid T{=}1] - \mathbb{E}[Y(0) \mid T{=}0]}_{\text{selection bias}}\]

The selection bias term asks: without any treatment, would the treated group have behaved differently from the untreated group anyway?

The bonus offer, decomposed

An offer was sent to members who flew at least twice last year. They booked at 12% during the campaign; everyone else booked at 4%. The naive "effect" is +8 points.

But frequent members book more anyway. Had they not received the offer, they would have booked at 10.5%. So the effect on them (the ATT) is 12 − 10.5 = 1.5 points, and selection bias is 10.5 − 4 = 6.5 points. Most of the naive number is selection bias, and nothing in the observed data reveals it.

Every method in the rest of this page is a strategy to make the selection bias term zero, or to estimate and remove it. Randomization makes it zero by design. The quasi-experimental methods of Chapters 07 to 10 make it small under assumptions that have to be argued for.

Chapter 02: Randomized Experiments

If a coin decides who gets the offer, then the treated and untreated groups are, on average, identical in everything: past spend, tier, home city, age, mood. Anything that differs between them afterwards is either the offer or chance. The selection bias term disappears, and the simple difference in means becomes an unbiased estimate of the effect. That is why randomized experiments, also called A/B tests or holdout tests, are the gold standard.

Designing an experiment

Decision	Question to answer	Loyalty example
Unit	What gets randomized?	Members
Population	Who is eligible?	Members active in the last 24 months
Split	How many in each group?	90% offer, 10% holdout
Primary metric	What single number decides success?	Incremental revenue per member net of points cost
Window	How long do we measure?	Campaign period plus 60 days, to catch pull-forward
Sample size	Is the test big enough to see the effect we care about?	Chapter 04

Two rules make the difference between an experiment and a story. First, decide the primary metric and the analysis before looking at results; with enough metrics and segments, something will always look significant (Chapter 12). Second, analyze everyone as randomized: members in the offer group who never opened the email still count in the offer group. Dropping them would bring selection bias straight back, because the members who open emails are not like the ones who don't. This is called an intention-to-treat analysis.

A quick sanity check

If you planned a 90/10 split and ended up with 88/12, something in the assignment or the data pipeline is broken. This sample ratio mismatch is one of the most common reasons experiment results are wrong, and it takes one line of code to check.

Chapter 03: Noise & Confidence

Randomization removes bias, but not chance. Even if the offer did nothing, the two groups would not book at exactly the same rate, because each member's decision is a bit random. Statistics is the tool for telling a real effect from that noise.

The difference in means and its standard error

Estimate and standard error

\[\hat{\tau} = \bar{Y}_T - \bar{Y}_C \qquad\qquad \text{SE}(\hat{\tau}) = \sqrt{\frac{s_T^2}{n_T} + \frac{s_C^2}{n_C}} \qquad\qquad \text{95\% CI} = \hat{\tau} \pm 1.96\,\text{SE}\]

$s^2$ is the variance of the outcome within each group. For a booking rate $p$, $s^2 = p(1-p)$. The standard error shrinks with the square root of the sample: four times the members, half the noise.

A 95% confidence interval is a range built by a procedure that captures the true effect in 95% of experiments. The p-value answers a narrower question: if the true effect were zero, how often would we see an estimate at least this far from zero? A p-value below 0.05 is the usual (and arbitrary) threshold for calling a result "statistically significant". Neither number tells you whether the effect is big enough to matter; that is a business question.

The best way to build intuition is to run the same experiment many times. The simulation below does exactly that: 200 identical loyalty experiments, each comparing an offer group with a holdout, each giving a slightly different answer.

Simulation · 200 identical experiments

Baseline booking rate True lift from the offer Members per group

Each bar counts experiments that produced that estimate. With 10,000 members per group and a true lift of half a point, the estimates scatter widely and many experiments miss the effect. Move the sample size up and watch the spread tighten.

Two things to notice. The estimates are centred on the truth: randomization works. But any single experiment can be far off, and with small samples a real effect is often missed entirely. A non-significant result in an underpowered test does not mean "no effect"; it means "we couldn't tell".

Chapter 04: Power & Sample Size

The power of an experiment is the probability that it detects an effect of a given size, if that effect is real. The standard target is 80%. Before running a test, you should know the smallest effect worth detecting, the minimum detectable effect (MDE), and whether your sample can see it.

Members per group for a difference in rates

\[n \;=\; \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2 \,\big[p_0(1-p_0) + p_1(1-p_1)\big]}{(p_1 - p_0)^2}\]

$p_0$ is the baseline rate, $p_1 = p_0 + \text{MDE}$, $z_{1-\alpha/2} = 1.96$ for a 5% significance level and $z_{1-\beta} = 0.84$ for 80% power. Halving the MDE quadruples the sample.

Calculator · how big does the test need to be?

Baseline booking rate Smallest lift worth detecting

Why revenue is harder than bookings

Booking rates are well-behaved. Revenue is not: most members spend nothing in a given month, and a few spend thousands. That long tail makes the variance, and therefore the required sample, much larger for revenue metrics than for conversion metrics. Common remedies: measure over a longer window, cap extreme values (winsorize) before analysis, analyze bookings and revenue per booking separately, and use the variance reduction of the next chapter.

In RM, the unit is the problem

A loyalty test can have a million members. An RM test often has only a few hundred flights, or a few dozen markets, and each one is noisy. Power calculations there are usually sobering, and they are the main reason RM teams need designs like switchbacks and the methods in Chapters 06 and 07.

Chapter 05: Variance Reduction

Much of the noise in an experiment comes from differences between members that existed before the test began: some spend a lot every year, some very little. Since we know what each member spent before the test, we can remove that predictable part of the variation and look at what is left. The effect estimate stays unbiased, and the noise falls, sometimes a lot.

The most widely used version is CUPED (Controlled-experiment Using Pre-Experiment Data), developed at Microsoft and now standard in online experimentation:

CUPED adjustment

\[\tilde{Y}_i = Y_i - \theta\,(X_i - \bar{X}) \qquad \theta = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} \qquad \operatorname{Var}(\tilde{Y}) = \operatorname{Var}(Y)\,(1 - \rho^2)\]

$X_i$ is a pre-experiment covariate, typically the same metric in the months before the test, and $\rho$ its correlation with the outcome. Because $X$ was fixed before randomization, it cannot be affected by the treatment, so the adjustment does not bias the estimate. Estimate the effect on $\tilde{Y}$ exactly as before.

Simulation · CUPED on 150 revenue experiments

How well past spend predicts future spend (ρ) True effect on spend per member

Each experiment has 5,000 members per group with skewed, lognormal spend averaging about $370. Both methods are centred on the truth; CUPED is just much tighter when past spend predicts future spend.

The same idea can be applied with several covariates at once through a regression of the outcome on the treatment indicator plus pre-period variables. Another option is stratification: randomize separately within groups (tier, region, spend band), which guarantees the groups are balanced on those variables from the start.

Chapter 06: Experiments in RM

RM is harder to test than loyalty. A loyalty offer can be sent to one member and not another. An RM decision, such as which fare classes are open on a flight, applies to everyone who searches that flight, and the customers who see one flight can easily book another. That creates two problems.

Problem one: what to randomize

Unit	How it works	Watch out for
Flight-departure	Each future departure is assigned to the new or old method	Customers moving between treated and control flights on the same route
Market (O&D or route)	All flights in a market get the same method	Few units, so low power; connecting traffic links markets
Switchback (time)	The whole system alternates between methods by day or week	Carry-over: bookings for a flight accumulate over weeks, across both regimes
Customer or session	Different shoppers see different prices or offers	Fairness and regulation; only feasible for some decisions such as ancillary offers

Problem two: interference

The key assumption behind a clean experiment is that one unit's treatment doesn't affect another unit's outcome (statisticians call it SUTVA). In RM that often fails. If the treated flight is cheaper, some of its extra passengers come from the control flight an hour later on the same route. The treated flight looks great, the control flight looks weak, and the comparison between them exaggerates the effect. Rolled out everywhere, there is no control flight left to steal from.

Simulation · a fare sale tested flight by flight

Share of the sale's extra passengers who would otherwise have taken a control flight

Setup: each flight normally carries 100 passengers at $200. Treated flights cut the fare 10% and gain 15 passengers. With enough cannibalization, a test that shows a double-digit gain hides a rollout that loses money.

Remedies: randomize at a level where customers rarely cross between groups (markets rather than flights), use switchback designs, or measure the control group's loss explicitly and include it. When none of that is possible, RM teams fall back on the quasi-experimental methods that follow.

Revenue opportunity models

RM systems often report a revenue opportunity metric: where actual revenue falls between a no-control benchmark (first come, first served) and a perfect-hindsight optimum. It is a useful health check, but it is not an impact evaluation: both benchmarks are computed from a model of demand, so it inherits that model's mistakes. It complements experiments rather than replacing them. The booking-limit simulator shows the same comparison on simulated flights, where the truth is known.

Chapter 07: Difference-in-Differences

Sometimes a change is rolled out to one part of the business and not another, without randomization: a new forecaster goes live on transatlantic routes first, a new earn structure launches in one country. The treated and untreated groups differ, so comparing them directly is biased. Comparing the treated group before and after is also biased, because other things changed at the same time. Difference-in-differences (DiD) combines the two comparisons so that each removes the other's bias.

Difference-in-differences

\[\hat{\tau}_{\text{DiD}} = \big(\bar{Y}_{T,\text{after}} - \bar{Y}_{T,\text{before}}\big) - \big(\bar{Y}_{C,\text{after}} - \bar{Y}_{C,\text{before}}\big)\]

The change in the comparison group estimates what would have happened to the treated group without the treatment. The key assumption is parallel trends: absent the treatment, both groups would have moved together.

Parallel trends cannot be proven, but it can be checked partially: if the two groups moved together for a long time before the change, it is more believable that they would have kept doing so. Plotting both series, and estimating "effects" in the pre-period where there should be none, is the standard test.

Simulation · a new forecaster on one set of routes

True effect (index points) Extra trend on treated routes Noise

Both groups share seasonality, a growth trend and a market-wide boost of 3 points from month 13. The before/after comparison credits the forecaster with all of it. Give the treated routes their own trend and watch DiD go wrong too: that is what a violation of parallel trends looks like.

In practice DiD is run as a regression with group and time fixed effects, which handles many groups and periods and gives proper standard errors (clustered by group). One modern caution: when different groups adopt the change at different times (staggered rollouts), the simple two-way fixed effects regression can give misleading answers, and newer estimators designed for staggered adoption should be used instead.

Chapter 08: Synthetic Control

DiD needs a comparison group that moves in parallel with the treated one. Often no single group does. Synthetic control builds one: a weighted average of untreated units chosen so that, before the change, it tracks the treated unit as closely as possible.

Synthetic control

\[\hat{Y}_{1t}(0) = \sum_{j=2}^{J+1} w_j\, Y_{jt}, \qquad w_j \ge 0, \quad \sum_j w_j = 1\]

Unit 1 is treated; units 2 to $J+1$ form the "donor pool". Weights are chosen to match the treated unit's pre-period path. After the change, the gap between the treated unit and its synthetic twin is the estimated effect.

A new co-brand card in one country

An airline launches a co-brand credit card in one country and wants to know how much it lifted member revenue there. No other country is a good match on its own. A synthetic control might combine 50% of country B, 30% of country C and 20% of country D, because that blend reproduced the treated country's monthly revenue closely for three years before the launch. After the launch, the treated country pulls away from its synthetic twin by 4%. That gap is the estimate.

Inference works differently from a normal experiment. The usual approach is a placebo test: pretend each donor country was treated, build its own synthetic control, and see how large its fake "effect" is. If the real country's gap is larger than almost all the placebo gaps, the effect is unlikely to be chance. In RM and network work, the simpler cousin of this idea, matched markets, is common: pair each treated market with an untreated market that looked similar before, and apply DiD to the pairs.

Chapter 09: Regression Discontinuity

Loyalty programs are full of thresholds: 25,000 points for Silver, 50,000 for Gold, a certain spend for the next tier. Thresholds are a gift to anyone evaluating impact. A member who finished the year with 50,200 points and one who finished with 49,800 are almost identical: same kind of traveler, same habits, one extra flight between them. But one becomes Gold and the other doesn't. Comparing members just above and just below the line is close to a randomized experiment. That is regression discontinuity (RD).

Sharp regression discontinuity

\[\hat{\tau}_{\text{RD}} = \lim_{x \downarrow c} \mathbb{E}[Y \mid X = x] \;-\; \lim_{x \uparrow c} \mathbb{E}[Y \mid X = x]\]

$X$ is the running variable (qualifying points), $c$ the threshold. In practice, fit a line on each side using only members within a bandwidth $h$ of the threshold, and measure the jump between the two lines at $c$.

Simulation · does Gold status change next year's spend?

True effect of Gold on next-year spend Bandwidth around the threshold

4,000 members, dots are averages per 1,000 points. The naive Gold vs non-Gold comparison mostly measures that Gold members fly more. A narrow bandwidth is less biased but noisier; widen it and the straight lines start missing the curvature, pulling the estimate away from the truth.

When RD breaks

The design relies on members not being able to place themselves precisely on one side of the line. Airline status is a well-known exception: members close to a threshold take extra flights, sometimes just for the points (the famous "mileage runs"). Those who make the effort are not like those who don't, so the members just above the line are no longer comparable to those just below. The warning sign is bunching: an unusual pile of members just above the threshold and a gap just below. Checking the density of the running variable around the cutoff (a McCrary test) should be the first step of any RD analysis on loyalty data.

When crossing the threshold only raises the probability of treatment rather than guaranteeing it (for example, status that must be claimed), the method becomes a fuzzy RD: the jump in the outcome is divided by the jump in the probability of treatment.

Chapter 10: Matching & Propensity Scores

When there is no experiment, no rollout to exploit and no threshold, the remaining option is to compare treated and untreated units that look alike on everything we can observe. If two members have the same tier, tenure, spend history, home airport and trip patterns, and one has the co-brand card while the other doesn't, perhaps the difference in their behavior is the card.

Propensity score and inverse probability weighting

\[e(x) = P(T = 1 \mid X = x) \qquad\qquad \hat{\tau}_{\text{IPW}} = \frac{1}{n}\sum_i \left[\frac{T_i Y_i}{\hat{e}(X_i)} - \frac{(1-T_i)\,Y_i}{1-\hat{e}(X_i)}\right]\]

The propensity score $e(x)$ summarizes all the observed characteristics in one number: the probability of being treated. Members with the same score can be compared, either by matching them or by weighting each one by the inverse of its probability of receiving the treatment it received.

All of this rests on one assumption, unconfoundedness: once we account for the observed variables, who got treated is as good as random. In loyalty, that is often a stretch. Members who sign up for a co-brand card may be planning a big trip, have just changed jobs, or simply like the airline more, and none of that is in the data. Matching removes the bias you can see and leaves the bias you can't.

How to use it honestly

Check overlap: if some treated members have no comparable untreated members, no method can estimate their effect. Check balance after matching: the groups should look alike on every observed variable. And run a sensitivity analysis: how strong would an unobserved factor have to be to explain the whole effect? If the answer is "not very", the result is fragile.

Chapter 11: Who Responds: Uplift

An average effect hides a lot. A bonus-points offer might raise booking rates by 1 point on average, but by 4 points among occasional travelers and by nothing among top-tier members who were going to fly anyway. The effect conditional on member characteristics is the conditional average treatment effect:

Conditional average treatment effect

\[\tau(x) = \mathbb{E}\big[Y(1) - Y(0) \mid X = x\big]\]

Marketers often sort customers into four groups based on how they respond:

Group	Without the offer	With the offer	What to do
Persuadables	Don't book	Book	Target them: this is where the lift is
Sure things	Book	Book	Don't pay them: the offer is pure cost
Lost causes	Don't book	Don't book	Don't bother
Sleeping dogs	Book	Don't book	Avoid: the contact backfires

The trap is that the usual targeting model predicts who will book (propensity), which mostly finds sure things. Uplift modeling predicts who will book because of the offer. The simplest version trains two models on experiment data, one on the treated group and one on the control group, and targets members where the predicted difference is largest. Uplift models need randomized data to learn from, which is one more reason to keep holdouts in every campaign.

Uplift models are evaluated with a Qini curve: rank members by predicted uplift, and plot the cumulative incremental bookings captured as you target more of them. A good model captures most of the lift in the first few deciles; a random ranking gives a straight line.

Chapter 12: Traps

Most bad impact numbers are not caused by exotic statistics. They come from a handful of recurring traps.

Regression to the mean

Members selected because their spend was unusually low last quarter will, on average, spend more next quarter anyway, because part of what made last quarter low was bad luck that won't repeat. A "win-back" campaign aimed at them will look successful with no effect at all. The only protection is a randomized holdout drawn from the same targeted group.

Simulation · a win-back email to the bottom 10%

Share of spend variation that is luck True effect of the email per member

20,000 members; the email goes to the lowest-spending 10% last quarter, half of whom are randomly held out. With a true effect of zero, the before/after comparison still shows a large "lift".

The rest of the list

Trap	What happens	Defence
Peeking	Checking results daily and stopping when significant inflates false positives far beyond 5%	Fix the duration in advance, or use sequential testing methods built for it
Many metrics, many segments	Test 20 things and one will be "significant" by chance	One primary metric; correct for multiple comparisons; treat segment findings as hypotheses
Pull-forward	A promotion moves bookings earlier instead of creating new ones	Measure over a window that includes the weeks after the campaign
Novelty effects	A new feature gets attention because it's new, then fades	Run long enough; look at the effect over time
Wrong metric	Bookings rise, but revenue per booking or margin falls; points cost is ignored	Measure net value: incremental margin minus program cost
Simpson's paradox	Yield rises in every cabin, but falls overall because the mix shifted to economy	Look at both the total and the components
Winner's curse	The winning variant's estimate is inflated, because it won partly by luck	Expect smaller effects on rollout; confirm with a holdout

Chapter 13: From Estimate to Decision

An impact estimate is an input to a decision, not the decision itself. Turning one into the other means converting the effect into money, accounting for costs, and being honest about uncertainty.

Net incremental value of a loyalty campaign

\[\text{Net value} = \underbrace{\Delta\text{bookings} \times \text{margin per booking}}_{\text{incremental margin}} \;-\; \underbrace{\text{points issued} \times \text{cost per point}}_{\text{all of it, not just on incremental bookings}} \;-\; \text{fixed costs}\]

Points paid to sure things are a cost with no return. The promotion ROI calculator works through this for a bonus campaign, with a confidence band around the result.

Carry the confidence interval all the way through. A campaign whose lift is somewhere between 0.2 and 1.8 points may be clearly profitable at the top of that range and clearly unprofitable at the bottom. Saying so, rather than reporting only the middle, is what makes an analysis trustworthy.

Choosing a method

Situation	Method	Key assumption
You control who gets it	Randomized experiment (02–05)	Randomization done right, no interference
Units interact (RM)	Cluster or switchback experiment (06)	Clusters contain the interference
Rolled out to some groups first	Difference-in-differences (07)	Parallel trends
One treated unit, many untreated	Synthetic control (08)	Pre-period fit carries forward
Treatment given above a threshold	Regression discontinuity (09)	No precise sorting around the cutoff
Nothing else available	Matching / propensity scores (10)	No unobserved confounders

What's next

The direction of travel is towards evaluation that is built in rather than bolted on. Permanent holdouts in the loyalty program, where a small random share of members never receives promotions, measure the long-run value of the whole marketing effort. Machine learning methods estimate heterogeneous effects from experiment data at scale. And in pricing and merchandising, bandit algorithms blur the line between testing and optimizing: they shift traffic towards the better option as evidence accumulates, trading a little statistical cleanliness for less revenue lost to the losing variant.

Anyway, that's the overview. If something here was unclear or you want to go deeper on any chapter, feel free to reach out. I find this stuff genuinely fascinating, and I'm always happy to talk about it.