Causal Inference
Causal inference uses assumptions, study design, and statistical methods to estimate what an intervention changed beyond observed association.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/causal-inference/
01What it means
Causal inference is the discipline of reasoning about how an outcome would differ under alternative actions or exposures. Because the same unit cannot be observed simultaneously under treatment and control, teams need a credible comparison, an explicit causal model, and assumptions connecting observed data to the counterfactual question. Randomized experiments are one design; observational methods require additional care about confounding and selection.[1][2]
- 01Causal questionIntervention, comparison, population, outcome, time
- 02Causal modelConfounders, mediators, selection, ordering
- 03DesignRandomization or justified observational method
- 04Assumption checksOverlap, measurement, interference, sensitivity
- 05Effect + uncertaintySeparate data-supported and assumed conclusions
The same unit cannot reveal both treatment and control outcomes at once; design and assumptions connect observed data to the counterfactual.
02Why it matters
Product data contains abundant correlations that can mislead decisions. Highly engaged users may adopt a feature because they were already likely to retain, making feature use look causal. Causal reasoning forces teams to state the intervention, population, outcome, time horizon, and assumptions, reducing the chance that prediction, segmentation, or before-and-after movement is presented as impact.
03When to use it
Use causal inference when a decision asks what will happen if the product changes an action, policy, price, message, or experience. Prefer randomized designs when feasible and ethical. For observational analysis, involve appropriate expertise and use domain knowledge to identify confounders, mediators, selection mechanisms, and measurement problems before choosing a method.
04How it works
- Formulate a precise causal question specifying intervention, comparison, population, outcome, and time horizon.
- Draw or otherwise state the assumed relationships among treatment, outcome, confounders, mediators, and selection variables.
- Choose a design—randomization, natural experiment, matching, weighting, regression discontinuity, difference-in-differences, or another justified approach.
- Assess identification assumptions, overlap, measurement, interference, missingness, and sensitivity to plausible violations.
- Estimate effects with uncertainty and communicate which conclusions follow from data versus untestable assumptions.
05A practical example
Users who enable collaborative editing retain more often, but the team does not conclude that enabling it causes retention. Larger teams are both more likely to collaborate and more likely to renew. The company defines the intervention, randomizes an onboarding prompt among eligible accounts, and estimates the effect of encouraging adoption rather than comparing self-selected users.
06Where it breaks
Causal estimates are only as credible as their design, measurements, and assumptions. Statistical adjustment cannot guarantee that every confounder was measured, and complex methods can conceal fragile identification. Effects may vary across people and time, while interference between users can violate standard assumptions. Product judgment is still needed to decide whether an estimated effect is valuable or ethical.
Common mistakes
- Describing correlation, prediction, or temporal order as proof of causation.
- Selecting an advanced estimator before defining the intervention and causal assumptions.
- Controlling for variables caused by the treatment or restricting analysis to a selected post-treatment population.