VIII · Statistics & inference
Hypothesis testing & p-values
What it is
Set a null hypothesis, compute the probability of observing your data (or more extreme) under it. Reject if p < α (typically 0.05).
Where it lives
A/B tests, regression analysis for performance hypotheses, root-cause analysis on outage metrics.
The key insight
p-values are not the probability that the null is true. They are easily abused — selection bias, multiple comparisons, and underpowered tests give meaningless "significant" results.