Peeking

Short answer

Peeking is the practice of repeatedly checking A/B test results and stopping as soon as they look statistically significant. In fixed-horizon tests, each extra look adds another chance for random noise to cross the threshold, inflating the false-positive rate. Pre-setting the sample size or using sequential testing methods avoids the problem.