Is that A/B result real?

Noise ≈ √(conversions).

Variant B got 58 conversions, A got 50. Ship it? Most "winning" tests in marketing are noise. A ten-second check will save you from celebrating coin flips.

The move

For two variants with similar traffic, the noise in the difference ≈ √(A + B) conversions. Only trust a difference bigger than about 2× that noise.

Worked example

A: 50 conversions, B: 58

  1. Noise ≈ √(50 + 58) = √108 ≈ 10.4
  2. Difference: 8
  3. 8 is less than 1× the noise → could easily be chance
  • This is a back-of-envelope version of a real significance test, great for spotting nonsense, not for publishing papers.
  • What matters is the number of conversions, not visitors. 10,000 visitors with 12 conversions is a tiny test.