Quick check: the gap in conversions should be at least 2 × √(total conversions)
21 − 15 = 6, and 2 × √36 = 12, so not yet
What does significant mean in plain terms?
Imagine the two versions were identical. Conversions would still not split evenly, the same way ten coin tosses often land six and four, or seven and three. A result is called significant when a gap as big as yours would turn up by chance fewer than about 1 time in 20 if the versions were the same. It says luck is an unlikely explanation, which is weaker than proof that the winner is better.
The quick check above approximates that standard test when both versions get similar traffic and the conversion rate is low, which describes most small business tests. Online calculators give a more exact answer.
Has a Parramatta barber's video ad beaten the photo? (example figures)
A barber in Parramatta runs a Meta A/B test between two ads with the same budget: a photo of a fresh skin fade (A) and a 10-second video of a cut in the chair (B). Both send people to the online booking page. After three weeks:
| A: photo | B: video | |
|---|---|---|
| Visitors from the ad | 300 | 300 |
| Bookings | 15 | 21 |
| Conversion rate | 5 percent | 7 percent |
B looks 40 percent better. The quick check says wait: 36 bookings in total, a square root of 6, so the gap needs to reach 12, and it is 6. A gap that size turns up by chance often enough to mean nothing yet.
If the rates hold at 5 and 7 percent, the gap reaches the bar at about 1,200 visitors from each ad, around 60 bookings against 84. At this budget, that is about three months. The owner keeps both ads running, because neither is losing money, and waits for the count to settle it.
How many conversions does a test need?
Enough that the win you are hoping for can reach the bar. The table shows the same quick check at different sizes.
| Total conversions in the test | Gap needed | A split that reaches it | How much better the winner is |
|---|---|---|---|
| 20 | about 9 | 15 against 5 | 3 times |
| 50 | about 14 | 33 against 17 | 94 percent |
| 100 | 20 | 60 against 40 | 50 percent |
| 400 | 40 | 220 against 180 | 22 percent |
Read it from the bottom up and the lesson for a small business is plain: with a few dozen conversions, only very big differences can be confirmed. So test big changes, such as a different offer or a different way to book, and leave the button colours to businesses with a hundred times your traffic.
Why are 7 days of dashboard figures too few to call a test?
Because a week holds too few conversions for most owner-led businesses, and the CoraVision dashboard cards cover exactly that: the last 7 days, a window better suited to spotting trouble. CoraVision reads clicks and conversions from Google Ads and Meta every night, so each side's count keeps growing for as long as a test runs. Judge a test over its full planned length and against the quick check, never on one good week.
Questions
Is 1 in 20 the only standard?
It is a convention, not a law. For a low-stakes change that is cheap to undo, accepting 1 in 10 is reasonable, as long as you know you will back the wrong version more often. For a change that costs real money, such as a new website, hold out for the stricter bar.
Are online significance calculators worth using?
Yes. They do the same sum as the quick check on this page, more precisely, and handle versions with unequal traffic. Enter the visitors and the conversions for each version, and read the result before you look at which one is ahead.
Does a significant result mean the win is big?
No. Significant means the difference is probably not luck. With enough traffic, a difference too small to matter can be significant. For a small business the opposite problem is more common: big differences that cannot be confirmed because there are too few conversions.