The first rule of post-click marketing is that you should test multiple landing experiences to find the ones that give you the highest conversion rate. (Note that this will vary according to the traffic source and audience segment, as has been blogged here before.)
The second rule, however, is to not jump to conclusions too quickly.
If you have two paths that you're testing, A and B, and you see path A is converting at 5% and B is converting at 8%, you might be tempted to immediately shut off A and drive everyone to B. But don't be hasty.
Landing experiences/conversion paths are like presidential primary candidates -- the early front-runner isn't necessarily the winner.
Before you can accurately predict how a path is going to perform in the future, you must have had enough respondents to make sure that your conversion rate is "statistically significant". In stats lingo, this means having a large enough sample.
A straightforward way to estimate the predictive accuracy of a path's conversion rate is to calculate a "confidence interval". Confidence intervals are usually stated like this: with 95% certainty, the conversion rate is between 3% and 7%. The 95% certainty is known as your level of confidence, and the range of 3% to 7% is your confidence interval. This interval could also be presented as a conversion rate of 5% with a margin of +/- 2%.
The way statistics work, the larger sample you have -- in this case, the more respondents you have sent to a path -- the more narrow your confidence interval becomes. When you start with only a few dozen respondents, the estimate of your conversion rate can be as wide as +/- 40%! Once you have enough respondents though, your margin can be very small, less than 1%. At that point, you have a pretty solid prediction of how that path is going to perform.
Here's the catch: the true conversion rate of a path could fall anywhere inside that confidence interval. So if you have path A converting with a confidence interval of 3-7% and path B with a confidence interval of 6-10%, you can't conclude that path B is the winner yet. It's just as likely that path A could turn out to convert at 7% as path B could convert at 6%.
The secret is to get enough respondents to each of the paths you're testing that the margin becomes small enough and the confidence intervals no longer overlap. If the interval for A tightens to 4-6% and B is 7-9%, then you can safely declare B the winner.
(Disclaimer: statistics can be crazy complex, so I've omitted some nuances above for clarity. If this bothers you, feel free to consult the Great Wiki for more information.)
-Scott Brinker
