There's always a little buzz (or is that confusion?) surrounding multi-variate testing (MVT), and recently some of the discussion seems to be centered around using MVT on smaller sample sizes. I was recently reading some views on MVT that assert if you have enough traffic to conduct A/B testing, you have enough traffic to conduct multivariable tests. I've always understood that MVT needed large sample sizes, so I was surprised at this new slant. I went straight to my favorite math wizard to see what's what. In case you have a burning desire to know if MVT can in fact be used for smaller sample sizes, I'll share what I learned....
This type of MVT testing is called "fractional factorial." And in theory, it requires a very small amount of traffic to generate the "answer" or optimal landing experience. But this type of MVT strategy could lead to premature conclusions--either the right conclusion, or quite possibly, the wrong one.
It's worth noting that Google avoided the whole fractional factorial approach for their Website Optimizer--no doubt due to the questionable nature of the results. Google noted that "the benefit of reducing combinations comes at the cost of limiting the conclusions that can be drawn from the experiment." To be fair, Google does also avoid the technical questions raised by the math, since most of the technical questions tend to be a bit over the top (have you ever read this MVT stuff? It's both fascinating and mind-numbingly complex). If you have a Ph.D. in statistics and this type of debate interests you, you can check out the Wikipedia article on fractional factorial designs.
At the end of the day, if you're toying with the idea of running a few tests using the fractional factorial approach, keep in mind that MVT testing in general can miss the big picture regardless of how much traffic you have. MVT can empower you to test lots of different combinations of graphics, offers, headlines, etc., but it is very one-dimensional and doesn't reveal the true value in analyzing your audience segments and traffic sources.
Of course MVT can be a very powerful option, particularly for high-traffic ecommerce sites. But if your traffic is moderate or if your sale is complex--as in B2B lead generation, B2B transactional and even B2C high-ticket/complex products & services, you may find there's more opportunity for dramatic improvements with well designed A/B testing.
If you're still trying to figure out what type of testing is right for your organization and you're not a math wizard, check out my blog from a few months ago that takes a really simple look at MVT vs A/B.
To be very honest, all the MVT stuff can be fascinating. But, we like everyday, easy testing. Testing that marketers can do without needing an advanced mathematics degree. And while we're partial to LiveBall's elegant testing solution, I should point out LiveBall supports third-party MVT solutions as well.
And no matter how you decide to go about it, your online landing experiences can certainly benefit from testing. So, I hope this blog post gave you some food for thought and motivation to get started with your own testing program.
-- Susan Delz
