Is MVT the only way to go and How much testing is too much?

I am not a statistician, but questioning whether multi-variate testing (MVT) is the best way to test online is a valid question!

Multi-variate tests are familiar and they allow one to test typically two to three variations of a few variables at the same time. These could be page layouts, images or creatives, messaging, or different products and offers. The goal of the test is to provide one with learning about their page - see what elements of the layout work better and what message works best at this particular time. The learnings could be either very specific, a certain message for a particular product, or they could be broad like a very different layout of the page that you are typically used to. Concequent actions may be taken to improve your old page and replace with with the new refreshed page. Having the business knowledge and your customer and product isight you can create a test that can answer very specific questions that you have for your current page.

However, one caviat of multi-variate testing is that when you design the test you need to design it is such a way so that you can detect change. Thus, if the elements that you are testing are not intuitively different, then you may not even get a read. Particularly if you test using a smaller sample size than needed to detect those changes. If your site is a heavy-traffic site, this may not be such a big issue, you may get your answers relatively fast, but you should still do your math so that your test is a true test.

Another caviat to MVT is the fact that in order to get that volume and get the faster read you may run the test on 100% of your traffic. However, internet being the active space that it is, constantly changes. Your test may be running for say one-two months, so you may benefit from the winning page for a little while longer after the test is over before the variables that you have tested no longer have the same impact on your online visitors who are constantly changing. And this is an ideal scenario, when you used your excellent business and industry sense that helps you stay ahead of the game to design various new pages that you tested - and I can't emphasise this point enough. This would be Possibility 1 on the graph below:

But regardless if you are the Possibility 1 or Possibility 3 you will once again plateau until you run the new and hopefully better test. And if you're good and do your industry research and customer research prior to the test so as to only test the most important learnings, you will do well, i.e. testing intelligently and not testing for the sake of testing. In any test you typically get out the quality of answers as what you put in.

However, you still won't get the full picutre. You know what you showed and you know which one of the things you showed turned out to be the success for that particular period of time. However, you don't know indirect impact of your test. How can you be sure that one of your other pages got more traffic because of a certain test element on the page you have tested? How can you be sure that the changes you saw were not due to external environment? And the list can go on... The MVT is almost there to confirm your beliefs and perhaps suprise you from time to time.

The MVT seems good, but there must be something better - something that learns on a continuum and not periodically. This is something that you see emerge in the web optimization the industry. A way to do the learnings continuously by creating a closed-loop system. It also requires a certain knowledge of yor customer and your product to yield you the best bang for your buck, however, it takes it one step further and allows your customers to tell you what they like / dislike about your entire website not just see the response for a page in question. An example of such an optimization vendor, which is very well known would be Omniture TouchClarity, now known as Test & Target 1:1. I am not here to advertise this company, but you have to give them credit for venturing into new territory. Besides looking at a particular page, they made their optimization platform very flexible, almost modular, where you now can group as many images across pages as you like to see a bigger picutre. Optimization, of course if it works, is superior to an MVT test. The whole meaning of optimization is to take the inputs and optimize or maximize to a given response, say a purchase on your site. The more well-selected imputs you have and the better you can sift through them and organize them in a meaningful manner using the optimization engine the better your optimization will be and this will consequently lead to more purchases, or your bottom line!

Of course if you have doubt in the optimization being the unfamiliar territory, you can split the traffic, if you have the luxury to do so from time to time and do some MVTs on the side to keep checking your gut feeling and the optimization model.