Multivariate Testing Strategies, Tips, & Examples

Multivariate testing for conversion rate optimization (CRO) is kind of like trying to create a new pizza for your pizzeria.

You want to make the best pizza so your customers feel compelled to buy it or inevitably come back to try if they didn’t the first time.

There are some pizza elements that entice most people, like tomato sauce or mozzarella cheese.

Once you get into the specifics of your prospective customers, you may be unsure of what they like best. Your experience tells you that they might like all different kinds of crusts, plant-based protein over traditional meat, or only green vegetables.

But, which combination of these elements would sell most?

Plant-based sausage and green peppers with a thin crust? Plant-based pepperoni and spinach with a stuffed crust? A combination of these two or something entirely different?

This is where multivariate testing comes in.

With multivariate testing, you can find exactly which combination of elements your target audience responds to best, from digital ads to landing pages.

In this article we’ll give you an understanding of the different types of multivariate testing and how that works, as well as the recipe for creating your own.

So without further ado, let’s get cooking.

Like other tests, multivariate tests (MVT) allow companies to test their optimization hypotheses by putting challenger variants against a control variant.

However, MVT tests a combination of elements within a single variant and puts that to the test against all other possible combinations, along with a control variant. We’ll talk more about the exceptions later on though.

This test type is different from A/B tests or A/B/n tests, as these tests compare a control and a challenger variant, or multiple challenger variants, with one differing variable, instead of multiple differing variables.

There are two types of multivariate tests:

1. Full factorial
2. Fractional factorial

Full factorial testing is the method that is usually used when it comes to MVT. This method tests all combinations equally amongst your traffic.

For example, if you’re running 9 tests, each variation will receive roughly 11.11% of your traffic.

The concept of this testing method is similar to split tests, in which these tests are defined by the fact that traffic is equally split between variants.

Fractional factorial pertains to a fraction of all the combinations of variables tested against your traffic. With this method, you can statistically deduce how the untested combinations would fare, based on sample size data from the live tests.

Fractional factorial is usually less accurate than full factorial, but it requires less traffic. So, it may be an option for lower traffic pages.

MVT for search engine optimization (SEO) is the same conceptually for CRO.

Some common use cases for SEO MVT are testing combinations of headlines and images and combinations of copy and buttons.

While MVT can have great results for CRO, it can have some negative impacts on SEO.

Although search engines like Google shouldn’t drive your strategy, it’s worth looking into what happens between CRO testing and SEO.

Here are some negative impacts multivariate testing could have on your SEO.

Cloaking

Cloaking occurs when test versions are drastically different. 

If Google sees a version of a page that is very different from a variant that users may be seeing, your site’s ranking may drop or your site may be removed from organic search. 

As you know, you may have multiple variations of your landing page when you’re conducting tests.

Cloaking goes against Google’s Webmaster Guidelines. Google thinks your website is routing to a different page than presented, and you are “cloaking” your true content.

This is all because search engines favor stability and consistency.

Page duplication 

Having very different versions of a page can be a problem, but so can having multiple similar pages, as this can create duplication issues for search engines.

Inconsistent redirects 

For all test types, search engines see the main URL but may also sometimes redirect to another test URL.

This may cause confusion as to which page is the main page and which one needs to be indexed.

Let’s say, you want to see what the best combination of a CTA button, hero image, and headline are to maximize your conversions on your landing page. Creating A/B test after A/B test to get these results would take a much longer time than if you did a multivariate test.

You’ll have several tests going on simultaneously, as you can test a surplus number of variables which leads to a surplus number of combinations or number of variations, but you will also get answers more efficiently with this test type.

The decision to implement MVT heavily depends on the amount of traffic.

We recommend using this test type for landing pages with a lot of traffic to make sure you have enough traffic to run your tests and procure data of statistical significance.

The test time is typically shorter for this test than the other test types, but it can be exponentially longer for pages with lower traffic, as you are splitting your traffic multiple times to send web visitors across several web page versions. 

Conducting MVT is very similar to the other test types.

Start by asking “what can be improved?”

Identify a problem

Before you can start testing, you need to examine your site and identify possible problems. These findings will be the basis of your tests.

Areas for improvement may be centered around usability or page layout. If you’re stuck, checking your website metrics in Google Analytics or referencing our testing guide may help you.

Formulate a hypothesis

You may recognize this step from the scientific method.

For example, you realize that your CTA button blends in with the rest of the hero section, your headline isn’t as concise as it could be, and your hero image isn’t that interesting.

You hypothesize that changing the button color so it contrasts with the hero section more will make it easier to see, using stronger verbs in the headline will communicate your offer better, and showing a product shot in the hero will grab more attention.

After you’ve determined your testing ideas, start creating your tests.

Create your variants

Let’s keep going with our current example.

You have two button colors, two headline options, and two different product shots you want to try. You’ll end up with 8 combinations with these different page elements if you’re doing full factorial MVT for your page designs.

Check your tests 

Remember to check your page variations. There’s nothing worse than making all your variants live, just to find out that things aren’t working how you thought like your new pages look different in different browsers or your changes didn’t carry over to mobile.

Now, you can start sending traffic to your page variants.

As we touched on before, MVT requires a lot of traffic and even more depending on how many tests you’re running.

We generally run our tests for about two weeks before calling the results.

After all the waiting and prep work, you can finally see how your tests did.

Analyze your results

The great thing about MVT is that you don’t have to guess which combination of elements works best.

With full factorial testing, you can see right away which combination of your ideas resonated with users the most and improved your conversion rate.

Learn from your results

Keeping in mind what you learned from your tests can be extremely useful for future tests.

For example, if a “fear-based” headline outperformed your “values-based headline” you can implement that knowledge for future tests or other pages.

Now you know all about what MVT is, why you would use it, and how you would use it.

We mentioned that MVT can be helpful for CRO, but how so?

There are three main benefits of MVT:

1. You can quickly get results by conducting multivariate tests simultaneously instead of several A/B tests sequentially.
2. You can clearly determine the contribution of several variables to your conversion goals.
3. You can conveniently measure how normally independent testing variables (i.e. an image and a headline) work together.

For clarification, we’re not saying that you shouldn’t keep running A/B tests.

If you think testing your original pizza, the one with plant-based ham, zucchini, and a thick crust, against the same pizza with plant-based pepperoni instead of ham, will give you the best results, you should.

Certain test types may be more compatible with your testing practices, and there are significant results to using MVT when you can.