Intro to A/B testing

Posted on 6/27/2022, 12:33:59 PM

Say that you want to increase conversions on your website. Based on the data, it seems that people are not responding well to the headline on your main page. How do you test the headline? An A/B test is the best way to test the headline and see which headline resonates better with customers. During an A/B test, you'll test the current headline (control group) against another headline at the same time. 50% of users will get the recent headline, while 50% will get the other headline. Visitors who get the control group will always see the control group until the experiment ends. Visitors who see the other headline will always see the other headline until the experiment ends. After a few weeks, you'll determine which headline led to an increase in conversion.

A/B testing allows you to test everything from emails, copy, and ads. It will enable you to find the best-performing asset before spending money on something that doesn't work. After testing the two different headlines, you can implement the one that works better and drive traffic to the new landing page. This makes it possible to determine where and how to invest your marketing budget.

How does it work?

During an A/B test, you randomly assign visitors to two versions of your website. One version is the control (how things are currently), and the second version is what you want to test. Usually, the second version changes one element. You could change multiple elements on the site, but it will be hard to know what caused the lift in conversion. For example, a simple A/B test might be to test two variations of a button on a page. 50% of website traffic will always see a blue button until the experiment ends, while the other 50% will always see a red button until the experiment ends. At the end of the experiment - usually, at least two weeks later, you will look at the analytics to see which version had a higher conversion rate. 

Why is A/B testing important?

A/B testing allows you to figure out what strategies work for a product. When launching Facebook ads, for example, it's essential to A/B test different variations to see which one resonates well with users. A/B testing helps you understand your target audience. It allows you to determine what works and what doesn't. It also enables you to reduce the bounce rate. 

How do you plan an A/B test?

Before launching an A/B test, knowing what you want to test is essential. Think of all the sales-related pieces of your website. You might want to test the headings, CTAs, images, copy, or forms. You might also want to send two versions of an email and see which performs better. The goal of an A/B test is to determine what version your audience responds better to. What version has a higher conversion?

A/B testing checklist

When launching an A/B test, be mindful of the following:

• What is the result that you are currently getting? Have you looked at the data and determined what needs to be tested. 
• Tests should run at the same time. You should split the test between two versions and distribute traffic between the two versions simultaneously. 
• Decide what features to test
• Create two variations of that feature
• Decide how long to test. You need to have enough traffic to be statistically significant. You can use the tools discussed below to determine whether you've reached a sample size that is statistically significant
• Choose a testing tool to help you launch the test.
• Launch the test to the world
• Look at the results and determine what version won.
• Rinse and repeat

Examples

Here are some examples of A/B tests:

What: Unbounce wanted to increase landing page optins. It had a page asking users for an email address in exchange for an asset. It launched a test where instead of an email address, it asked people to tweet about the product. 

Results: Asking for an email address led to higher optins than asking people to tweet about the product.

What: Wall Monkeys, an online wall decal website, wanted to increase the conversion rate and customer experience. They decided to move the webpage search bar, hoping that it would increase conversions. Using heatmaps, they were able to see where people looked at first.

Results: By moving the search bar, conversions increased by 550%. 

How much time?

You should run one test at a time. You can test one variable at a time. For example, test blue vs red button. You could also test multiple variables simultaneously; for example, red vs blue vs yellow button. Make sure it's part of the same test, not two different ones. When running a test, you need enough traffic to reach statistical significance. Some tools will help you determine whether you reached stats sig. You should also run tests for at least two whole weeks. Running a test for too long can lead to skewed results. When in doubt, rerun the test. 

When running tests, ensure you have at least 30 conversions and at least two weeks of data. Your results need to be statistically significant for you to say that one variation was better or worse than the other. You might want to use tools like vwo.com and optimizely.com to determine whether your results are statistically significant. Optimizely provides an SDK that allows you to implement the test in code and a dashboard that will enable you to decide whether or not you reached statistical significance and the results. 

For email tests, you can try mail chimp or constant contact. These tools will allow you to test different emails at a time.

A/B testing is an excellent way to determine whether you're going in the right direction when making decisions that will affect your website's conversion rate. You do not want to implement things without testing. Test first and implement. If you need help running an A/B test, our mentors have years of experience with experimentation. They can help you get started. Book a call with a mentor today.