9 A/B testing myths: busted

By Ankit Prakash | April 2018


You make so many marketing decisions day-in and out. You focus on making your email copy refreshing, your landing pages intuitive, your social updates engaging, and continuously think of ways to keep your leads engaged. But how many times do you consider experimenting with each of these before they go live?

I know what you are thinking. We seldom pause to focus on creating tests to nullify all hypotheses. In a bid to keep up in the race, we almost miss out on one of the most effective ways to make our strategy a sure-shot winner: A/B testing.

A/B testing or Split testing enables you to make decisions that are data-based and will contribute in bringing in more leads for your business. Using A/B testing can help in optimizing your marketing assets and increase lead conversions.

Unfortunately, this is not what marketers think about when A/B testing or Split testing is mentioned. Most marketers tend to give in into common beliefs, which are nothing but myths. These myths often boggle down the confidence of marketers, preventing them from making well-informed decisions. So, before you head into that camp, here are 9 A/B testing myths busted for you.

AB Testing Myths Cover Image

Myth #1: You cannot do A/B testing if you don't have massive website traffic.


A/B testing is about assessing and comparing two treatments. You create two versions of the same marketing asset and test them with a similar-sized audience. The winning test is statistically significant with a level of confidence (usually 95%+). To do this, you don't need huge visitors' traffic. In fact, you just need enough to touch the statistical significance (i.e., a point where you have minimum 95% confidence).

I agree that having more visitors increases the chance of getting a more accurate picture of what can work and what cannot. However, there is no fixed number to support A/B testing.

PS: To know what is Statistical Significance and Confidence, you can read 15 A/B testing mistakes to avoid.

Myth #2: Between A/B test and Marketers' instincts, instincts always win.

Not really. Putting it more practically: on the one hand, you think a certain treatment will work and on the other hand, you make decisions based on accurate data. Now, isn't it clear which one takes the winning cup?

Your years of experience can surely give you a better judgment capability, but relying on pure instinct can result in low conversions. It makes more sense to club your experience with data-backed results and then make an informed decision. It is proven that split testing can increase lead generation by 30-40% for B2B websites.

Myth #3: What worked for others will work for you as well.

How many times have you believed that changing the color or the font of your CTA can give you better results? You have probably come to this conclusion only after doing one or two such tests. But it isn't necessarily true that this will be a set rule. While it may have worked for you, it may not work for others.

For instance, changing headlines on their homepage did nothing for Groove. When they tested two different headlines, the result was inconclusive.

groove-headline-testOn the contrary, Tulos, a Finnish SEO and optimizing agency, revealed that testing headlines has helped them achieve 127% more conversions. They made simple changes to a headline for a Scandinavian outdoor store and saw amazing results.

English Translation of the image headline below: Order Men’s Clothing easily for bargain prices


Taking inspiration from others is great, but believing that you will have the same results is not. You can likely find other inconclusive tests such as the one above by Groove, mostly listed as 'A/B testing best practices.'

Myth #4: Only Multivariate testing rules. Forget everything else.

Umm no. A/B testing and Multivariate testing are not same, nor are they opposites. They both are effective tests and help in making data-based decisions. The only difference is, the purpose of each of  these tests is different.

If you want to know if changing the color of your CTA button on your landing page makes a difference, you change the color and keep every other element of your landing page the same. In this test, you are trying to find an answer to a direct question: “Does color alteration impact your landing page conversion or not?” Everything else, starting from the content to audience size to element placement, remains same. This is A/B testing. Unlike Multivariate testing, you are not trying to find out how the change of color combined with the number of fields in your form and content tone affects your conversion.

So, hailing one test over the other makes no sense because the purpose of both these tests is different.

Myth #5: A/B testing requires huge budget.

Not necessarily. You can conduct A/B tests for free as well (depending on the tool that you are using). You can try Google Analytics' Content Experiments tool, which is free. However, for this tool, you need to be more of a tech expert.

Most marketing automation tools come with A/B testing features so that you don't need to spend additional funds for a  single feature. In the end, though paid tools are more expensive than free tools (like you didn't know that!),using a comprehensive marketing automation tool can help you cut down overhead costs.

The other part of the story is, to do split testing you need to have a good grip on your mathematics. All winning tests should be statistically significant, and to know this, you need to calculate this for proper interpretation.

In short, some level of technological knowledge and mathematical understanding is needed (depending on your resources), but the budget is not necessarily a hindrance when you want to do some A/B testing.

Myth #6: Every single marketing decision needs to be tested.

No. This might sound contradictory to the "not relying only on instincts" part, however while it is true that decisions should be data-based, it is also true that not every element or every action requires testing.

Testing something insignificant can unnecessarily prolong your tests, taking a toll on your time and other resources. Such tests don't offer anything valuable other than just doing the test for the heck of it.

Let's say you are confused if your current CTA button's color will give you a higher conversion rate or not. You want to change the color from blue to any other color.

Now, when you Google 'CTA button color for conversions' you get ample information to refer (like below). Hence, testing multiple colors can be avoided in this case because you already have ready-to-use information at hand.


Stop worrying about testing every single thing. Some decisions can be made without doing an A/B test.

Myth #7: A/B testing will hurt my SEO

People think that testing multiple versions of one page can result in getting content marked as duplicate by Google. As a consequence, they believe they will  get penalized in the SERPs.

Good news. This is a false alarm. On the contrary, Google encourages marketers with a handful of guidelines to test your content to get more web traffic. So, next time someone gives you “good advice” about A/B testing, Google's algorithms and SEO, just shoo them away.

Myth #8: The winning result always looks beautiful. BEAUTIFUL.

Alas! All that's pretty is not necessarily worthy. A/B testing is not about choosing a final result that looks beautiful. It doesn't need to be. All it needs is to give you a result that shows improved conversions. Your landing page can be beautiful, but if it is out of context, it doesn't matter to your leads.

For instance, if you follow the case study of The Olympic Store's checkout page, you will see it was not about making that page look good. Yes, a clean and navigable design is mandatory, but it doesn't mean you have to work round the clock to make it look pretty. Here is the final result. They tested two-page checkout vs. single-page checkout.


Myth #9: A/B testing is the end of all

You run a sample test. You find the winning version... but you are not done yet. A/B testing is not about testing two variables only. It is about testing continuously and optimizing your content for better conversions and leads. In short, no A/B test can be a single experiment. It has to happen consistently, be adjusted every single time and improve with each test.

So, what commonly heard A/B testing myth you've bunked recently?

This a guest post from Artic PinPoint, a full-service marketing automation platform. They use advanced tracking and fingerprint technology to help clients form stronger relationships with their leads. You can learn more about how they compare to other CRMs with this detailed marketing automation comparison spreadsheet

About the author:

Ankit is a technology entrepreneur at Aritic—Full Stack Marketing Automation software platform for digital business at Aritic with Sales CRM, Transactional Mail, and Desk. Ankit also spends a majority of his time reviewing various marketing tools and platforms on his marketing blog- Sprout24. Follow Ankit on Twitter at @ankitpr89


Topics: Digital, Digital Marketing