Introduction to A/B testing
A/B testing is a popular method used by online marketers to test different variations of a given message. For email marketing, the most common scenarios of A/B testing are to test two or more subject lines (i.e. the title of the email) and to test different calls to action (e.g. text or image call to action). By sending the same email - but with slight variations of it - you can figure out which email is the best performing one in terms of open rate or click through rate.
In most cases, you will take advantage of the automated A/B testing tools that exist in MailUp, but there are rare scenarios in which you may also need to run a "manual" A/B test.
For email marketing best practices and tips, view the Email Marketing Resources section.
Creating an Automatic A/B Test
Before working on the A/B test page, you should have a pre-existing email message saved in the Email tab. Decide what changes you want to make to the message.
Create as many versions of the original email as you wish. To do this, select the original email under Messages > Email > Saved and click on Create a copy from the actions drop down. Once you have created a copy of the message, go ahead and make the changes to that email. For example, your first version may have a different subject line. Continue to create as many copies of the original email for each change.
It is recommended that you change only one variable (e.g. subject line, images, text, etc) per email at a time so that you can accurately track its result.
Once you have one or more variations of the email, you are ready to start your A/B test.
- Go to Send > New A/B Test. Select which emails will perform in the A/B test.
- Choose the recipient size of the test mailing. The test mailing group is the percentage you choose of your mailing list.
- Select the performance metric for the A/B test: choose either open rate or click through rate. To have results that are statistically relevant:
- You should test with at least 1,500 recipients if you are using the Open rate
- You should test with at least 6,000 recipients if you are using the Click Through rate
- Select the number of hours before the best performing email is determined and sent out to the remaining recipients.
- If you want to judge the test based on a variable that is outside of the message (e.g. number of orders, number of signups, etc.), please see Creating a manual A/B test below
As a rule of thumb, choose the open rate metric if your email versions vary only on subject line. In fact, a more relevant subject line leads to a higher open rate with all else constant. On the other hand, choose the click through rate metric if your email versions vary in the layout, text, or images of the body of your email. In this case, for example, a brighter image (vs a darker one) may lead to a higher click through rate.
To learn more about email performance metrics (e.g. open rate (OR), click through rate (CTR)) view this knowledge base article.
Click on Continue.
Known limitation: The A/B test cannot be scheduled. Please choose an appropriate time to send out the mailing. After the A/B test emailing is finished, the best performing email will be sent out to the remaining recipients.
In the Mailing Setup page, choose your mailing list (i.e. all subscribers or selected groups). Once you click on Send now, the A/B test will run instantly. When the A/B test is finished, the system will send the best performing email to the remaining recipients in your mailing list.
There are instances in which an automatic A/B test might not fit your needs. For example you may want to determine the winner of your test considering other criteria out of the automatic test (for examples conversion, purchase....).
The "manual" AB TEST option allows you to do that. Check here how to run a manual AB test