Create a New A/B Test

Owned by Product team
Last updated: Apr 26, 2022
2 min read

Preliminary step

Before starting with the A / B test make sure you have at least one email message saved in the Messages> Email> List menu.

Now decide what variation to make of your email: you can test the subject, the summary or an element in the body of the message, such as the main CTA.

Step 1: Choose the message you want to A/B test on

Create as many versions of the original email as you like. To do this, select the original message from Messages> Email> List and click "Create a Copy" from the Actions menu. You can also create a new email with similar characteristics, but remember that to learn something from an A/B test, it is often better to focus on the single element.

We have created two different variations for our message: "Chocolate cookies for breakfast?" and "Chocolate biscuits?" (with the addition of Emoji) and on these we will perform the A/B test.

Step 2: Configure the A/B test

Once you've created two or more variations for your message, you're ready to go A/B test. Go to Messages> Email>A/B test and click on "NEW A/B TEST" at the top right.

  1. Give your test a name (it only serves to identify it internally on the platform)

  2. Choose the variations of the message, from 2 to 5.

  3. Choose how to divide the mailings. You can choose as a test sample a percentage of recipients between 5 and 100% (attention, if you use 100% of the sample as recipients you will have more accurate statistics on performance, but the winning message will be sent to the same percentage of recipients as the others) .
  4. Select the type of metric - Best Open Rate or Best Click-through Rate - for automatic determination of the winning message, or select the manual choice if you prefer to manually determine the winner.

For statistically significant results, select enough recipients to evaluate the effectiveness of the versions. It is usually not recommended to carry out an A / B test with less than 1000 total recipients. If your selection criterion is the opening rate, we suggest you send the message to at least 1500 recipients, if the criterion is clicks, the number rises to at least 6000 recipients.

Step 3: choose the recipients

At this point, after clicking on "Choose recipients", the usual campaign sending interface will appear, to select the recipients of your A/B test.

Step 4: queue and schedule mailings

In the last screen of the wizard, the queuing and loading of the tests takes place. Once completed:

  1. See the test summary
  2. Decide when to start them. Normally, you will not be able to select a date or time that is 5 minutes or less after the current time.
  3. Decide how long after the end of the tests the winning message should start (only if you chose to send it automatically in step 1).
  4. We suggest that you wait a sufficient number of hours (at least 4) from the end of sending.
  5. Press "SAVE" to schedule your A / B test.