In this 2nd Part, series on experimentation. Let’s start with the A/B testing,
In this course I learned the following:
Fundamentals of A/B testing.
The ROAR model for deciding when to run A/B tests.
Key Performance Indicators.
The FACT and ACT research model.
The 6V Conversion Canvas for conducting research.
1. Fundamentals of A/B testing:
A/B testing helps you make better, more well-thought, and trustworthy decisions.
“But how much data do I need to run A/B tests?”
A great way to think about this question is using the ROAR model.
2. The ROAR (Risk, Optimization, Automation, Re-think) Model:
In this phase, you’re starting your business and don’t have a lot of customers. Likely, you won’t have enough data to run A/B tests.
As a general rule of thumb, under 1000 conversions, you can’t do split testing.
That leads you to try a lot of different ideas for increasing your conversions.
If you find something that has a 15% impact on your conversions, then you have discovered a winning idea that will help you expand your business.
To get to the optimization phase, you need to have at least 1000 conversions per month to do proper A/B testing.
After you get above the 1000 conversions threshold, you can experiment with different A/B tests.
At this level, the winner of your A/B test needs to have at least a 5% impact.
To get to the automation phase, you need to get at least 10.000 conversions per month.
At this point, your company is at a mature level where you’re making a lot of sales per month.
Remember, to get to the last phase of the ROAR model, you’ll need to have at least 100 000 conversions per month.
At this level, you’re a huge company like Amazon or Apple, that can run thousands and thousands of experiments each month.
Here, the winners of A/B tests have a small impact. However, due to the scale of the company that impact results in a lot of profit.
Here’s an excellent A/B test calculator from CXL,
Here’s how to use it:
Use at least 80% power to understand who are your winners.
Put your significance at 90% to remove too many false positives.
“Statistical power is the likelihood that an experiment will detect an effect when there is an effect there to be detected.”
3. Key Performance Indicators (KPI)
Each A/B test has different performance indicators like:
Clicks: Easy to improve but don’t have a huge impact on your business.
Behaviour: Easy to optimize but also doesn’t make a great impact.
Transactions: Start optimizing here.
Revenue per user: Harder to optimize, but it has a positive impact on your business.
Potential Lifetime Value: Hardest indicator to improve but very important one.
If you’re a mature company, you should optimize things like revenue per user or potential lifetime value.
If you’re not a mature company then focusing on clicks, behaviour, transactions are a better idea.
Key things to keep in mind:
A/B test tools and calculators are only comparable with binary variables (0/1) meaning you either convert or you don’t.
KPIs can’t be user satisfaction, number of page views, etc. You should look for success indicators and avoid vanity metrics.
Your company should have a good OEC (Overall Evaluation Criteria) for everyone in the company. OEC is defined by using short-term metrics that predict long-time value.
4. The FACT and ACT Model
FACT stands for Find, Analyze, Create, Test.
In the finding and analyzing phase, the main question is “What to optimize?”
Your customer behaviour research goals should give you:
Insights in the most important customer journey.
Understanding of the basic behaviour of your customers.
An input for setting a hypothesis.
In the creating and testing phase of the model, you’re doing a lot of experiments based on the information you analyzed.
Then, you start implementing the ACT model:
Analyzing the results.
Telling the results.
5. SIX V Conversion Canvas
Here, you should ask yourself two questions:
- What company values are important and relevant?
- What focus delivers the most business impact?
Here, you’re doing a full analysis of your competitors and their best practices.
You can buy from them to understand more about their customer journey or track their website changes. Also, you can learn which tools they are using on their website and even find their A/B tests.
Here, you’re looking at your own web analytics and behaviour data to learn more about your customers.
Questions to ask yourself when looking at your web analytics:
Where do visitors stay on the site?
Where do they come from?
What is the flow of those visitors?
Are there notable differences between segments or products?
What’s the behaviour on the most important pages?
Questions to ask yourself when looking at your landing pages:
Where do visitors start their journey?
What’s the difference between new and existing customers?
What’s the difference per device?
Questions to ask yourself when looking at your traffic sources:
Where do visitors come from?
Do they already have a product in mind?
Do they already know the brand?
Questions to ask yourself when looking at different customer journeys:
What’s the CTR (and return rate) to the next step?
What’s the exit rate and time spent on a page per step?
Here’s a typical e-commerce flow example:
All users on your website with enough time to take action.
All users on your website with at least some interaction.
All users on your website with heavy interaction.
All users on your website with clear intent to buy.
All users on your website are willing to buy.
All users on your website that succeed in buying.
All users on your website that return with the intent to buy more.
Here are different measurements you can use when conducting research:
The number of templates.
The time on a page.
Heat and scroll maps.
Ask yourself “what insights can be taken from the voice of the customer data such as surveys, feedback, and service contact?”
Here’s what you can do to gather more data:
Talk to your customer service.
Check chat logs.
Check social media feedback.
Ask for feedback online.
Create focus groups.
Make usability studies.
Questions to ask yourself here are:
What scientific research, insights, and models are available?
What do we know from scientific literature?
“If there’s an industry, there are studies done about them.”
Here, useful resources are things like Google Scholar, Semantic Scholar, DeepDyve.
A question to ask yourself here is:
- What insights are validated in previous experiments and analyses?
At this point, you take into account things like assumptions on behaviour and motivation as well as assumptions on emotion & rationality.
Hope you have learned about the Experimentation Process. I strongly recommend to take up CXL’s Growth Marketing MiniDegree if you want to get more deeper.