Statistical significance: a fundamental part of B2B website experimentation

As B2B marketers we love to experiment, try new things, and test what does and doesn’t work. Most importantly, we measure results to ensure all our efforts are making a difference when it comes to converting potential clients. When applying experimentation to a website, multiple variants can be changed and tested. Including copy and imagery, colors or position of call-to-action buttons and placement of logos and other social proof like awards and badges. Whilst we monitor an abundance of stats and figures such as traffic, engagements, tracking, clicks, conversions, and bounce rates. Statistical significance is the measure we use to determine if the changes we made impacted performance or if the changes affected the results by chance.

We do this within B2B website personalization by creating a traffic split on a campaign, where 50% of the traffic receives the generic, one size fits all website page (the control) and the remaining 50% receives the personalized experience. We measure the results to decide which version is winning (the one that has the most conversions). Once the data meets statistical significance, we know that the personalized experience is outperforming the non-personalized experience. Then we can create a new campaign where 100% of that audience receives the highest performing experience, because statistically speaking we know this version delivers more conversions.

What is statistical significance and what does it measure?

Known as stat sig for short (let’s face it, the full name is a bit of a mouthful), this measure is used in the academic world to determine the reliability of a statistic. The most common representation of the reliability comes in the form of a metric called a ‘confidence interval.’ MarTech explains; this means there is a 95% confidence interval that if we repeated a test, the observed result would hold true 95% of the time. MarTech also debates that whilst a 95% confidence level may be required at an academic level, within the business world we can adjust this to reflect our risk tolerance, so some marketers may reduce the confidence interval to 90% or even as low as 80%.

Here at Webeo we focus on a confidence interval of 95%. There are of course some exceptions to the rule, but we’ll cover that later.

Why is statistical significance important in B2B experimentation?

The long and short of it is; it reduces risk! Basing your decisions on data rather than a gut feeling means you are allocating resource, including budgets, time, and manpower, to the activity that will yield the most significant improvements and maximize ROI. In B2B website experimentation, particularly when thinking about website personalization, this means the traffic you’re driving to your website will receive the best online experience. One that is relevant to them and their needs, and will increase the likelihood that they convert into an MQL.

What are the challenges using statistical significance for decision making?

Statistical significance is great for large sets of data, enabling quick decision making once your results prove that one experiment outweighs the other. On the flip side, a drawback of using stat sig can be if your sample size is small, it will take some time to reach a confidence level that enables you to push forward with a decision. For example, if your experiment is targeting a particularly niche audience, such as manufacturers of composite welding contracts based in rural areas of England. It will take some time to attract enough of those businesses to your website and collect enough data to determine the best experience. In instances like this where volume is low, at Webeo we recommend you do not do an experiment.

Instead, you should create a tailored experience on your website for that audience because you understand enough about your ICP (Ideal Customer Profile) to know by presenting the correct information for their needs, will increase the conversion rate. Plus you always have the results of your generic website as a benchmark anyway.

Another challenge marketers may come across when measuring the effectiveness of their B2B website experiments is what to measure and use as the main data comparison point. The overarching objective of website personalization campaigns is to generate ROI. To achieve this they need their website to deliver enquiries. Therefore, all stat sig comparisons of generic versus personalized experiences are based on the number of conversions delivered.

Following best practice

Some additional elements need to be considered when it comes to using stat sig for business decisions, including defining clear objectives, randomization and control groups, and length of the experiment. When it comes to B2B website personalization the objective is clear. I want to test which experience performs best when it comes to generating conversions – my generic website that presents the same information to everyone, or my personalized, tailored experience that presents the most relevant and important information to my website visitors based on everything I know about them. For other B2B website experiments, the objective may be different, so it’s important to define that at the outset.

Having control groups in place and randomizing which prospects receive the variations of the experiment is also particularly important. When using experimentation in Webeo, our portal enables you to split an audience equally into variants A and B. This is important because to be a true test the audience must be like for like. So you’re comparing healthcare companies that received a generic website experience versus healthcare companies that received a personalized website experience for example. Rather than healthcare companies versus logistics companies.

Monitoring the results of your experiment over time is also important. The B2B sales cycle is particularly long compared to B2C. Analyzing results over time allows for any fluctuations and seasonality that may affect your findings and therefore this enhances the reliability of your results even further.

How to kick start your B2B website experimentation journey

We have endless examples of experiments our clients are undertaking to ensure the decisions they are making for their business are the right ones. If you would like to learn more about how you can use stat sig and data led decision making for the optimization of your B2B website and business growth, get in touch.