How Split Ratios Affect Your A/B Test Results

This guide explains how changing the split of your A/B test (e.g., from a 50/50 split to a 70/30 split based on impressions) affects the statistical significance and performance of your test.

What is a Split Ratio in A/B Testing?

The split ratio determines how traffic (or impressions) is divided between your test variants. For example:

• 50/50 Split: Each variant gets an equal share of impressions.

• 70/30 Split: One variant gets 70% of impressions, while the other receives 30%.

How Does the Split Ratio Impact Your Test Results?

1. Balanced Split (50/50):

   • Best for Accuracy: Both variants receive equal traffic, minimizing variability and maximizing statistical power.

   • Quick to Detect Small Differences: Because both groups have the same sample size, it’s easier to detect subtle changes in performance.

2. Uneven Split (70/30):

   • Favors the Dominant Variant: More impressions go to the higher-priority group, making it a good choice if you’re leaning toward one variant but still want to test another.

   • Slightly Less Efficient: The smaller group (30%) may need more impressions or time to produce reliable data, especially for detecting small differences.

Does an Uneven Split Affect Statistical Significance?

No matter the split ratio, the threshold for statistical significance (e.g., p < 0.05) remains the same. However:

• A 50/50 split is generally the most statistically powerful option, as it ensures balanced and consistent data across both groups.

• A 70/30 split may slightly reduce the efficiency of your test because the smaller group has fewer impressions, which can increase variability.

When to Use an Uneven Split

A 70/30 split can be useful when:

• You Have a Strong Favorite: If you believe one variant will outperform the other, allocating more impressions to the dominant variant can be strategic.

• Business Priorities Favor One Variant: For example, if you’re testing a new dynamic message but want the majority of customers to see the control version for consistency.

Tips for Running A/B Tests with Uneven Splits

1. Increase Total Sample Size:

   To compensate for the smaller group, aim to gather more total impressions. This ensures the 30% group has enough data for accurate results.

2. Focus on Effect Size:

   Uneven splits are better suited for larger differences in performance. If you’re testing subtle changes, consider a 50/50 split instead.

3. Plan for Test Duration:

   With a 70/30 split, the smaller group takes longer to reach statistical significance. Be prepared to run the test for a slightly longer period.

4. Leverage Tools for Uneven Tests:

   Use Because’s dynamic testing capabilities to track performance and ensure both groups are measured accurately, even with an uneven split.

Need Help? Let’s Optimize Together!

If you’re unsure which split ratio is best for your goals, we’re here to help. Reach out to our support team at [e-mail] or explore our A/B testing best practices for tips on maximizing your results.