An A/B test shows two versions of something, such as a headline or a button, to similar visitors at the same time, and measures which version gets more of the action you want. That is the whole method. What makes it powerful is that it replaces opinions with evidence, including the opinions of people who are very sure they know what works.
The phrase "test everything" needs one honest caveat, which this article covers with real numbers: you can only test what your traffic is large enough to measure. Below, you will see a documented test that paid off enormously, a worked sample-size example, and a simple process you can follow.
- A split test compares two versions on similar visitors at the same time, so the result reflects the change and not the calendar.
- Change one thing at a time, decide your sample size before you start, and do not stop the test early because one version looks ahead.
- Small improvements need large samples. A site with low traffic should test bigger changes.
- Write down every result, including the ones that fail, so you learn something from each test.
A Documented Test: What the Obama 2008 Campaign Learned
The most cited case comes from Dan Siroker, who was director of analytics for the 2008 Obama campaign and later co-founded the testing company Optimizely. In his own account of the experiment, the team tested the splash page, the first page visitors saw, in December 2007.
They tested four button labels ("Sign Up," "Sign Up Now," "Join Us Now," and "Learn More") against six pieces of media, three images and three videos, which made 24 combinations. About 310,000 visitors saw the page during the test, roughly 13,000 for each combination.
The winner was the "Learn More" button paired with a family photo. It had a sign-up rate of 11.6%, compared with 8.26% for the original page, an improvement of 40.6%. Siroker also noted that campaign staff expected a video to win, and that every video did worse than every image. Without the test, they would likely have used a video.
One caution: the larger numbers in that article, about 2.88 million additional email sign-ups and an estimated $60 million in donations, are Siroker's own estimates. They assume that the improvement held for the rest of the campaign. The test result itself is solid, but treat the money figure as an estimate.
The team's favorite option lost. That is exactly why they tested.
How Many Visitors You Actually Need
Here is the part most guides skip. The smaller the improvement you want to detect, the more visitors you need. These figures use common statistical settings (95% confidence and 80% power) and are approximate, based on the standard formula for comparing two conversion rates.
3% conversion, detecting a 40% relative lift (3% to 4.2%)
About 3,800 visitors per version, or around 7,600 in total.
3% conversion, detecting a 20% relative lift (3% to 3.6%)
About 13,900 visitors per version, or around 27,800 in total.
3% conversion, detecting a 10% relative lift (3% to 3.3%)
About 53,200 visitors per version, or around 106,000 in total.
If your page gets 2,000 visitors a month, a 10% lift is out of reach for a single test. That does not mean testing is useless for you. It means you should test big changes, such as a completely different headline or offer, and use free qualitative checks for the small ones. Our guide to heatmaps shows how to see where people click without needing huge traffic.
What to Test First
Start where a change can move the result the most. For most small sites, that means:
1. The headline
It is the first thing visitors read. Test a version that names the result against a version that names the product. Our article on hooks covers headline ideas worth testing.
2. The button label
Compare a specific action with a generic one, such as "Get my free checklist" against "Submit." Our guide to a clear call to action helps you write the options.
3. The main image
The Obama test shows that a simple photo can beat a video the team loved. Test a photo against a different photo before you spend on production.
4. The email subject line
Email tools often include a split test for subject lines. It is a fast place to practice, because each send gives you a result.
A 5-Step Process for Reliable Tests
Step 1: Write a hypothesis
Use this format: "If we change X to Y, then Z will go up, because ___." The last part forces you to explain why the change should help, which makes the result easier to learn from.
Step 2: Change only one thing
If you change the headline, button, and image together and the page wins, you do not know which change mattered. The Obama team varied two elements on purpose and tested every combination, which needs far more visitors. For a smaller site, test one element at a time.
Step 3: Decide the sample size before you start
Use the figures above, or a free sample-size calculator, to decide how many visitors each version needs. Write the number down before you look at any results.
Step 4: Run the test for full weeks and do not peek
Visitor behavior often differs between weekdays and weekends, so run the test for whole weeks. Do not stop the moment one version looks ahead. Early leads often disappear, and stopping early is a common reason tests give false winners.
Step 5: Record the result and the lesson
Add every test to a simple log, including tests that fail. A lost test still teaches you what your audience does not respond to.
Use this template for each entry:
3 Mistakes That Produce False Winners
Mistake 1: Stopping when you see a lead
A version can look 30% better after two days and end the test equal to the other. Wait for the sample size you set.
Mistake 2: Testing tiny changes on a small site
Changing a button color on a page with 2,000 visitors a month will not produce a result you can trust. Save small tweaks for pages with much more traffic.
Mistake 3: Declaring a winner and never testing again
Audiences and offers change. A winning headline is the new baseline to beat, not the final answer.
Replace Assumptions With Evidence
The strongest lesson from the Obama test is not the button label. It is that the people who felt sure were wrong, and that a simple test showed it. You do not need that team's traffic to apply the lesson. Pick one page that matters, write a hypothesis, test a bold change, and let the numbers decide.
Start with your headline this week. Decide the sample size first, run the test for full weeks, and record what you learn.
Frequently Asked Questions
What is an A/B test?
It is an experiment in which half of your visitors see version A and the other half see version B, at the same time. You then compare which version produced more of the action you want, such as sign-ups or purchases.
How long should I run an A/B test?
Run it until you reach the sample size you decided in advance, and for at least one full week so that weekday and weekend behavior are both included. Do not stop early because one version is ahead.
Can I A/B test with low traffic?
You can, but only for large changes. A small site cannot reliably detect small improvements, so test completely different headlines or offers and use heatmaps and short user tests for smaller questions.
Should I test more than one change at once?
For most small sites, no. Changing several things together makes it hard to know what caused the result. Large teams with a lot of traffic can test combinations, as the Obama campaign did.
