A/B testing headlines means showing two versions of a headline to similar visitors at random and keeping the one that converts better. The framework below covers what to test, how to set up the test and the mistakes that make results worthless.
Why does A/B testing headlines matter?
Your first headline is almost never your best one. Every headline you write is a hypothesis, and the one that feels right is not always the one that converts best. Testing is the only way to know.
Headline tests are among the highest-return work in conversion optimization. A 5% improvement in click-through rate can mean thousands of additional visitors, and that gain keeps adding up for as long as the page runs.
A badly run test, though, wastes traffic and gives you misleading results. The steps below show how to run a clean A/B test, and our wider guide to headline testing covers multivariate and bandit tests.
What should you test in a headline?
Some parts of a headline are more worth testing than others. Start with the ones that have the most impact, and run a quick headline analysis first to find the weakest part of your current headline.
Element 1: The core promise
The core promise is the main benefit or outcome your headline communicates. "Cut churn by 30%" is a core promise. "Stop losing customers" is the same promise put another way.
Testing the core promise tells you the most about what your audience responds to.
Element 2: Specificity level
"Cut churn by 30%" is more specific than "reduce customer loss." The more specific version often converts better, but not always, so test how specific your audience wants it.
Element 3: Emotional tone
Headlines can be serious ("Analytics for data-driven decisions") or emotional ("Know what your best customers do before they churn"). Different audiences respond to different tones, so test the emotional version against the rational one.
Element 4: Format
Try a question against a statement, a numbered list against a direct promise, or a command against a question. Format changes how the headline feels and whether it creates urgency. Our overview of the types of headlines lists the formats worth putting against each other.
How do you set up a headline A/B test?
Step 1: Define your metric
For a landing page headline, success might be clicks on the primary CTA. For an email subject line, it might be the open rate. On a demo request page, count booked demos rather than clicks.
Pick your primary metric before you start. Changing it mid-test invalidates the results.
Step 2: Write one variable change
Test one thing at a time. If you change both the format and the promise, you won't know which change caused the result, so write two headlines that differ in exactly one way.
Step 3: Split traffic evenly
Divide your traffic at random, so that half your visitors see headline A and half see headline B. An equal split keeps the results clean.
Step 4: Set a minimum sample size
Small samples produce noisy results. A 10% difference with 100 visitors could be random noise; the same 10% difference with 10,000 visitors is significant. Use a sample size calculator to work out how many visitors you need before you call a test.
Step 5: Run long enough
Run your test for at least two full business cycles. Traffic patterns differ by day of the week, and a test that stops too soon gives skewed results.
What are the common headline testing mistakes?
Mistake 1: Testing without traffic
Without enough traffic, the results are noise. If you're getting fewer than 1,000 visitors per week, work on increasing traffic before you test.
Mistake 2: Calling tests too early
Stopping a test because one headline is ahead feels efficient, but early results often flip, so the early winner can mislead you. Run the test to statistical significance before you call it.
Mistake 3: Ignoring secondary metrics
Your primary metric might be click-through rate and your secondary metric bounce rate. If headline A gets more clicks but also more bounces, it isn't a clear winner. Track secondary metrics to catch side effects like that.
Mistake 4: Not documenting results
Write down what you tested, what the results were and what you learned. Otherwise you'll forget what worked and why. A written record turns single tests into knowledge the whole team can use.
FAQ: A/B testing headlines
Q: How much traffic does a headline test need?
A: Enough for each version to reach about 100 conversions. Below that, the result is noise.
Q: How long should a test run?
A: At least one full week, so that weekday and weekend visitors are both included, and until the sample size is reached.
Q: Can I test more than two headlines?
A: Yes, but each extra version needs the same traffic. With low traffic, test two at a time.
What should you do now?
- Identify one high-traffic page to test.
- Write two headline variations that differ in one specific way.
- Set up a split test with equal traffic distribution.
- Define your primary metric and minimum sample size.
- Run the test for at least two weeks.
- Document the results and apply the winning variation.
Sources
- Wikipedia: A/B testing
- Wikipedia: Sample size determination
- Nielsen Norman Group: How Long Do Users Stay on Web Pages?
Need variations to test? Punchd's AI headline generator writes 20 headlines in different styles from one brief.