Use Math, Not Gut, When Making Adwords Decisions

mad men cast

Marketing is no longer the domain in which the Visionary Creative rules.

Don Draper is dead.

The internet has changed advertising. Traditional advertising such as print, radio and TV commercials are expensive, slow, and subject to a lot of noise when trying to measure their result. What is worse, A/B testing traditional advertising – on the exact same geographic sample,  over the exact same period of time – is pretty much impossible. When you advertise on the internet, however, you can serve every other visitor either ad A or B, and land them on page A or B, and measure the results.

As such, advertising is now more science than art.  But how do you know, not guess, if the result of an A/B test is valid? Pretend that we have two ads, A and B, that both are seen 100 times. Ad A got 10 clicks, for a Click Through Ratio (CTR) of 10%. Ad B got just 1 click (CTR=1%). You may not need a calculator to see ad A is the winner. But what if the spread is less? What if Ad B got 4 clicks or 5 clicks? If 4 clicks, Ad A is still a winner. If 5 clicks, you need more data. How do you figure it out? Use a Significance Calculator.

A Significance Calculator is a statistical tool that allows you to calculate if you have an adequate sample size in your A/B test to determine if the results are valid. The smaller the spread in results, the greater the sample size needed.

Let’s see a demonstration where the eye may not be able to know if you have a statistically valid result.

Below we have three examples in which we have identical CTRs. In this example we are telling the calculator that we want to be 95% certain that we have found the winner in our A/B test.

Significance Calculator

Here we see that NO, we lack adequate data to make a decision.


Significance Calculator

Again we see that NO, we still need more data.


Significance Calculator

Finally we have adequate impressions and clicks for a valid and actionable test.


Do not get me wrong. Creativity is still has its place when creating a successful Google Adwords or Bing Adcenter campaign. It is essential to get into the potential customer’s head and to create a Persona of the target audience, and speak their language in both the ad and on the website. This is the domain of insight, creativity, and inspiration. But the advertiser that wins most is not the most creative. It’s the one who is continually testing ideas and improving based on data – accurate data.

And if you are going to channel your inner Don Draper, do the booze part at night and not when you are putting the inspiration to the actual test.

Thanks for reading,

Ted Zuhlsdorf

Leave a Reply

Your email address will not be published.