Win The Internet With A/B Testing

An Introduction To A/B Testing

Our first item for this lesson is a quick and simple one. This post is designed to give you a very high view of what testing is and why you should consider doing it. Before moving on to our next item, take some time and research these two testing platforms: Visual Website Optimizer and Optimizely, and find a third option of your choosing. Once you look into them, you should create an account on your preferred platform and get familiar with their features and tools.

Optimization at the Obama campaign

In this article, Kyle mentions quite a few important notes about testing. Be sure to explore the tools he mentions, as well as the different types of tests that were run: structural changes, copy changes, color & aesthetic tests, etc. After reading through this case study, you should take a product or website that you have built and try to compile 2-3 different a/b tests that you would consider running. For each, list out the following things:

First: What is your hypothesis for this test? (IE: “Changing the layout of this sidebar will increase clicks to the ‘Related Articles’ button).

Second: What type of analytics events can you create to measure changes? (IE: An event that fires when a user clicks any link in the sidebar. This could show what percentage of clicks in the sidebar are being lost to other elements in each layout).

Third: What outside variables could affect this test? (IE: Changing the sidebar layout also happened to make more ads visible on the page, which could affect the data for the test)

Testing Your Messages: Framing for Increased Conversions

This article is a deep dive into the endless ways you can change & test copy. After reading this, take a headline that exists on a project that you created and write 3 sets of variations that could become with a test. Make sure to refer back to the article and ask yourself the right questions about how you’re ‘framing the value proposition.’ This exercise will help you think about your copy and it’s iterations in a brand new light. Changing a few words or the perspective of the sentence can completely change how people perceive and interact with your product.

Statistical Significance & Other A/B Test Pitfalls

Statistical significance is critical to your optimization success. It is far too easy to judge results from a test too early or incorrectly. Understanding and being able to judge statistical significance on your own tests is an essential skill for anyone trying to optimize their product. During early stages of testing, use a Statistical Significance Calculator to help judge your data’s validity. You can also use this duration calculator to help you assess how long your test should run.

In Defense Of A/B Testing

You should now have a good grasp of what A/B testing is, you can start diving deeper into the theory of testing. Along with the lucrative successes from testing, there are also quite a few pitfalls that come with it. Knowing about these pitfalls ahead of time will let you consciously avoid them as you prepare your hypothesis, craft your tests and analyze your data. At this point, you’ll also want to understand what multivariate testing is and when you should be using this instead of A/B testing.

Wrap-up