Win The Internet With A/B Testing

Designing websites and building products for people is an exciting job. You get to use creativity and problem solving skills to create, in some cases, world-changing solutions. The most exciting part of this, however, is being able to run user-research sessions and A/B tests to see how your customers actually use your product. In most cases, results from these tests are incredibly eye-opening and the learning opportunities are endless.

This lesson should give you an understanding of what A/B testing and Optimization are, why you should apply them to your work and how to properly conduct A/B and multi-variate tests.

  1. An Introduction To A/B Testing

    Our first item for this lesson is a quick and simple one. This post is designed to give you a very high view of what testing is and why you should consider doing it. Before moving on to our next item, take some time and research these two testing platforms: Visual Website Optimizer and Optimizely, and find a third option of your choosing. Once you look into them, you should create an account on your preferred platform and get familiar with their features and tools.

  2. Optimization at the Obama campaign

    In this article, Kyle mentions quite a few important notes about testing. Be sure to explore the tools he mentions, as well as the different types of tests that were run: structural changes, copy changes, color & aesthetic tests, etc. After reading through this case study, you should take a product or website that you have built and try to compile 2-3 different a/b tests that you would consider running. For each, list out the following things:

    First: What is your hypothesis for this test? (IE: “Changing the layout of this sidebar will increase clicks to the ‘Related Articles’ button).

    Second: What type of analytics events can you create to measure changes? (IE: An event that fires when a user clicks any link in the sidebar. This could show what percentage of clicks in the sidebar are being lost to other elements in each layout).

    Third: What outside variables could affect this test? (IE: Changing the sidebar layout also happened to make more ads visible on the page, which could affect the data for the test)

  3. Testing Your Messages: Framing for Increased Conversions

    This article is a deep dive into the endless ways you can change & test copy. After reading this, take a headline that exists on a project that you created and write 3 sets of variations that could become with a test. Make sure to refer back to the article and ask yourself the right questions about how you’re ‘framing the value proposition.’ This exercise will help you think about your copy and it’s iterations in a brand new light. Changing a few words or the perspective of the sentence can completely change how people perceive and interact with your product.

  4. Statistical Significance & Other A/B Test Pitfalls

    Statistical significance is critical to your optimization success. It is far too easy to judge results from a test too early or incorrectly. Understanding and being able to judge statistical significance on your own tests is an essential skill for anyone trying to optimize their product. During early stages of testing, use a Statistical Significance Calculator to help judge your data’s validity. You can also use this duration calculator to help you assess how long your test should run.

  5. In Defense Of A/B Testing

    You should now have a good grasp of what A/B testing is, you can start diving deeper into the theory of testing. Along with the lucrative successes from testing, there are also quite a few pitfalls that come with it. Knowing about these pitfalls ahead of time will let you consciously avoid them as you prepare your hypothesis, craft your tests and analyze your data. At this point, you’ll also want to understand what multivariate testing is and when you should be using this instead of A/B testing.


    Now that you’ve completed this lesson, you should be all set to start testing your products. You have your platform of choice from #1, a set of test options and relevant hypothesis from #2, a set of copy tests from #3, and tools to properly manage and analyze your data form #4. Lastly, #5 should have given you the understanding of difficulties with A/B testing, along with the alternative: Multivariate tests.

    So, what are you waiting for? The data is out there just waiting for you to gather it.

Finally, an easy to follow design course

Receive a design lesson in your inbox each week, hand crafted by a design pro. Learn at your own pace, and apply it to your real life work – no fake projects here.

No spam ever. We promise.