Identify Test Pages
Tests take a lot of traffic. When you are testing variations you need to select a sample size that allows you to evaluate the outcomes with statistical certainty. In the example below I have a current conversion rate of 20% and I want to see if the changes I’ve made increase the conversion rate by 10% or more (so from 20% to 22+%). I’ll need 6,347 visitors per variation in order to complete my test. So I will want to select pages (or sites if you have multiple) that allows me to test in a decent timeframe. Keep in mind that tests do go stale and can introduce inaccuracies if they run too long so running a test for 12 months is not a perfect way to make up for low traffic volume.
You can find the tool I use on Evan Miller’s site. Just make sure to select the “Relative” radio button, not “Absolute”.
Get everyone’s buy in before running the test. Stopping the test before you reach the agreed on sample size will result in more false results. If you stop a test early be prepared to throw out the data and draw no conclusions. This mostly happens when people see a strongly positive result and want to switch to the new variation (false positive) or when they see a strongly negative result, panic, and pull the test. In either case the test is not conclusive.