2. How Long Should My Test Run???
• The data rate (number of conversions per
day)
• Size of improvements found (Percentage
Improvement)
• Size of your test (Number of alternative
designs)
• The confidence in your answer ( How sure you
need to be )
3. Data Rate
• The data rate decribes how quickly you collect
data during your test.
• The volume of traffic for landing page
optimization tests is best measured in the
number of conversion actions per day ( and
not the number of unique visitors).
• You can decrease the size of the test. This can
be done by decreasing the granularity of your
test elements.
4. Size of Improvements Found
• If you have managed to uncover a clearly
superior version of your landing page, the performance improvement would quickly become
apparent.
• often, an initial round of changes will fix some of
the obvious problems and improve performance
significantly.
• Since you do not know the size of the possible
improvements ahead of tme, the length of the
time required for the tuning test may vary
signifantly.
5. Size of your Test
• The size of your test can be measured by the size
of the search space that you are considering.
• The search space is the whole universe of the
alternative designs possible in your test.
• A simple head to head test has a search space
size of 2 ( the originl, and the alternative landing
page version that you are testing.)
• If you are testing multiple elements on the page,
you need to multiply together the number of
alternative versions for each one.
6. Variable Interactions
• When the setting for one variable in your test
positively or negatively influences the setting of
another variable is variable interaction.
• If they have no effect then they are said to be
independent.
• In a positive interaction, 2 variables create a
synergistic effect.
• In a negative interaction, 2 variables undercut
each other and cancel out some of the individual
effects.
