This document provides an overview of an optimization process for testing and improving marketing efforts. It discusses 13 steps in the process: 1) defining business goals, 2) understanding the target audience, 3) analyzing web data, 4) performing heuristic analysis, 5) capturing challenges, 6) forming hypotheses from challenges, 7) designing experiments, 8) creating variations, 9) building variations, 10) implementing experiments, 11) analyzing results, 12) reporting findings, and 13) determining next steps. Following a scientific process helps ensure statistically valid results that can add to the bottom line through reasonable testing efforts.

1. Testing & Optimization:
A Deeper Look
January 16, 2013
Brendan Regan
Marketing Optimization Specialist
Analytics Pros

2. BestPractices – Seattle, WA – 12.6.12
Late last year, I presented “Building a Successful
Optimization Program.”

3. TO GET THE SLIDES:
http://www.slideshare.net/analyticspros/best-
practices-optimizationpresentationbrendanregan

4. TODAY, we’re focusing in for a deeper look…
People | Process | Tools
Process | Tools

5. Why is Optimization
“process” worth a
webinar?
I’m glad you asked.

6. “Process is boring.”

7. Process is the stuff of superheroes!

8. Process lets you…
1. Achieve valid, effective tests
2. Take full advantage of business
opportunities

9. Without Process With Process
Unclear, misleading results  Statistically valid results 
Results that don’t add $  Results that add to the bottom line 
High effort, low return  Reasonable effort, huge return 
Opinion-based testing  Data-driven testing 
“Inconclusive” results (a costly tie)  Clear winners and losers 

10. Take advantage of opportunities
1. Time to market
– How fast can you get
improvements in place?
2. Test the right stuff, in the right
order
– Low Investment, high Return

11. Maximize Return Over Time

12. Today, You Will…
1. Learn the Scientific Method…for
Marketers
2. Learn a proven Optimization
process
3. Discover 4 hidden BestPractices
nuggets 

13. The Scientific Method…for Marketers
1. Define the problem
2. Research/observe the problem
3. Form a hypothesis
4. Conduct experiment
5. Analyze experiment results Look familiar?
6. Form a conclusion
7. Socialize results

14. Optimization Process
Optimization
Science Process Marketing

15. 1) Business Goals
Image credit: Square2Marketing.com

16. 1) Business Goals
• In science-speak, this is “define
the problem”
• Resist temptation to:
– skip it
– assume
• Multiple goals OK
– prioritize

17. 1) Business Goals
• Example:
– Conversion Rate = primary business goal
– Forward to a friend engagement =
secondary
– Test for Conversion Rate
– Analyze/monitor F2F engagement

18. Hidden BestPractice Nugget #1:
Test as “close to” revenue as you can.
Testing further upstream can be
misleading.

19. 2) Understand Target Audience
Image credit: Closed Loop Marketing

20. 2) Understand Target Audience
• In science-speak, this is “research the
problem”
• Generally qualitative data (“market
research”)
– personas
– feedback forms
– focus groups
– social media listening
– chat transcripts
– call center monitoring
– demographics/psychographics

21. 3) Web Data Analysis

22. 3) Data Analysis (Quantitative)
• In science-speak, this is
“research the problem”
• Web Analytics
– High traffic points
– Decision/conversion points
– Persuasion points
– High abandonment points
• Survey Data

23. 4) Heuristic Analysis
Image credit: Printed Circuit University

24. 4) Heuristic Analysis
• In science-speak, this is
“research the problem”
• Are “best practices” (rules of thumb)
applied?
• Are you at parity with competitors
and industry leaders?

25. 5) Capture Challenges (roadblocks)
No, not
this kind of
Roadblock.
Image credit: YoJoe.com

26. 5) Capture Challenges (roadblocks)
Image credit: ArtsJournal.com

27. 5) Capture Challenges (roadblocks)
• Challenges presented by your
site/marketing to your
prospects
• Ex: “Landing page lacks ‘social
proof’ content.”

28. 6) Challenges » Hypotheses

29. 6) Challenges » Hypotheses
• Every challenge/roadblock has
potential solution(s)
• Best guess, best solution =
hypothesis
• Ex: “Adding ‘social proof’
content will increase conversion
rate.”

30. Hidden BestPractice Nugget #2:
• Hypothesis: “Adding ‘social proof’
content will increase conversion rate.”
• Research Question: “Which version of
social proof content will perform
best?”

31. 7) Design Experiment

32. 7) Design Experiment
• Formal documentation
• Capture:
– Goal(s)
– Problem/challenge
– Quantitative/Qualitative proof
– Hypothesis
– Specifications

33. Hidden BestPractice Nugget #3:
“Confidence level” - the statistical probability
that your results are not due to chance.
Own your confidence level. Most tools are 95%.
Get buy-in on tool’s default or other confidence
level.

34. 8) Design Variations

35. 8) Design Variations
• Sketch/wireframe
• Copy deck
• Designs/mockups
• Approval process
–Per experiment design
–Marketing/Legal blessing

36. 9) Build Variations

37. 9) Build Variations
• You may have to write code 
• You may “throw away” code
• Page load speed needs to be
consistent!

38. 10) Implement Experiment

39. 10) Implement Experiment
• Integrate w/ testing tool
• Test before you test
– X-browser
– Acceptance testing?
• Launch & Monitor
– 7 days minimum to mitigate “day-of-week”
– Patience, please 

40. 11) Analyze Results
No, not
this kind of
analysis.

41. 11) Analyze Results

42. 11) Analyze Results
• Google Analytics integrations
– Optimizely, Visual Website Optimizer, etc.

43. 11) Analyze Results
• What does the result mean
about the target audience?
• Are the learnings applicable
elsewhere?
– Other sites?
– Other campaigns?
– Other channels?

44. 12) Report Findings

45. 12) Report Findings
• If +, tie it to revenue
– “estimated”

46. 12) Report Findings
• If +, tie it to revenue
– “estimated”
• If -, what did you learn?

47. 13) Next Steps
• Install winning version
– You don’t make $ until you install
the winner!
– Production-ready code, SEO, etc.
• Follow-up test ideas
• Add to Marketing knowledge
base

48. Image credit: http://rahuleink.wordpress.com/2010/04/23/what-is-six-sigma/
Hidden
BestPractice
Nugget #4:
Optimize your Optimization. Stay lean,
learn as you go, smooth out the process
using Six Sigma or other methodologies.

49. Key Takeaways
1. Optimization Process is your friend
2. Optimization Process maps to science…
– …but bows to Marketing
3. Optimize your own Optimization efforts
– “…the best is the enemy of the good.”
Voltaire (1694–1778)

50. Questions?