After graduated from Institut Teknologi Bandung in 2008, Universidade Nova de Lisboa and Freie Universität Bozen in 2012, Zakka worked as a system analyst / software developer in his hometown, Bandung, for a year before deciding to join Bukalapak in 2013 as the Head of Product. With no or little prior knowledge and a huge shifting from project-based to product-based development, he learned a lot to build Bukalapak reaching more than 7x increasing of CVR within 3.5 years of his time there. He believed the easiest way to gain respect from the team is by knowing at least a bit (hard skill) regarding the team's skill. *** This slide was shared at Tech in Asia Product Development Conference 2017 (PDC'17) on 9-10 August 2017. Get more insightful updates from TIA by subscribing techin.asia/updateselalu

1. Analytics & Testing for PM
Zakka Fauzan
- prepared exclusively for PDC’17 -

2. Brief Information
´ All presented here is related to (quantitative) data.
´ However, analytics can be either qualitative or quantitative data (or both).

3. Analytics
´ Why, as PM, do we need to do analysis?
´ Because analytics is one of MANY skills needed by a PM.
´ Here I will just explain two experiments to be analyze:
´ Results of feature developed
´ Results of feature enhanced

4. Principle of Doing Analytics
´ Always test it, more about this later.
´ MVP! You can measure better with MVP.
´ Beware of MVP, it should be Minimum and Viable both.

5. Principle of Doing Analytics
Easy way to understand
MVP, what it should be.

6. Principle of Doing Analytics
´ Put the tracker everywhere so you have better analytics.
´ When it’s written everywhere, it does mean everywhere.
´ “Easy” case is when seeing the whole funnel, not acquisition vs activation only
(e.g., sessions vs transactions), but also sessions for each page.
´ Difficult case, when seeing a form, not only see “how many visiting the form” vs
“how many finished filling the form”, but also see the drop/exit rate for each field.

7. Principle of Doing Analytics
´ Be skeptical
´ Be aware of correlation trap, a.k.a. cum hoc ergo propter hoc.
´ When we find any correlation, make sure that it’s causation (or not).
´ Case in point, Bukalapak in 2014, #pelapak vs #trx.
´ When having a causation, there are 3 possible cases.

8. Principle of Doing Analytics
´ Remember of the pirate metrics of a startup
´ AARRR – Acquisition, Activation, Retention, Revenue, Referral
´ There are two approaches on this:
´ If you know exactly where you want to fix a thing, get focused on it
´ If you are not sure or don’t know, check every part of the metrics, fix the one having lots
of low-hanging-fruit enhancement.
´ At one time (define it by yourself, be it week, 2 weeks, sprint, month, or whatever)
just get focused on one or two out of those five metrics.
´ Too much focuses = no focus at all.

9. Tools for Analytics
´ Google Analytics
´ Google Tag Manager
´ Google Search Console (GWT) especially if your products involve search
engine a lot
´ Mixpanel
´ Some analytics tool for qualitative data can as well be used if you’re
executing qualitative analytics

10. Testing
´ Same questions for analytics skill applies here as well, PM does need testing
skill as one of a lot of skills he needs to master.
´ There are other advantages:
´ Being more independent from data people.
´ Getting respected. One of the easiest ways to gain respect from people around
is by speaking their language. Hence, understanding the (general) language of
data is important.

11. Types of Testing
´ Historical Testing
´ On-off Testing
´ Quite similar to historical testing
´ AB testing / Multivariate testing
´ Multivariate testing is just AB testing with more variants (more than two)
´ When not to use AB testing?

12. Some tips for doing A/B Testing
´ Make sure that your A/B testing is 100%
random.
´ A/B testing should be done at least
one week, because each day is
unique.
´ Wait for enough confidence level
(usually 90/95/99%) before deciding.
´ If it’s possible, always run the test AFTER
the winner is decided.
´ Measure MORE than one metric.

13. Some extra notes regarding A/B Testing
´ To be able to do A/B testing properly, make sure these four property
supports:
´ Resource – time, money, etc
´ Tool - next slide
´ Focus – because sometimes doing A/B testing will make tasks executed a bit
slower
´ People – it’s not enough to have data-mindset on only data, product or even
engineering people, it needs to be spread widely on every part of the company,
even to the C-levels.

14. Best Resources for AB Testing (in my
opinion)

15. Tools for Testing
´ Visual Web Optimizer
´ Optimizely
´ Mixpanel
´ Growthhackers.com
´ Build it yourself!

16. Thank you