Presentation given to the Gener8tor portfolio companies on 1/31/2013.

1. STARTUP ANALYTICS
Getting Started Down the Path to Understanding
Your Business and Your Users
Dale Beermann
Chief Technology and Analytics Officer
dale@studyblue.com

2. THE GOAL OF ANALYTICS:
IMPROVING YOUR BUSINESS BY
ANSWERING AND ACTING ON QUESTIONS
With every question answered, ask yourself if it is the
desired result. If not, determine what needs to be
done to improve it.

3. BUSINESS METRICS
VERSUS
USAGE METRICS
You should always be reporting on your business metrics.
Analytics is the way to understand what is driving them.
Effectively, business metrics are the aggregate result of
your usage metrics.

4. BUSINESS METRICS
The ultimate goal of business metrics is to evaluate the health
of your business. Examples:
How fast is your business growing?
What is your churn rate?
What is your cost per acquisition for each channel?
What is your Average Revenue per Active User?

5. USAGE METRICS
The ultimate goal of usage metrics is to evaluate the health of
your product. Examples:
What percentage of your users are realizing your value
propositions?
Is your new feature reaching the expected audience?
What percentage of users make it through the onboarding
process?
What percentage of users are using social channels?

6. INFLUENCING BUSINESS
METRICS
Know the answers to your high-level business metrics before
digging into your usage.
Use your usage metrics to determine how you can influence
your business metrics.

7. BEST PRACTICES

8. THE RIGHT TIME TO START
Have you found your product/market fit?
There may be some high level business metrics that help
you get there, but don’t start your analysis on a product
that is going through a massive amount of change.

9. WHERE TO START
Have you filled out a Business Model Canvas? What are your
business’ most important metrics?
How well are each of your customer segments doing
when it comes to realizing your value propositions?
Take your value propositions and work backwards through
the paths that your users take to get there.

10. DEVELOP GOOD HABITS
Make Analytics a core part of your development workflow.
Ensure you are creating both good behavioral habits as well as
good programming habits.

11. GOOD BEHAVIORAL
HABITS
Review your metrics on a regular basis
Continually log changes that are going into your product
There will inevitably be a point in the future where you ask
yourself what happened six months ago to influence a
particular metric.

12. GOOD PROGRAMMING
HABITS
Create guidelines and tools that require you to implement
metrics as you build out your software
E.g. Use abstract click handlers that can be easily refactored:
display.addClassesHandler(new SBClickHandler(SBAnalytic.HOME_FIND_CLICK) {
@Override public void doOnClick(ClickEvent event) {
...
}
});

13. AVOID VANITY METRICS
Page views don't matter (impressions may).
Time On Site can be interesting, but doesn't necessarily
convey usage.
It’s very difficult to influence metrics like Page Views or Time
on Site. Attempting to do so will be a waste of your time.

14. FOCUS ON ACTIONABLE
METRICS
These are going to be different for every business.
Again, you want to find the metrics that mean the most to
your company and determine how you can influence them.

15. MAXIMIZING A METRIC
CAN HAVE SIDE EFFECTS
Providing multiple options splits your usage between them.
Similarly, forcing users down one particular path means they
can’t take another. This can arise in subtle ways.
In some cases, such as with a payment page, you may be able
to find the optimal solution without many side affects.
Ask yourself: What user segments are affected by
this change? Will any side effects be worth it?

16. IMPLEMENTATION

17. CAVEAT: I DO NOT SUBSCRIBE TO THE IDEA
THAT YOU SHOULD LIMIT WHAT YOU TRACK.
If you are smart about how you’re doing your analysis,
you will not fall into the trap of “analysis paralysis.”

18. START WITH
GOOGLE ANALYTICS
It’s free and you can throw everything at it without worrying
about usage tiers.
We don’t use the high level (vanity) metrics for much. Rather,
by sending our events through Google Analytics, we have the
ability to answer a lot of questions.

19. GETTING THE MOST OUT OF
GOOGLE ANALYTICS
Track all of your events (views, clicks, actions).
This isn’t limited to your click stream. Track final events for
workflows (e.g. completed_onboarding). This allows you to
create Advanced Segments for those events.
Set up profiles for each platform (web, iOS, Android, etc.).
You’re going to have very different usage patterns for each
platform, and they should be analyzed separately.

20. GETTING THE MOST OUT OF
GOOGLE ANALYTICS
Make use of custom variables.
At the very least, you should be setting your (non
personally-identifiable) user ID as one of the variables.
This will let you find some per-user data that is otherwise
difficult with Google Analytics.
If you have organizational data, or if your users are
segmented in pre-defined ways, this can help look at those
segments more closely.

21. I’M TRACKING MY EVENTS.
NOW WHAT?
Funnel Analysis
The goal of a funnel analysis
is to determine where your
users are falling off.
Take one of your core
metrics and walk through
the steps it takes to get
there.

22. FUNNEL ANALYSIS EXAMPLE
StudyBlue and Indexable
Content
We want to maximize the
amount of content created
that is “paired” with a class.
How does that happen?

23. HOW DO YOU IMPROVE
YOUR FUNNELS?
Think about how can you change an experience to improve
the end result.
Sometimes this is as simple as changing a button’s color or
using a modal popup (while thinking about the side effects).
A/B Testing
A/B Testing can be a reliable way to evaluate multiple paths.
Caveat: Do your homework and understand statistical
significance. Learn what a chi-squared test is.

24. TOOLS FOR FUNNEL
ANALYSIS
Google Analytics does make it possible to do some of this.
Their goal conversions are annoying if you don’t use page
views the way they expect.
Create advanced segments for users with particular events.
Other good for-pay tools are KissMetrics and Mixpanel.
Roll your own.
In all honesty, doing this stuff yourself isn’t that hard.

25. TANGENT: YOUR OWN
IMPLEMENTATION
You’ll want to use partitioned tables under the hood (if your
data store supports it).
In postgresql, we use triggers to write data to the correct
table. Queries then only hit the necessary tables for the time
span you’ve defined.
We got away with a table per week for about 5 years. Our
table schema:
user_id, session_id, platform, activity_id, activity_timestamp, activity_detail

26. ADVANCED ANALYTICS

27. COHORT ANALYSIS
A cohort is a set of users grouped in a particular fashion.
Typical cohorts are time-based (week of registration). Cohorts
can also be based on acquisition campaigns (e.g. Adwords vs.
Direct vs. SEO).
The purpose of a cohort analysis is to understand user
retention and if your changes are making an impact between
cohorts.

28. WHY COHORT ANALYSIS
Most educated investors are going to ask for cohort analyses.
Cohort analyses, and their corresponding retention rates help
determine:
Engagement levels. Are you a one-and-done sort of site?
Churn rates. If users aren’t coming back to your site, or if
churn is higher than acquisition, your site will not grow.
Quantifying the value of your existing userbase.

29. A COHORT ANALYSIS
EXAMPLE
Sadly, I can’t provide some of our own data here. But this is
what your cohort analyses will look like:

30. COHORT ANALYSIS QUERIES
In Posgtresql: crosstab.
In MySQL: Pivot Tables (still pretty manual).
In everything else: pull your data into one of the above. Or
write a lot of code.

31. THE HOLY GRAIL
A full fledged Customer Relationship Management system
driven from your analytics solutions:
Adaptive in-app user education
Drip email campaigns
Churn prediction
Re-engagement

32. QUESTIONS?