The art of data analysis

The Art Of Data Analysis

Karthik Shashidhar
Quant Consultant
karthik.shashidhar@gmail.com

© Karthik Shashidhar

Introduction

Six-step process

Case Study

Common Pitfalls


Why do you need this workshop?

We are moving to an increasingly data-driven world

Ability to use data for day-to-day decision-making
can prove to be a massive competitive advantage

This workshop equips managers with basic tools for
dealing with data

Who needs this workshop?

What is the optimal level of sales
Sales Managers commissions in order to maximize
profitability?

Production How do we set daily production targets
Managers given probabilities of line shut downs?

What are the factors that determine
HR Managers
employee attrition?

This workshop is suitable for personnel in middle to
senior management roles across functions

Frame a clear and concise problem
statement

Break down your problem into
smaller problems, and then use
those to generate hypotheses

Gather, clean and prepare data
A structured, iterative
approach to data-driven
decision making Test hypotheses. In the process,
generate additional hypotheses

Consolidate results to solve the
main problem

Make the data tell a story


The Rs. 32 Poverty Line

Based on data from the 66th NSSO Survey, the Planning
Commission fixed the “Poverty Line” at Rs. 32 per person
per day for people living in urban areas. This has led to
much controversy and protests. The Prime Minister has
asked for your inputs. What do you recommend?


statement



For your reference
Test hypotheses. In the process,

main problem



statement
How would you frame the problem
statement for this one? Break down your problem into
• Your client may not have framed the
question precisely. You need to do
that job and frame a precise problem Gather, clean and prepare data
statement
• “Solving this problem” should tell
you everything you want to know Test hypotheses. In the process,
from your analysis generate additional hypotheses
• Be concise, so that you remain
focused towards answering your
question Consolidate results to solve the
• Frame your question such that it has main problem
an objective answer. Yes/No
questions or questions with
numerical answers are preferred Make the data tell a story


statement
Has the poverty line been set
too low at Rs. 32 per day? Break down your problem into
• This problem statement has an
objective answer (yes/no)
• The solution to this will be necessary Gather, clean and prepare data
and sufficient to answer the
question our client (the PM)
demands Test hypotheses. In the process,
• The question addresses directly the generate additional hypotheses
situation (people complaining that
the poverty line has been set too
low) Consolidate results to solve the
• This problem statement is to the main problem
point and doesn’t take on additional
responsibilities (such as defining an
alternate poverty line) Make the data tell a story


What problems do we need to statement

solve in order to solve the
main problem? smaller problems, and then use

• The set of “level two problems” must be
precise and complete, in that: Gather, clean and prepare data
• The combination of solution of all
level two problems leads to the
solution of the main problem
• The solution of each level two Test hypotheses. In the process,
problem directly impacts the main generate additional hypotheses
problem
• Once again, it is key to frame problems
concisely and with objective answers
• We need not stop at two levels. Some Consolidate results to solve the
main problem
level two problems might require
solution of deeper problems. Add them
to the list of sub-problems


What do we need to know to statement

answer “Has the poverty line been
set too low at Rs. 32 per day?” smaller problems, and then use
• How is “poverty line” defined?
• What are the implications of poverty
line? Gather, clean and prepare data
• What is the distribution of income in
India?
• Does the distribution of income vary Test hypotheses. In the process,
across states? If it varies significantly generate additional hypotheses
does it make sense to have a state-
wise poverty line?
• What are the essential goods that Consolidate results to solve the
most people need? main problem
• For a given income level, what
essential goods can a person afford?


Problems generate sub-problems, statement

and some of these will lead to
hypotheses. smaller problems, and then use

• Hypothesis1: There is significant
difference in income level across
states
• Hypothesis2: Essential goods are Test hypotheses. In the process,
those that the poorest people
consume. Also, their use flattens out
as income goes up
main problem



Some problems, however, are statement
direct, and don’t need hypotheses.
Some are qualitative while others Break down your problem into
need data smaller problems, and then use
• Question1: How is “poverty line”
defined?
• Poverty line is the minimum Gather, clean and prepare data
income level that is deemed
adequate
• If a family is “below poverty Test hypotheses. In the process,
line” it qualifies for additional generate additional hypotheses
state benefits
• Question2: What is the distribution
of incomes in each state? Consolidate results to solve the
• Question3: Is there some kind of a main problem
threshold about the proportion of
population that can be below
poverty line? Make the data tell a story


statement

What data do you need here?

• It is important to frame problem and Gather, clean and prepare data
break it down into components
before listing data requirements,
else data could bias you Test hypotheses. In the process,
• Define data requirements in a generate additional hypotheses
general fashion, to allow you to
easily access proxies
• Remember to gather data that both Consolidate results to solve the
answers your questions and will main problem
allow you to test your hypotheses



Once you’ve identified data statement

requirements, identify sources and
gather data smaller problems, and then use

• Here we need
• Distribution of a measure of
income for India
• Distribution of a measure of Test hypotheses. In the process,
income for each state
• Spending patterns for different
income levels
• Data on household sizes in Consolidate results to solve the
main problem
different states



Once you’ve identified data statement

requirements, identify sources and
gather data smaller problems, and then use

• The National Sample Survey Gather, clean and prepare data
Organization (NSSO) conducts
surveys every 5 years about income
and expenditure, so we could Test hypotheses. In the process,
perhaps use this generate additional hypotheses
• However, income data gathered from
surveys are notorious with respect to
quality Consolidate results to solve the
• Poor have little savings so their total main problem
consumption is a better indicator of
income than the income data


statement
Data cleaning is an ugly but
important step Break down your problem into
• It is important to make sure names
from data procured from different
sources match Gather, clean and prepare data
• For example, some government
sites say “AndhraPradesh”, while
others say “Andhra Pradesh”. Test hypotheses. In the process,
Fails if you want to do a join generate additional hypotheses
• If data set is small, go through it
once to check numbers for
consistency. For example, if you have Consolidate results to solve the
data on percentages, make sure it main problem
adds up to 100%
• For larger data sets, try write scripts
to do basic cleaning Make the data tell a story


statement
Understand and prepare data
before you dive into analysis Break down your problem into

• Get a general feel for the numbers Gather, clean and prepare data
before getting into the analysis
• Simple visualization techniques such
as scatter plots and density plots Test hypotheses. In the process,
help generate additional hypotheses
• Use simple summary statistics
(mean, median, SD, quartiles) to get
a better feel for the data Consolidate results to solve the
• Check out what different functional main problem
forms of your data look like



While testing hypotheses, be on the statement

lookout for anything
interesting/unusual smaller problems, and then use
• It is impossible to generate all
possible hypotheses before you
begin the analysis Gather, clean and prepare data
• Usually, as you test out some
hypotheses, something in the data
will stand out which will lead to Test hypotheses. In the process,
further hypotheses generate additional hypotheses
• It is ok to generate these
hypotheses, which is what makes it
an iterative process Consolidate results to solve the
• However, one needs to be careful to main problem
not stray from the original objective
– each new hypothesis should
directly tie in to the original question Make the data tell a story


statement

Consolidate results

• Build up your case in a bottom-up
manner Gather, clean and prepare data
• Sometimes different pieces of
analysis can throw up contradictory
inferences. Check, and reconcile Test hypotheses. In the process,
before you integrate generate additional hypotheses
• Make sure all components of the
solution that you required are
available Consolidate results to solve the
• Don’t include results in the final main problem
analysis unless it makes a definite
contribution to the final solution


statement

Use graphics intelligently!

• A picture is worth a thousand words, so
use clear and easy-to-use visualizations Gather, clean and prepare data
to communicate your findings
• Use visualizations that make the solution
self-evident, rather than something that
requires a lot of explanation Test hypotheses. In the process,
• Use your graphics to communicate, not generate additional hypotheses
to confuse. If the intent of a graphic is to
confuse, it is better to leave out that
graphic
• Sometimes all it takes to solve the Consolidate results to solve the
main problem
problem is to visualize the data from a
different perspective!



This graphic shows the decile in statement

which Rs. 32 per day (Rs. 960 per
month) would fall in each state smaller problems, and then use


Test hypotheses. In the process,

main problem



Correlation does Beware of
not imply anecdotal
causality evidence
Beware of Don’t overfit
Outliers models

Data-driven inference is
fraught with pitfalls. Drawing Contradictory
Start with getting
a feel for the data
the wrong conclusion out of inferences from
data is easier than drawing the same data

right conclusion.
Don’t simply Don’t over-
throw everything complicate
into the mix Models can graphics
Graphics can
deceive misbehave


Outliers can
significantly distort
inferences


“Throwing
everything into the
mix” may not
always produce an
accurate model


It could lead to
multicollinearity,
for example

According to this regression, the tallest person should have an
extremely large right foot and a tiny left foot! That makes no sense!

Over-fitting can
lead to spurious
models

It helps to keep your models as simple as possible. A simple rule of thumb – a
good model is one that can be easily explained in simple English


Diving into model
fitting without first
understanding the
data can lead to
suboptimal results

People are prone to doing regressions without actually
looking at the data. Here, a simple linear regression gives a
reasonable fit (R^2 = 42%).
However, a simple scatter plot would suggest a clear Y=
1/X kind of relationship which the regression completely
misses out on

Contradictory
inferences can be
derived from the
same data


160

140

120

100

80

60

40

20
150
0
0 2 4 6 8 10 12 14 16 140

130

Choice of axes and 120

scales can have a 110

significant impact on 100

the message your 90

graphic conveys 80
0 2 4 6 8 10 12 14 16


Correlation does not imply causality


Mistaking correlation for
causality can lead to
hilarious conclusions


Readers get turned
off by overly
complicated graphics


Anecdotal/
insufficient data
can lead to false
conclusions


A model is just
that: a model. It is
not a substitute
for reality


The Art of Data Analysis will be further illustrated
by means of a detailed Case Study relevant to your
company/industry

For a half-day workshop on The Art of Data Analysis
(including a case study), contact Karthik Shashidhar at
karthik.shashidhar@gmail.com


The art of data analysis

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à The art of data analysis

Similaire à The art of data analysis (20)

Plus de Karthik Shashidhar

Plus de Karthik Shashidhar (17)

Dernier

Dernier (20)

The art of data analysis