Python Notes for mca i year students osmania university.docx
Module 1 the power of data
1. This programme has been funded with
support from the European Commission
Module 1:
The Power of Data
Why data skills matter
Complete this module in conjunction
with Module 1 notes
2. GENERATION DATA | USING DATA FOR PROFIT
Discover the importance of
data as a driver of business
growth and profitability.
a. Comprehend the role of big data
for business.
b. Understand and be able to apply
the key terms relating to data.
c. Know how big data can be turned
into smart data.
d. Be able to articulate a case for
data as a route to competitive
advantage
3. This programme has been funded with support from the European Commission. The author
is solely responsible for this publication (communication) and the Commission accepts no
responsibility for any use that may be made of the information contained therein.
1
3
4
From Big Data to Smart Data
A brief history of data
Benefits of data for business
2 What is Big Data
2.1 The 5 Vs of data
2.2 Understanding data
Overview
4. GENERATION DATA | USING DATA FOR PROFIT
2. WHAT IS BIG DATA?
• Data of a very large size, typically to the extent that its manipulation
and management present significant logistical challenges. Oxford English
Dictionary.
• A new attitude by businesses, non-profits, government agencies, and
individuals that recognises that combining data from multiple sources
could lead to better decisions. Gill Press in Forbes, 2014.
• High-volume, high-velocity and high-variety information assets that
demand cost-effective, innovative forms of information processing for
enhanced insight and decision-making. Gartner, 2014
5. GENERATION DATA | USING DATA FOR PROFIT
Before we start, let’s set
the scene by reading
The Model T Ford
(resource 1)
Limitations of the one size fits
all model in the era of digital
data.
6. GENERATION DATA | USING DATA FOR PROFIT
1. A BRIEF HISTORY OF DATA
1997
The first documented use of the
term “big data” appeared in a
1997 paper by scientists at
NASA. They described the
problem with the visualization of
large data sets which do not fit on
local disk memories as “the
problem of big data”.
https://www.forbes.com/sites/gilpress/2014/09/03/12-big-
data-definitions-whats-yours/#79b0a05813ae
7. GENERATION DATA | USING DATA FOR PROFIT
VOLUME
The magnitude
of data being
generated.
VELOCITY
The speed at
which data is
being
generated and
aggregated.
VARIETY
The types of
data available
to us.
VERACITY
The accuracy or
trustworthiness
of the data.
VALUE
The extent to
which data
generates
economically
valuable insights.
2.1 The 5 Vs of Big Data
9. 2. Velocity
GENERATION DATA | USING DATA FOR PROFIT
Big data technology allows
us to analyze the data
while it is generated,
without ever putting it
into databases.
For many businesses, the
speed of data creation is
even more important than
the volume.
10. GENERATION DATA | USING DATA FOR PROFIT
Real time insights
MIT Media Lab used location data
from phones to infer how many
people were in Macy’s parking lots
on Black Friday.
They could estimate the retailer’s
sales on that critical day even before
Macy’s itself had recorded those
sales.
Rapid insights provide an obvious
competitive advantage to analysts
and managers.
11. 3. Variety
GENERATION DATA | USING DATA FOR PROFIT
Before data was predominantly
structured – it was numerical
and highly organized. Today 80%
of the world’s data is
unstructured, including photos,
social media updates, readings
from sensors etc.
Today’s big data technology
allows structured and
unstructured data to be
harvested, stored, and used
simultaneously.
12. 4. Veracity
GENERATION DATA | USING DATA FOR PROFIT
Big data can be a crucial part of
business strategy and growth, but
high volumes of data are of no use
if the data is not accurate.
The most common problems are
data incompleteness and
inconsistencies. When these are
known and accounted for, data can
be cleaned or issues can be taken
into account.
13. 5. Value
GENERATION DATA | USING DATA FOR PROFIT
With so much data around, it is easy
to fall into the buzz trap and embark
on big data initiatives without a clear
understanding of the business value
it will bring.
Adapting data to suit your business
needs will enable you to unlock the
hidden potential within the
information you’ve collected, which
means you will get the most value
out of your data.
14. GENERATION DATA | USING DATA FOR PROFIT
MACHINE GENERATED DATA
Includes financial systems transactions, cloud applications, call detail records, medical
devices, GPS data and sensor data. It is valuable because it contains a definitive, real
time record of the behaviour of users and their transactions.
SOCIAL DATA
Information that social media users publicly share, including metadata such as the user's
location, language spoken, biographical data and/or shared links. It is valuable to
marketers looking for customer insights that may increase sales.
HUMAN GENERATED DATA
Exists as nonnumeric, unstructured data sets from online surveys, social media posts,
even phone calls. It is valuable because it describes a person’s interests the social
aspects of human interaction, but it can be very difficult to analyse.
META DATA
Data that provides information about other data. For example, information about the
title, subject, author and size of a document constitute metadata about that document.
2.2 UNDERSTANDING DATA
15. GENERATION DATA | USING DATA FOR PROFIT
STRUCTURED
High degree of
organization, such as
relational database.
Examples: Dates, phone
numbers, customer
names, transaction
information...
UNSTRUCTURED
Information that is
difficult to organize using
traditional mechanisms.
Examples: Images, social
media...
SEMI STRUCTURED
Information not in a
database but that does
have some organizational
properties that make it
easier to analyze
Examples: Websites, XML,
e-mails...
2.2 UNDERSTANDING DATA
17. GENERATION DATA | USING DATA FOR PROFIT
3. FROM BIG DATA TO SMART DATA
Smart data describes data that has valid, well-defined, meaningful
information with an added layer of intelligence or interpretation. This
enables decisions to be made more quickly, and even in some
cases, without human intervention or processing from a centralized
system.
DATA
Monthly
sales report
Sales report synthesisng peaks and
troughs according to day/time, creating
recommendations on how to adjust
staffing.
SMART DATA
18. GENERATION DATA | USING DATA FOR PROFIT
3. FROM BIG DATA TO SMART DATA
1 3 42
Can we
sense
this
data?
Can we
generate a
sensible
result
from it?
Can we
use it for a
better
service?
Can we
convert it
into a
profit?
Source: Sen, Ozturk, Vayvay 2016.
19. GENERATION DATA | USING DATA FOR PROFIT
Is your
business smart?
Datavores Data Builders Data Mixers
16% 22% 31%
Make strong
use of data
and analysis
for decision–
making.
Use big datasets
requiring
dedicated servers
for parallel
processing.
Combine data from a
variety of sources.
Dataphobes
30%
Work with small
datasets and few
data sources, and do
not use data or analysis
to make decisions.
In 2015 NESTA investigated
the data practices of 404
medium and large UK
businesses in six sectors.
They found four main types of
companies.
Econometric analysis reveals
that Datavores and Data
Builders are over 10% more
productive than the
Dataphobes after controlling
for other factors.
20. GENERATION DATA | USING DATA FOR PROFIT
Don’t rely on HiPPOs
Many companies still rely on
HiPPOs, the Highest-Paid
Person’s Opinion.
Throughout the business world
today, people rely too much on
experience and intuition and
not enough on data.
Source: Harvard Business Review, October 2012
21. GENERATION DATA | USING DATA FOR PROFIT
4. BENEFITS OF DATA FOR BUSINESS
The importance of big data does not revolve around how
much data a company has but how a company utilizes the
collected data.
Being able to analyze and predict market and customer
behavior with Big Data is a new paradigm shift for SMEs.
When it is implemented correctly, it can yield increased
flexibility, productivity, responsiveness, anticipation and
ability to meet customer need through capturing blind spots
and making better decisions.
22. GENERATION DATA | USING DATA FOR PROFIT
4. BENEFITS OF DATA FOR BUSINESS
Product Innovation
• Market insights
• New products
• Real time feedback
• Maximise profit
Process improvements
• Time reductions
• Cost savings
• Improved Quality
• Manage online reputation
• Customer satifaction
23. GENERATION DATA | USING DATA FOR PROFIT
FOURS WAYS
BUSINESSES USE
DATA TO GROW
Reduce Time To Market
Introducing new products or services involves many life cycle stages. A
large pharmaceutical company reduced the time it takes to run clinical trial
simulations by 98% by moving its work into a dedicated data services
environment in the cloud.
Before moving, scientists were using a shared internal environment where
it took 60 hours to run hundreds of jobs. Now that each scientist has a
dedicated environment, 2,000 jobs can be processed in 1.2 hours without
causing an impact to other members of the team.
Improve Financial Performance
Corporate finance departments are moving beyond periodic reporting
using big data to reduce risks and costs, identify opportunities, and
improve the accuracy of forecasts. As it’s easier to retain an existing client
than acquire a new one, a common application is to analyse clients’
payment history to identify those most at risk of ceasing to use a service,
before they do so.
By predicting the risk, the company can boost the customer service to this
client, replacing emails with calls, for example, to retain their custom.
24. GENERATION DATA | USING DATA FOR PROFIT
FOUR WAYS
BUSINESSES USE
DATA TO GROW
Minimize Equipment And Asset Failures
Sensors can be embedded into just about everything, enabling
companies to use the data to determine when maintenance is
required. Ideally, when an issue has arisen, companies want to
understand the issue, what caused it, and how it can be resolved,
preferably before a maintenance professional or crew is dispatched.
Companies that collect more information can enable more proactive
maintenance, save valuable staff hours and improve customer
satisfaction.
Optimize staffing and supply chain for weather
The weather has a huge economic impact at macro level but also affects
businesses of all sizes because of its unpredictable impact on the
demand of certain products and services.
Companies that relate data points on weather conditions to sales and
customer service data are better placed to adjust staffing and supply
chain strategies ahead of time.
25. GENERATION DATA | USING DATA FOR PROFIT
GLOSSARY
UNITS OF DATA
Byte 100 1
Kilobye KB 103 1,000
Megabyte MB 106 1,000,000
Gigabyte GB 109 1,000,000,000
Terabyte TB 1012 1,000,000,000,000
Petrabyte TB 1015 1,000,000,000,000,000
Exabyte EB 1018 1,000,000,000,000,000,000
DATA
Information, especially facts or numbers, collected,
examined and used to help decision-making.
DATASET
A collection of data, typically in tabular form.
DATABASE
A digital collection of data and the structure around
which the data is organized.
DISTRIBUTED FILE SYSTEM
Data storage system that stores large volumes of data
across multiple storage devices to decrease the cost
and complexity of storing large amounts of data.
26. GENERATION DATA | USING DATA FOR PROFIT
ACTIVITY 1
How much
data do you
generate?
Imagine your typical day from when you wake up from when you
go to bed. You might check your smart phone, log on to your
computer and several app, make phone calls, respond to emails,
purchase items on line, post content on social media……
Write a list of all the types of data that you generate, what type of
data is it, who can see it and what value it might be to them.
Activity Data
generated
Type of data Who sees it Business value
Added item to
cloud-based
personal
calendar
Location of my
appointment
Semi-
structured
Google Can target ads
to my location
27. GENERATION DATA | USING DATA FOR PROFIT
ACTIVITY 2
True or false?
“The evidence is clear:
Data-driven decisions tend to be better
decisions. Leaders will either embrace this
fact or be replaced by others who do.”
Do you believe this statement is true or false? Are there times
when a HiPPO is better placed to make a decision?
28. GENERATION DATA | USING DATA FOR PROFIT
ACTIVITY 3
What does it
mean to be
data driven?
Select an industry from above and carry out investigation to find
two ways that businesses in that sector have used data to drive
their business growth.
1. What did they do?
2. How did using data modify their internal processes or product
innovation?
3. What were the tangible benefits to profitability?
Add icons to illustrate each industry:
Financial Services
Retail
Food and beverages
Manufacturing
Healthcare
Tourism
Transportation