Contenu connexe
Similaire à Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson, Managing Director, InfoReady (20)
Plus de IT Network marcus evans (20)
Australian CIO Summit 2012: Big Data, New Physics, and Geospatial Super-Food by Tristan Sternson, Managing Director, InfoReady
- 1. Big Data,
New Physics, and
Geospatial Super-Food
Tristan Sternson, InfoReady Managing Director
1 © 2012 Infoready Pty Ltd
- 2. My Background – Tristan Sternson
Past 12 years focussed purely on IM / BI Solutions
Started InfoReady in 2008
Prior Roles – Accenture Data Management & Architecture / IM
Lead, PWC Consulting / IBM
Personally designed and deployed and led many large IM and
DW application in Australia and UK
Thought leader in Information Management in Australia and
APAC
Early adopter Data Governance, Big Data, Industry Data
Models, Appliance DW solutions
2 © 2012 Infoready Pty Ltd
- 3. Who is InfoReady?
Pure-Play Information Management and Business Intelligence
Consulting firm
Team InfoReady career IM
and BI Experts
• One of the fastest growing
consulting firms in
Australia.
• IM Focused Tier One
Consulting capability.
• Focus - people, process and
technology
• Assisting companies turn
valuable information into
actionable intelligence.
• Strategy, Architecture,
Solution Design & Delivery
3 © 2012 Infoready Pty Ltd
- 4. Big Data Definition
Datasets that grow so large that they become difficult to work with,
including; capture, storage, search, sharing, analytics, and visualization.
Benefits of working with larger and larger datasets allowing
analysts to "spot business trends, prevent diseases, combat crime.”
We haven’t seen anything yet, as more devices come online, eg;
mobile, airborn, logs, cameras, microphones etc…
4 Wikipedia - 2012
© 2012 Infoready Pty Ltd
- 6. Big Data – Why the hype?
By 2015, nearly 3B people will be online, pushing the data
created and shared to nearly 8 zettabytes.
30 billion pieces of content were added to Facebook this past
month by 600M plus users.
More than 2B videos were watched on YouTube … yesterday.
In the US mobile phone users between the ages of 18 and 24
send an incredible 110 text messages per day.
32B searches were performed last month … on Twitter.
Worldwide IP traffic will quadruple by 2015.
6 © 2012 Infoready Pty Ltd
- 7. Business Value
Business leaders frequently make
1 in 3 decisions based on information they
don’t trust, or don’t have
1 in 2 access to leaders say theythey ’need to
Business
the information
don’t have
do their jobs
of CIOs cited “Business intelligence &
83% Analytics” as part of their visionary
plans to enhance competitiveness
of CEOs recognise they need to better
60% understand information more rapidly
in order to make swift decisions
7 7 © 2012 Infoready Pty Ltd
- 9. What the Industry Analysts say
Gartner predicts Big Data to be
one of the top-10 strategic initiatives
for 2012
9 © 2012 Infoready Pty Ltd
- 10. What the Industry Analysts say
Key take-aways from Analyst perspectives Gartner TDWI
Data will grow exponentially
Fusion of structured and unstructured data
The connection between big data and advanced
analytics will get even stronger
Future users will not be able to put all useful
information into a single data warehouse
10 © 2012 Infoready Pty Ltd
- 12. Trend: Organizations Are Getting Dumber
Available
Observation
Computing Power Growth
Space
Context
Enterprise
Amnesia
Sensemaking
Algorithms
Time
12 © 2012 Infoready Pty Ltd
- 13. Trend: Organizations Are Getting Dumber
Available
Observation
Computing Power Growth
Space
WHY?
Context
Sensemaking
Algorithms
Time
13 © 2012 Infoready Pty Ltd
- 14. Algorithms at Dead End.
You Can’t
Squeeze Knowledge
Out of a Pixel.
14 © 2012 Infoready Pty Ltd
- 15. No Context
scrila34@msn.com
15 © 2012 Infoready Pty Ltd
- 16. Context, definition
Better understanding
something by taking into
account the things around it.
16 © 2012 Infoready Pty Ltd
- 17. Information in Context … and Accumulating
scrila34@msn.com
Job
Applicant Top 200
Customer
Criminal
Investigation
Identity
Thief
17 © 2012 Infoready Pty Ltd
- 18. The Puzzle Metaphor
Imagine an ever-growing pile of puzzle pieces of varying sizes,
shapes and colors
What it represents is unknown – there is no picture on hand
Is it one puzzle, 15 puzzles, or 1,500 different puzzles?
Some pieces are duplicates, missing, incomplete, low quality, or
have been misinterpreted
Some pieces may even be professionally fabricated lies
Until you take the pieces to the table and attempt assembly,
you don’t know what you are dealing with
18 © 2012 Infoready Pty Ltd
- 19. Puzzling
1000 pieces 100 pieces
100% 10%
(duplicates)
12 pieces 12 pieces
100% 100%
(pure noise)
66 pieces
66%
19 © 2012 Infoready Pty Ltd
- 20. 20 © 2012 Infoready Pty Ltd
- 21. 21 © 2012 Infoready Pty Ltd
- 25. 10 Mins – Completed Dora Puzzels
25 © 2012 Infoready Pty Ltd
- 28. Incremental Context – Incremental Discovery
10:00am START
1min “I can see Dora”
1min “How many puzzles are there?”
8min “Are there 1000 pieces and 3 or 4 puzzles?”
10min 2 x Dora puzzles complete
12min “I have blue sky and an animal”
18mins “The other puzzle is more colourful – maybe a red
motorbike”
23min “we’ve found Jenny Sanders – can I search google on my
iPhone for the picture?”
35min “How can we have 2 pieces the same?”
28 © 2012 Infoready Pty Ltd
- 31. Quickly we find meaning (90mins)
66 pieces
of
1190 pieces
only 5.5%
31 © 2012 Infoready Pty Ltd
- 32. Wow 1%
11 pieces
of
1190 pieces
only 1%
32 © 2012 Infoready Pty Ltd
- 37. Incremental Context – Incremental Discovery
55min “Second puzzle is definitely a motorbike – I can see a
wheel and seat”
65min Motorcycle coming together very quickly
70min “It’s definitely a koala”
75min “The koala has a baby”
83min “The middle piece of the bike is missing – do I really
need it, I know what it is”
88min “These are both Australian puzzles”
114min One of the kids starts isolating pieces that are causing her
“noise”
130min 7 chunks emerge from 7 piles of SORTED pieces
165min Pieces beginning to come together quite quickly and
picture starts to really emerge
37 © 2012 Infoready Pty Ltd
- 38. How Context Accumulates
With each new observation … one of three assertions are made:
1) Un-associated; 2) placed near like neighbors; or 3) connected
Must favor the false negative
New observations sometimes reverse earlier assertions
Some observations produce novel discovery
As the working space expands, computational effort increases
Given sufficient observations, there can come a tipping point
Thereafter, confidence improves while computational effort
decreases!
38 © 2012 Infoready Pty Ltd
- 40. Counting Is Difficult
Mark R Smith
(614) 13-123-123
DL: 00001234
Mark Smith
6/12/1978
0413123123
File 2
File 1
40 © 2012 Infoready Pty Ltd
- 41. The Rise and Fall of a Population
Unique Identities
True Population
Observations
41 © 2012 Infoready Pty Ltd
- 42. Data Triangulation
New Record
Mark R Smith
(614) 13-123-123
DL: 00001234
Mark Smith
6/12/1978
0413123123
Mark Randy Smith
0413123123
DL: 00001234
File 2
File 1
42 © 2012 Infoready Pty Ltd
- 43. Big Data [in context]. New Physics.
More data: better the predictions
– Lower false positives
– Lower false negatives
More data: bad data good
– Suddenly glad your data is not perfect
More data: less compute
43 © 2012 Infoready Pty Ltd
- 44. Big Data
Pile of ____ In Context
44 © 2012 Infoready Pty Ltd
- 45. One Form of Context: “Expert Counting”
Is it 5 people each with 1 account … or is it 1
person with 5 accounts?
Is it 20 cases of H1N1 in 20 cities … or one
case reported 20 times?
If one cannot count … one cannot estimate
vector or velocity (direction and speed).
Without vector and velocity … prediction is
nearly impossible.
45 © 2012 Infoready Pty Ltd
- 46. Expert Counting: Degrees of Difficulty
Deceit
Bob Jones Ken Wells
123455 550119
Incompatible
Features
Bob Jones bjones@hotmail
Fuzzy 123455
Bob Jones Robert T Jonnes
Exactly 123455 000123455
Same
Bob Jones Bob Jones
123455 123455
46 © 2012 Infoready Pty Ltd
- 47. Key Features Enable Expert Counting
People Cars Router
Name Make Device ID
Address Model Make
Date of Birth Year Model
Phone License Plate No. Firmware Vers.
Passport VIN Asset ID
Nationality Owner Etc.
Biometric Etc.
Etc.
47 © 2012 Infoready Pty Ltd
- 48. Consider Lying Identical Twins
PASSPORT #123
Sue PASSPORT #123
3/3/84 Sue
Uberstan 3/3/84
Exp 2011 Uberstan
Exp 2011
“Same
person –
trust me.”
Fingerprint
DNA
Most Trusted
Authority
48 © 2012 Infoready Pty Ltd
- 49. The same thing cannot be in
two places … at the same
time.
Two different things cannot
occupy the same space … at
the same time.
49 © 2012 Infoready Pty Ltd
- 50. Space Time Enables Absolute Disambiguation
People Cars Router
Name Make Device ID
When When When
Address Model Make
Where Where Where
Date of Birth Year Model
Phone License Plate No. Firmware Vers.
Passport VIN Asset ID
Nationality Owner Etc.
Biometric Etc.
Etc.
50 © 2012 Infoready Pty Ltd
- 51. “Life Arcs” Are Also Telling
Bill Smith Bill Smith
13/4/67 13/4/67
Melbourne, Victoria Brisbane, Queensland
Address History Address History
Melbourne, Vic 2008-2008 Carina, QLD 2005-2009
St Kilda, Vic 2005-2008 Brisbane, QLD 2005-2005
Hampton, Vic 1996-2005 Bondi, NSW 1990-2005
Brighton, Vic 1984-1996 Carina, QLD 1982-1990
51 © 2012 Infoready Pty Ltd
- 52. OMG
52 © 2012 Infoready Pty Ltd
- 53. Space-Time-Travel
Cell phones are generating a staggering amount of
geo-locational data – 600B transactions per day
being created in the US alone
This data is being “de-identified” and shared with
third parties – in volume and in real-time
Your movement quickly reveals where you spend
your time (e.g., evenings vs. working hours)
Re-identification (figuring out who is who) is
somewhat trivial
53 © 2012 Infoready Pty Ltd
- 54. Powerful Predictions
Prediction with 87% certainty where you will be
next Thursday at 5:35pm
Names of the top 10 people you co-locate with,
not at home and not at work
Intelligence service preempts the next mass
protest in real-time
Robbery of a convenience store is about to
happen at 10:42pm
54 © 2012 Infoready Pty Ltd
- 55. Consequences
Space-time-travel data is the ultimate
biometric
It will enable enormous opportunity
It will unravel one’s secrets
It will challenge existing notions of privacy
And, it’s here now and more to come
55 © 2012 Infoready Pty Ltd
- 57. The Greater the Context, the Greater the Value
Data
in Context
Value of Data
Pile of Data
(Big) Records Managed (Ludicrous Big)
57 © 2012 Infoready Pty Ltd
- 58. Time Is Of The Essence
The better the
predictions … the
Batch faster they will be
wanted.
Day
Willingness to Wait
“Why did we have
to wait until the
Hour end of the day for
the smart answer?”
200ms Real-Time
(Iffy) Relevance (Totally)
58 © 2012 Infoready Pty Ltd
- 60. Sense and Respond
Observation
Space
New
Observations
What you know
60 © 2012 Infoready Pty Ltd
- 61. Sense and Respond
Observation
Space
Data Finds
Data
Relevance
Finds the Sensor
(200ms)
?
Decide
61 © 2012 Infoready Pty Ltd
- 62. Sense and Respond Explore and Reflect
Observation
Space Deep
Reflection
Curated
Data
Data Finds Pattern
Data Discovery
Directed
Attention
Relevance
Finds the Sensor
(200ms)
?
Relevance
Decide Find You
62 © 2012 Infoready Pty Ltd
- 63. Sense and Respond Explore and Reflect
Observation
Space Deep
Reflection
Curated
Data
Data Finds Pattern
Data Discovery
Directed
Attention
Relevance NEW
Finds the Sensor
(200ms)
? INTERESTS
Decide
63 © 2012 Infoready Pty Ltd
- 64. Sense and Respond Explore and Reflect
Observation
Space Deep
Reflection
Curated
Data
Data Finds Pattern
Data Discovery
Directed
Attention
Relevance NEW
Finds the Sensor
(200ms)
? INTERESTS
Decide
64
Report and Manage
© 2012 Infoready Pty Ltd
- 66. The most competitive organizations
are going to make sense of what they are observing
fast enough to do something about it
while they are observing it.
66 © 2012 Infoready Pty Ltd
- 67. Wish This On The Enemy
Available
Observation
Computing Power Growth
Space
Context
Enterprise
Amnesia
Sensemaking
Algorithms
Time
67 © 2012 Infoready Pty Ltd
- 68. The Way Forward: Enterprise Intelligence
Available
Observation
Computing Power Growth
Space
Context
Sensemaking
Algorithms
Time
68 © 2012 Infoready Pty Ltd
- 69. Questions?
Email: tristan.sternson@infoready.com.au
Twitter: http://www.twitter.com/tsternson
Blog: www.infoready.com.au
LinkedIn: http://www.linkedin.com/in/tristansternson
69 © 2012 Infoready Pty Ltd