This document discusses big data applications (BDAs) and their importance. [1] BDAs harness vast amounts of data through cloud computing and machine learning to deliver improved outcomes for users, such as more traffic, conversions or revenue. [2] They are characterized by their use of big data, network effects, action-orientation, and measurable results. [3] The document encourages embracing BDAs to enhance business and customer experience by unlocking the value in data through continuous improvement based on outcomes.
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Anatomy of a Big Data Application (BDA)
1. ANATOMY OF A BIG DATA APP
What are they? And why should you care?
2. WHAT WE’LL COVER TODAY
• Context
• Challenges
• Why should you
care?
• Big Data
Applications
(BDAs)
• How to leverage
BDAs
3. HOW BIG ARE WE TALKING?
• 1,000,000,000,000,000,000
bytes = “exabyte”
• Library of Congress
estimates
• 1,800 exabytes in 2011
• 7,900 exabytes by 2015
4. HOW BIG ARE WE TALKING?
“Data sets so large and complex that they become
awkward to work with using on-hand database
management tools” – Wikipedia
5. CHALLENGES
• Collection & storage
• 80s and 90s –
IBM, Oracle, SAP, SAS, EM
C
• Now this is cheap enough for
the masses
• Analysis
• ERPs, IBM, Google
• 90s, 2000s, now
• Acting on it!
• Now (for the leaders)
6. RISE OF THE MACHINES
Cheap, scalable computing power enabled the big data revolution
10. DIY APPROACH TO BIG DATA ANALYSIS
More datapoints and more
analysis yields:
• More analysis paralysis
• No guarantee analysis is correct
• Fails to get deep into data
• Slow
• Reconciling multiple sources is
nearly impossible
• Ex. Lead attribution
11. DIGGING DEEP
Q: How many numbers can a typical
human store in their “working memory”?
12.
13. BIG DATA APPS: WHY SHOULD YOU CARE?
• Improve your business and people
• Unlocks value of data (rather than dumping it in a
datawarehouse)
• BDAs are becoming ever-present part of life
• You can tie back to outcomes and actions!!!!
14. BIG DATA APPS DELIGHT (MORE) CUSTOMERS
• BDAs
• Anticipate their needs
• Provide excellent results
• Improve their day
18. BIG DATA APPLICATIONS
They harness the world’s data to deliver you a
better outcome –more traffic, action, conversions
and/or revenue.
19. KEY CHARACTERISTICS OF BIG DATA APPS
• Big Data
• Cloud
• Network effect
• Action
• Results
20. NETWORK EFFECT
• Value of BDA increases with # of end-users/customers
• More users mean mean more frequency of usage
• More users means better data
• Network Effects from Data:
• Data enables them to generate "news feed" events which in turn
drives more data
• Example: I update my profile once, everyone sees the change
• Data as a Source of Competitive Advantage
• Selling the milk not the store.
• The DBA itself is not the value. It’s the data (and what you can do with
it) that counts.
21. NETWORK EFFECT: LINKEDIN
• 3 degrees of separation
• Accurate
• Both parties win
• Timely
• Content
• Groups
• Answers
• Status
• Company
• Becoming a destination for recruiters AND professionals
22. ACTION
• No more analysis
paralysis
• Machine Learning
• Continuous cycle
• Real-time
23. ACTION: PANDORA
• Music Genome Project
• Give it a little, it
predicts a lot
• Responds to feedback
in real time
• How often do you
change the station?
• Knows you better than
you know yourself
24. RESULTS
The Old Way The BDA Way
1. Collect data 1. Create great content
2. Store data 2. Use BDA
3. Rationalize data 3. Get results
4. Analyze data 4. Repeat infinitely
5. Select data to act on
6. Act on data
7. Measure results
8. Rationalize with
revenue impact
25. RESULTS: BLOOMREACH
• Websites trust the
machine
• Trust rewarded with
results
• Clicks
• Conversions
• BloomReach is paid
on measurable
impact to revenues
26. HOW TO WORK WITH A BDA EFFECTIVELY
• Play your part
• Realize more
goes on outside
you walls than
inside
• Evaluate based
on outcomes
27. PLAY YOUR PART
• Do what you do best
• Creative
• Content
• Relationships
28. OUTSIDE YOUR WALLS
• Realize more goes on outside
your company than within
• Make peace with that
• For example
• The web knows more about
customer than you do
• And more about traffic
• And even more about songs
you'd like
31. IN SHORT…
BDAs improve the customer experience
and therefore, the company bottom line.
32. QUESTIONS TO ASK YOURSELF
• What does the web know that I
don’t about my customers and
prospects?
• Can I scale my current approach
to be BIG?
• What changes in my go to
market would yield the biggest
impact?
Definition from WikipediaIt’s a very common sense, intuitive way to describe it
The challenges of big data have been, in this chronological order: collection/storage, analysis and action. Most are still stuck on Analysis – vendors and practitioners
Big data not possible without cheap, scalable computer power
Q: Why don’t we have rooms full of telephone switchboard operators connecting calls any more? A: Slow, expensive and error proneQ: Index the web by hand?A: It’s impossibleQ: Why isn’t there a DJ picking my next song on Pandora?A: The Music Genome Project is better at it. I’ll bet you change the FM station in the car more often than you switch Pandora stations.
Most people working with big data – vendors and practitioners – are stuck on data analysisThis graph of temperatures in NYC over 6 days is pretty easy to analyze, but what about when we’re looking at a more complicated dataset – something like we’d see with Big Data…
This is a “Palustris Network”, a map of protein-protein interactions from a bioinformatics, mass spectrometer experiment. This is “big data” analysis. This is something humans alone simply cannot analyze.So if your big data strategy is hung up on the analysis step rather than acting on the real data you have at your disposal, remember that even the smartest scientists in the world have learned to leverage machine learning and big data apps to deal with things like mass spectrometer analysis.
Many are attacking the symptoms rather than the problem.
In analysis or action, how much can a person get their head around at one time?
On average, people can have between 3 and 4 numbers in their “working memory”If that’s the case, how could they possibly analyze and act on datasets with dozens, hundreds, thousands or more trends?
Make customers forget they're being delighted by a machine and make them think the company just read their mindNext time you’re in your car, notice how many times you change the stationThen next time you stream Pandora, notice how few times you change it.That’s the power a BDA has to delight customers.
Remember when your living room was full of alphabetized CD racks?Now with BDAs like Spotify, can find anything, subscribe to playlists others create and take it anywhere
Where do you go for traffic information now?
LinkedIn not only helps you stay in touch with your connections, you can do cool things like map how their relationships to you and one another.
BDAs don’t just repackage your data in a cool interface or offer productivity improvements in data scalability, they harness the world’s data to deliver you a better outcome –more traffic, action, conversions or revenue.
“We won’t spend more time talking about Big Data since we’ve covered it already. And I think by now everyone knows what the cloud is, so we’ll take that as a given as well. So let’s focus on the later 4 points – network effect, action, and results of BDAs”
Value increases with more usersNetwork effect of data – change once, seen everywhere Big Data is a competitive advantage for those platforms
How many people are you 3 degrees away from?Accurate and timelyLI is a destination
Don’t just analyze, ACT at scale and in real-time
Music Genome Project maps connections between songsThen you personalize it
The old way, you had to pay vendors up front because no one was sure what the results would beThe BDA way, the proof that the “machine” works IS the results themselves. So that’s what you pay for. Let me give you an example…
We have some advice for how best to work with a BDAPlay your partRealize more goes on outside your walls than insideEvaluate success based on outcomes
Focus on what people do best
The web and BDAs know more about your customers than you doBe at peace with that
Trust the BDAs are workingVerify based on the results
You know when each of these BDAs works because…air traffic control is a bdaspunk error linkedin find peoplebloomreach get revenue
Improve customer experienceThereby improving bottom line