My talk on Big Data from Dallas Day of .NET 2014

2. https://github.com/glennblock
https://twitter.com/gblock
“I should be
tweeting"

3. 3
Make machine data accessible, usable
and valuable to everyone.

4. Platform for Machine Data
Any Machine Data
HA Indexes
and Storage
Search and
Investigation
Proactive
Monitoring
Operational
Visibility
Real-time
Business
Insights
Commodity
Servers
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID

5. DATA

6. 15,000 BC – Pictures
Lascaux, France

7. 6000 BC – Symbols

8. 3,500 BC – Language

9. 1,275 BC – Papyrus

10. 1st - 13th Century - Codex

11. 13th Century – Movable type

12. 15th Century – Printing press

13. 19th to 20th century
Babbage Analytical engine

14. 1936 – Turing machine

15. 1945 – ENIAC

16. 1947 – The first bug

17. 1977 - Arpanet

18. 1990s Internet

19. Phones and Tablets

20. RFID

21. Cloud

22. Services

23. New consumer devices
23

25. 90 percent of all the data in the
world has been generated over
the last two years
source: sciencedaily.com

26. Every day 2.5 quintillion bytes
of data is generated
1 quintillion = 1 + 18 zeros!
57.5 billion 32 GB iPads
source: storagenewsletter.com

27. 2.7 zettabytes exist in the digital
universe
1 zettabyte = 1 + 21 zeros!
42zb = All human speech digitized
source: highscalability.com

28. How big is big?

29. That’s A LOT of data!

30. How do you harness it?

31. This is what big data
is really about.

32. Asking questions and
getting answers

33. Massive amounts of data.
Machine generated
VOLUME

34. Data is coming from a multitude of sources
Mix of structured and un-structured
(JSON, XML, CSV, Plain Text)
Need a way to store it and and query it
VARIETY

35. VARIETY
Log files
Activity Feeds
Emails
Device Streams
Audio Files
Videos

36. Data arrives at many different frequencies
Need to be able to process real time.
VELOCITY

37. Not all data that is stored is useful.
Need to identify the useful data
Need to wade through all the noise
VERACITY

38. SOLUTIONS

39. Map/Reduce
function map(String name, String document):
// name: document name
// document: document contents
for each word w in document:
emit (w, 1)
function reduce(String word, Iterator partialCounts):
// word: a word
// partialCounts: a list of aggregated partial counts
sum = 0
for each pc in partialCounts:
sum += ParseInt(pc)
emit (word, sum)

40. Hi scale and availability databases

41. Distributed processing
of large datasets

42. Data Visualization and analysis

43. End to end tools

44. More information
www.mongodb.org
www.memsql.com
cassandra.apache.org
hadoop.apache.org
www.tableausoftware.com
www.elasticsearch.org
splunk.com

45. @gblock http://github.com/glennblock
http://www.flickr.com/photos/11812960@N04/4050576435