Slides from my Microsoft HDInsight session at the dotnet Cologne 2013

1. Sascha Dittmann
Blog: http://www.sascha-dittmann.de
Twitter: @SaschaDittmann
Microsoft HDInsight für .NET Entwickler
Big Data Analysen mit JavaScript und C#

2. Large Hadron Collider (CERN Schweiz)
http://public.web.cern.ch/public/en/lhc/Computing-en.html
Der LHC Teilchenbeschleuniger
produziert 15 PB Messdaten pro Jahr*

3. Woher kommt Big Data
70% of U.S.
smartphone owners
regularly shop online
via their devices.
44% of users
(350M people)
access Facebook via
mobile devices.
50% of
millennials use
mobile devices to
research products.
60%of U.S.
mobile data will be
audio and video
streaming by 2014.
Mobility
2/3of the world's
mobile data traffic will
be video by 2016.
33%of BI will
be consumed via
handheld devices
by 2013.
Gaming consoles are
now used an average of
1.5 hrs/wk
to connect to the
Internet.
80%growth of
unstructured data is
predicted over the
next five years.
1.8 zettabytes
of digital data were
in use
worldwide in
2011, up 30%
from 2010.
1 in 4
Facebook users
add their location
to posts
(2B/month).
500M Tweets
are hosted on
Twitter each day.
38% of people
recommend a brand
they “like” or follow
on a social network.
100M
Facebook
“likes” per day.
Brands get
Big
Data
Social
Mobility Cloud

4. Big Data Szenarien
Web app
optimization
Smart meter
monitoring
Equipment
monitoring
Advertising
analysis
Life sciences
research
Fraud
detection
Healthcare
outcomes
Weather
forecasting
Natural resource
exploration
Social network
analysis
Churn
analysis
Traffic flow
optimization
IT infrastructure
optimization
Legal
discovery

5. Big Data ist sexy
http://hbr.org/

6. Apache Hadoop Ecosystem
MapReduce (Job Scheduling/Execution System)
HDFS
(Hadoop Distributed File System)
HBase (Column DB)
Pig (Data
Flow)
Hive
(Warehouse
and Data
Access)
Oozie
(Workflow)
Sqoop
Traditional BI Tools
HBase / Cassandra
(Columnar NoSQL Databases)
Avro(Serialization)
Zookeeper(Coordination)
Apache
Mahout
Cascading
(programming
model)
Hadoop = MapReduce + HDFS
Flume

7. Microsoft HDInsight
MapReduce (Job Scheduling/Execution System)
HDFS
(Hadoop Distributed File System)
HBase (Column DB)
Pig
(Data
Flow)
Hive
(Warehous
e and Data
Access)
Oozie
(Workflow)
Sqoop
Traditional BI Tools
HBase / Cassandra
(Columnar NoSQL Databases)
Avro(Serialization)
Zookeeper(Coordination)
Apache
Mahout
Cascading
(programmin
g model)
Hadoop = MapReduce + HDFS
Flume
Windows
SystemCenter
ActiveDirectory
Visual Studio

8. Hadoop Distributed File System (HDFS)
Bootvorgang
Ausfallsicherheit
Benutzeranfrage

9. Hadoop Distributed File System (HDFS)
Bootvorgang
Ausfallsicherheit
Benutzeranfrage

10. Bootvorgang
Ausfallsicherheit
Benutzeranfrage
Hadoop Distributed File System (HDFS)

11. Hadoop Distributed File System (HDFS)
 Portable Operating System Interface (POSIX)
 Replikation auf mehrere Datenknoten
js> #ls /user/Sascha/input/ncdc
Found 9 items
drwxr-xr-x - Sascha supergroup 0 2013-04-24 13:09 /user/Sascha/input/ncdc/all
drwxr-xr-x - Sascha supergroup 0 2013-04-24 13:01 /user/Sascha/input/ncdc/all2
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/metadata
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/micro
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/micro-tab
-rw-r--r-- 3 Sascha supergroup 529 2013-04-23 13:06 /user/Sascha/input/ncdc/sample.txt
-rw-r--r-- 3 Sascha supergroup 168 2013-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

12. HDInsight Dashboard Demo

13. Map/Reduce am Beispiel von Messdaten
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
Jahr Lufttemperatur

14. Map/Reduce am Beispiel von Messdaten
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
Messqualität

15. Map/Reduce
Map
Sort
Shuffle
DataNode
Map
Sort
Shuffle
DataNode
Map
Sort
Shuffle
DataNode
Reduce
0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1950,22
1950,55
1952,-11
1950,33
1949,0
1950,[22,33,55]
1952,-11
1949,0
1950,55
1952,-11

16. Map/Reduce mit Combine Methode
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Reduce
0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1950,22
1950,55
1952,-11
1950,33
1949,0
1950,55
1952,-11
1950,33
1949,0
1950,[33,55]
1952,-11
1949,0
1950,55
1952,-11

17. Map/Reduce am Beispiel von Messdaten

18. Wörter zählen mit JavaScript (Map)

19. Wörter zählen mit JavaScript (Reduce)

20. Map/Reduce mit JavaScript

21. Verfeinern mit Pig Latin
pig
.from("/user/Sascha/input/texte")
.mapReduce("/user/…/WordCount.js"
, "Woerter, Anzahl:long")
.orderBy("Anzahl DESC")
.take(15)
.to("/user/Sascha/output/Top15Woerter")

22. Pig Latin

23. Wörter zählen mit C# (Map - Classic)

24. Wörter zählen mit C# (Reduce - Classic)

25. Map/Reduce mit C#

26. .NET Job Submission Framework (Map)

27. .NET Job Submission Framework (Reduce)

28. Externe Hive-Tabelle erzeugen
CREATE EXTERNAL TABLE twitter_raw
(
tweet_json STRING
)
COMMENT 'Twitter Sample Data'
ROW FORMAT DELIMITED LINES TERMINATED
BY '10'
STORED AS TEXTFILE
LOCATION '/example/twitterdata';

29. Twitter JSON
{
"possibly_sensitive_editable":true,
"place":null,
"text":"Pre - #ConvCloud chat insights. " #Cloud Security, are we missing the point?" from
@christianve http://t.co/Smo0CPvb #HP #cloudsource”,
"id_str":"223418953114984448”,
"favorited":false,
"possibly_sensitive":false,
"created_at":"Thu Jul 12 14:10:04 +0000 2012",
"retweeted":false,
"retweet_count":0,
"user":{
"is_translator":false,
"profile_use_background_image":true,
"profile_image_url_https":"https://si0.twimg.com/profile_images/640456324/
Paul_Calento_normal.jpg",
"id_str":"103006513",
"profile_text_color":"333333",
"statuses_count":5984,
"following":null,
"followers_count":744,
"default_profile_image":false,
"profile_link_color":"FF3300",
}, …..
}

30. JSON in Hive interpretieren
FROM twitter_raw
INSERT OVERRIDE TABLE twitter_temp
SELECT get_json_object(tweet_json, '$.created_at'),
substr(get_json_object(tweet_json, '$.created_at'),9,2),
substr(get_json_object(tweet_json, '$.created_at'),12,8),
get_json_object(tweet_json, '$.in_reply_to_user_id_str'),
get_json_object(tweet_json, '$.text'),
get_json_object(tweet_json, '$.contributors'),
get_json_object(tweet_json, '$.retweeted'),
get_json_object(tweet_json, '$.truncated'),
get_json_object(tweet_json, '$.favorited'),
cast(get_json_object(tweet_json, '$.retweet_count') as int),
/* … */
get_json_object(tweet_json, '$.user.profile_image_url_https'),
cast(get_json_object(tweet_json, '$.user.followers_count') as int),
get_json_object(tweet_json, '$.user.location'),
get_json_object(tweet_json, '$.user.time_zone'),
get_json_object(tweet_json, '$.user.created_at');

31. Hive

32. RDBMS vs. Hadoop
RDBMS Hadoop
Volumen Gigabyte Petabyte
Verarbeitung Ad-Hoc und batch Batch
Updates Viele Lese- und
Schreibzugriffe
Einmal schreiben,
Viele Lesezugriffe
Schema Statisches Schema Dynamisches Schema
Datenintegrität Hoch Niedrig
Skalierverhalten Nicht-Linear Linear

33. Polybase / SQL Server PDW

34. Fragen
? ?
?
?
?