.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

•

0 j'aime•484 vues

Wir leben in einem Datenzeitalter! Nach Schätzungen in 2006, betrug das weltweite "Datenuniversum" ca. 0.18 Zettabytes (1 ZB => 10 hoch 21 Bytes bzw. 1 Mrd. Terabytes). In 2011 hatte sich dieses Volumen sogar verzehnfacht (1,8 Zettabytes). Somit wird in vielen Anwendungsszenarien das Thema Big Data und Big Processing immer wichtiger. Klassische relationale Datenbanksysteme, sowie Statistik und Visualisierungstools, sind oft nicht in der Lage, derart große Datenmengen zu verarbeiten. Für Big Data kommt daher eine neue Art von Software zum Einsatz, die massiv parallel auf bis zu hunderten oder tausenden von Prozessoren bzw. Servern arbeitet.

Technologie

Apache Hadoop-based Services für Windows Azure

Sascha Dittmann
Software Developer / Solution Architect
Twitter: @SaschaDittmann
Blog: http://www.sascha-dittmann.de

Hadoop Distributed File System

Cluster Startvorgang

Hadoop Distributed File System
Ausfall des Namenodes (Failover)

Hadoop Distributed File System
Benuteranfrage

①

② ② ②

Hadoop Distributed File System
 Portable Operating System Interface (POSIX)
 Replikation auf mehrere Datenknoten
js> #ls input/ncdc
Found 9 items
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/_distcp_logs_g0dedn
drwxr-xr-x - Sascha supergroup 0 2012-04-24 12:04 /user/Sascha/input/ncdc/_distcp_logs_ofj0u6
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:09 /user/Sascha/input/ncdc/all
drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/all2
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/metadata
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro
drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro-tab
-rw-r--r-- 3 Sascha supergroup 529 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt
-rw-r--r-- 3 Sascha supergroup 168 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

Map/Reduce
DataNode DataNode DataNode 0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300

1949,0
1952,-11
1950,22
Map Map Map 1950,55
1950,33

Sort Sort Sort 1949,0
1950,[22,33,55]
Shuffle Shuffle Shuffle 1952,-11

Reduce
1949,0
1950,55
1952,-11

RDBMS vs. Map/Reduce
RDBMS Map/Reduce
Datenmenge Gigabytes Petabytes
Zugriff Interaktiv und Batch Batch
Lese- / Schreibzugriffe Viele Lese- und Einmaliges Schreiben
Schreibzugriffe Viele Lesezugriffe
Datenstruktur Statisches Schema Dynamisches Schema
Datenintegrität Hoch Niedrig
Skalierverhalten Nicht-Linear Linear

Demos
 Hadoop Dashboard
 Interactive Console
 Remote Desktop
 Nutzung des WA Storage
 Map/Reduce via JavaScript
 C# Streaming
 Power Pivot

Cloud Bloggers

Die Blogs der deutschen Cloud Computing-Community

Link: http://cloudbloggers.de

Contenu connexe

Plus de Sascha Dittmann

Wer heutzutage in Big Data Projekten involviert ist, kommt an Hadoop-Tools, wie beispielsweise Hive, Spark oder Storm kaum noch vorbei. Doch wie sieht es mit den Microsoft Bordmittel, genauer gesagt Visual Studio, C# und SQL, aus? Dieser Frage geht Sascha Dittmann in dieser Session nach. Er zeigt anhand von Codebeispielen, und den Azure Data Lake Diensten, wie eine Umsetzung auch mit reinen Microsoft Tools möglich ist.

C# + SQL = Big Data

Sascha Dittmann

Die meisten Softwareprojekte fangen mit einer einfachen relationalen Datenbank an. Doch was soll man machen, wenn die Anforderungen an die Datenbank wachsen? Was, wenn die Datenbankgröße und die Anzahl der gleichzeitigen Abfragen stetig wächst? Was, wenn aus einzelnen Datenbanken ein Data Warehouse entsteht? Sascha Dittmann zeigt in dieser Session wie solche und ähnliche Szenarien mit Microsoft Azure umsetzbar sind. Er stellt dabei Technologien wie Azure SQL Database, Azure Elastic Pools und Azure SQL Data Warehouse im Detail vor und gibt Beispiele, sowie Tipps & Tricks aus der Praxis.

Hochskalierbare, relationale Datenbanken in Microsoft Azure

Sascha Dittmann

Spark auf Hadoop ist hochskalierbar. Cloud Computing ist hochskalierbar. R, die erweiterbare Open Source Data Science Software, eher nicht. Aber was passiert, wenn wir Spark auf Hadoop, Cloud Computing und den Microsoft R Server zu einer skalierbaren Data Science-Plattform zusammenfügen? Stellen Sie sich vor wie es sein könnte, wenn Sie das Erkunden, Transformieren und Modellieren von Daten in jeder beliebigen Größe aus Ihrer Lieblings-R-Umgebung durchführen könnten. Stellen Sie sich nun vor, wie man anschließend die erzeugten Modelle - mit wenigen Klicks - als skalierbare, cloud basierte Web-Services-API bereitstellt. In dieser Session zeigt Sascha Dittmann, wie Sie Ihren R-Code, tausende von Open-Source-R-Pakete sowie die verteilte Implementierungen der beliebtesten Maschine-Learning-Algorithmen nutzen können, um genau dies umzusetzen. Dabei zeigt er wie man ein HDInsight Spark-Cluster inkl. eines Microsoft R Server-Clusters erstellt, sowie das daraus entstandene Model im SQL Server oder als swagger-based API für Anwendungsentwickler bereitstellt.

Microsoft R - Data Science at Scale

Sascha Dittmann

Seit dem SQL Server 2000 hielt Stück für Stück die XML-Unterstützung Einzug in die Microsoft RDBMS Welt. Mit der Azure DocumentDB kam die zweite, hauseigene NoSQL-Datenbank in der Microsoft Cloud hinzu, welche die Daten im JSON-Format verarbeitet. In dieser Session werden wir anhand eines Praxisbeispiels step-by-step, d.h. von den vorbereitenden Schritten, über das Schreiben bis hin zum Lesen der Daten, diese beiden Technologien gegenüberstellen. Dabei arbeiten wir die Vor- und Nachteile der einzelnen Ansätze heraus und zeigen Best Practices auf.

SQL Server vs. Azure DocumentDB – Ein Battle zwischen XML und JSON

Sascha Dittmann

Wer hoch-skalierbare Anwendungen in Microsoft Azure bauen wollte, kam bislang an den Azure Cloud Services nicht vorbei. Doch diese Technologie hatte ihre Ecken und Kanten. Die an der BUILD 2015 veröffentlichte Azure Service Fabric Preview soll hierbei Abhilfe schaffen. Mit ihr sollen Entwickler schneller und unkomplizierter Anwendungen für die Cloud, wie auch fürs eigene Rechenzentrum, konstruieren können. Sascha Dittmann durfte bereits seit Ende letzen Jahres mit dem Dienst Erfahrungen sammeln und stellt diesen, anhand einiger Code-Beispiele, in dieser Session vor.

dotnet Cologne 2015 - Azure Service Fabric

Sascha Dittmann

Das Programmiermodel MapReduce, welches vor einigen Jahren von Google veröffentlicht wurde, hat Einzug in zahlreiche Systeme erhalten. Dabei wurde es sowohl als eigenständiges System, wie beispielweise bei Hadoop, Disco oder Amazon Elastic MapReduce, aber auch als Abfragesprache innerhalb größerer Systeme, wie beispielweise bei MongoDB, Greenplum DB oder Aster Data, implementiert. Diese Session stellt gängige Problemstellungen aus der Praxis vor und wie diese mit dem MapReduce Framework von Microsoft HDInsight umgesetzt werden können.

SQL Saturday #313 Rheinland - MapReduce in der Praxis

Sascha Dittmann

Microsoft HDInsight Podcast #001 - Was ist HDInsight

Sascha Dittmann

dotnet Cologne 2013 - Windows Azure Mobile Services

Sascha Dittmann

dotnet Cologne 2013 - Microsoft HD Insight für .NET Entwickler

Sascha Dittmann

Der Einbezug von Cloud Computing in die tägliche Entwicklungsarbeit gewinnt weiter an Bedeutung. Ob hybride Ansätze oder Greenfield-Engineering, Windows Azure bietet die entsprechenden Building Blocks um eine Anforderung zielführend umzusetzen. In dem Workshop werden wir gemeinsam eine klassische on-premise Anwendung „cloudifizieren“ und Schritt für Schritt Cloud-Ressourcen zielführend einsetzen.

Developer Open Space 2012 - Cloud Computing Workshop

Sascha Dittmann

PASS Camp 2012 - Big Data mit Microsoft (Teil 1)

Sascha Dittmann

CloudOps Summit 2012 - 3 Wege in die Cloud

Sascha Dittmann

Big Data & NoSQL

Sascha Dittmann

NoSQL mit RavenDB und Azure

Sascha Dittmann

Windows Azure für Entwickler V1

Sascha Dittmann

Plus de Sascha Dittmann (15)

C# + SQL = Big Data

Hochskalierbare, relationale Datenbanken in Microsoft Azure

Microsoft R - Data Science at Scale

SQL Server vs. Azure DocumentDB – Ein Battle zwischen XML und JSON

dotnet Cologne 2015 - Azure Service Fabric

SQL Saturday #313 Rheinland - MapReduce in der Praxis

Microsoft HDInsight Podcast #001 - Was ist HDInsight

dotnet Cologne 2013 - Windows Azure Mobile Services

dotnet Cologne 2013 - Microsoft HD Insight für .NET Entwickler

Developer Open Space 2012 - Cloud Computing Workshop

PASS Camp 2012 - Big Data mit Microsoft (Teil 1)

CloudOps Summit 2012 - 3 Wege in die Cloud

Big Data & NoSQL

NoSQL mit RavenDB und Azure

Windows Azure für Entwickler V1

Dernier

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

DBX First Quarter 2024 Investor Presentation

Dropbox

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Dernier (20)

Axa Assurance Maroc - Insurer Innovation Award 2024

Corporate and higher education May webinar.pptx

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Boost Fertility New Invention Ups Success Rates.pdf

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

CNIC Information System with Pakdata Cf In Pakistan

[BuildWithAI] Introduction to Gemini.pdf

FWD Group - Insurer Innovation Award 2024

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

Why Teams call analytics are critical to your entire business

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

How to Troubleshoot Apps for the Modern Connected Worker

Manulife - Insurer Transformation Award 2024

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

1. Apache Hadoop-based Services für Windows Azure Sascha Dittmann Software Developer / Solution Architect Twitter: @SaschaDittmann Blog: http://www.sascha-dittmann.de

2. Apache Hadoop & Co Zookeeper Pig

3. Hadoop Distributed File System Cluster Startvorgang

4. Hadoop Distributed File System Ausfall des Namenodes (Failover)

5. Hadoop Distributed File System Benuteranfrage ① ② ② ②

6. Hadoop Distributed File System  Portable Operating System Interface (POSIX)  Replikation auf mehrere Datenknoten js> #ls input/ncdc Found 9 items drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/_distcp_logs_g0dedn drwxr-xr-x - Sascha supergroup 0 2012-04-24 12:04 /user/Sascha/input/ncdc/_distcp_logs_ofj0u6 drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:09 /user/Sascha/input/ncdc/all drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/all2 drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/metadata drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro-tab -rw-r--r-- 3 Sascha supergroup 529 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt -rw-r--r-- 3 Sascha supergroup 168 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

7. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[22,33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11

8. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 1949,0 1952,-11 Combine Combine Combine 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11

9. RDBMS vs. Map/Reduce RDBMS Map/Reduce Datenmenge Gigabytes Petabytes Zugriff Interaktiv und Batch Batch Lese- / Schreibzugriffe Viele Lese- und Einmaliges Schreiben Schreibzugriffe Viele Lesezugriffe Datenstruktur Statisches Schema Dynamisches Schema Datenintegrität Hoch Niedrig Skalierverhalten Nicht-Linear Linear

10. Apache Hadoop & Co Zookeeper Pig

11. Demos  Hadoop Dashboard  Interactive Console  Remote Desktop  Nutzung des WA Storage  Map/Reduce via JavaScript  C# Streaming  Power Pivot

12. Cloud Bloggers Die Blogs der deutschen Cloud Computing-Community Link: http://cloudbloggers.de

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure

Recommandé

Recommandé

Contenu connexe

Plus de Sascha Dittmann

Plus de Sascha Dittmann (15)

Dernier

Dernier (20)

.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure