Zh Tw Introduction To Hadoop And Hdfs

•

9 j'aime•1,904 vues

Hadoop is an open-source software framework for distributed storage and processing of large datasets using the MapReduce programming model. It includes HDFS for data storage and MapReduce for data processing across clusters of compute nodes. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. It provides reliability through data replication and distributed architecture.

Technologie Formation

• Hadoop
• Hadoop (HDFS)
–
–
–
•

Copyright 2009 - Trend Micro Inc.

Hadoop ?

• Hadoop

• Apache top-level Cloud Applications

• Hadoop
– (HDFS) MapReduce HBase

– MapReduce
• Java Hadoop Distributed File System
(HDFS)
• C++/Java/Shell/
Command…
A Cluster of Machines
•
– Linux Mac OS/X Windows Solaris
–

Copyright 2009 - Trend Micro Inc.

Hadoop

• 2003 2
– Google MapReduce
• 2003 10
– Google Goofle File System (GFS)
• 2004 12
– Google MapReduce
• 2005 7
– Doug Cutting Nutch MapReduce
• 2006 2
– Hadoop Nutch Lucene
• 2006 11
– Google Bigtable

Copyright 2009 - Trend Micro Inc.

Hadoop

• 2007 2
– Mike Cafarella Hbase
• 2007 4
– Yahoo! 1000 Hadoop
• 2008 1
– Hadoop Apache

Copyright 2009 - Trend Micro Inc.

Who use Hadoop?
• Yahoo!
– Hadoop 2 CPU 10
• Google
– Hadoop
• Amazon
– Amazon Hadoop
–
• IBM
– Blue Cloud
• Trend Micro
– Hadoop

• Hadoop …
– http://wiki.apache.org/hadoop/PoweredBy

Copyright 2009 - Trend Micro Inc.

Hadoop (HDFS)

Copyright 2009 - Trend Micro Inc.

HDFS

• (Single
Namespace)
•
– 1 1 10 Peta Bytes
•
– Write-once-read-many
–
• (block)
– 128 MB
– (replica)
(DataNode)

Copyright 2009 - Trend Micro Inc.

HDFS

•
–

• (File replication)
– 3 .
–
•
–
–
•
– (low latency)

– (Batch processing)

Copyright 2009 - Trend Micro Inc.

(NameNode)

• NameNode HDFS (File System
Namespace)
– (blocks)
– (block) Data Node
• Hadoop cluster
•

Copyright 2009 - Trend Micro Inc.

NameNode (Metadata)

• Name node Metadata

– Metadata

–

• Metadata
– (files)
– (blocks)

– (block)
(Data Node)
–
• : (creation time),
(replication factor)

Copyright 2009 - Trend Micro Inc.

NameNode (Metadata)
• ( EditLog)
–

• FsImage
– Name Node

• (Name Space)
• (Block) (File)

•
– NameNode
FsImage EditLog

• Checkpoint
– NameNode
– FsImange
EditLog EditLog
FsImange

Copyright 2009 - Trend Micro Inc.

(Secondary NameNode)

• NameNode FsImage EditLog NameNode

• FSImage EditLog FSImage
• FSImage NameNode
– NameNode EditLog
• Secondary NameNode NameNode (Fail over)
– Hadoop Name Node

FsImage
FsImage
(new)

EditLog

Copyright 2009 - Trend Micro Inc.

NameNode

• NameNode SPOF (single point of failure)
• (High Availablity)

SPOF!!

Copyright 2009 - Trend Micro Inc.

(DataNode)

• (Blocks)

– ( ext3)

– block metadata
• (CRC), block

–
• Block
– Blocks
NameNode
– NameNode
block
NameNode
block

Copyright 2009 - Trend Micro Inc.

HDFS – (Replication)

• 3
• (block size)
(replication factor)
• (rack- aware)
.

Copyright 2009 - Trend Micro Inc.

Block Placement

• Policy (v0.19)
–
–
–
–
•

Copyright 2009 - Trend Micro Inc.

Heartbeats

• DataNode Heartbeats NameNode
– 3
• NameNode Heartbeats DataNode

Copyright 2009 - Trend Micro Inc.

(Data Correctness)

• Checksum
– Cyclic Redundancy Check (CRC32 )
•
– 512 Checksum
– DataNode Checksum
•
– Checksum
–

Copyright 2009 - Trend Micro Inc.

(User Interface)

• API
– Java API
– C language wrapper for the Java API is also avaiable

• POSIX like command
– hadoop dfs -mkdir /foodir
– hadoop dfs -cat /foodir/myfile.txt
– hadoop dfs -rm /foodir myfile.txthadoop dfs -rm /foodir myfile.txt

• DFSAdmin
– bin/hadoop dfsadmin –safemode
– bin/hadoop dfsadmin –report
– bin/hadoop dfsadmin -refreshNodes

• Web
– http://host:port/dfshealth.jsp

Copyright 2009 - Trend Micro Inc.

Web
(http://172.16.203.136:50070)

Classification Copyright 2009 - Trend Micro Inc.

POSIX Like command

Copyright 2009 - Trend Micro Inc.

Java API

Copyright 2009 - Trend Micro Inc.

• Hadoop document and installation
– http://hadoop.apache.org/
• Hadoop Wiki
– http://wiki.apache.org/hadoop/
• Google File System Paper
– http://labs.google.com/papers/gfs.html

Copyright 2009 - Trend Micro Inc.

Contenu connexe

Similaire à Zh Tw Introduction To Hadoop And Hdfs

Zh Tw Introduction To Cloud Computing

kevin liao

Zh Tw Introduction To H Base

kevin liao

Introduction to hadoop and hdfs

TrendProgContest13

App cap2956v2-121001194956-phpapp01 (1)

outstanding59

Inside the Hadoop Machine @ VMworld

Richard McDougall

App Cap2956v2 121001194956 Phpapp01 (1)

outstanding59

Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon

DataWorks Summit/Hadoop Summit

Big Data/Hadoop Infrastructure Considerations

Richard McDougall

Introduction to Hadoop and Big Data Processing

Sam Ng

Gregor Hohpe Track Intro The Cloud As Middle Ware

deimos

Big Data: Introduction to Hadoop

tokopedia

Hadoop 101 (v1) (20150730)

Fahmi Fachreza

Hadoop 101 - Big Data Technology

Firman Gautama

Is your cloud ready for Big Data? Strata NY 2013

Richard McDougall

Information Retrieval, Applied Statistics and Mathematics onBigData - German ...

Romeo Kienzler

Zh Tw Introduction To Map Reduce

kevin liao

NameNode Analytics - Querying HDFS Namespace in Real Time

Plamen Jeliazkov

The YUI Library (Yahoo! Course @NCU)

Joseph Chiang

S Cain - GMOD in the cloud

Jan Aerts

S Cain - GMOD in the cloud

Jan Aerts

Similaire à Zh Tw Introduction To Hadoop And Hdfs (20)

Zh Tw Introduction To Cloud Computing

Zh Tw Introduction To H Base

Introduction to hadoop and hdfs

App cap2956v2-121001194956-phpapp01 (1)

Inside the Hadoop Machine @ VMworld

App Cap2956v2 121001194956 Phpapp01 (1)

Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon

Big Data/Hadoop Infrastructure Considerations

Introduction to Hadoop and Big Data Processing

Gregor Hohpe Track Intro The Cloud As Middle Ware

Big Data: Introduction to Hadoop

Hadoop 101 (v1) (20150730)

Hadoop 101 - Big Data Technology

Is your cloud ready for Big Data? Strata NY 2013

Information Retrieval, Applied Statistics and Mathematics onBigData - German ...

Zh Tw Introduction To Map Reduce

NameNode Analytics - Querying HDFS Namespace in Real Time

The YUI Library (Yahoo! Course @NCU)

S Cain - GMOD in the cloud

Dernier

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Developing An App To Navigate The Roads of Brazil

V3cube

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Dernier (20)

A Year of the Servo Reboot: Where Are We Now?

What Are The Drone Anti-jamming Systems Technology?

How to Troubleshoot Apps for the Modern Connected Worker

HTML Injection Attacks: Impact and Mitigation Strategies

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Advantages of Hiring UIUX Design Service Providers for Your Business

AWS Community Day CPH - Three problems of Terraform

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Finology Group – Insurtech Innovation Award 2024

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

presentation ICT roal in 21st century education

Powerful Google developer tools for immediate impact! (2023-24 C)

Automating Google Workspace (GWS) & more with Apps Script

Developing An App To Navigate The Roads of Brazil

A Domino Admins Adventures (Engage 2024)

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Zh Tw Introduction To Hadoop And Hdfs

1. Hadoop Hadoop (HDFS) Public 2009/5/13

3. Hadoop ? • Hadoop • Apache top-level Cloud Applications • Hadoop – (HDFS) MapReduce HBase – MapReduce • Java Hadoop Distributed File System (HDFS) • C++/Java/Shell/ Command… A Cluster of Machines • – Linux Mac OS/X Windows Solaris – Copyright 2009 - Trend Micro Inc.

4. Hadoop • 2003 2 – Google MapReduce • 2003 10 – Google Goofle File System (GFS) • 2004 12 – Google MapReduce • 2005 7 – Doug Cutting Nutch MapReduce • 2006 2 – Hadoop Nutch Lucene • 2006 11 – Google Bigtable Copyright 2009 - Trend Micro Inc.

6. Who use Hadoop? • Yahoo! – Hadoop 2 CPU 10 • Google – Hadoop • Amazon – Amazon Hadoop – • IBM – Blue Cloud • Trend Micro – Hadoop • Hadoop … – http://wiki.apache.org/hadoop/PoweredBy Copyright 2009 - Trend Micro Inc.

14. NameNode (Metadata) • ( EditLog) – • FsImage – Name Node • (Name Space) • (Block) (File) • – NameNode FsImage EditLog • Checkpoint – NameNode – FsImange EditLog EditLog FsImange Copyright 2009 - Trend Micro Inc.

15. (Secondary NameNode) • NameNode FsImage EditLog NameNode • FSImage EditLog FSImage • FSImage NameNode – NameNode EditLog • Secondary NameNode NameNode (Fail over) – Hadoop Name Node FsImage FsImage (new) EditLog Copyright 2009 - Trend Micro Inc.

22. (User Interface) • API – Java API – C language wrapper for the Java API is also avaiable • POSIX like command – hadoop dfs -mkdir /foodir – hadoop dfs -cat /foodir/myfile.txt – hadoop dfs -rm /foodir myfile.txthadoop dfs -rm /foodir myfile.txt • DFSAdmin – bin/hadoop dfsadmin –safemode – bin/hadoop dfsadmin –report – bin/hadoop dfsadmin -refreshNodes • Web – http://host:port/dfshealth.jsp Copyright 2009 - Trend Micro Inc.

28. • Hadoop document and installation – http://hadoop.apache.org/ • Hadoop Wiki – http://wiki.apache.org/hadoop/ • Google File System Paper – http://labs.google.com/papers/gfs.html Copyright 2009 - Trend Micro Inc.

Zh Tw Introduction To Hadoop And Hdfs

Recommandé

Recommandé

Contenu connexe

Similaire à Zh Tw Introduction To Hadoop And Hdfs

Similaire à Zh Tw Introduction To Hadoop And Hdfs (20)

Dernier

Dernier (20)

Zh Tw Introduction To Hadoop And Hdfs