Soumettre la recherche
Mettre en ligne
Ozone- Object store for Apache Hadoop
•
Télécharger en tant que PPTX, PDF
•
6 j'aime
•
4,218 vues
Hortonworks
Suivre
Slides from ApacheCon
Lire moins
Lire la suite
Technologie
Signaler
Partager
Signaler
Partager
1 sur 46
Télécharger maintenant
Recommandé
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
Erik Krogen
Ozone: An Object Store in HDFS
Ozone: An Object Store in HDFS
DataWorks Summit
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks
Hive: Loading Data
Hive: Loading Data
Benjamin Leonhardi
Hadoop Overview kdd2011
Hadoop Overview kdd2011
Milind Bhandarkar
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
RocksDB compaction
RocksDB compaction
MIJIN AN
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Recommandé
Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
Erik Krogen
Ozone: An Object Store in HDFS
Ozone: An Object Store in HDFS
DataWorks Summit
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks
Hive: Loading Data
Hive: Loading Data
Benjamin Leonhardi
Hadoop Overview kdd2011
Hadoop Overview kdd2011
Milind Bhandarkar
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
RocksDB compaction
RocksDB compaction
MIJIN AN
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Introduction to MongoDB
Introduction to MongoDB
Mike Dirolf
Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
DataWorks Summit
What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?
DataWorks Summit
Apache Spark Architecture
Apache Spark Architecture
Alexey Grishchenko
File Format Benchmark - Avro, JSON, ORC and Parquet
File Format Benchmark - Avro, JSON, ORC and Parquet
DataWorks Summit/Hadoop Summit
NiFi 시작하기
NiFi 시작하기
Byunghwa Yoon
High throughput data replication over RAFT
High throughput data replication over RAFT
DataWorks Summit
Introduction to Redis
Introduction to Redis
Dvir Volk
Kudu Deep-Dive
Kudu Deep-Dive
Supriya Sahay
Apache Spark At Scale in the Cloud
Apache Spark At Scale in the Cloud
Databricks
MyRocks Deep Dive
MyRocks Deep Dive
Yoshinori Matsunobu
Apache Kudu: Technical Deep Dive
Apache Kudu: Technical Deep Dive
Cloudera, Inc.
Apache Hudi: The Path Forward
Apache Hudi: The Path Forward
Alluxio, Inc.
AF Ceph: Ceph Performance Analysis and Improvement on Flash
AF Ceph: Ceph Performance Analysis and Improvement on Flash
Ceph Community
HBase Low Latency
HBase Low Latency
DataWorks Summit
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
The Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
Databricks
[211] HBase 기반 검색 데이터 저장소 (공개용)
[211] HBase 기반 검색 데이터 저장소 (공개용)
NAVER D2
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
DataWorks Summit/Hadoop Summit
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
Ozone and HDFS's Evolution
Ozone and HDFS's Evolution
DataWorks Summit
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Contenu connexe
Tendances
Introduction to MongoDB
Introduction to MongoDB
Mike Dirolf
Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
DataWorks Summit
What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?
DataWorks Summit
Apache Spark Architecture
Apache Spark Architecture
Alexey Grishchenko
File Format Benchmark - Avro, JSON, ORC and Parquet
File Format Benchmark - Avro, JSON, ORC and Parquet
DataWorks Summit/Hadoop Summit
NiFi 시작하기
NiFi 시작하기
Byunghwa Yoon
High throughput data replication over RAFT
High throughput data replication over RAFT
DataWorks Summit
Introduction to Redis
Introduction to Redis
Dvir Volk
Kudu Deep-Dive
Kudu Deep-Dive
Supriya Sahay
Apache Spark At Scale in the Cloud
Apache Spark At Scale in the Cloud
Databricks
MyRocks Deep Dive
MyRocks Deep Dive
Yoshinori Matsunobu
Apache Kudu: Technical Deep Dive
Apache Kudu: Technical Deep Dive
Cloudera, Inc.
Apache Hudi: The Path Forward
Apache Hudi: The Path Forward
Alluxio, Inc.
AF Ceph: Ceph Performance Analysis and Improvement on Flash
AF Ceph: Ceph Performance Analysis and Improvement on Flash
Ceph Community
HBase Low Latency
HBase Low Latency
DataWorks Summit
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
The Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
Databricks
[211] HBase 기반 검색 데이터 저장소 (공개용)
[211] HBase 기반 검색 데이터 저장소 (공개용)
NAVER D2
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
DataWorks Summit/Hadoop Summit
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
Tendances
(20)
Introduction to MongoDB
Introduction to MongoDB
Ozone: scaling HDFS to trillions of objects
Ozone: scaling HDFS to trillions of objects
What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?
Apache Spark Architecture
Apache Spark Architecture
File Format Benchmark - Avro, JSON, ORC and Parquet
File Format Benchmark - Avro, JSON, ORC and Parquet
NiFi 시작하기
NiFi 시작하기
High throughput data replication over RAFT
High throughput data replication over RAFT
Introduction to Redis
Introduction to Redis
Kudu Deep-Dive
Kudu Deep-Dive
Apache Spark At Scale in the Cloud
Apache Spark At Scale in the Cloud
MyRocks Deep Dive
MyRocks Deep Dive
Apache Kudu: Technical Deep Dive
Apache Kudu: Technical Deep Dive
Apache Hudi: The Path Forward
Apache Hudi: The Path Forward
AF Ceph: Ceph Performance Analysis and Improvement on Flash
AF Ceph: Ceph Performance Analysis and Improvement on Flash
HBase Low Latency
HBase Low Latency
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
The Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
[211] HBase 기반 검색 데이터 저장소 (공개용)
[211] HBase 기반 검색 데이터 저장소 (공개용)
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
Similaire à Ozone- Object store for Apache Hadoop
Ozone and HDFS's Evolution
Ozone and HDFS's Evolution
DataWorks Summit
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
DataWorks Summit
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
Chris Nauroth
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
DataWorks Summit/Hadoop Summit
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
DataWorks Summit/Hadoop Summit
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
DataWorks Summit
Hadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
DataWorks Summit/Hadoop Summit
Running Services on YARN
Running Services on YARN
DataWorks Summit/Hadoop Summit
Big data spain keynote nov 2016
Big data spain keynote nov 2016
alanfgates
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
DataWorks Summit/Hadoop Summit
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
Cloudera, Inc.
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
Big Data Spain
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
DataWorks Summit/Hadoop Summit
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
Edelweiss Kammermann
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
alanfgates
Apache Hive on ACID
Apache Hive on ACID
Hortonworks
Apache Hive on ACID
Apache Hive on ACID
DataWorks Summit/Hadoop Summit
Similaire à Ozone- Object store for Apache Hadoop
(20)
Ozone and HDFS's Evolution
Ozone and HDFS's Evolution
Ozone and HDFS’s evolution
Ozone and HDFS’s evolution
Evolving HDFS to a Generalized Storage Subsystem
Evolving HDFS to a Generalized Storage Subsystem
Hadoop & cloud storage object store integration in production (final)
Hadoop & cloud storage object store integration in production (final)
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Hadoop & Cloud Storage: Object Store Integration in Production
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
CBlocks - Posix compliant files systems for HDFS
CBlocks - Posix compliant files systems for HDFS
Hadoop 3 in a Nutshell
Hadoop 3 in a Nutshell
Running Services on YARN
Running Services on YARN
Big data spain keynote nov 2016
Big data spain keynote nov 2016
Evolving HDFS to Generalized Storage Subsystem
Evolving HDFS to Generalized Storage Subsystem
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBaseCon 2013: Apache HBase and HDFS - Understanding Filesystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
Apache Hadoop YARN: Past, Present and Future
Apache Hadoop YARN: Past, Present and Future
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
The Open Source and Cloud Part of Oracle Big Data Cloud Service for Beginners
Hive ACID Apache BigData 2016
Hive ACID Apache BigData 2016
Apache Hive on ACID
Apache Hive on ACID
Apache Hive on ACID
Apache Hive on ACID
Plus de Hortonworks
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Hortonworks
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Hortonworks
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Hortonworks
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Hortonworks
HDF 3.2 - What's New
HDF 3.2 - What's New
Hortonworks
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Hortonworks
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Hortonworks
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
Hortonworks
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
Hortonworks
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
Hortonworks
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
Hortonworks
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Hortonworks
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Hortonworks
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
Hortonworks
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Hortonworks
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
Hortonworks
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
Hortonworks
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Hortonworks
Plus de Hortonworks
(20)
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
HDF 3.2 - What's New
HDF 3.2 - What's New
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Dernier
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Enterprise Knowledge
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
Sinan KOZAK
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Rafal Los
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Delhi Call girls
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Malak Abu Hammad
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
V3cube
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Anna Loughnan Colquhoun
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
naman860154
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Puma Security, LLC
Dernier
(20)
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Ozone- Object store for Apache Hadoop
1.
1 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Object Store for Apache Hadoop Anu Engineer aengineer@apache.org Arpit Agarwal arp@apache.org
2.
2 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store ⬢ With workloads like IoT we are looking at scaling to trillions of objects. ⬢ Apache HDFS is designed for large objects – not for many small objects – Small files create memory pressure on namenode. ⬢ Each small file creates a block in the datanode. –Datanodes send all block information to namenode in BlockReports. ⬢ Both of these create scalability issues on Namenode. ⬢ Metadata in memory is the strength of the original GFS and HDFS design, but also its weakness in scaling number of files and blocks. ⬢ An object store has simpler semantics than a file system and is easier to scale Apache Hadoop, Hadoop, Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.
3.
3 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone – Why an Object Store (continued) ⬢ Ozone attempts to scale to trillions of objects – This presentation is about how we will get there. ⬢ Ozone is built on a distributed metadata store. ⬢ Avoids any single server becoming a bottleneck ⬢ More parallelism possible in both data and metadata operations ⬢ Build on well tested components and understood protocols –RAFT for consensus •RAFT is a protocol for reaching consensus between a set of machines in an unreliable environment where machines and network may fail. –Off-the-shelf Key-Value store like LevelDB •LevelDB is an open-source standalone key-value store built by Google.
4.
4 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Alternative solutions to NameNode scalability ⬢ HDFS federation aims to address namespace and Block Space scalability issues. –Federation deployment and planning adds complexity –Requires changes to other components in the Hadoop stack ⬢ HDFS-8286 - Partial Namespace In Memory. –Proposal to keep active working set of namespace in memory. ⬢ HDFS-5477 - Block Management as a Service. –Proposed solution for block space scalability issue. ⬢ Ozone borrows many ideas from these and is a super set of these approaches.
5.
5 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Presentation Outline Ozone Introduction Ozone Architectural Overview Containers Ozone - Bringing it all together Bonus Slides - if we have time.
6.
6 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Introduction ⬢ An Ozone URL –http://hostname/myvolume/mybucket/mykey ⬢ An S3 URL –http://hostname/mybucket/mykey ⬢ An Azure URL –http://hostname/myaccount/mybucket/key
7.
7 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions ⬢ Storage Volume –A notion similar to an account –Allows admin controls on usage of the object store e.g. storage quota –Created and managed by admins only ⬢ Bucket –Consists of keys and objects –Similar to a bucket in S3 or a container in Azure –ACLs
8.
8 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - Definitions (continued) ⬢ Storage Key –Unique in a bucket ⬢ Object –Values in a bucket –Each corresponds to a unique key within a bucket
9.
9 © Hortonworks
Inc. 2011 – 2016. All Rights Reserved Ozone - REST API ⬢ POST - Creates Volumes and Buckets –Only Admin creates volumes –Bucket can be created by owner of the volume ⬢ PUT - Updates Volumes , Buckets and creates keys –Only admin can change some volume settings –Buckets have ACLs –Creates Keys
10.
1 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone - REST API (continued) ⬢ GET - Lists volumes and buckets and allows reading of keys –Lists Volumes –List Buckets –Get Keys ⬢ DELETE - Deletes volumes, buckets and keys. –Delete Volumes –Delete Buckets –Removes the Key
11.
1 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Components ⬢ Containers – Actual storage locations on Datanodes. –We acknowledge the term container is overloaded. No relation to YARN containers or LXC. –Assume container means a collection of blocks on a datanode for now. –Containers deep dive to follow. ⬢ Ozone Handler - REST frontend for the Ozone rest protocol - deployed on datanodes. ⬢ Storage Container Manager (SCM) - Manages the container life cycle. ⬢ Ozone Key Space Manager (KSM) - Maps Ozone entities to Containers. ⬢ Container Client - Talks to KSM to discover the location of a container and sends IO requests to the appropriate container.
12.
1 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Overview
13.
1 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager
14.
1 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ Key to container mapping service. ⬢ Keeps the key ranges to containers mapping in memory. –Θ(number of containers) - 1 Exabyte cluster = 200M containers x 5GB each. –Memory usage scales with number of containers and not number of keys. ⬢ KSM does NOT know about all the keys in the system. ⬢ KSM state is replicated via RAFT, NameNode-like snapshots.
15.
1 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager ⬢ KSM knows about Ozone Volumes and Buckets. ⬢ KSM keeps a map of Volumes to container and buckets to containers. ⬢ KSM performs longest prefix match on a given string. ⬢ Example: The user wants to lookup a key - “/volume1/bucket1/key1” –KSM authenticates the user, maps this key to a container and looks up the container location. –Container client gets a token from the KSM and talks to the container on the data node. –Container client makes a getKey call to the datanode container with the full key path. –DataNode validates the access token and serves the value. ⬢ Contents of a bucket may span multiple containers.
16.
1 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved KSM - Bucket spanning multiple containers
17.
1 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved1 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Containers
18.
1 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Container Framework ⬢ A shareable generic block service that can be used by distributed storage services. ⬢ Make it easier to develop new storage services - BYO storage format and naming scheme. ⬢ Design Goals –No single points of failure. All services are replicated. –Avoid bottlenecks •Minimize central state •No verbose Block Reports ⬢ Lessons learned from large scale HDFS clusters. ⬢ Ideas from earlier proposals in HDFS community.
19.
1 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Container Framework Components
20.
2 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers
21.
2 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers ⬢ A container is the unit of replication –Size bounded by how quickly it can be re-replicated after a loss. ⬢ Each container is an independent key-value store. –No requirements on the structure or format of keys/values. –Keys are unique only within a container. ⬢ E.g. key-value pair could be one of –An Ozone Key-Value pair –An HDFS block ID and block contents •Or part of a block, when a block spans containers.
22.
2 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers (continued) ⬢ Each container has metadata – Metadata consistency maintained via the RAFT protocol – Metadata consists of keys and references to chunks. ⬢ Container metadata stored in LevelDB. – Exact choice of KV store unimportant. LevelDB is already used by other Hadoop components. ⬢ A chunk is a piece of user data. – Chunks are replicated via a data pipeline. – Chunks can be of arbitrary sizes e.g. a few KB to GBs. – Each chunk reference is a (file, offset, length) triplet. ⬢ Containers may garbage collect unreferenced chunks. ⬢ Each container independently decides how to map chunks to files – Allow reauthoring files for performance, compaction and overwrites.
23.
2 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers (continued)
24.
2 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Containers support simple client operations ⬢ Write chunks - streaming writes ⬢ Put(key, List<ChunkReference>) –The value is a list of chunk references. –Putting a key makes previously written chunk data visible to readers. –Put overwrites previous versions of the key. ⬢ Get(key) –Returns a list of chunk references ⬢ Read chunks - streaming reads ⬢ Delete(key) ⬢ List Keys
25.
2 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager
26.
2 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager ⬢ A fault-tolerant replicated service ⬢ Replicates its own state using RAFT protocol ⬢ Provides Container Location Service to clients –Given a container ID, return a list of nodes with replicas –Mapping a container ID to Data Nodes (and vice versa) ⬢ Provides Cluster Membership Management –Maintain list of live Data Nodes in the cluster –Handle heartbeats from DataNodes
27.
2 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager (continued) ⬢ Provides Replication services –Detect lost container replicas and initiate re-replication –Containers send a container report. •Unlike HDFS block reports which include details about each block , a container report is a summary of information. •This is used by KSM for placement of containers ⬢ If a node suffers from disk failure or if a node is lost, the reconstruction is a local activity which is coordinated via RAFT running on the data nodes.
28.
2 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Storage Container Manager ⬢ Maintains pre-created containers ⬢ Collects container operation statistics ⬢ Decides which Data Nodes form the replication set for a given container. –The number of replication sets in a cluster is bounded –Borrowing the work done by Facebook and RAMCloud (Copysets, Cidon et al. 2013). ⬢ Important - Knows nothing about keys –Does NOT provide Naming Service (mapping keys to containers) –e.g. KSM provides naming for Ozone.
29.
2 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Conceptual Representation of Ozone and Container State
30.
3 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 0 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ozone - Bringing it all together
31.
3 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Bringing it all together - Ozone createVolume operations Key Space Manager Replicated containers 1: createVolume Container Manager 2: Lookup(volName, Operation) 3: getContainer 4: putMetdata(VolumeName, Properties) Ozone HandlerClient Heartbeats & Reports DataNodes
32.
3 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Tracing an Ozone PutKey Key Space Manager Replicated containers 1: Ozone - putKey 2: Lookup(keyName, Operation) 4: putData(File, offset, Length, data) Client 5: putMetadata(key, List<chunks>) Ozone Handler DataNodes
33.
3 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Tracing an Ozone createVolume OzoneVolume vol = (new VolumeBuilder(pipeLine)) .setCreated(new Date()) .setOwnerName("bilbo") .setClient(client) .setName(“shire”) .build(); POST /shire keyData = {ContainerKeyData} keyName = "shire" containerName = "OzoneContainer" metadata = 0."Created" -> "1449533074362" 1."CreatedBy" -> "gandalf" 2."Key" -> "VOLUME" 3."Owner" -> "bilbo" Ozone REST Handler code Container wire and storage format
34.
3 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Metadata operations ⬢ Any metadata write to a container is Replicated via RAFT. ⬢ Machines forming the replication set for a container comprise a pipeline. ⬢ A createVolume call reduces to putKey operation on the container. ⬢ putKey is consistent, atomic and durable. ⬢ All metadata data operations are done via putKey, getKey and deleteKey. ⬢ Data is written to one or more chunks and a key is updated to point to those chunks. ⬢ Updating the key makes the data visible in the namespace.
35.
3 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Current State of Ozone ⬢ Stand alone container framework. ⬢ Single node ozone using container framework. ⬢ Full REST API -- Command Line Tools and Client Libs are fully functional. ⬢ Active development in branch HDFS-7240. ⬢ Work in progress: –SCM –KSM –Replication Pipeline –RAFT
36.
3 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Acknowledgements ⬢ Ozone is being designed and developed by Jitendra Pandey, Chris Nauroth, Tsz Wo (Nicholas) Sze, Jing Zhao, Suresh Srinivas, Sanjay Radia, Anu Engineer and Arpit Agarwal. ⬢ The Apache community has been very helpful and we were supported by comments and contributions from Kanaka Kumar Avvaru, Edward Bortnikov, Thomas Demoor, Nick Dimiduk, Chris Douglas, Jian Fang, Lars Francke, Gautam Hegde, Lars Hofhansl, Jakob Homan, Virajith Jalaparti, Charles Lamb, Steve Loughran, Haohui Mai, Colin Patrick McCabe, Aaron Myers, Owen O’Malley, Liam Slusser, Jeff Sogolov, Enis Soztutar, Andrew Wang, Fengdong Yu, Zhe Zhang & khanderao.
37.
3 7 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
38.
3 8 © Hortonworks Inc.
2011 – 2016. All Rights Reserved3 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bonus Slides
39.
3 9 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ KSM deals with dynamic partitioning of containers. ⬢ Let us say that a user starts by uploading all his photographs to a bucket in ozone. ⬢ Since all the photographs are called IMG_* (thanks Apple), we will soon overflow the 5GB capacity of the container. ⬢ At this point we need to split the container.
40.
4 0 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone KSM - Dynamic Container Partitioning ⬢ The container client attempts to write the Nth ozone key, IMG_N, gets a partition required error. ⬢ Container client will take that error and return that info to KSM. ⬢ That error contains the info -- about the proposed split -- That is IMG_0- IMG_200 will stay in this container and IMG_201-IMG_400 will move to next container. ⬢ Note: KSM initiates container partitioning but mechanics of the split are handled by the Container Layer
41.
4 1 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ One of the assumptions we have made about a container split is that the splits are on the same datanode as the original container. ⬢ This allows us to reduce a split operation to a copy of Keys from one LevelDB to another LevelDB. ⬢ if we need to move actual file data from one datanode to another -- we do support container moves. However they are slow. ⬢ A split on the other hand will complete in seconds in most cases. ⬢ The split point is chosen by the container so that we are able to pick the 50th percentile position that gives us reasonable chance at an equal partition of a container. ⬢ KSM does not know about the keys or the actual data sizes until much later. ⬢ So always relies on the container to tell it where the split should be.
42.
4 2 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Dynamic Container Partitioning ⬢ A container split is done in KSM via updating the Tree. The range partition key we maintain gets updated to reflect the fact that Keys - {IMG_0 - IMG_200} are on container C1, and keys {IMG_201-IMG_Z} are on C2. ⬢ Container will update the SCM when the split is done. ⬢ This information is learned and maintained by KSM.
43.
4 3 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Soft Quotas ⬢ In the HDFS world, a Quota is hard limit. It is actually conservative in quota management. ⬢ In the ozone world, Quotas are soft quotas. That is users can and will be able to violate it, but KSM/SCM will eventually learn about it and lock the volume out. ⬢ The key reason why this is different is because KSM/SCM is not involved in the allocation of chunks. ⬢ The containers have a partial -- that is an isolated view of the data in a volume. Since volumes can span many containers, it is possible for users to allocate chunks that violate the volume quota, but eventually KSM will learn and disable further writes to a volume.
44.
4 4 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone Key Space Manager - Missing namespace problem ⬢ One great thing about HDFS is Namenode. –Despite scalability issues, in most cases Namenode does a wonderful job. ⬢ Ozone - Subtle problem if we lose the all replicas of a container. ⬢ We will not only lose data -- just as if HDFS lost its all 3 replicas, but we will also lose information about which keys have been lost. ⬢ To solve this issue, we propose to have a on-disk eventually consistent log maintained by a separate service. –Records information about the keys that exist in the cluster. ⬢ This Scribe service logs the state of the cluster.
45.
4 5 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Ozone - Range reads ⬢ Ozone supports range reads and might support range writes like part upload in S3. –Ozone achieves this by using the chunk mechanism. –Chunks offer a stream like interface. –You can seek to any location and read as many bytes as you want. –This is used by ozone to support range reads ⬢ Periodic Scanner can reclaim unreferenced chunks.
46.
4 6 © Hortonworks Inc.
2011 – 2016. All Rights Reserved Scalability - What HDFS does well ⬢ HDFS NN stores all namespace metadata in memory (as per GFS) –Scales to large clusters (5K) since all metadata in memory •60K-100K tasks can share the Namenode •Low latency –Large data if files are large ⬢ Proof points of large data and large clusters –Single Organizations have over 600PB in HDFS –Single clusters with over 200PB using federation –Large clusters of over 4K multi-core nodes hitting a single NN
Télécharger maintenant