ORC File Introduction

•Télécharger en tant que PPTX, PDF•

13 j'aime•11,815 vues

Owen O'Malley

I present the Optimized Row Columnar (ORC) file format for Apache Hive.

Technologie

ORC Files

Owen O’Malley
owen@hortonworks.com

December 2012

© Hortonworks Inc. 2012 Page 1

Top Level

Page 2
© Hortonworks Inc. 2012

File Structure

Page 3
© Hortonworks Inc. 2012

Stripe Structure

Page 4
© Hortonworks Inc. 2012

File Layout

Page 5
© Hortonworks Inc. 2012

Integer Column Serialization

Page 6
© Hortonworks Inc. 2012

String Column Serialization

Page 7
© Hortonworks Inc. 2012

Compression

Page 8
© Hortonworks Inc. 2012

Projection and Predicate Filtering

Page 9
© Hortonworks Inc. 2012

Example File Sizes

Page 10
© Hortonworks Inc. 2012

Final notes

Page 11
© Hortonworks Inc. 2012

Comparison

RC File Trevni ORC File
Hive Type Model N N Y
Separate complex columns N Y Y
Splits found quickly N Y Y
Default column group size 4MB 64MB* 250MB
Files per a bucket 1 >1 1
Store min, max, sum, count N N Y
Versioned metadata N Y Y
Run length data encoding N N Y
Store strings in dictionary N N Y
Store row count N Y Y
Skip compressed blocks N N Y
Store internal indexes N N Y

Page 12
© Hortonworks Inc. 2012

Contenu connexe

Tendances

Apache phoenix: Past, Present and Future of SQL over HBAseenissoz

What is new in Apache Hive 3.0?DataWorks Summit

Using Apache Hive with High PerformanceInderaj (Raj) Bains

Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!Mich Talebzadeh (Ph.D.)

Apache Tez – Present and FutureDataWorks Summit

Redis introductionFederico Daniel Colombo Gennarelli

Hadoop and Enterprise Data WarehouseDataWorks Summit

Apache hadoop technology : BeginnersShweta Patnaik

File Format Benchmark - Avro, JSON, ORC & ParquetDataWorks Summit/Hadoop Summit

Apache Spark OverviewVadim Y. Bichutskiy

Hive+Tez: A performance deep divet3rmin4t0r

Hadoop introduction , Why and What is Hadoop ?sudhakara st

ORC File - Optimizing Your Big DataDataWorks Summit

LLAP: long-lived execution in HiveDataWorks Summit

Cost Savings at High Performance with Redis Labs and AWSAmazon Web Services

Performance Optimizations in Apache ImpalaCloudera, Inc.

Hadoop Overview & Architecture EMC

[211] HBase 기반 검색 데이터 저장소 (공개용)NAVER D2

Spark as a Platform to Support Multi-Tenancy and Many Kinds of Data Applicati...Spark Summit

Introduction to RedisArnab Mitra

Tendances (20)

Apache phoenix: Past, Present and Future of SQL over HBAse

What is new in Apache Hive 3.0?

Using Apache Hive with High Performance

Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!

Apache Tez – Present and Future

Redis introduction

Hadoop and Enterprise Data Warehouse

Apache hadoop technology : Beginners

File Format Benchmark - Avro, JSON, ORC & Parquet

Apache Spark Overview

Hive+Tez: A performance deep dive

Hadoop introduction , Why and What is Hadoop ?

ORC File - Optimizing Your Big Data

LLAP: long-lived execution in Hive

Cost Savings at High Performance with Redis Labs and AWS

Performance Optimizations in Apache Impala

Hadoop Overview & Architecture

[211] HBase 기반 검색 데이터 저장소 (공개용)

Spark as a Platform to Support Multi-Tenancy and Many Kinds of Data Applicati...

Introduction to Redis

En vedette

File Format Benchmarks - Avro, JSON, ORC, & ParquetOwen O'Malley

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon

File Format Benchmark - Avro, JSON, ORC & ParquetDataWorks Summit/Hadoop Summit

ORC File and Vectorization - Hadoop Summit 2013Owen O'Malley

Structor - Automated Building of Virtual Hadoop ClustersOwen O'Malley

Adding ACID Updates to HiveOwen O'Malley

Protecting Enterprise Data in Apache HadoopOwen O'Malley

Data protection2015Owen O'Malley

Plugging the Holes: Security and Compatability in HadoopOwen O'Malley

Next Generation MapReduceOwen O'Malley

Bay Area HUG Feb 2011 IntroOwen O'Malley

Next Generation Hadoop OperationsOwen O'Malley

Optimizing Hive QueriesOwen O'Malley

Hadoop Security ArchitectureOwen O'Malley

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

Mapreduce total order sorting techniqueUday Vakalapudi

Hive integration: HBase and Rcfile__HadoopSummit2010Yahoo Developer Network

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

ORC File & Vectorization - Improving Hive Data Storage and Query PerformanceDataWorks Summit

En vedette (20)

File Format Benchmarks - Avro, JSON, ORC, & Parquet

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

File Format Benchmark - Avro, JSON, ORC & Parquet

ORC File and Vectorization - Hadoop Summit 2013

Structor - Automated Building of Virtual Hadoop Clusters

Adding ACID Updates to Hive

Protecting Enterprise Data in Apache Hadoop

Data protection2015

Plugging the Holes: Security and Compatability in Hadoop

Next Generation MapReduce

Bay Area HUG Feb 2011 Intro

Next Generation Hadoop Operations

Optimizing Hive Queries

Hadoop Security Architecture

Strata London 2016: The future of column oriented data processing with Arrow ...

Sql on everything with drill

Mapreduce total order sorting technique

Hive integration: HBase and Rcfile__HadoopSummit2010

Strata NY 2016: The future of column-oriented data processing with Arrow and ...

ORC File & Vectorization - Improving Hive Data Storage and Query Performance

Similaire à ORC File Introduction

Optimizing Hive QueriesDataWorks Summit

Inside hadoop-devSteve Loughran

Availability and Integrity in hadoop (Strata EU Edition)Steve Loughran

HA Hadoop -ApacheCon talkSteve Loughran

Mobile Development Meets Semantic TechnologyBlue Slate Solutions

Orange County HUG - Agile Data on HDPHortonworks

An Introduction to Spring DataOliver Gierke

LA HUG - Agile Analytics Applications on HDPHortonworks

Hadoop: today and tomorrowSteve Loughran

Cloud Consolidation with Oracle (RAC) - How much is too much?Markus Michalewicz

Sentri SharePoint Performance webinarSentri

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?Laura Hood

Agile analytics applications on hadoopHortonworks

Hortonworks: Agile Analytics Applicationsrussell_jurney

Compaction and Splitting in Apache AccumuloHortonworks

Introduction to DDDRadosław Mejer

ORC: 2015 Faster, Better, SmallerDataWorks Summit

Ozone and HDFS’s evolutionDataWorks Summit

ORC 2015t3rmin4t0r

Storage Characteristics Of Call Data Records In Column Store DatabasesDavid Walker

Similaire à ORC File Introduction (20)

Optimizing Hive Queries

Inside hadoop-dev

Availability and Integrity in hadoop (Strata EU Edition)

HA Hadoop -ApacheCon talk

Mobile Development Meets Semantic Technology

Orange County HUG - Agile Data on HDP

An Introduction to Spring Data

LA HUG - Agile Analytics Applications on HDP

Hadoop: today and tomorrow

Cloud Consolidation with Oracle (RAC) - How much is too much?

Sentri SharePoint Performance webinar

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?

Agile analytics applications on hadoop

Hortonworks: Agile Analytics Applications

Compaction and Splitting in Apache Accumulo

Introduction to DDD

ORC: 2015 Faster, Better, Smaller

Ozone and HDFS’s evolution

ORC 2015

Storage Characteristics Of Call Data Records In Column Store Databases

Plus de Owen O'Malley

Running An Apache Project: 10 Traps and How to Avoid ThemOwen O'Malley

Big Data's Journey to ACIDOwen O'Malley

ORC Deep Dive 2020Owen O'Malley

Protect your private data with ORC column encryptionOwen O'Malley

Fine Grain Access Control for Big Data: ORC Column EncryptionOwen O'Malley

Fast Access to Your Data - Avro, JSON, ORC, and ParquetOwen O'Malley

Strata NYC 2018 IcebergOwen O'Malley

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and ParquetOwen O'Malley

ORC Column EncryptionOwen O'Malley

Plus de Owen O'Malley (9)

Running An Apache Project: 10 Traps and How to Avoid Them

Big Data's Journey to ACID

ORC Deep Dive 2020

Protect your private data with ORC column encryption

Fine Grain Access Control for Big Data: ORC Column Encryption

Fast Access to Your Data - Avro, JSON, ORC, and Parquet

Strata NYC 2018 Iceberg

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and Parquet

ORC Column Encryption

Dernier

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

A Year of the Servo Reboot: Where Are We Now?Igalia

GenCyber Cyber Security Day PresentationMichael W. Hawkins

What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Artificial Intelligence: Facts and MythsJoaquim Jorge

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

Dernier (20)

Presentation on how to chat with PDF using ChatGPT code interpreter

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Advantages of Hiring UIUX Design Service Providers for Your Business

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Data Cloud, More than a CDP by Matt Robison

A Year of the Servo Reboot: Where Are We Now?

GenCyber Cyber Security Day Presentation

What Are The Drone Anti-jamming Systems Technology?

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Handwritten Text Recognition for manuscripts and early printed texts

08448380779 Call Girls In Civil Lines Women Seeking Men

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

A Domino Admins Adventures (Engage 2024)

Artificial Intelligence: Facts and Myths

Boost Fertility New Invention Ups Success Rates.pdf

CNv6 Instructor Chapter 6 Quality of Service