Serialization (Avro, Message Pack, Kryo)

•

1 j'aime•2,716 vues

오석 한

2010년 세미나 자료

Technologie

Serialization
Avro, Message Pack, Kryo
Han O Seok

Avro
• Apache Foundation
• JSON deﬁneded Schema

Avro
• Created by Doug Cutting, the Creator of
Hadoop
• Data is always accompanied by a schema
- Support for dynamic typing-code
generation is not required

Performance of Avro
• Avro is not the fastest, But is in the top half

Message Pack
• Rich data structures - JSON
• Interface Deﬁnition Language(IDL) - thrift
• Create Schema Based Annotaion
• RPC
Sync,Async Support
Event-Driven I/O

Kryo
• Google Code
• Easy to Collect Serializers Case By Class
• Support Compression
• Kryo TCP & UDP client/server library

BenchmarkingV2
• http://code.google.com/p/thrift-protobuf-
compare/wiki/BenchmarkingV2

Contenu connexe

Tendances

SCHEDULING ALGORITHMSDhaval Sakhiya

Asynchronous programming in ASP.NETAlex Thissen

Basic Concepts of OOPs (Object Oriented Programming in Java)Michelle Anne Meralpis

deadlock avoidancewahab13

Process management os conceptpriyadeosarkar91

Process synchronization in Operating SystemsRitu Ranjan Shrivastwa

Deadlock Presentationsalmancreation

Introduction to OOP(in java) BY Govind Singhprabhat engineering college

Java ServletsBG Java EE Course

DeadlockMohd Arif

Basic concepts of object oriented programmingSachin Sharma

Structure of operating systemGayathriS578276

Java OOP Programming language (Part 6) - Abstract Class & InterfaceOUM SAOKOSAL

CPU Scheduling in OS Presentationusmankiyani1

7가지 동시성 모델 - 3장. 함수형 프로그래밍Hyunsoo Jung

System callsBernard Senam

Abstraction in javasawarkar17

Multithreading In Javaparag

Event Handling in javaGoogle

DDD eXchangeSkills Matter

Tendances (20)

SCHEDULING ALGORITHMS

Asynchronous programming in ASP.NET

Basic Concepts of OOPs (Object Oriented Programming in Java)

deadlock avoidance

Process management os concept

Process synchronization in Operating Systems

Deadlock Presentation

Introduction to OOP(in java) BY Govind Singh

Java Servlets

Deadlock

Basic concepts of object oriented programming

Structure of operating system

Java OOP Programming language (Part 6) - Abstract Class & Interface

CPU Scheduling in OS Presentation

7가지 동시성 모델 - 3장. 함수형 프로그래밍

System calls

Abstraction in java

Multithreading In Java

Event Handling in java

DDD eXchange

En vedette

Serialization and performance by Sergey MorenetsAlex Tumanoff

3 apache-avrozafargilani

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

Chapter 8 big data and privacyopeyemiatilola1992

おひろめ会：Javaにおけるデータシリアライズ手法moai kids

From a kafkaesque story to The Promised LandRan Silberman

Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)VMware Tanzu

Avro Data | Washington DC HUGCloudera, Inc.

Serialization and performance in JavaStrannik_2013

Avro introductionNanda8904648951

Hadoop I/O AnalysisRichard McDougall

Hadoop InternalsPietro Michiardi

Event Stream Processing with Kafka and SamzaZach Cox

Type safe, versioned, and rewindable stream processing with Apache {Avro, K...Hisham Mardam-Bey

Big Data, Security Intelligence, (And Why I Hate This Title) Coastal Pet Products, Inc.

Best Practices for Using Apache Spark on AWSAmazon Web Services

Avro introRandy Abernethy

Apache Avro and YouEric Wendelin

Big Data and Security - Where are we now? (2015)Peter Wood

Hadoop and Spark Analytics over Better StorageSandeep Patil

En vedette (20)

Serialization and performance by Sergey Morenets

3 apache-avro

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Chapter 8 big data and privacy

おひろめ会：Javaにおけるデータシリアライズ手法

From a kafkaesque story to The Promised Land

Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)

Avro Data | Washington DC HUG

Serialization and performance in Java

Avro introduction

Hadoop I/O Analysis

Hadoop Internals

Event Stream Processing with Kafka and Samza

Type safe, versioned, and rewindable stream processing with Apache {Avro, K...

Big Data, Security Intelligence, (And Why I Hate This Title)

Best Practices for Using Apache Spark on AWS

Avro intro

Apache Avro and You

Big Data and Security - Where are we now? (2015)

Hadoop and Spark Analytics over Better Storage

Similaire à Serialization (Avro, Message Pack, Kryo)

Berlin Buzzwords 2019 - Taming the language border in data analytics and scie...Uwe Korn

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022Timothy Spann

Ursa Labs and Apache Arrow in 2019Wes McKinney

End-to-end Data Governance with Apache Avro and AtlasDataWorks Summit

ApacheCon2022_Deep Dive into Building Streaming Applications with Apache PulsarTimothy Spann

High Performance Systems in Go - GopherCon 2014Derek Collison

How Apache Arrow and Parquet boost cross-language interoperabilityUwe Korn

Scaling with Symfony - PHP UKRicard Clau

Apache Arrow: Cross-language Development Platform for In-memory DataWes McKinney

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...Timothy Spann

NAVER Ceph Storage on ssd for ContainerJangseon Ryu

Deep Dive into Building Streaming Applications with Apache Pulsar Timothy Spann

Real time cloud native open source streaming of any data to apache solrTimothy Spann

OSS EU: Deep Dive into Building Streaming Applications with Apache PulsarTimothy Spann

High Concurrency Architecture and Laravel Performance TuningAlbert Chen

Apache Arrow at DataEngConf Barcelona 2018Wes McKinney

Scaling with swaggerTony Tam

NoSQL afternoon in Japan Kumofs & MessagePackSadayuki Furuhashi

NoSQL afternoon in Japan kumofs & MessagePackSadayuki Furuhashi

DBCC 2021 - FLiP Stack for Cloud Data LakesTimothy Spann

Similaire à Serialization (Avro, Message Pack, Kryo) (20)

Berlin Buzzwords 2019 - Taming the language border in data analytics and scie...

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Ursa Labs and Apache Arrow in 2019

End-to-end Data Governance with Apache Avro and Atlas

ApacheCon2022_Deep Dive into Building Streaming Applications with Apache Pulsar

High Performance Systems in Go - GopherCon 2014

How Apache Arrow and Parquet boost cross-language interoperability

Scaling with Symfony - PHP UK

Apache Arrow: Cross-language Development Platform for In-memory Data

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

NAVER Ceph Storage on ssd for Container

Deep Dive into Building Streaming Applications with Apache Pulsar

Real time cloud native open source streaming of any data to apache solr

OSS EU: Deep Dive into Building Streaming Applications with Apache Pulsar

High Concurrency Architecture and Laravel Performance Tuning

Apache Arrow at DataEngConf Barcelona 2018

Scaling with swagger

NoSQL afternoon in Japan Kumofs & MessagePack

NoSQL afternoon in Japan kumofs & MessagePack

DBCC 2021 - FLiP Stack for Cloud Data Lakes

Plus de 오석 한

Smart work오석 한

RPC protocols오석 한

Cassandra오석 한

Smart Phone CPU오석 한

Functional progrmming with scala오석 한

Linux tips오석 한

Apache Click오석 한

JAVA NIO오석 한

예제로 쉽게 배우는 Log4j 기초 활용법오석 한

Vi 단축키명령어오석 한

Perl Script Document오석 한

Perl Script오석 한

정규 표현식 기본 메타문자 요약오석 한

정규표현식의 이해와 활용오석 한

Plus de 오석 한 (14)

Smart work

RPC protocols

Cassandra

Smart Phone CPU

Functional progrmming with scala

Linux tips

Apache Click

JAVA NIO

예제로 쉽게 배우는 Log4j 기초 활용법

Vi 단축키명령어

Perl Script Document

Perl Script

정규 표현식 기본 메타문자 요약

정규표현식의 이해와 활용

Dernier

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

A Call to Action for Generative AI in 2024Results

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

Partners Life - Insurer Innovation Award 2024The Digital Insurer

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Developing An App To Navigate The Roads of BrazilV3cube

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Dernier (20)

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

Driving Behavioral Change for Information Management through Data-Driven Gree...

A Domino Admins Adventures (Engage 2024)

Salesforce Community Group Quito, Salesforce 101

Boost PC performance: How more available memory can improve productivity

Unblocking The Main Thread Solving ANRs and Frozen Frames

Handwritten Text Recognition for manuscripts and early printed texts

A Call to Action for Generative AI in 2024

Automating Google Workspace (GWS) & more with Apps Script

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Partners Life - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Developing An App To Navigate The Roads of Brazil

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

08448380779 Call Girls In Friends Colony Women Seeking Men

Data Cloud, More than a CDP by Matt Robison

The Codex of Business Writing Software for Real-World Solutions 2.pptx

GenCyber Cyber Security Day Presentation

Serialization (Avro, Message Pack, Kryo)

1. Serialization Avro, Message Pack, Kryo Han O Seok

2. What is Serialization?

3. What is Serialization?

5. Avro • Apache Foundation • JSON deﬁneded Schema

6. Avro • Created by Doug Cutting, the Creator of Hadoop • Data is always accompanied by a schema - Support for dynamic typing-code generation is not required

7. Performance of Avro • Avro is not the fastest, But is in the top half

8. Message Pack • Rich data structures - JSON • Interface Deﬁnition Language(IDL) - thrift • Create Schema Based Annotaion • RPC Sync,Async Support Event-Driven I/O

9. Format of Message Pack

10. Performance of Message Pack

11. Kryo • Google Code • Easy to Collect Serializers Case By Class • Support Compression • Kryo TCP & UDP client/server library

12. BenchmarkingV2 • http://code.google.com/p/thrift-protobuf- compare/wiki/BenchmarkingV2

13. Thanks :)