ソーシャルゲームログ解析基盤のHadoop活用事例

•Télécharger en tant que KEY, PDF•

25 j'aime•5,299 vues

知教本間

【エンジニアカフェEvent×gumiStudy】ソーシャルゲームの解析を支える技術-Hadoop編- http://www.facebook.com/event.php?eid=245262765524522 の発表資料です。

Technologie Business

1.

2. Elastic MapReduce EMR

3.

4. EMR

3

1 1,000 /
AP
APAP
AP DB
d ﬂuentd ﬂuentd mongos mongod(PRIMARY)

DB
conﬁg
mongod(SECONDARY)

DB
ﬂuentd mongos
mongod(SECONDARY)

conﬁg ReplicaSets & Sharding
NFS

6

8.5GB 1.4GB /

ID

Nov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345
[BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45

Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345
[BeforePower] 25 [AfterPower] 20 [ConsumePower] 5

8

MongoDB
MongoDB ( )

"app" : "hogehoge",

"userid" : "12345",
ID
"dateint" : 20111101,

"hourint" : 23,

"actions" : [

"CONSUME_POWER", MongoDB Sharding
"ADD_MONEY"

],

"records" : [

"action" : "ADD_MONEY",

"timeint" : 235959,

]

10

EMR
Hive Pig Hadoop Streaming

Hadoop
Streaming
(Python)

11

m2.4xlarge × 1 4.9GB 85

EMR(m2.xlarge) × 5 4.9GB 44

m2.4xlarge × 1 7.2GB 138

EMR(m2.xlarge) × 5 7.2GB 69

(Macbook Air) 3.6GB 30 …

12

( )

NFS Amazon S3 EMR

S3

EMR

S3 EMR

conﬁg
MongoDB mongos

15

S3
boto

S3
c3cmd S3

EMR
Mapper,Reducer,Python2.7

MongoDB
pymongo MongoDB

EMR
Client Tool(Ruby) EMR

16

EMR
S3
EC2⇔S3 20MB/sec

Hadoop

HadoopStreaming

EMR

18

Recommandé

MongoDB Basic ConceptsMongoDB

企業・業界情報プラットフォームSPEEDAにおけるElasticsearchの活用Akira Kitauchi

Redis深入浅出iammutex

MongoFr : MongoDB as a log CollectorPierre Baillet

Java/Spring과 Node.js의공존동수 장

NoSQL 동향NAVER D2

Cache in API GatewayGilWon Oh

PostgreSQLReuven Lerner

Recommandé

MongoDB Basic ConceptsMongoDB

企業・業界情報プラットフォームSPEEDAにおけるElasticsearchの活用Akira Kitauchi

Redis深入浅出iammutex

MongoFr : MongoDB as a log CollectorPierre Baillet

Java/Spring과 Node.js의공존동수 장

NoSQL 동향NAVER D2

Cache in API GatewayGilWon Oh

PostgreSQLReuven Lerner

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico

PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico

MesosCon 2018Pablo Delgado

MongodbScott Motte

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종NAVER D2

CuPy v4 and v5 roadmapPreferred Networks

Как PostgreSQL работает с дискомPostgreSQL-Consulting

To Hire, or to train, that is the question (Percona Live 2014)Geoffrey Anderson

XtraDB 5.7: key performance algorithmsLaurynas Biveinis

Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Ontico

Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico

PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016Tomas Vondra

Embulk and Machine Learning infrastructureHiroshi Toyama

PostgreSQL performance archaeologyTomas Vondra

PostgreSQL is the new NoSQL - at Devoxx 2018Quentin Adam

Chainer v4 and v5Preferred Networks

Sun jdk 1.6 gc english versionbluedavy lin

Ops Jumpstart: Admin 101MongoDB

Advanced backup methods (Postgres@CERN)Anastasia Lubennikova

GlusterFS As an Object StorageKeisuke Takahashi

Apache NemoNAVER Engineering

Berkeley Performance TuningGeorge Ang

Contenu connexe

Tendances

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico

PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico

MesosCon 2018Pablo Delgado

MongodbScott Motte

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종NAVER D2

CuPy v4 and v5 roadmapPreferred Networks

Как PostgreSQL работает с дискомPostgreSQL-Consulting

To Hire, or to train, that is the question (Percona Live 2014)Geoffrey Anderson

XtraDB 5.7: key performance algorithmsLaurynas Biveinis

Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Ontico

Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico

PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016Tomas Vondra

Embulk and Machine Learning infrastructureHiroshi Toyama

PostgreSQL performance archaeologyTomas Vondra

PostgreSQL is the new NoSQL - at Devoxx 2018Quentin Adam

Chainer v4 and v5Preferred Networks

Sun jdk 1.6 gc english versionbluedavy lin

Ops Jumpstart: Admin 101MongoDB

Advanced backup methods (Postgres@CERN)Anastasia Lubennikova

GlusterFS As an Object StorageKeisuke Takahashi

Tendances (20)

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...

PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)

MesosCon 2018

Mongodb

[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종

CuPy v4 and v5 roadmap

Как PostgreSQL работает с диском

To Hire, or to train, that is the question (Percona Live 2014)

XtraDB 5.7: key performance algorithms

Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...

Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...

PostgreSQL na EXT4, XFS, BTRFS a ZFS / FOSDEM PgDay 2016

Embulk and Machine Learning infrastructure

PostgreSQL performance archaeology

PostgreSQL is the new NoSQL - at Devoxx 2018

Chainer v4 and v5

Sun jdk 1.6 gc english version

Ops Jumpstart: Admin 101

Advanced backup methods (Postgres@CERN)

GlusterFS As an Object Storage

Similaire à ソーシャルゲームログ解析基盤のHadoop活用事例

Apache NemoNAVER Engineering

Berkeley Performance TuningGeorge Ang

Parallel Computing for Econometricians with Amazon Web Servicesstephenjbarr

Deploying with JRubyJoe Kutner

3rd meetup - Intro to Amazon EMRFaizan Javed

Cost effective BigData Processing on Amazon EC2Sujee Maniyam

料理を楽しくする画像配信システムIssei Naruta

BedCon 2013 - Java Persistenz-Frameworks für MongoDBTobias Trelle

Putting Kafka Together with the Best of Google Cloud Platform confluent

mongodb tutorialJaehong Park

BDT201 AWS Data Pipeline - AWS re: Invent 2012Amazon Web Services

Lessons learned scaling big data in cloudVijay Rayapati

Antonios Giannopoulos Percona 2016 WiredTiger Configuration VariablesAntonios Giannopoulos

Flume-Cassandra Log ProcessorCLOUDIAN KK

COOKPADでのHadoop利用Tatsuya Sasaki

Introduction To Elastic MapReduce at WHUGAdam Kawa

R Jobs on the CloudJohn Doxaras

KVSの性能、RDBMSのインデックス、更にMapReduceを併せ持つAll-in-One NoSQL: MongoDB Rakuten Group, Inc.

(SDD401) Amazon Elastic MapReduce Deep Dive and Best Practices | AWS re:Inven...Amazon Web Services

Cloud Computing BP-Study 20090319Yukio Andoh

Similaire à ソーシャルゲームログ解析基盤のHadoop活用事例 (20)

Apache Nemo

Berkeley Performance Tuning

Parallel Computing for Econometricians with Amazon Web Services

Deploying with JRuby

3rd meetup - Intro to Amazon EMR

Cost effective BigData Processing on Amazon EC2

料理を楽しくする画像配信システム

BedCon 2013 - Java Persistenz-Frameworks für MongoDB

Putting Kafka Together with the Best of Google Cloud Platform

mongodb tutorial

BDT201 AWS Data Pipeline - AWS re: Invent 2012

Lessons learned scaling big data in cloud

Antonios Giannopoulos Percona 2016 WiredTiger Configuration Variables

Flume-Cassandra Log Processor

COOKPADでのHadoop利用

Introduction To Elastic MapReduce at WHUG

R Jobs on the Cloud

KVSの性能、RDBMSのインデックス、更にMapReduceを併せ持つAll-in-One NoSQL: MongoDB

(SDD401) Amazon Elastic MapReduce Deep Dive and Best Practices | AWS re:Inven...

Cloud Computing BP-Study 20090319

Plus de 知教本間

gumiにおける、海外支社とのAtlassian製品利用事例知教本間

GitHubEnterpriseからBitbucket(Stash) への移行事例知教本間

AWSアカウント開設からインスタンスを立ち上げるまでの作業自動化について知教本間

Use case for using the ElastiCache for Redis in production知教本間

チームでChef serverを運用するには知教本間

Redisへと至る、gumiデータストアの歴史知教本間

ソーシャルゲームのEMR活用事例知教本間

MongoDBざっくり解説知教本間

ソーシャルゲームログ解析基盤のMongoDB活用事例知教本間

Plus de 知教本間 (9)

gumiにおける、海外支社とのAtlassian製品利用事例

GitHubEnterpriseからBitbucket(Stash) への移行事例

AWSアカウント開設からインスタンスを立ち上げるまでの作業自動化について

Use case for using the ElastiCache for Redis in production

チームでChef serverを運用するには

Redisへと至る、gumiデータストアの歴史

ソーシャルゲームのEMR活用事例

MongoDBざっくり解説

ソーシャルゲームログ解析基盤のMongoDB活用事例

Dernier

How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe

TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

UiPath Community: Communication Mining from Zero to HeroUiPathCommunity

A Journey Into the Emotions of Software DevelopersNicole Novielli

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

Scale your database traffic with Read & Write split using MySQL RouterMydbops

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein

Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

How to write a Business Continuity PlanDatabarracks

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

Sample pptx for embedding into website for demoHarshalMandlekar2

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda

The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech

Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal

Dernier (20)

How AI, OpenAI, and ChatGPT impact business and software.

TeamStation AI System Report LATAM IT Salaries 2024

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

UiPath Community: Communication Mining from Zero to Hero

A Journey Into the Emotions of Software Developers

DevEX - reference for building teams, processes, and platforms

Scale your database traffic with Read & Write split using MySQL Router

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24

Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx

How to write a Business Continuity Plan

The State of Passkeys with FIDO Alliance.pptx

Sample pptx for embedding into website for demo

Time Series Foundation Models - current state and future directions

[Webinar] SpiraTest - Setting New Standards in Quality Assurance

So einfach geht modernes Roaming fuer Notes und Nomad.pdf

The Ultimate Guide to Choosing WordPress Pros and Cons

Potential of AI (Generative AI) in Business: Learnings and Insights

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...

ソーシャルゲームログ解析基盤のHadoop活用事例

1. Hadoop gumi 1

2. : Twitter:@CkReal gumi 2 AWS S3,EMR 2

3. 1. 2. Elastic MapReduce EMR 3. 4. EMR 3

4. 4

5. 2 ( 1 ) ﬂuentd 5

6. 1 1,000 / AP APAP AP DB d fluentd fluentd mongos mongod(PRIMARY) DB config mongod(SECONDARY) DB fluentd mongos mongod(SECONDARY) config ReplicaSets & Sharding NFS 6

7. EMR 7

8. 8.5GB 1.4GB / ID Nov 1 23:59:59 hogehoge-ap1 hogehoge ADD_MONEY 12345 [BeforeMoney] 67979 [AfterMoney] 68024 [Money] 45 Nov 1 23:59:59 hogehoge-ap2 hogehoge CONSUME_POWER 12345 [BeforePower] 25 [AfterPower] 20 [ConsumePower] 5 8

9. NFS NFS 9

10. MongoDB MongoDB ( ) "app" : "hogehoge", "userid" : "12345", ID "dateint" : 20111101, "hourint" : 23, "actions" : [ "CONSUME_POWER", MongoDB Sharding "ADD_MONEY" ], "records" : [ "action" : "ADD_MONEY", "timeint" : 235959, ] 10

11. EMR Hive Pig Hadoop Streaming Hadoop Streaming (Python) 11

12. m2.4xlarge × 1 4.9GB 85 EMR(m2.xlarge) × 5 4.9GB 44 m2.4xlarge × 1 7.2GB 138 EMR(m2.xlarge) × 5 7.2GB 69 (Macbook Air) 3.6GB 30 … 12

13. EMR CPU 13

14. 14

15. ( ) NFS Amazon S3 EMR S3 EMR S3 EMR conﬁg MongoDB mongos 15

16. S3 boto S3 c3cmd S3 EMR Mapper,Reducer,Python2.7 MongoDB pymongo MongoDB EMR Client Tool(Ruby) EMR 16

17. EMR 17

18. EMR S3 EC2⇔S3 20MB/sec Hadoop HadoopStreaming EMR 18

19. GB/ 19

20. 20

21. 21

ソーシャルゲームログ解析基盤のHadoop活用事例

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à ソーシャルゲームログ解析基盤のHadoop活用事例

Similaire à ソーシャルゲームログ解析基盤のHadoop活用事例 (20)

Plus de 知教 本間

Plus de 知教 本間 (9)

Dernier

Dernier (20)

ソーシャルゲームログ解析基盤のHadoop活用事例

Notes de l'éditeur

Plus de 知教本間

Plus de 知教本間 (9)