Q4M Microblogcon

Q4M a high-performance message queue for MySQL Cybozu Labs, Inc. Kazuho Oku

Introduction of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Design Goals of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object]

What is a Message Queue? ,[object Object],[object Object],[object Object],[object Object]

Message Queue (cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Minimal Configuration of a MQ ( 最小構成 ) ,[object Object],[object Object],Sender Receiver Queue

MQ and Relays ,[object Object],[object Object],[object Object],[object Object],Sender Receiver Relay Queue Queue

Merits of Message Relays ( リレーの意義 ) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Message Brokers ,[object Object],[object Object],[object Object],[object Object],[object Object]

Q4M as a Message Queue (MQ としての Q4M) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Internal Architecture of Q4M ,[object Object],[object Object],[object Object],[object Object]

Benchmark ,[object Object],[object Object],[object Object],[object Object],$ VAR_LENGTH=504 CONCURRENCY=50 MESSAGES=100000 DBI='dbi:mysql:test;mysql_socket=/tmp/mysql51.sock' DBI_USER=root MYSQL_SOCKET='/tmp/mysql51.sock' USE_C_CLIENT=1 t/05-multirw.t 1..4 ok 1 - check number of messages ok 2 - min value of received message ok 3 - max value of received message ok 4 - should have no rows in table Multi-reader-writer benchmark result: Number of messages: 100000 Number of readers: 50 Elapsed: 13.063 seconds Throughput: 7655.070 mess./sec.

Benchmark (cont.) ,[object Object],[object Object],[object Object]

Sending a Message ,[object Object]

Receiving a Message ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

OWNER Mode and NON-OWNER Mode (cont.) ,[object Object],[object Object],[object Object],[object Object]

Function for Entering OWNER Mode ,[object Object],[object Object],[object Object],[object Object]

queue_wait ,[object Object],[object Object],[object Object]

queue_wait(tbl_cond) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

queue_wait(tbl_cond) (cont.) ,[object Object]

queue_wait(tbl_cond,[tbl_cond…,timeout]) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Functions for Exiting OWNER Mode ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Receiving a Message (revisited) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Restrictions in SQL of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Why no Indexes Support? ,[object Object],[object Object]

Why no Indexes Support? ,[object Object],[object Object],[object Object],[object Object]

Configuration Options of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

My Build Options ,[object Object],[object Object]

Connecting Distant Servers ,[object Object],[object Object],Content Analysis Service Pathtraq DB MySQL conn. over SSL,gzip Queue Queue

User Notifications ,[object Object],[object Object],DB App. Logic SMTP Agent IM Agent Queue(s)

Asynchronous Updates ,[object Object],[object Object],DB App. Logic DB Queue

Scheduling Web Crawlers ,[object Object],[object Object],[object Object],URL DB Spiders Re- scheduler Store Result Read URL If failed to fetch, store URL in retry queue Request Queue Retry Queue

Prioritized Task Scheduling ,[object Object],[object Object],[object Object],Content & Keyword DB Request Queue Keyword Analyzer Store Keywords Read URL ID Read Content URL ID Prio 0 1 0

Prioritized Task Scheduling (cont.) ,[object Object],[object Object],[object Object],Content & Keyword DB Request Queue Keyword Analyzer Store Keywords Read URL ID Read Content URL ID Prio 0 1 0

Q4M Microblogcon

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (7)

En vedette

En vedette (20)

Similaire à Q4M Microblogcon

Similaire à Q4M Microblogcon (20)

Plus de Kazuho Oku

Plus de Kazuho Oku (20)

Dernier

Dernier (9)

Q4M Microblogcon