Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Zeromq anatomy & jeromq
Similaire à Zeromq anatomy & jeromq (20)
Dernier
Dernier (20)
Zeromq anatomy & jeromq
- 2. ZeroMQ MQ 가 아니무니다.. TCP on Steroids
- 3. ZMQ Socket ROUTER DEALER XPUB XSUB PUSH PULLREP REQ PUB SUB
- 5. io_thread_t ZMQ Socket Mailbox Signaler ypipe_t <cmd_t> Session Socket Session Socket poller r fd fd ypipe_t <msg_t> inpipe ypipe_t <msg_t> outpipe ZMQ Socket ZMQ Socket ZMQ Socket ZMQ Socket SessionSessionSession Context Mailbox r
- 6. Msg union { ... struct { unsigned char data [29]; unsigned char size; unsigned char type; unsigned char flags; } vsm; struct { content_t *content; unsigned char unused [29 + 1 - sizeof (content_t*)]; unsigned char type; unsigned char flags; } lmsg; DATA S T F T F S DATA vsm lmsg
- 7. writer thread reader thread YQueue yqueue_t <msg_t, 256> yqueue_t <command_t, 16> struct chunk_t { T values [N]; chunk_t *prev; chunk_t *next; }; atomic_ptr_t <chunk_t> spare_chunk ... begin back • Only single thread can read from begin_chunk • Only single thread can write to yqueue_t <msg_t> to back chunk • Only single thread can write to yqueue_t <command_t> at a time
- 8. YPipe atomic_ptr_t <T> c Reader Writer T* r T* w T* f bool flush () { if ( c.cas ( w, f ) != w ) { c.set ( f ) ; w = f ; return false ; // reader is sleeping } w = f ; return true ; } bool check_read () { if ( queue.front () != r ) return true ; r = c.cas ( queue.front () , NULL ) ; if ( queue.front () == r ) return false ; // nothing to read return true; } read() => queue<T>.pop write() => queue<T>.push r w f
- 9. Signaler socketpair ( AF_UNIX, SOCK_STREAM, 0, sv[2] ) int w = sv[0]; // writer fd int r = sv[1]; // reader fd, Used at IOThread.poll Reader recv () { :: recv ( r , dummy[1] ); } wait ( timeout ) { poll ( r , timeout ); } Writer send () { :: send ( w, dummy[1] ); }
- 10. Mailbox ypipe_t <command_t, 16> cpipe signaler_t signaler Reader Writer recv (timeout) { signaler.wait (timeout) return cpipe.read () } send (cmd) { lock () cpipe.write (cmd) cpipe.flush () unlock () signaler.send () }
- 11. ZMQ Socket ZMQ Socket mailbox_t signaler_ typipe_t <cmd_t> Session Socket Session Socket r fd fd ypipe_t <msg_t> inpipe ypipe_t <msg_t> outpipe • Session is create when connect / accept •Worker thread owns ZMQ socket • IOThread owns Sessions • recv msg from socket and en-queue inpipe • recv msg from outpipe and send to socket • zsocket.send en-queue to outpipe • zsocket.recv de-queue from inpipe if any
- 13. in_event () { // read from socket size = 8192 // *** buffer = decoder . get_buffer (size) ::recv (buffer, size) decoder . process_buffer(buffer) session . flush() } out_event () { // write to socket size = 8192 // *** buffer = encoder . get_data (size) ::send (buffer, size) } msg = parse (buffer) session.write (msg) Stream Engine msg = session.read () buffer.fill (msg)
- 14. Summary inline lock-free well-defined threading model less system calls • malloc / free • bulk socket recv / send • balance copy overhead
- 15. JeroMQ - jeromq.org Hand-made Compatible with ZeroMQ-3 tcp:// inproc:// ipc:// pgm:// same dev experience 95%~ performance
- 16. JeroMQ - diff Java NIO SelectableChannel (Direct) ByteBuffer Signaler Pipe.SourceChannel / Pipe.SinkChannel Mixed with counter YQueue - circular linked list Plain Socket Proxy Persistence - Kafka alike MappedByteBuffer ZeroCopy
- 17. Future Zero Persistence Queue http://github.com/miniway/zper Faster than Kafka 50% Less code Configurable topology Java 7 - AsynchronousChannel
- 18. Conclusion Why reinvent the wheel “The only limitation is your imagination and sobriety”
- 19. Q/A