Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Networking in Java with NIO and Netty

5 627 vues

Publié le

Let's dive under the hood of Java network applications. We plan to have a deep look to classic sockets and NIO having live coding examples. Then we discuss performance problems of sockets and find out how NIO can help us to handle 10000+ connections in a single thread. And finally we learn how to build high load application server using Netty.

https://github.com/kslisenko/java-networking

Publié dans : Logiciels
  • Hallo , how can I connect to Mikrotik API by Netty
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Networking in Java with NIO and Netty

  1. 1. 1CONFIDENTIAL NETWORKING IN JAVA WITH NIO AND NETTY KANSTANTSIN SLISENKA JUN 1, 2017
  2. 2. 2CONFIDENTIAL ABOUT ME Java Backend engineer Speaker at Java Tech Talks, SEC Online, CMCC Tech Talks, IT Week I’m interested in Complex Java backend, SOA, databases High load, fault-tolerant, distributed systems KANSTANTSIN SLISENKA EPAM Systems, Lead Software Engineer
  3. 3. 3CONFIDENTIAL WHY IS IMPORTANT TO DIG INTO JAVA NETWORKING Classic Java project Tomcat + Spring + Hibernate + … High load, latency-sensitive Custom TCP/UDP solution
  4. 4. 4CONFIDENTIAL AGENDA Blocking vs non-blocking resources1 Blocking I/O2 Non-blocking I/O3 Netty: high performance networking framework4
  5. 5. 5CONFIDENTIAL BLOCKING VS NON-BLOCKING RESOURCES client thread read/write response Blocking resources Database transactions, RPC/WS calls blocked
  6. 6. 6CONFIDENTIAL BLOCKING VS NON-BLOCKING RESOURCES client thread read/write response client thread read/write nope read/write response client thread read/write callback with response read/write nope register callback Blocking resources Non-blocking resources Database transactions, RPC/WS calls Queues, WebSockets, Non-blocking data storage blocked
  7. 7. 7CONFIDENTIAL NETWORK SOCKETS Socket is blocking API over OS and Network hardware not a physical object PERMANENT KNOWN PORT ANY UNUSED PORT SOCKET SOCKET Max ports = 65535
  8. 8. 8CONFIDENTIAL JAVA SOCKETS = BLOCKING IO Client Server
  9. 9. 9CONFIDENTIAL SOCKET SERVER LIVE CODING EXAMPLE
  10. 10. 10CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) *until keep alive timeout Fixed thread pool Requests queue
  11. 11. 11CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) Client 4 *until keep alive timeout Fixed thread pool Requests queue new connection
  12. 12. 12CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) Client 4 *until keep alive timeout Fixed thread pool Requests queue new connection drop by timeout
  13. 13. 13CONFIDENTIAL SOCKET SERVER CONCLUSION 1. Ineffective thread utilization – many threads in waiting/blocked state – extra memory for stack and thread local resources 2. Poor performance – context switching overhead 3. Limited number of connections – when thread pool is fully consumed Benefits Drawbacks 1. Simple development well known Input/OutputStream API 2. No check for partially received messages thread is blocked until fully reads message
  14. 14. 14CONFIDENTIAL IO VS NIO IO (blocking) Thread Thread Thread socket socket socket BLOCKING READ/WRITE WAIT WAIT WAITWAIT
  15. 15. 15CONFIDENTIAL IO VS NIO IO (blocking) NIO (non-blocking) NON BLOCKING READ/WRITE Thread buffer buffer buffer socket channel socket channel socket channel Thread Thread Thread socket socket socket BLOCKING READ/WRITE WAIT WAIT WAITWAIT ALWAYS RUNNING NOTIFICATIONS selector
  16. 16. 16CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector socket channel
  17. 17. 17CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector socket channel 1 WRITE READY
  18. 18. 18CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector PUT DATA2 socket channel 1 WRITE READY
  19. 19. 19CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector PUT DATA2 SEND DATA3 socket channel 1 WRITE READY
  20. 20. 20CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector socket channel
  21. 21. 21CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector RECEIVE DATA1 socket channel
  22. 22. 22CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector RECEIVE DATA1 socket channel 2 READ READY
  23. 23. 23CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector GET DATA3 RECEIVE DATA1 socket channel 2 READ READY
  24. 24. 24CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 Empty buffer BYTE ARRAY
  25. 25. 25CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 LIMIT CAPACITY Empty buffer Writing mode BYTE ARRAY BYTE ARRAY POSITION 0 1 2 3 4 5 6 7 8 9
  26. 26. 26CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 LIMIT CAPACITY 1 0 1 1 0 POSITION LIMIT CAPACITY Empty buffer Writing mode Reading mode BYTE ARRAY BYTE ARRAY BYTE ARRAY buffer.flip()buffer.clear() POSITION 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
  27. 27. 27CONFIDENTIAL DIRECT BUFFERS 0 1 2 3 4 5 6 7 8 JVM Network Interface Card 0 1 2 3 4 5 0 1 2 3 4 5 Physical memory 9 Shared memory address space Shared memory address space Memory should be continuous because NICs are block- oriented devices
  28. 28. 28CONFIDENTIAL NIO SERVER LIVE CODING EXAMPLE
  29. 29. 29CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
  30. 30. 30CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
  31. 31. 31CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer get ready channels socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
  32. 32. 32CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress get buffer put read write get ready channels socket channel #2 socket channel #1 Business logic WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer
  33. 33. 33CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE new connection buffer Client #2 Client #4 Client #1 no data data transfer in progress new connection socket channel #3 Client #3 data transfer in progress get buffer put register in selector read write get ready channels socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD Business logic buffer
  34. 34. 34CONFIDENTIAL CAN WE GO TO PRODUCTION WITH A SINGLE THREAD? https://www.pinterest.com/banjolele/one-man-band/
  35. 35. 35CONFIDENTIAL PRODUCTION-READY NIO SERVER new connection
  36. 36. 36CONFIDENTIAL SELECTOR PRODUCTION-READY NIO SERVER ServerSocket Channel acceptor thread new connection Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded
  37. 37. 37CONFIDENTIAL SELECTOR SELECTOR PRODUCTION-READY NIO SERVER ServerSocket Channel selector thread … acceptor thread Socket Channel SELECTOR selector threadSocket Channel SELECTOR selector threadSocket Channel new connection Selector threads Multiple processing threads to get benefit from multi core CPU Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded
  38. 38. 38CONFIDENTIAL SELECTOR SELECTOR Application thread pools (long running tasks) PRODUCTION-READY NIO SERVER ServerSocket Channel selector thread … worker thread worker thread worker thread application thread acceptor thread Socket Channel SELECTOR selector threadSocket Channel SELECTOR selector threadSocket Channel new connection Selector threads Multiple processing threads to get benefit from multi core CPU Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded Application threads For long tasks (back-end, DB, computation), to avoid blocking of selector threads
  39. 39. 39CONFIDENTIAL PARTIAL MESSAGE RECEIVING PROBLEM Message reader splits/joins data blocks into messages Data block Message Message Message Message Message Message message reader Data block Data block MessageMessage Data block
  40. 40. 40CONFIDENTIAL NIO CONCLUSION Complexity • complex code • learning curve for developers • handling partially received messages • combining async with sync code (need different thread groups) Benefits Drawbacks Single shared thread for many connections • less memory • less context switching overhead
  41. 41. 41CONFIDENTIAL NETTY ONE FRAMEWORK TO RULE THEM ALL
  42. 42. 42CONFIDENTIAL NETTY: JAVA NETWORK FRAMEWORK Some projects using Netty
  43. 43. 43CONFIDENTIAL NETTY SERVER ARCHITECTURE Network
  44. 44. 44CONFIDENTIAL transport Boss EventLoopGroup NIO NETTY SERVER ARCHITECTURE Network Socket Native JNI Thread pool for new connections
  45. 45. 45CONFIDENTIAL transport Boss EventLoopGroup channel channel pipeline Worker EventLoopGroup EventExecutorGroup NIO NETTY SERVER ARCHITECTURE Inbound Handler Inbound Handler Inbound Handler Outbound Handler Outbound Handler Outbound Handler Network Socket Native JNI Separate thread pool for blocking logic I/O thread pool For non-blocking logic Thread pool for new connections
  46. 46. 46CONFIDENTIAL channel channel pipeline NETTY HANDLERS SSL Handler encrypted encrypted
  47. 47. 47CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder SSL Handler encrypted encrypted bytes
  48. 48. 48CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder HTTP Object Aggregator SSL Handler encrypted encrypted bytes http messages buffer
  49. 49. 49CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder HTTP Object Aggregator SSL Handler encrypted encrypted bytes http messages Your Business Logic Handler full http messages buffer
  50. 50. 50CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder HTTP Object Aggregator HTTP Request Encoder SSL Handler encrypted encrypted bytes http messages bytes Your Business Logic Handler full http messages buffer full http messages
  51. 51. 51CONFIDENTIAL NETTY SERVER LIVE CODING EXAMPLE
  52. 52. 52CONFIDENTIAL WHAT IS FASTER: IO OR NIO?
  53. 53. 53CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * ? ? ? ? NIO is faster Context switching is very slow Threads take up too much memory Thread synchronization will kill you
  54. 54. 54CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? ? ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster
  55. 55. 55CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster
  56. 56. 56CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster Don’t overuse thread-local objects!
  57. 57. 57CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster Don’t overuse thread-local objects! Use non-blocking data structures
  58. 58. 58CONFIDENTIAL CONCLUSION • Huge number of connections • Medium or slow data rate per connection • Asynchronous world (event-based) • Non blocking database • Queues • Web-sockets • Callbacks, listeners • Data streaming • Don’t use standard NIO, use frameworks like Netty • Highly skilled team because of complex code Use sockets* Use NIO* *Not a strict rule, always point to holy-wars • Small number of connections • Huge data rate per connection • Synchronous world (procedure-based) • Classic enterprise applications • Request-response protocols • Blocking database • Blocking RPC or web-service calls • Bottleneck is not server, but blocking resource • Make servers stateless and scale horizontally
  59. 59. 59CONFIDENTIAL Java Network Programming, 4’th edition O’REILLY Elliotte Rusty Harold BUY A BOOK, MAKE THEM RICH Netty in Action Manning Norman Maurer Java NIO O’REILLY Ron Hitchens • Code examples: github.com/kslisenko/java-networking • NIO performance: www.mailinator.com/tymaPaulMultithreaded.pdf • NIO tutorial: tutorials.jenkov.com/java-nio/index.html • ITT 2015 - Heinz Kabutz - The Multi-threading, Non Blocking IO: youtube.com/watch?v=uKc0Gx_lPsg • Netty - One Framework to rule them all by Norman Maurer: youtube.com/watch?v=DKJ0w30M0vg • Scalable IO in Java: http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf
  60. 60. 60CONFIDENTIAL THANK YOU! QUESTIONS? kslisenko@gmail.com kslisenko linkedin.com/in/kslisenko/ Konstantin Slisenko kanstantsin_slisenka@epam.com

×