Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
​ Joel Koshy
​ Sr. Staff Software Engineer
​ LinkedIn
Outline
Background
A look back at Kafka 0.7
Kafka 0.8, 0.9 and shifting bottlenecks
Kafka 0.10 message format changes
Mirr...
Kafka at LinkedIn
Kafka 0.7
Kafka 0.7 message format (uncompressed)
Kafka 0.7 message format (compressed)
Kafka 0.7 message format
Message offset is
the physical offset
Kafka 0.7 message format
Internal messages of compressed
message sets are not addressable
via a scalar offset
Kafka 0.7 message format
Consumer checkpoints offset M
for this message
•  Tricky to checkpoint within
compressed message set
•  Hard to rewind by N
messages
•  Unsuitable for features
such as l...
B U T ! ! ! 
Very efficient
(broker did not need to modify messages)
Predominantly network in 0.7
S H I F T I N G B O T T L E N E C K S O V E R T I M E
Kafka 0.8
Replication
Replication in Kafka 0.8
Replication in Kafka 0.8
S H I F T I N G B O T T L E N E C K S O V E R T I M E 
Predominantly network in 0.7
Still network in 0.8, gradually tiltin...
Kafka 0.8 message format (uncompressed)
NOT a
physical offset!
Kafka 0.8 message format (compressed)
Envelope offset is the
largest offset in the set
Broker now needs to assign logical offsets
even to internal messages
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Kafka 0.9
Security (and many other features)
SSL
•  Forego zero-copy optimization
•  CPU overhead to decrypt/encrypt
•  Minor impact
•  (Used only on our mirroring pip...
Mirroring topology and storage policy changes
Mirroring topology and storage policy changes
Mirroring topology and storage policy changes
•  Reduced retention period
across the board
•  File system tuning
2 0 1 5 - 2 0 1 6 
New hardware
10Gbps NICs, bigger disks, XFS
S H I F T I N G B O T T L E N E C K S O V E R T I M E 
Predominantly network in 0.7
Storage and network in 0.8, 0.9 (1Gbps...
Kafka 0.9 broker profile
Kafka 0.9 broker profile
Cluster expansion
Kafka 0.10
New message format
Kafka 0.10 message format (uncompressed)
Kafka 0.10 message format (compressed + append time)
Kafka 0.10 message format (compressed + create time)
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Handling produce requests
Handling fetch requests
Handling fetch requests
Handling fetch requests
Handling fetch requests
Handling fetch requests
Migrate clients before switching to 0.10 message format
Ideal Less ideal Worse Worst
Majority
producer version
0.10 0.9 0....
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
Migrating to the new message format
U S E C A U T I O N ! ! 
Severe performance degradation with older clients
and there is no roll-back after switching
So know your clients!
•  Useful to have a shepherding
system in your service infra
•  EOL older libraries
•  Check API ver...
Kafka 0.10 broker profile
Impact of new message format
•  Fine-grained time-based offset
lookup
•  Facilitates correctness of time-
based retention
Other benefits of
0.10 format
“Time” under 0.9 message format
Message time is mtime of
containing segment so time
granularity is at segment-level
“Time” under 0.9 message format
Time resets on
partition reassignment
Time in 0.10 message format
Time indexes built from
timestamps in messages
Time in 0.10 message format
Mirroring pipelines
(embracing the 0.7 way)
0.7 mirror maker was a dumb fast pipe
0.7 mirror maker was a dumb fast pipe
0.7 mirror maker was a dumb fast pipe
0.7 mirror maker was a dumb fast pipe
… limited only by network
0 . 7 M I R R O R M A K E R
0.8+ mirror maker
•  Needs to preserve order of
keyed messages
0.8+ mirror maker
•  Needs to preserve order of
keyed messages
0.8+ mirror maker
•  Needs to preserve order of
keyed messages
0.8+ mirror maker
•  Needs to preserve order of
keyed messages
•  0.8+ consumers do not support
shallow iteration (KAFKA-7...
Kafka 0.8+ mirror maker profile
Handling keyed messages in pass-through mode
•  Need to preserve order of keyed messages…
but pass-through mirror maker ca...
Handling keyed messages in pass-through mode
•  Need to preserve order of keyed messages…
but pass-through mirror maker ca...
•  Restore shallow iteration in consumer
(KAFKA-1895)
•  “Todd’s trick” – introduce an identity
compression codec in produ...
•  Restore shallow iteration in consumer
(KAFKA-1895)
•  “Todd’s trick” – introduce an identity
compression codec in produ...
•  Jiangjie Qin (KIP-3[1,2,3])
•  Todd Palino (pass-through mirroring)
•  Kafka open source community
Acknowledgments
+
Redlining Kafka Pipelines
Prochain SlideShare
Chargement dans…5
×

Redlining Kafka Pipelines

LinkedIn’s deployment of Kafka and its use cases have grown tremendously over the last couple of years. Given our large scale deployments, we keep careful watch on performance and capex. In this talk we will take a close look at some of the shifting performance bottlenecks and cost considerations that we have had to grapple with over the years, and how newer features in Kafka 0.10 and hardware improvements have helped address these.

  • Soyez le premier à commenter

Redlining Kafka Pipelines

  1. 1. ​ Joel Koshy ​ Sr. Staff Software Engineer ​ LinkedIn
  2. 2. Outline Background A look back at Kafka 0.7 Kafka 0.8, 0.9 and shifting bottlenecks Kafka 0.10 message format changes Mirroring – the 0.7 way Q&A
  3. 3. Kafka at LinkedIn
  4. 4. Kafka 0.7
  5. 5. Kafka 0.7 message format (uncompressed)
  6. 6. Kafka 0.7 message format (compressed)
  7. 7. Kafka 0.7 message format Message offset is the physical offset
  8. 8. Kafka 0.7 message format Internal messages of compressed message sets are not addressable via a scalar offset
  9. 9. Kafka 0.7 message format Consumer checkpoints offset M for this message
  10. 10. •  Tricky to checkpoint within compressed message set •  Hard to rewind by N messages •  Unsuitable for features such as log compaction Drawbacks of 0.7 message format
  11. 11. B U T ! ! ! Very efficient (broker did not need to modify messages)
  12. 12. Predominantly network in 0.7 S H I F T I N G B O T T L E N E C K S O V E R T I M E
  13. 13. Kafka 0.8 Replication
  14. 14. Replication in Kafka 0.8
  15. 15. Replication in Kafka 0.8
  16. 16. S H I F T I N G B O T T L E N E C K S O V E R T I M E Predominantly network in 0.7 Still network in 0.8, gradually tilting toward storage
  17. 17. Kafka 0.8 message format (uncompressed) NOT a physical offset!
  18. 18. Kafka 0.8 message format (compressed) Envelope offset is the largest offset in the set
  19. 19. Broker now needs to assign logical offsets even to internal messages
  20. 20. Handling produce requests
  21. 21. Handling produce requests
  22. 22. Handling produce requests
  23. 23. Handling produce requests
  24. 24. Kafka 0.9 Security (and many other features)
  25. 25. SSL •  Forego zero-copy optimization •  CPU overhead to decrypt/encrypt •  Minor impact •  (Used only on our mirroring pipelines at the time) Kafka 0.9
  26. 26. Mirroring topology and storage policy changes
  27. 27. Mirroring topology and storage policy changes
  28. 28. Mirroring topology and storage policy changes •  Reduced retention period across the board •  File system tuning
  29. 29. 2 0 1 5 - 2 0 1 6 New hardware 10Gbps NICs, bigger disks, XFS
  30. 30. S H I F T I N G B O T T L E N E C K S O V E R T I M E Predominantly network in 0.7 Storage and network in 0.8, 0.9 (1Gbps NICs) Increasingly CPU in 0.9 (10Gbps NICs)
  31. 31. Kafka 0.9 broker profile
  32. 32. Kafka 0.9 broker profile Cluster expansion
  33. 33. Kafka 0.10 New message format
  34. 34. Kafka 0.10 message format (uncompressed)
  35. 35. Kafka 0.10 message format (compressed + append time)
  36. 36. Kafka 0.10 message format (compressed + create time)
  37. 37. Handling produce requests
  38. 38. Handling produce requests
  39. 39. Handling produce requests
  40. 40. Handling produce requests
  41. 41. Handling produce requests
  42. 42. Handling produce requests
  43. 43. Handling produce requests
  44. 44. Handling produce requests
  45. 45. Handling fetch requests
  46. 46. Handling fetch requests
  47. 47. Handling fetch requests
  48. 48. Handling fetch requests
  49. 49. Handling fetch requests
  50. 50. Migrate clients before switching to 0.10 message format Ideal Less ideal Worse Worst Majority producer version 0.10 0.9 0.10 0.9 Majority consumer version 0.10 0.10 0.9 0.9
  51. 51. Migrating to the new message format
  52. 52. Migrating to the new message format
  53. 53. Migrating to the new message format
  54. 54. Migrating to the new message format
  55. 55. Migrating to the new message format
  56. 56. Migrating to the new message format
  57. 57. Migrating to the new message format
  58. 58. Migrating to the new message format
  59. 59. Migrating to the new message format
  60. 60. Migrating to the new message format
  61. 61. Migrating to the new message format
  62. 62. Migrating to the new message format
  63. 63. Migrating to the new message format
  64. 64. Migrating to the new message format
  65. 65. U S E C A U T I O N ! ! Severe performance degradation with older clients and there is no roll-back after switching
  66. 66. So know your clients! •  Useful to have a shepherding system in your service infra •  EOL older libraries •  Check API versions in public access logs •  Add API version metrics to the Kafka broker PRODUCERS CONSUMERS
  67. 67. Kafka 0.10 broker profile
  68. 68. Impact of new message format
  69. 69. •  Fine-grained time-based offset lookup •  Facilitates correctness of time- based retention Other benefits of 0.10 format
  70. 70. “Time” under 0.9 message format Message time is mtime of containing segment so time granularity is at segment-level
  71. 71. “Time” under 0.9 message format Time resets on partition reassignment
  72. 72. Time in 0.10 message format Time indexes built from timestamps in messages
  73. 73. Time in 0.10 message format
  74. 74. Mirroring pipelines (embracing the 0.7 way)
  75. 75. 0.7 mirror maker was a dumb fast pipe
  76. 76. 0.7 mirror maker was a dumb fast pipe
  77. 77. 0.7 mirror maker was a dumb fast pipe
  78. 78. 0.7 mirror maker was a dumb fast pipe
  79. 79. … limited only by network 0 . 7 M I R R O R M A K E R
  80. 80. 0.8+ mirror maker •  Needs to preserve order of keyed messages
  81. 81. 0.8+ mirror maker •  Needs to preserve order of keyed messages
  82. 82. 0.8+ mirror maker •  Needs to preserve order of keyed messages
  83. 83. 0.8+ mirror maker •  Needs to preserve order of keyed messages •  0.8+ consumers do not support shallow iteration (KAFKA-732) •  0.8+ producers do not support pass-through mode
  84. 84. Kafka 0.8+ mirror maker profile
  85. 85. Handling keyed messages in pass-through mode •  Need to preserve order of keyed messages… but pass-through mirror maker cannot repartition
  86. 86. Handling keyed messages in pass-through mode •  Need to preserve order of keyed messages… but pass-through mirror maker cannot repartition •  Work around is to require identical partition counts across all clusters and do identity partitioning •  i.e., Pinput= Poutput
  87. 87. •  Restore shallow iteration in consumer (KAFKA-1895) •  “Todd’s trick” – introduce an identity compression codec in producer •  Uniform partition counts across clusters 0.10 pass-through mirroring how-to
  88. 88. •  Restore shallow iteration in consumer (KAFKA-1895) •  “Todd’s trick” – introduce an identity compression codec in producer •  Uniform partition counts across clusters •  … and a few more subtleties (future talk) 0.10 pass-through mirroring how-to
  89. 89. •  Jiangjie Qin (KIP-3[1,2,3]) •  Todd Palino (pass-through mirroring) •  Kafka open source community Acknowledgments
  90. 90. +

×