Zipkin is a distributed tracing system that helps us gather timing data for all the disparate services at Twitter, and manages collection and lookup of data through a Collector and a Query service. With Zipkin, we can trace a subset of all requests made to the site, and collect detailed data about the path taken through our systems, as well as timings. Then, we can visualize and ultimately pinpoint where and possibly why a response took longer than expected.

1. Z I P K I N
A distributed tracing framework

2. Why Zipkin?

3. Google
0.5 sec slower
20% traffic drop
@skr | @thisisfranklin 3

4. Google
0.5 sec slower 1.5 sec faster
20% traffic drop CTR up 12%
@skr | @thisisfranklin 3

5. Performance matters

6. Front end
@skr | @thisisfranklin 5

7. Front end Back end
@skr | @thisisfranklin 5

8. @skr | @thisisfranklin 6

9. @skr | @thisisfranklin 6

10. @skr | @thisisfranklin 7

11. @skr | @thisisfranklin 8
Picture from http://www.flickr.com/photos/jpellgen

12. @skr | @thisisfranklin 9

13. • Collects traces from production requests
• Low overhead
• Minimum of extra work for developers
@skr | @thisisfranklin 9

14. Finagle
“Finagle is an asynchronous network stack for the JVM that you can use to build
asynchronous Remote Procedure Call (RPC) clients and servers in Java, Scala, or
any JVM-hosted language.”
github.com/twitter/finagle

15. What to capture?

16. @skr | @thisisfranklin 12

17. Finagle Http service
@skr | @thisisfranklin 12

18. Finagle Http service
@skr | @thisisfranklin 12

19. Finagle Http service
Finagle Thrift Service
@skr | @thisisfranklin 12

20. Finagle Http service
Finagle Thrift Service
@skr | @thisisfranklin 12

21. Finagle Http service
Finagle Thrift Service
@skr | @thisisfranklin 12

22. Finagle Http service
Finagle Thrift Service Service
@skr | @thisisfranklin 12

23. Finagle Http service
Finagle Thrift Service Service
@skr | @thisisfranklin 12

24. Finagle Http service
Finagle Thrift Service Service
@skr | @thisisfranklin 12

25. Zipkin terminology
‣ Annotation: string data associated with a particular timestamp, service, and
host
Time
time: 2012-01-21 22:37:01
value: “something happened”
server: 135.34.53.2
service: “timelineservice”
@skr | @thisisfranklin 13

26. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send
Span
@skr | @thisisfranklin 14

27. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send
Span
T:10ms Server Receive
@skr | @thisisfranklin 14

28. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send
Span
T:10ms Server Receive T:90ms Server Send
@skr | @thisisfranklin 14

29. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send T:100ms Client Receive
Span
T:10ms Server Receive T:90ms Server Send
@skr | @thisisfranklin 14

30. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send T:100ms Client Receive
Span
T:20ms Read 30 kbytes from file
T:10ms Server Receive T:90ms Server Send
@skr | @thisisfranklin 14

31. ‣ Span: represents one specific method call; made up of a set of annotations.
Has a name and an id.
Time
T:0ms Client Send T:100ms Client Receive
Span
T:20ms Read 30 kbytes from file
T:10ms Server Receive T:90ms Server Send
‣ Trace: a set of spans all associated with the same request
@skr | @thisisfranklin 14

32. Finagle
http
service
@skr | @thisisfranklin 15

33. • Generate a random i64 trace id
Finagle
http
service
@skr | @thisisfranklin 15

34. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http
service
@skr | @thisisfranklin 15

35. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http
service
Finagle
thrift
service
@skr | @thisisfranklin 15

36. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http
service
• Generate new span id
Finagle
thrift
service
@skr | @thisisfranklin 15

37. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http
service
• Generate new span id
• Pass trace header
Finagle
thrift
service
@skr | @thisisfranklin 15

38. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http struct RequestHeader {
service i64 trace_id,
i64 span_id,
• Generate new span id optional i64 parent_span_id,
• Pass trace header
optional bool sampled
}
Finagle
thrift
service
@skr | @thisisfranklin 15

39. • Generate a random i64 trace id
• Decide if we should sample the
trace or not Finagle
http struct RequestHeader {
service i64 trace_id,
i64 span_id,
• Generate new span id optional i64 parent_span_id,
• Pass trace header
optional bool sampled
}
Finagle
• Thrift service adopts trace id from thrift
header if it exists service
@skr | @thisisfranklin 15

40. Finagle
http
service
Finagle
thrift
service
@skr | @thisisfranklin 15

41. Finagle
http
service
Finagle Finagle
thrift thrift
service service
@skr | @thisisfranklin 15

42. Finagle
http
service
Finagle Finagle
thrift thrift
service service
Finagle
thrift
service
@skr | @thisisfranklin 15

43. Finagle
http
service
S
Finagle Finagle
thrift thrift
S
service service
S
Finagle
thrift
S
service
@skr | @thisisfranklin 15

44. Finagle
http
service
S
Zipkin
collector
Finagle Finagle
thrift thrift
S
service service
S
Cassandra
Finagle
thrift
S
service
@skr | @thisisfranklin 15

45. Finagle
http
service
S
Zipkin
collector
Finagle Finagle
thrift thrift
S
service service
S Zipkin Zipkin
Cassandra
Query UI
Finagle
thrift
S
service
@skr | @thisisfranklin 15

46. Finagle ♥ Zipkin

47. Finagle Thrift server
ServerBuilder()
.bindTo(address)
.codec(ThriftServerFramedCodec())
.name("servicename")
.build(someService)
@skr | @thisisfranklin 17

48. Finagle Thrift server
ServerBuilder()
.bindTo(address)
.codec(ThriftServerFramedCodec())
.name("servicename")
.build(someService)
@skr | @thisisfranklin 17

49. Finagle Thrift server
ServerBuilder()
.bindTo(address)
.codec(ThriftServerFramedCodec())
.name("servicename")
.build(someService)
@skr | @thisisfranklin 17

50. Finagle Thrift server
ServerBuilder()
.bindTo(address)
.codec(ThriftServerFramedCodec())
.name("servicename")
.build(someService)
@skr | @thisisfranklin 17

51. ServerBuilder()
.bindTo(address)
.codec(ThriftServerFramedCodec())
.name("servicename")
.tracerFactory(ZipkinTracer())
.build(someService)
@skr | @thisisfranklin 18

52. ClientBuilder()
.cluster(hosts)
.codec(ThriftClientFramedCodec())
.name("clientname")
.tracerFactory(ZipkinTracer())
.build(someService)
@skr | @thisisfranklin 19

53. Trace.record("doing stuff")
@skr | @thisisfranklin 20

54. Trace.record("doing stuff")
@skr | @thisisfranklin 20

55. Trace.record("doing stuff")
time: 2012-01-21 22:37:01
value: “doing stuff”
server: 135.34.53.2
service: “timelineservice”
@skr | @thisisfranklin 20

56. Trace.recordBinary("key", data)
@skr | @thisisfranklin 21

57. Key Value
Trace.recordBinary("key", data) responsecode 500
cache:somekey Hit
sql.query select *...
@skr | @thisisfranklin 21

58. Platform Protocol Client Server
Finagle Thrift Yes Yes
Finagle HTTP Yes Yes
Finagle Memcache Yes No
Finagle Redis Yes No
Cassie Thrift Yes No
Querulous JDBC Yes No
Ruby Thrift Yes Yes
@skr | @thisisfranklin 22

59. Zipkin UI

60. @skr | @thisisfranklin 24

61. @skr | @thisisfranklin 24

62. @skr | @thisisfranklin 24

65. What did we find?

66. github.com/twitter/zipkin
@zipkinproject
@skr | @thisisfranklin 27