call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
Microservices Application Tracing Standards and Simulators - Adrians at OSCON
1. Microservices Application Tracing Standards
and Simulators
From Zipkin to Greater Tracing: Involving a wider group of people in
distributed tracing
@adrianfcole
@adrianco
#oscon
5. Distributed Tracing commoditizes knowledge
Distributed tracing systems collect end-to-end latency graphs
(traces) in near real-time.
You can compare traces to understand why certain requests
take longer than others.
6. Zipkin is like Chrome DevTool’s network panel!
http://zipkin.io/
• .. except you see your whole architecture
8. commit 92c941890c2009a401b777093342dc4f28955640
Author: Johan Oskarsson <johan@oskarsson.nu>
Date: Tue Nov 15 10:09:47 2011 -0800
[split] Enable B3 tracing for TFE. Filter out finagle-http headers from incoming requests
BigBrotherBird is silently born
9. Zipkin is less silently born
commit 2b7acead044e71c744f39804abe564383eb5f846
Author: Johan Oskarsson <johan@oskarsson.nu>
Date: Wed Jun 6 11:28:34 2012 -0700
Initial commit
12. So what happened?
Zipkin development at Twitter was in short bursts, centered on other work
Many experienced Zipkin engineers don’t work at Twitter (or in the Bay Area)
Platform diversity is a reality for many
Having the same goals was our opportunity
13. How’s OpenZipkin doing now?
Zipkin's now releasable (maybe too releasable)
We’re working on understandability on usability
We’re making the community easier to find
We hit bumps, and sometimes reverse change
15. The “greater” tracing
Many groups are solving similar problems
Some focus on stacks, others on instrumentation
By collaborating more, we can make tracing greater
16. Instrumentation portability
Interop through shared trace pipelines.
Practical matters, like categorization and tactical designs
Moving R&D to implementation
Simulation and system testing
distributed-tracing google group
Distributed Tracing Workgroup
17. OpenTracing is an effort to clean-up and de-risk
distributed tracing instrumentation
OpenTracing Interfaces decouple instrumentation from
vendor-specific dependencies and terminology. This
allows applications to switch products with less effort.
http://opentracing.io/
OpenTracing: Go, Python, Java, JavaScript..
18. A single configuration change to bind a Tracer
implementation in main() or similar
import opentracing "github.com/opentracing/opentracing-go"
import "github.com/tracer_x/tracerimpl"
func main() {
// Bind tracerimpl to the opentracing system
opentracing.InitGlobalTracer(
tracerimpl.New(kTracerImplAccessToken))
... normal main() stuff ...
}
How does it work?
Clean, vendor-neutral instrumentation code that
naturally tells the story of a distributed operation
import opentracing "github.com/opentracing/opentracing-go"
func AddContact(c *Contact) {
sp := opentracing.StartSpan("AddContact")
defer sp.Finish()
sp.LogEventWithPayload("Added contact: ", *c)
subRoutine(sp, ...)
...
}
func subRoutine(parentSpan opentracing.Span, ...) {
...
sp := opentracing.StartChildSpan(parentSpan, "subRoutine")
defer sp.Finish()
sp.Info("deferred work to subroutine")
...
}
Thanks, @el_bhs for
the slide!
19. Pivot Tracing is applied research from Brown University
(the one that brought us X-Trace).
Pivot tracing allows you to dynamically query systems at
runtime, grouping on “Baggage” which propagates across
service boundaries.
pivottracing.io
Pivot Tracing
20. Start writing queries including the fancy
happened-before join
From incr In DataNodeMetrics.incrBytesRead
Join cl In First(ClientProtocols) On cl -> incr
GroupBy cl.procName
Select cl.procName, SUM(incr.delta)
How does it work?
Services need to be in Java and be able to talk to
a provided PubSub broker.
// Add a library
<dependency>
<groupId>edu.brown.cs.systems</groupId>
<artifactId>pivottracing-agent</artifactId>
<version>4.0</version>
</dependency>
// Initialize it on bootstrap
PivotTracing.initialize(); @brownsys_jmace
made this!
22. What does @adrianco do?
@adrianco
Technology Due
Diligence on Deals
Presentations at
Conferences
Presentations at
Companies
Technical
Advice for Portfolio
Companies
Program
Committee for
Conferences
Networking with
Interesting PeopleTinkering with
Technologies
Maintain
Relationship with
Cloud Vendors
Previously: Netflix, eBay, Sun Microsystems, CCL, TCU London BSc Applied Physics
23. @adrianco
Testing Flow Monitors
Monitoring tools often “explode on impact” with
real world use cases at scale
Interestingly large complex environments are
expensive to create or hard to get access to
Free, open source tools don’t have a budget…
24. OSS Microservice Simulator
Model and visualize microservices
Simulate interesting architectures
Generate large scale configurations
Stress test real tools like Zipkin
Code: github.com/adrianco/spigo
Simulate Protocol Interactions in Go
Visualize with D3, Neo4j or Guesstimate
See for yourself: http://simianviz.surge.sh
Follow @simianviz for updates
ELB Load Balancer
Zuul
API Proxy
Karyon
Business Logic
Staash
Data Access Layer
Priam
Cassandra Datastore
Three
Availability
Zones
Denominator
DNS Endpoint
25. POST Spigo flows to zipkin
# collect flows, duration 2 seconds, architecture lamp
$ ./spigo -c —d 2 -a lamp
—snip—
# clean out zipkin database and post newly created data
$ misc/zipkin.sh lamp
26. Spigo Nanoservice Structure
func Start(listener chan gotocol.Message) {
...
for {
select {
case msg := <-listener:
flow.Instrument(msg, name, hist)
switch msg.Imposition {
case gotocol.Hello: // get named by parent
...
case gotocol.NameDrop: // someone new to talk to
...
case gotocol.Put: // upstream request handler
...
outmsg := gotocol.Message{gotocol.Replicate, listener, time.Now(),
msg.Ctx.NewParent(), msg.Intention}
flow.AnnotateSend(outmsg, name)
outmsg.GoSend(replicas)
}
case <-eurekaTicker.C: // poll the service registry
...
}
}
}
Skeleton code for sideways replicating a Put message
Instrument incoming requests
Instrument outgoing requests
update trace context
32. Migrating to Microservices
See for yourself: http://simianviz.surge.sh/migration
Endpoint
ELB
PHP
MySQL
MySQL
Next step Controls node
placement distance
Select models
36. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
See for yourself: http://simianviz.surge.sh/riak
37. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Load Balancer
Load Balancer
See for yourself: http://simianviz.surge.sh/riak
38. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
39. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
40. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Ingest Message Queue
Load Balancer
Load Balancer
Stream Service
Analytics Service
See for yourself: http://simianviz.surge.sh/riak
41. Single Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
Load Balancer
Normalization Services
Enrich Message Queue
Riak KV
Enricher Services
Ingest Message Queue
Load Balancer
Load Balancer
Stream Service Riak TS
Analytics Service
Ingester Service
See for yourself: http://simianviz.surge.sh/riak
42. Two Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
East Region Ingestion
West Region Ingestion
Multi Region TS Analytics
See for yourself: http://simianviz.surge.sh/riak
46. @adrianco
Conclusion
Monitoring tools can be stressed at scale by
simulating their inputs with metrics, dependency
graphs and flows
Spigo can be extended to produce any format
output at very large scale from a laptop