Tracing and Profiling of App Deployed on Kubernetes

1. Tracing and Profiling App
Deployed on Kubernetes
Pradipta Banerjee (@pradipta_kr, github.com/bpradipt,
www.cloudgeekz.com )
Hemant Shaw (github.com/hkshaw1990)
IBM Systems and Development Labs

2. Agenda
• Sample Application
• DistributedTracers
• Advanced Concepts

3. Sample Application
Front
End
User
Service
Catalog
Service
Shipping
Service
User
DB
Catalog
DB
Shipping
DB
O
Order
Service
Cart
Service
Cart
DB
Payment
Service

4. What can go wrong ?

5. Sample Application
Front
End
User
Service
Catalog
Service
Shipping
Service
User
DB
Catalog
DB
Shipping
DB
O
Order
Service
Cart
Service
Cart
DB
Payment
Service
Slow DB query
Increased
Latency
Service Failed
Slow DB writes

6. How to identify the problem ?

7. DistributedTracers
• Visibility in the application layer
• Report Latencies
• Call Graph
• Platform instrumentation
• Tracers:
• Zipkin (Twitter)
• Jaeger (Uber)
• Appdash
• Lightstep

8. DistributedTracers

9. Demo
Sample app with platform tracing
https://www.youtube.com/watch?v=bCnQHZwX3UM

10. What if my application doesn’t have tracing
code?

11. One possible solution
• Leverage Kubernetes Sidecar pattern
• Examples
• Istio service mesh
• Perf Sidecar Injector

12. Perf Sidecar Injector
• Key Concepts
• Dynamically add sidecar with tracing logic to a Kubernetes POD
• Leverage POD PID Namespace Sharing
• Want to try ?
• https://github.com/bpradipt/perf-sidecar-injector

13. Demo
Using Linux ’Perf’ via Sidecar
https://asciinema.org/a/ROb7NNyibXXV1sRPg1ZG8jXjg

14. References
• Try out DistributedTracing: https://medium.com/opentracing/distributed-
tracing-in-10-minutes-51b378ee40f1
• OpenTracing docs: http://opentracing.io/documentation/
• Zipkin docs: http://zipkin.io/pages/architecture.html
• Platform metrics with OpenTracing: https://github.com/opentracing-
contrib/perfevents
• Opentracing with Platform Instrumentation:
http://cloudgeekz.com/1477/distributed-tracing-platform-
instrumentation.html
• Tracing and Profiling Microservices Application Deployed on IBM Cloud
Private: https://medium.com/ibm-cloud/tracing-and-profiling-
microservices-application-deployed-on-ibm-cloud-private-fe1f4c274329

15. Thanks!
Disclaimer: Logos and images used are registered trademarks and copyrights of their respective
companies and organizations.Views expressed are of the author’s and not necessarily of the employer’s.

16. DistributedTracers
Application
Application
code
+
ZipkinClient
push
collector
storage
API
UI
DB
Zipkin
Backend
What is missing here ?

17. DistributedTracers with PlatformVisibility
Application
Application using
OpenTracingAPI
push
collector
storage
API
UI
DB
Zipkin
Backend
Zipkin
Client
perfevents
perf interface
Kernel
Hardware (PMCs)