In the past, applications where created as monolithic entities running on a single server. If this is the past for you, too, you will have experienced the downside of modern distributed and cloud applications, as debugging, troubleshooting, and monitoring is not easily accomplished with traditional approaches. Within this session, we will explore different possibilities for collecting and analyzing the needed information to solve issues on modern distributed application and discuss the advantages and disadvantages of each approach like debugger, log files, performance counter and third party solutions. The focus of this session will be on Developer and DevOps need, as increased release cycles and third party dependency more and more result in the need for troubleshooting also on production system, rather than in an isolated test environment. This session requires a solid understanding of distributed applications and knowledge of SOA, but most principles also apply to and can be beneficial for more traditional application design approaches. The used code examples are in .NET but the shown principles generally apply to other languages, too, and shown software is often available for a variety of environments.

1. Debugging,
Troubleshooting
Distributed Web
& Cloud Applications
San Diego, July 27th, 2013
SolCal
Code Camp

2. Theo Jungeblut
• Engineering manager & lead by day
at AppDynamics in San Francisco
• Coder & software craftsman by night
• Architects decoupled solutions
tailored to business needs & crafts
maintainable code to last
• Worked in healthcare and factory
automation, building mission critical
applications, framework & platforms
• Degree in Software Engineering
and Network Communications
• Enjoys cycling, running and eating
theo@designitright.net
www.designitright.net

3. Warning
• Contains also proprietary
software
• This areas are pointed out
• Please walk away now,
if that is not acceptable
for you, it’s okay

4. Overview
• Inquiry about the Status Quo
• Some Facts -> Dev Process & Ops Reality
• Monolithic Applications
• Debugging & Logging
• Distributed Applications
• Side Effects you need to troubleshot
• Performance Counter
• Microsoft IntelliTrace
• AppDynamics
• Summary
• Q & A

5. We need YOU!

6. How agile are You?

7. The Impact of Process
* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

8. How many releases
do your company
have a year?

9. Typical Releases a Year*
58%
17%
21%
3%
2+ months
monthly
weekly
daily
http://www.appdynamics.com/blog/2011/12/14/storm-clouds-in-2012-summary-of-
appdynamics-apm-customer-survey/

10. Development + Operations = DevOps
* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

11. Difference between Dev & Ops
Dev
• Getting / being agile: need for frequent updates
with new feature and bug fixes
• Achieves stability trough running latest
Ops
• Highest goal: stable execution flawless
execution, by knowing the environment and
software
• Achieves stability through careful rollouts

12. “The Past/Present”*
Web
Server DBUser
* or maybe still your present?

13. “The Past/Present” - Architecture
Data Access Layer
Business Logic Layer
Presentation Layer
DB

14. How are you
troubleshooting?

15. Traditional Debugging

16. Traditional Debugging
Advantages
• Visibility for executed code path
Disadvantages
• Performance impact
• Requires software installation
• Often not possible in production
• Typical does not find easily distributes issues
• Only real time results

17. Log Files
Advantages
• Output adjustable without restart
• Accepted in Production
Disadvantages
• Manual, time intense creation of log statements
• Manual, time intense process of result review
• Limited scope & visibility
• Potentially high performance impact based on IO

18. “The Present/Future”- Architecture
Microsoft MSDN: http://msdn.microsoft.com/es-es/architecture/gg189193

19. “The Present/Future”
ASP.NET
MVC 3/4
(IIS)
User
WAS Tomcat
Azure
Web Role
SQL Server MySQL
Active
Directory
Azure
Worker Role
WAS
3rd Party
Web Services
Java
Application
MongoDB
3rd Party
Web Services
WCF HTTP

20. The (Netflix) Distributed Reality *
* http://techblog.netflix.com/2012/06/netflix-operations-part-i-going.html

21. What has this to do
with
Debugging
and
Troubleshooting?

22. Who does really know
how their application
is performing?

23. What impacts
Application
Performance?

24. Concurrency
http://en.wikipedia.org/wiki/File:Smithposter.jpg
• Deadlocks
• Raise Conditions
• Unexpected complexity
• …

25. Data Volume
http://en.wikipedia.org/wiki/File:Smithposter.jpg
• constantly increasing
• Volume hides details
• Handling volume
impacts resources
• …

26. Resources
http://blog.laptopmag.com/amd-cries-foul-but-tests-show-battery-life-inferior-to-intels/battery_morpheus
• are limited
• Danger of Starvation
• …

27. Who is
Performance Testing
every release?

28. http://www.slideshare.net/appdynamics/what-can-devops-learn-from-formula-1

29. Dev + Ops = Success
• DevOps has to be a combined approach
• Mostly no visibility in real performance
& production state

30. IntelliTrace™
• “Flight Recorder” for your Process
• Records configurable details
• Allows offline analysis
• Microsoft specific Tool in Visual Studio
Let’s take a look
http://msdn.microsoft.com/en-us/library/hh398365.aspx

31. Summary IntelliTrace
Advantages
• Allows great visibility into single process/AppPool
• Allows to play back test or production issue
Disadvantages
• Limited to Single Process / AppPool Visibility
• Should be only temporally enable in production
• Requires configuration effort for balance between
details & performance

32. Performance Counter
• Collect machine wide and process
specific data
• Allows on- and offline analysis
• Microsoft specific tool build into
Windows
http://www.codeproject.com/Articles/8590/An-Introduction-To-Performance-Counters

33. Summary Performance Counter
Advantages
• Huge OOTB support in Windows/IIS/etc.
• Low overhead
• Extensible
Disadvantages
• Collecting application specific data requires
source code changes

34. AppDynamics
• Application Performance Management
• Optimized for Production
• Monitoring instead of investigating
• Providing the “Right-Data”
Let’s take a look
http://www.appdynamics.com/products-editions.php

35. Summary AppDynamics
Advantages
• Works in production on high performance sides
• Monitors, Notify, Troubleshoot
• Allows instance handling in cloud environments
• Free lite version
Disadvantages
• Full version not free
• Does not include every application detail

36. Build your own Opinion (for Free)
• Lite Version
free but
limited
• 30 days free
Trial of Pro &
Azure Version
http://www.appdynamics.com/products-free-download.php

37. Summary
• Understand Dev Impact to Ops Execution
• Understand the business impact of
production issues
• Monolithic is simpler to Troubleshoot
• Distributed is more complex
• The right tools can make a Big Difference

38. Downloads,
Feedback & Comments:
Q & A
Graphic by Nathan Sawaya
courtesy of brickartist.com
theo@designitright.net
www.designitright.net
www.speakerrate.com/theoj
www. slideshare.net/theojungeblut

39. Blog, Rating, Slides
http://www.DesignItRight.net
www.speakerrate.com/theoj
www.slideshare.net/theojungeblut

40. … thanks for you attention!
And visit and support the
www.sandiegodotnet.com
Please fill out the
feedback, and…
www.speakerrate.com/theoj