The connection between sub-second web application performance and revenue is becoming more and more apparent with established companies regularly reporting the benefits of reducing page load times. This talk will cover: * Designing for performance * Approaches to instrumenting and measuring application performance * Areas of focus for both front-end and back-end improvement * Techniques, tools and modules available in Django-land for improving performance * New and emerging technologies, for example SPDY protocol and Django 1.5's StreamingHttpResponse

1. The Life of a Web Request
Measuring and improving
Django application performance
PyCon Australia 2013
Roger Barnes
@mindsocket
roger@mindsocket.com.au
http://slideshare.net/mindsocket/

2. What are you here for
Aspects of site performance
Measuring everything
Fixing the right thing
AND HOW

3. Performance focus for this talk
✔ user perspective
✔ seconds per request
✗ benchmarks
✗ users per server
✗ requests per second

5. Why should we care?
● Web application performance affects
– engagement

6. Why should we care?
● Web application performance affects
– engagement
– bounce rate
http://www.slideshare.net/joshfraz/sept-2012rumtalk

7. Why should we care?
● Web application performance affects
– engagement
– bounce rate
– conversion/abandonment

8. Why should we care?
● Web application performance affects
– engagement
– bounce rate
– conversion/abandonment
– search engine ranking

9. Why should we care?
● Web application performance affects
– engagement
– bounce rate
– conversion/abandonment
– search engine ranking
– revenue

10. Why should we care?
Web sites are
larger
&
than ever

11. Why should we care?
Average web page:
1.3Mb
85 requests
- HTTPArchive.org

12. Why should we care?
Users are increasingly mobile
Their expectations are up
Many "responsive" sites are
no smaller than their desktop
version

13. Let's look at a
web request!

15. There's a redirect!

16. How about
some
CSS?

17. Let's get
some
JavaScript
and images
Start Render:
1.6s

18. Document complete: 8.6s
Fully loaded: 9.0s
Requests: 177
Bytes in: 2,689kb
This is on a good connection

19. The Performance Golden Rule
"80-90% of the end-user response time
is spent on the frontend.
Start there."
- Steve Souders
Source: http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/

21. Front-end Measurement
● Instrument your live application for Real User
Monitoring (RUM)
– Google Analytics
– New Relic

22. Front-end Measurement
● Synthetic testing
● Lots of tools available
● 3 in particular...

23. Chrome Developer Tools

24. webpagetest.org

25. Browser navigation timing
http://www.w3.org/TR/navigation-timing/

26. django-debug-toolbar
Not in PyPi yet,
see GitHub version
Back end Front end

27. Front-end offenders
● Static resources – size and number
● Lack of caching and compression
● 3rd-party resources
● Overdownloading

28. Static Resources
● Minification/combination of CSS/JavaScript
– django-compressor django-pipeline
● Images
– Appropriate format and compression level
– Correct size
– Remove metadata
● jpegoptim, optiPNG
– Consider sprites

29. Static Resources
● Web server/accelerator config
– Caching/expires/compression headers
● Static files to CDN
– see Django's storage hooks
● Deferred loading
● Silver bullet (?)
– mod_pagespeed (apache)
– ngx_pagespeed (nginx)

30. Ok, now what?

31. Back-end measurement
Production instrumentation with New Relic

32. django-statsd and graphite

33. django-debug-toolbar

34. runprofileserver in
django-extensions

35. Web server
WSGI application (WSGIHandler.__call__)
Initialise Request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template Rendering
Backend Services
Typical Django Request

36. Caching Dynamic Requests
● Potentially easy wins
● Beware cache invalidation
● Different levels
– Caching headers
– Web Server/Web Accelerator
– Caching middleware
– Per view caching
– Template fragment caching
– Low level object caching

37. Web server / Reverse proxy
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Typical Django Request
Caching
Browser

38. Compressing Dynamic Content
Web server
Web accelerator
gzip middleware

39. Query Performance
● Use debug toolbar
● Avoid unnecessary queries
– eg exists followed by separate get
● select_related to get FKs in same query
– Be wary of optional foreign keys
● prefetch_related to get collections
– 1 additional query instead of N+1

40. Defer work
● Defer user-irrelevant workloads
● Target
– Views
– Signals (eg post_save)
– Middleware

41. Defer work with celery
def login(request):
# Your login code here
# ...
user = form.save()
upload_to_nsa.delay(user.id)
return HttpResponse(...)

42. Ok, now what?

43. Web server
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Django Works Then Responds
Bytes on the wire
(Time to First Byte)
Back end

44. Can we stream?
● StreamingHttpResponse in 1.5
● Patch for streaming templates (#13910)
● Caveats
– Middleware compatibility
– Exception handling
– YAGNI?

45. What about eager streaming?
Hack: View middleware that streams the top of a
page before even calling the view
Proof of concept:
https://github.com/mindsocket/
django-perf-example

46. Regular
Streaming
Eager streaming

47. Visual appearance

48. Future context
● APIs, websockets and JavaScript, oh my!
● SPDY
● Mobile
● Full stack ownership

49. Summary
● Measure, measure, measure
● Don't ignore the front-end
● Beware diminishing returns
● Technology is changing

50. Resources
● General web performance
– http://www.slideshare.net/Strangeloopnet/37-lessons-
ive-learned-on-the-performance-front-lines
– http://www.webpagetest.org/
– https://developers.google.com/speed/pagespeed/
– http://www.mnot.net/cache_docs/
– http://newrelic.com/
– https://dvcs.w3.org/hg/webperf/raw-
file/tip/specs/NavigationTiming/Overview.html
– https://www.varnish-cache.org/

51. Resources
●
Django
– https://docs.djangoproject.com/en/dev/topics/cache/
– https://code.djangoproject.com/wiki/ProfilingDjango
– https://code.google.com/p/django-command-extensions/wiki/RunProfileServer
– https://developers.google.com/chrome-developer-tools/
– https://docs.djangoproject.com/en/dev/howto/static-files/deployment/#staticfiles-
from-cdn
– https://docs.djangoproject.com/en/dev/howto/custom-file-storage/
– https://github.com/django-debug-toolbar/django-debug-toolbar/issues/319
– https://github.com/sorl/sorl-
thumbnail/commit/e5f493a340dea8a33703c05fd66ee76d2f019881
– https://github.com/aaugustin/django-c10k-demo
– https://github.com/mindsocket/django-perf-example
– https://www.djangopackages.com/grids/g/asset-managers/

52. Questions?
http://slideshare.net/mindsocket
Thank You!