1. www.day.com
Top Support Issues
Dispatcher & OutOfMemory Situations
Speaker: Honwai Wong
Duration: 45 min
Feedback: techsummit@day.com
Day Technical Summit 2008 1

2. www.day.com
Agenda
 Dispatcher
 Basics
 Request Handling
 Best-Practice
 Feature: Permission Sensitive Caching
 OutOfMemory
 Symptoms
 Causes
 Measures
 Analysis
 DayCare
Day Technical Summit 2008 2

3. www.day.com
Dispatcher
Basics
 Webserver module, e.g. Apache, IIS, iPlanet
 Reverse proxy
 Loadbalancing
 Caching
 Simple caching and invalidation rules
 No cache dependencies resolution
 Only caches GET requests
 with an extension
 without query parameters
Day Technical Summit 2008 3

4. www.day.com
Dispatcher
Architecture
www.day.com partnernet.day.com
Render farm www.day.com
Communiqué
Render farm partnernet.day.com
Dispatcher Module
Webserver
Day Technical Summit 2008 4

5. www.day.com
Dispatcher
Request Handling - Chain
deliver page
Dispatcher Module
Is NO
Communiqué
request
Webserver
cacheable?
Incoming GET
YES
request
NO
Already in cache?
YES
YES
NO
Is document
up-to-date?
cache and deliver page
Day Technical Summit 2008 5

6. www.day.com
Dispatcher
Flush request process
Dispatcher Module
Communiqué Webserver
touch .stat-files up to configured level
FLUSH request
to invalidate files in respective folders
flush cache: remove all files linked to
handle of replicated page
Day Technical Summit 2008 6

7. www.day.com
Dispatcher
Best-Practice
 Always use the dispatcher module, even for author
 Apply tree-based cache-invalidation if possible
 configuration of /statfileslevel in dispatcher.any
 Distribute load among multiple redundant publish instances
 dispatcher module or native loadbalancer (hardware)
 Initiate flush requests from publish instance
 avoid potential timing conflicts (ensures that cache flush is initiated
AFTER page has actually reached publish instance)
 performance gain of author by shifting cache management to publish
Day Technical Summit 2008 7

8. www.day.com
Dispatcher
Feature: Permission Sensitive Caching
 Problem: public site contains content with restricted access
 Solution (so far): exclude such pages from being cached to
force authentication and authorization on a CQ level
 potential performance impact
 New feature: Permission Sensitive Caching
 Check whether user has access to cached content prior
delivery
 Benefit: lightweight query sent to CQ to determine access,
static delivery of actual content from webserver cache
Day Technical Summit 2008 8

9. www.day.com
Dispatcher
Feature: Permission Sensitive Caching
 Requires CQ to handle authentication requests from
Dispatcher by returning appropriate response status codes
 Additional configuration section in dispatcher.any
 Fully integrated in upcoming CQ version 4.3.0
 Will work with Dispatcher version 4.0.1 and above
Day Technical Summit 2008 9

10. www.day.com
Dispatcher
Feature: Permission Sensitive Caching
Dispatcher Module
Is NO
request
Webserver CQ Instance
cacheable?
Incoming GET
YES
request
NO
Already in cache?
YES
NO
Is document
up-to-date?
YES
Send HEAD
request
CQ returns
YES
200 ?
NO, return 40x HTTP status
Day Technical Summit 2008 10

11. www.day.com
Dispatcher
Feature: Permission Sensitive Caching
DEMO
Day Technical Summit 2008 11

12. www.day.com
OutOfMemory
Symptoms
 CQ instance crashes
 Performance degradation
 Seemingly endless loop of repeated garbage collections
 Instance restart restores normal operation
Day Technical Summit 2008 12

13. www.day.com
OutOfMemory
Causes
 Maximum heap size is insufficient for the application
 Application code makes heavy use of in-memory operations
 e.g. caching objects/data in memory for fast access
 Bug in product which causes memory leaks
 e.g. maintaining references to unused objects, thus preventing proper
garbage collection by the JVM
Day Technical Summit 2008 13

14. www.day.com
OutOfMemory
Measures
 Inspect OutOfMemory error message
 Java heap space -> increase maximum heap-size
 PermGen space -> increase maximum PermGen size
 Add GC specific JVM options to monitor garbage collection
 –verbose:gc –xloggc:gc.log –XX:+PrintGCDetails –XX:
+PrintGCTimeStamps
 Add JVM options to record heap-dump once OutOfMemory
occurs
 -XX:HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>
Day Technical Summit 2008 14

15. www.day.com
OutOfMemory
Analysis
 Check application code for memory intensive operations
 Inspect heapdump generated by JVM to identify potential
memory leaks
 Available tools:
 jhat, the Java Heap Analysis Tool
 YourKit, Java Profiler with powerful GUI (needs license)
 SAP Memory Analyzer, Eclipse Open Source Project
 Open DayCare ticket if potential memory leak discovered
Day Technical Summit 2008 15

16. www.day.com
OutOfMemory
DayCare
 Provide Day support with detailled information:
 Recent changes introduced to the system that might be related
 JVM options
 Any problem related log-files (server-log files, error.log)
 Heapdump, ideally compressed and uploaded to ftp.day.com
Day Technical Summit 2008 16

17. www.day.com
Questions?
Day Technical Summit 2008 17