More and more clients are looking to understand the capabilities of the OTM/G-Log architecture and configuration in order better tune OTM. Usually, this is required because of poor OTM performance or as preparation for significant changes to OTM configuration, volume, or platform. The client may be experience poor performance throughout the entire system or for a very specific use cases. The primary objective of a Performance Tuning Exercise is to understand how OTM is being utilized and to recommend solution to improve the performance of OTM. We recommend and will take the audience through a “ground-up” performance tuning exercise, starting with hardware and infrastructure, moving to Java and App server tuning, then to OTM technical tuning and finally to the OTM functional tuning (data, agents, etc). These audits may identify hardware constraints at each tier, networking, or other infrastructure constraints causing sub-optimal system performance. Simply stated, the performance audit will identify all bottlenecks in the system if they exist. In many cases the largest performance is impacts are not hardware, but rather how the data is configured within the application. So as part of the exercise we will analyze database performance, individual SQL queries, OTM Queues, bulk planning parameters, agents, rates and the settlement process. Understanding the methods which will best identify these bottlenecks will help you avoid performance issues early in your project and save considerable time and expense as you near go-live. This presentation will guide you through the steps necessary to better understand what is impacting performance and how to best handle it. It will provide lessons learned and tools that are available to you better manage and maintain a healthy OTM environment. Presented by Chris Plough at MavenWire

1. OTM Performance Review and
Benchmarking
OTM SIG User Conference ‘08

2. Agenda
Performance Analysis Methodologies
Our Holistic Bottom-Up Approach
Benchmarking
Other Options
Q&A

3. Methodologies
The “Top-Down” Approach
Great for determining the root-cause of specific
performance issues.
The “Bottom-Up” Approach
Great for doing a site review and identifying issues
and bottlenecks (current and potential).
The Holistic Approach
Ensure that all components (technical, functional and
external) are taken into consideration.

4. MavenWire’s Holistic Bottom-Up Approach
Our Approach
Hardware / Platforms
Operating System
Java Tuning
Application Server Tuning
OTM Thread Tuning
OTM Diag Servlets
Agents
Rates
Itineraries
Logs
Benchmarking

5. Hardware / Platforms
CPU and Hardware Platform Matters!
CPU Speed – Not a Good Indicator of
Performance
Other factors (cores, memory bandwidth, on-chip
cache) necessitate benchmarking
OTM Requires both high multi-threading and
high single-thread performance
Lots of cores and high per-core performance
Performance of Current Platforms
Linux / x86-64
Windows / x86-64 (note: memory limitations)
Solaris
HP-UX / PA-RISC
Note: HP-UX / Itanium currently unknown
AIX / POWER

6. Operating System / Stats
Review system performance under production
load for the previous 2 weeks
Utilize System Tools to Monitor
sar / kSar
top / prstat / topas / etc
Utilize Tools to Trend
Nagios / Munin / etc

7. Oracle DB
The following have improved DB performance
Ensure you’re updating DB stats regularly
Patched to 10.2.0.3
Partitioning is enabled
CURSOR_SHARING from EXACT to SIMILAR
OPTIMIZER_MODE = CHOOSE
STATISTICS_LEVEL = ALL
In some cases the following helped
OPTIMIZER_FEATURES_ENABLE = 9.2.0
Otherwise – Tune it like a normal Oracle DB
Standard DBA skill set and best practices
Utilize tools like the Oracle DB Statspack
Increase memory (PGA / SGA / etc) allocation
Pin frequently used packages/procedures to memory
Decrease storage IO Wait (more spindles, etc)
Separate out indexes, tablespaces, logs

8. Java Tuning
OTM is HIGHLY Dependent on JVM
Performance
JRockit Performs considerably faster than
other JVMs
Many Current-Generation JVMs Self-Tune
(including JRockit)
Platform Specific Parameters
Allocate as much Memory as Possible
2-3GB depending on platform
For both Web and App server
Monitor Garbage Collection
Most frequent JVM performance issue
OTM v6.0 will utilize a 64-bit JVM
NOT a silver bullet!

9. Application Server Tuning
Shouldn’t need to tune Apache or Tomcat
WebLogic vs. OAS
OTM has been running on WL since 1999
May no longer be an issue with BEA acquisition
WebLogic Tuning
Utilize the WebLogic Console
http://otmapp.mavenwire.com:7001/console
Number of Execute Threads
$OTM_HOME/weblogic/config/gc3domain/config.xml.template
ExecuteThreads
Should be 70-90, depending on load
Percentage of Socket Reader Threads
$OTM_HOME/weblogic/config/gc3domain/config.xml.template
ThreadPoolPercentSocketReaders
May have to increase to 75
DB Connection Pool
Now Tuned within the OTM Application

10. OTM Tuning
Ensure You’re Running the Latest RU
Thread Tuning
Take your time – verify results
Tune iteratively – avoid contention
If you don’t know what a thread pool does – ask first
Pay attention to Queue Size and Wait Time
Temporarily Tune Threads via the
EventDiagServlet
Careful about killing threads!
Permanently Set Threads via the
glog.properties file
OTM Thread Tuning
No Bottleneck? Check the
MediatorDiagServlet

11. OTM Diagnostic Servlets
OTM Event Diag Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.event.E
ventDiagServlet
OTM Topic Queue Assignments Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.event.T
opicQueueAssignmentsServlet
OTM Mediator Diag Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.mediat
or.MediatorDiagServlet
OTM Object Lock Diag Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.synch.
object.ObjectLockDiagServlet
OTM Connection Pool Diag Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.dataso
urce.CPDiagServlet
OTM Bean Cache Servlet
http://otmweb.mavenwire.com/GC3/glog.webserver.beanca
che.BeanCacheServlet

12. Agents
Try to model agents so only one agent fires
for any given event
Review your saved conditions for optimal
performance
Leverage other functionality than Agents –
Auto Assignment Rules, Contact Notifications

13. Rates
Ensure that rate offerings that are expired are
also inactive
If you have rate offerings active ensure that
rate records are associated to them.
Avoid redundancy in your rate structure. If
you rate offering can only handle once piece
of equipment there is no need to also define it
at the rate record level.

14. Itineraries
The more itineraries valid for a given move
the longer OTM will take to plan. Test using
the order planning action “Show Routing
Options”
Simply your itineraries where ever possible

15. Logs
To much logging turned on will impact
performance.
Turn logging on as needed to troubleshoot
Log features that can impact performance
SQL
RatingEngine
RatingEngineDetail
RatingEngineDebug
Persistence

16. MavenWire Benchmark Suite
Suite of Benchmarks compiled to allow
comparative hardware / platform testing
Does not require OTM installation, decreasing
setup complexity and testing time
Freely available to the OTM Community
All Benchmarks utilized are Open Source
Software (OSS), allowing for free use and
modification as necessary
Full Details, including download, installation,
runtime and comparison data available at:
http://www.otmfaq.com/forums/blogs/chrisplough/

17. Benchmarking - VolanoMark
VolanoMark
Java-based benchmark that simulates high
transactional and multi-threaded load
Reflects the performance of the following OTM
activities
Web UI, Agents, Integration, General Workflow, General OTM
Activities (not including optimization and planning based)
Higher numbers are better

18. Benchmarking - DaCapo
DaCapo
Java-based benchmark that simulates highly
computational, algorithmic, single-threaded
processing
Reflects the performance of the following OTM
activities
Optimization and Planning / Bulk Planning
Lower numbers are better

19. Benchmarking – Soap Stone
Soap Stone
Java-based benchmark that tests data throughput
between servers and replicates application protocols,
such as HTTP, RMI and RAW.
Reflects the throughput and protocols utilized
between the various OTM Tiers
Browser / Web: HTTP
Web / App: RMI
App / DB: RAW
Higher numbers are better

20. Benchmarking – Hammerora
Hammerora
Benchmark based on the TPC-C and TPC-H
benchmarks.
Reflects the performance and scalability of the DB
Tier
Lower numbers are better

21. Other Options
Web Tier
Load Balancing
SSL Accelerator
WAN / Web App Accelerator
App Tier
OTM SCA
DB Tier
Oracle RAC

22. Online Resources
Performance
kSar
http://ksar.atomique.net/
Nagios
http://www.nagios.org/
Munin
http://munin.projects.linpro.no/
Benchmarking
Full Replication Details
http://www.otmfaq.com/forums/blogs/chrisplough/
VolanoMark
http://www.volano.com/benchmarks.html
DaCapo
http://dacapobench.org/
Soap Stone
http://soap-stone.sourceforge.net/
Hammerora
http://hammerora.sourceforge.net/

23. Q & A and Discussion
Questions?

24. Thank You!
chris.plough@mavenwire.com
866.343.4870 x701
www.MavenWire.com