"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
OpenDS_Jazoon2010
1. Building a high
performance directory
server in Java
Lessons learned and tips
from the OpenDS project.
Ludovic Poitou
Matthew Smith
Sun Microsystems
#118
2. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
2
3. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
3
4. The OpenDS Project
> Released in Open Source in July 2006
– CDDL
– Source code at https://opends.dev.java.net/
> Sponsored by Sun Microsystems
> Written in Java by LDAP experts
5. What is it ?
> OpenDS is effectively a Java based Server supporting the LDAPv3 protocol and
services
– Objet Oriented, Hierarchical Data Model
– CRUD operations
> It comes with its own embedded database
– Based on Berkeley DB Java Edition
– Not accessible from outside
> It has all security, access controls, password management features to safely
store the information about Users
6. What is it for ?
> Generic object oriented data store
> White pages and Email Address Book
> Mostly the data store for Identities
– For Authentication and Authorization
– For profiles and personalization
> The underlying infrastructure in all Enterprises
– Leveraged by Web and Mail infrastructure products
– Cornerstone of Identity Management products:
Access Management and Federation
Provisioning and De-provisioning tools
7. Who for ?
> Telecom service providers, financial institutions use LDAP directories for
customer related services (Portal)
– Storing customers identities, phones, services associated
– Building highly available services for 10 Millions, up to 200 Millions users
> But OpenDS can be used as OS naming service, or for SMB
– OpenSolaris, Solaris, Linux...
– Coupled with SAMBA, as a Domain controler
– Integrated with Kerberos
– White Pages...
> And being 100% pure Java, OpenDS can be embedded in other Java
applications or Web applications
– OpenSSO
7
8. OpenDS 2.2
> Released in December 2009
> LDAPv3 directory server fully standard compliant
– Supports many LDAP standard and experimental extensions
– Supports Multi Master Replication with 3 different levels of data consistency
– Extensive security features
> Improved performances, reliability over OpenDS 1.0
> Installs in 6 clicks and less than 3 minutes
> Several GUI and CLI to manage, monitor the OpenDS server
> Extensive documentation
> Localized in 6 different languages
9. Performance characteristics
> As for most servers, scalability is extremely important
– Up to hundreds million of entries
– Up to thousands connections
– Maximize use of CPUs
> What is the operations throughput ?
> What is the average response time ? The maximum response time ?
> Our basic test
– 10 M entries, with an average size of 2.6K
– 2 servers, with Multi-Master replication between them
12. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
12
13. How to reach those results ?
> 2 main aspects
– Architecture and code
– Run-time : JVM and Garbage Collector optimization
> There is a strong relationship between code design and memory optimization
15. Patterns
> Use of Asynchronous I/O
– Exception for write disk transactions
> Use of Immutable Objects
– Intrinsic thread safety
– Avoid need for defensive copies
> Use of “Factories” over Constructors
– Avoid creating an object
– Ease optimization for common cases
Example : Most AttributeDescriptions have 0 options
Example : Attributes generally have 1 value
– For immutable objects.
15
17. Anti-Patterns
> String concatenation
– Make sure to use a StringBuilder
– Compiler now optimize simple “Aaa” + “Bbb” concatenation
> Avoid very long methods (thousands of lines of code)
> Avoid exposing the concrete representation of an object
– Set vs LinkedHashSet.
– Not a performance issue, but will require more work when optimizing code
for performance later
> Try to define only the methods you need.
17
18. Java Collections
> Vector and Hashtable are synchronized for all methods
– Pay the price even if not necessary
> Some Java collection classes are not synchronized by default
– ArrayList, LinkedList replace Vector
– HashSet, HashMap replace Hashtable
> To synchronize, wrap in a class
> With a “static factory”
– Collections.synchronizedList(new ArrayList())
> ConcurrentHashMap, for concurrency
– But watch when using the iterator
18
19. Critical Sections
> Try to minimise the code and time spent in the critical sections
> But the throughput is limited by the time spent in the largest critical section
– Example : LinkedBlockingQueue
200 000 operations on x64 processor
20 000 operations on the T2000 processor
– We use it for the WorkQueue and Access Logs
20. Caching data
> Using Caches reduce the disk access thus should provide better performances
> But cache eviction add pressure to the GC
– When modifying entries
– When the cache is too small to hold all data
> A cache is also a contention point
> If you want to cache objects, make sure you cache those that will be reused.
> Alternate possibility, use “thread local” cache.
– But watch out for the cost (with 1000 threads?)
21. Server monitoring
> Getting statistics for a server is mandatory
> Beware of contention
– Stats are updated frequently
– But seldom read
> A strategy could be to keep per thread statistics and collect them on demand
– Not yet implemented in OpenDS !
22. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with tuning Sun JVM
> Conclusion
22
23. Performance Tuning
> When dealing with performances, you should consider the whole system
– Java VM
– OS
– Hardware : CPU, Memory, Disks, Network...
> In our case, we try to avoid disk I/Os
– And try to cache as much of the database
> We also want deterministic response times
– Avoid any Full GC (Stop The World)
– Make sure minor GC pauses are as small as possible
23
24. JVM Tuning for OpenDS
> Super Size The Heap !
– We use 32GB Heaps, sometime
up to 96GB
– 2GB for the New Generation (or
¼ of heap if < 8GB)
– -Xms32768M -Xmx32768M
-Xmn2048M
> Use CMS
– -XX:+UseConcMarkSweepGC
– -XX:+UseParNewGC
> -XX:MaxTenuringThreshold=1
– Avoid copy of objects in New Gen
24
25. Some interesting JVM Options
> -XX:CMSInitiatingOccupancyFraction=70
– Define the amount of occupancy in Old Gen before starting to collect
– Larger = better throughput but higher full GC risk
> -XX:+UseCompressedOops
– For 64bits JVM, less than 32BG of heap
– Will be the default in coming Java 6 updates
> If running on processor with NUMA architecture
– -XX:+UseNUMA
> -XX:+AggressiveOpts
– Enables aggressive JIT optimizations, not related to GC
25
26. The Garbage First (G1) GC
> Introduced in the Java HotSpot VM in JDK 7.
> An experimental version of G1 has also been released since Java SE 6 Update
14.
> G1 is the long-term replacement for HotSpot's low-latency Concurrent Mark-
Sweep (CMS) GC
> Should be officially supported with Java SE 6 Update 21
26
27. G1 Characteristics
> Future CMS Replacement
– Server “Style” Garbage Collector
– Parallel, Concurrent
– Generational
– Good Throughput
– Compacting
– Improved ease-of-use
– Predictable (though not hard real-time)
> The main stages consist of remembered set (RS) maintenance, concurrent
marking, and evacuation pauses.
27
28. JVM Options With G1
> -XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
> PauseTime (Hints, Goal with no promise, otherwise use Java Real Time )
– -XX:MaxGCPauseMillis=50 (target of 50 milliseconds)
– -XX:GCPauseIntervalMillis=1000 (target of 1000 msecs)
> Generation Size
– -XX:+G1YoungGenSize=512m (for a 512 MB young gen)
> Parallelism
– -XX:+G1ParallelRSetUpdatingEnabled
– -XX:+G1ParallelRSetScanningEnabled
28
29. OpenDS et G1
> Goal: Avoid any Full GC, best control of pauses
> Collaboration between the HotSpot and the OpenDS teams
– OpenDS is used as a “Large” reference application
– Between 10 and 20 enhancements integrated in G1 following the tests
– Performance with large heaps improved by a factor of 10
> We're still discovering it
– When doing read operations, we see pauses between 10 and 20 ms with
32GB JVM
– But we're still seeing Full GC when doing Write operations (More garbage,
stresses more the Old Gen)
– Hopefully this will be resolved in next builds
29
31. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
31
32. Summary
> OpenDS
– A open source LDAP directory server, 100% pure Java
– Easy to install and use
– Designed for high performance and high scalability
> We saw some patterns and tips used in the OpenDS project
> Knowing and understanding the JVM and GC is required to build high
performance server
– Tuning JVM and GC is an art
– Performance engineering is a profession
> Who said that Java is slow ?!
32
33. The Art of GC Tuning
JavaOne Presentation: GC Tuning in HotSpot JVM
http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-4887&yr=2009&track=javase
33
34. Now...
> Give OpenDS a try
– http://www.opends.org
> Join our community:
– Join/Login sur Java.net
– http://opends.dev.java.net
– Request a Role
– Subscribe to the mailing lists
– IRC: #opends on freenode.net
> OpenDS is localized in several
languages. It's community based,
through online tools. An easy way to
participate.
34
35. Ludovic Poitou blogs.sun.com/Ludo
Sun Microsystems Ludovic.Poitou@sun.com
Matthew Swift
Sun Microsystems Matthew.Swift@sun.com
