building a high performance directory server in...
TRANSCRIPT
![Page 1: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/1.jpg)
Building a high performance directory server in JavaLessons learned and tips from the OpenDS project.Ludovic Poitou
Matthew Smith
Sun Microsystems
#118
![Page 2: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/2.jpg)
2
AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
![Page 3: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/3.jpg)
3
AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
![Page 4: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/4.jpg)
The OpenDS Project
> Released in Open Source in July 2006
– CDDL
– Source code at https://opends.dev.java.net/
> Sponsored by Sun Microsystems
> Written in Java by LDAP experts
![Page 5: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/5.jpg)
What is it ?
> OpenDS is effectively a Java based Server supporting the LDAPv3 protocol and services
– Objet Oriented, Hierarchical Data Model
– CRUD operations
> It comes with its own embedded database
– Based on Berkeley DB Java Edition
– Not accessible from outside
> It has all security, access controls, password management features to safely store the information about Users
![Page 6: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/6.jpg)
What is it for ?
> Generic object oriented data store
> White pages and Email Address Book
> Mostly the data store for Identities
– For Authentication and Authorization
– For profiles and personalization
> The underlying infrastructure in all Enterprises
– Leveraged by Web and Mail infrastructure products
– Cornerstone of Identity Management products: Access Management and Federation Provisioning and De-provisioning tools
![Page 7: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/7.jpg)
7
Who for ?
> Telecom service providers, financial institutions use LDAP directories for customer related services (Portal)
– Storing customers identities, phones, services associated
– Building highly available services for 10 Millions, up to 200 Millions users
> But OpenDS can be used as OS naming service, or for SMB
– OpenSolaris, Solaris, Linux...
– Coupled with SAMBA, as a Domain controler
– Integrated with Kerberos
– White Pages...
> And being 100% pure Java, OpenDS can be embedded in other Java applications or Web applications
– OpenSSO
![Page 8: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/8.jpg)
OpenDS 2.2
> Released in December 2009
> LDAPv3 directory server fully standard compliant
– Supports many LDAP standard and experimental extensions
– Supports Multi Master Replication with 3 different levels of data consistency
– Extensive security features
> Improved performances, reliability over OpenDS 1.0
> Installs in 6 clicks and less than 3 minutes
> Several GUI and CLI to manage, monitor the OpenDS server
> Extensive documentation
> Localized in 6 different languages
![Page 9: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/9.jpg)
Performance characteristics
> As for most servers, scalability is extremely important
– Up to hundreds million of entries
– Up to thousands connections
– Maximize use of CPUs
> What is the operations throughput ?
> What is the average response time ? The maximum response time ?
> Our basic test
– 10 M entries, with an average size of 2.6K
– 2 servers, with Multi-Master replication between them
![Page 10: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/10.jpg)
Searchrate on Sun x4170 box
![Page 11: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/11.jpg)
Modrate on a Sun x4170 box
![Page 12: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/12.jpg)
12
AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
![Page 13: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/13.jpg)
How to reach those results ?
> 2 main aspects
– Architecture and code
– Run-time : JVM and Garbage Collector optimization
> There is a strong relationship between code design and memory optimization
![Page 14: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/14.jpg)
Architecture Overview
![Page 15: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/15.jpg)
15
Patterns
> Use of Asynchronous I/O
– Exception for write disk transactions
> Use of Immutable Objects
– Intrinsic thread safety
– Avoid need for defensive copies
> Use of “Factories” over Constructors
– Avoid creating an object
– Ease optimization for common cases Example : Most AttributeDescriptions have 0 options Example : Attributes generally have 1 value
– For immutable objects.
![Page 16: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/16.jpg)
16
Patterns
> Producers / Consumers
– Queues
– Thread Pool
– Monitors
> Strategies
– Queue Strategies : ConcurrentLinkedQueue vs LinkedBlockingQueue
![Page 17: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/17.jpg)
17
Anti-Patterns
> String concatenation
– Make sure to use a StringBuilder
– Compiler now optimize simple “Aaa” + “Bbb” concatenation
> Avoid very long methods (thousands of lines of code)
> Avoid exposing the concrete representation of an object
– Set vs LinkedHashSet.
– Not a performance issue, but will require more work when optimizing code for performance later
> Try to define only the methods you need.
![Page 18: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/18.jpg)
18
Java Collections
> Vector and Hashtable are synchronized for all methods
– Pay the price even if not necessary
> Some Java collection classes are not synchronized by default
– ArrayList, LinkedList replace Vector
– HashSet, HashMap replace Hashtable
> To synchronize, wrap in a class
> With a “static factory”
– Collections.synchronizedList(new ArrayList())
> ConcurrentHashMap, for concurrency
– But watch when using the iterator
![Page 19: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/19.jpg)
Critical Sections
> Try to minimise the code and time spent in the critical sections
> But the throughput is limited by the time spent in the largest critical section
– Example : LinkedBlockingQueue 200 000 operations on x64 processor 20 000 operations on the T2000 processor
– We use it for the WorkQueue and Access Logs
![Page 20: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/20.jpg)
Caching data
> Using Caches reduce the disk access thus should provide better performances
> But cache eviction add pressure to the GC
– When modifying entries
– When the cache is too small to hold all data
> A cache is also a contention point
> If you want to cache objects, make sure you cache those that will be reused.
> Alternate possibility, use “thread local” cache.
– But watch out for the cost (with 1000 threads?)
![Page 21: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/21.jpg)
Server monitoring
> Getting statistics for a server is mandatory
> Beware of contention
– Stats are updated frequently
– But seldom read
> A strategy could be to keep per thread statistics and collect them on demand
– Not yet implemented in OpenDS !
![Page 22: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/22.jpg)
22
AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with tuning Sun JVM
> Conclusion
![Page 23: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/23.jpg)
23
Performance Tuning
> When dealing with performances, you should consider the whole system
– Java VM
– OS
– Hardware : CPU, Memory, Disks, Network...
> In our case, we try to avoid disk I/Os
– And try to cache as much of the database
> We also want deterministic response times
– Avoid any Full GC (Stop The World)
– Make sure minor GC pauses are as small as possible
![Page 24: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/24.jpg)
24
JVM Tuning for OpenDS
> Super Size The Heap !
– We use 32GB Heaps, sometime up to 96GB
– 2GB for the New Generation (or ¼ of heap if < 8GB)
– -Xms32768M -Xmx32768M -Xmn2048M
> Use CMS
– -XX:+UseConcMarkSweepGC
– -XX:+UseParNewGC
> -XX:MaxTenuringThreshold=1
– Avoid copy of objects in New Gen
![Page 25: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/25.jpg)
25
Some interesting JVM Options
> -XX:CMSInitiatingOccupancyFraction=70
– Define the amount of occupancy in Old Gen before starting to collect
– Larger = better throughput but higher full GC risk
> -XX:+UseCompressedOops
– For 64bits JVM, less than 32BG of heap
– Will be the default in coming Java 6 updates
> If running on processor with NUMA architecture
– -XX:+UseNUMA> -XX:+AggressiveOpts
– Enables aggressive JIT optimizations, not related to GC
![Page 26: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/26.jpg)
26
The Garbage First (G1) GC
> Introduced in the Java HotSpot VM in JDK 7.
> An experimental version of G1 has also been released since Java SE 6 Update 14.
> G1 is the long-term replacement for HotSpot's low-latency Concurrent Mark-Sweep (CMS) GC
> Should be officially supported with Java SE 6 Update 21
![Page 27: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/27.jpg)
27
G1 Characteristics
> Future CMS Replacement
– Server “Style” Garbage Collector
– Parallel, Concurrent
– Generational
– Good Throughput
– Compacting
– Improved ease-of-use
– Predictable (though not hard real-time)
> The main stages consist of remembered set (RS) maintenance, concurrent marking, and evacuation pauses.
![Page 28: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/28.jpg)
28
JVM Options With G1
> -XX:+UnlockExperimentalVMOptions-XX:+UseG1GC
> PauseTime (Hints, Goal with no promise, otherwise use Java Real Time )
– -XX:MaxGCPauseMillis=50 (target of 50 milliseconds)
– -XX:GCPauseIntervalMillis=1000 (target of 1000 msecs)
> Generation Size
– -XX:+G1YoungGenSize=512m (for a 512 MB young gen)
> Parallelism
– -XX:+G1ParallelRSetUpdatingEnabled
– -XX:+G1ParallelRSetScanningEnabled
![Page 29: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/29.jpg)
29
OpenDS et G1
> Goal: Avoid any Full GC, best control of pauses
> Collaboration between the HotSpot and the OpenDS teams
– OpenDS is used as a “Large” reference application
– Between 10 and 20 enhancements integrated in G1 following the tests
– Performance with large heaps improved by a factor of 10
> We're still discovering it
– When doing read operations, we see pauses between 10 and 20 ms with 32GB JVM
– But we're still seeing Full GC when doing Write operations (More garbage, stresses more the Old Gen)
– Hopefully this will be resolved in next builds
![Page 30: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/30.jpg)
30
OpenDS G1 and Searches
![Page 31: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/31.jpg)
31
AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
![Page 32: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/32.jpg)
32
Summary
> OpenDS
– A open source LDAP directory server, 100% pure Java
– Easy to install and use
– Designed for high performance and high scalability
> We saw some patterns and tips used in the OpenDS project
> Knowing and understanding the JVM and GC is required to build high performance server
– Tuning JVM and GC is an art
– Performance engineering is a profession
> Who said that Java is slow ?!
![Page 33: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/33.jpg)
33
The Art of GC Tuning
http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-4887&yr=2009&track=javase
JavaOne Presentation: GC Tuning in HotSpot JVM
![Page 34: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/34.jpg)
34
Now...
> Give OpenDS a try
– http://www.opends.org
> Join our community:
– Join/Login sur Java.net
– http://opends.dev.java.net
– Request a Role
– Subscribe to the mailing lists
– IRC: #opends on freenode.net
> OpenDS is localized in several languages. It's community based, through online tools. An easy way to participate.
![Page 35: Building a high performance directory server in Javajazoon.com/history/Portals/0/Content/slides/th_a6_1330-1420_poitou.pdf · Building a high performance directory server in Java](https://reader031.vdocument.in/reader031/viewer/2022021901/5b899fc07f8b9a287e8c9f65/html5/thumbnails/35.jpg)
Ludovic Poitou blogs.sun.com/Ludo
Sun Microsystems [email protected]
Matthew Swift
Sun Microsystems [email protected]