Dhruv Kumar's talk from http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/124095182/

1. Building Low Latency Java Applications with
Ehcache
Dhruv Kumar
Software Engineer, Terracotta Inc.
Dhruv.Kumar@terracottatech.com

2. Why In-Memory?
2

3. 3
Data Center Machines
3

4. 4
Data Center Machines
4

5. 5
Data Center Machines
5

6. 6
Memory Capacity Trends
6

7. 7
Disk Capacity Trends
7

8. 8
SSD Capacity Trends
8

9. 9
CPU Speed Trends
9

10. 10
Network I/O Trends
10

11. 11
Memory Throughput Trends
11

12. 12
SSD Throughput Trends
12

13. 13
Disk Throughput Trends
13

14. 14
Trend Summary
14

15. 15
Trend Summary
15
§ Storage is cheap and increases exponentially
§ Most transfer rates increase exponentially except
disk throughput.
– Gap between disk capacity and transfer rate is increasing
§ Accessing disk is very slow: transferring 100GB
@ 100 MB/s takes 1000s, or approx. 17 minutes.
– Will get worse, as 512GB/node RAM will be common in 2
years.
– Faster to access Network Attached Memory!
– SSDs still not common due to unfavorable price performance

16. 16
In-Memory Computing Opportunity
16
§ 256 GB RAM * 4 nodes = 1 TB In-Memory Cluster
§ 1 TB can hold many large datasets: 1 billion users
@ 1 KB each
§ Number of CPU cores double every 24 months:
memory/core increases exponentially
§ Bottomline: judiciously using RAM is key.

17. Ehcache System Architecture
17

18. 18
Terracotta Server Array
Scale with Data and Processing Needs
18
BigMemory
App Server
Ehcache
App Server
Ehcache
Traditional
3-Tier Architecture
Scale Vertically Scale Horizontally
Elastic Scale
In the Cloud
App Server
Ehcache
Quartz
App Server
Ehcache
Quartz
Database
Database
Database
Database
App Server
Ehcache
App Server
Ehcache
BigMemory
Terracotta Server Array
BigMemory
Increase Data in Memory
Reduce Database Reliance

19. 19
Hello Ehcache!
Database
Traditional
3-Tier Architecture
App Server
Ehcache
§ Reduction in database reliance
– Store repetitively used data and access them
in memory speeds
§ Reduction of new objects created
– Stored objects will be reused constantly thus
reducing the creation of new objects
§ Less network traffic
– Stored data is already at the application layer
so no need to make a network call to get it
§ Reduce CPU usage
– Objects in cache are already in the format you
need for use, no additional marshaling needed.

20. 20
Example Configuration
Configuration via Ehcache.xml
20
Example Configuration
<ehcache>!
!
<cache name=”UserPreferencesCache"!
maxElementsInMemory="10000"!
timeToIdleSeconds="300”!
! ! !memoryStoreEvictionPolicy="LRU”/>!
!
<cache name=”ShoppingCartCache"!
maxElementsInMemory=”2500"!
timeToLiveSeconds=”6000”!
! ! !memoryStoreEvictionPolicy=”LFU"/>!
!
</ehcache>!
20

21. 21
Simple API
21
Example Code
public Object testCache(String key) throws Exception {!
!CacheManager cacheManager = !
! !new CacheManager( “<path to my ehcache.xml>”);!
!Cache myCache = cacheManager.getCache("MyCache");!
!Object value;!
! !!
!Element element = myCache.get(key);!
!if (element == null) {!
! !value = "go get it from somewhere like DB or service,
! ! ! etc";!
! !myCache.put(new Element(key, value));!
!} else {!
! !value = (Object) element.getValue();!
!}!
! !!
!return value;!
}!

22. 22
Ehcache Search
§ Intuitive, full-featured Search API
§ Fast, efficient implementation
§ Make cache searchable, then specify attributes
22
<cache name=”peopleCache”>
<searchable>
<searchAttribute name="age" expression="value.getAge()"/>
<searchAttribute name="gender" expression="value.getGender()"/>
</searchable>
</cache>
Example: Search for 32-year-old males
results = cache.createQuery().includeKeys()
.addCriteria(age.eq(32)).and (gender.eq("male"))
.execute();
Example Configuration and Code

23. 23
Ehcache Search Using SQL-like Syntax
§ Can also search Ehcache using a SQL like syntax
§ String Queries mapped to Ehcache Search API
23
Example: Search for 32-year-old males
results = cache.createQuery().includeKeys()
.addCriteria(age.eq(32)).and (gender.eq("male"))
.execute();
// or using Ehcache Query Language
QueryManager qm = (new QueryManagerBuilder()).addEhcache(peopleCache).build();
String statement = “select age, gender from peopleCache
where age = 32 and gender =‘male’;”
results = qm.createQuery(statement).end().execute();
Example Configuration and Code

24. 24
Ehcache Persistence
§ Persist actions in a log-append store (Fast Restart Store).
§ Each mutate operation is recorded. Optionally synchronous.
§ On restart, Ehcache entries and Index data are reconstructed from the
log.
24
Example ConfigurationExample Configuration
<ehcache>!
<cache name=”UserPreferencesCache"!
maxElementsInMemory="10000"!
timeToIdleSeconds="300”!
! ! persistenceStrategy=“localRestartable”>!
</ehcache>!

25. 25
In-memory Data and Speed
25
Milliseconds
App Response Time
Microseconds
App Response Time
Memory
90% of Data in
Database
Database
90% of Data in
Memory

26. 26
Storing Data away from Java Heap
(BigMemory)
Off-heap memory allocator. Uses NIO
DirectByteBuffers.
Bypasses Java Garbage Collection.
Only Serializable keys and values can be
placed in Off-heap.
Pure Java implementation compatible with
all popular JVMs
Requires no changes to application code,
JVM or OS
26
Database
Vertical Scaling
App Server
Ehcache
BigMemory

27. 27
BigMemory Configuration
<ehcache>
<cache name= UserPreferencesCache"
maxBytesOnHeap=”512M"
timeToIdleSeconds="300
!memoryStoreEvictionPolicy="LRU
!overflowToOffHeap="true
! !maxBytesOffHeap= 30G"/>
<cache name= ShoppingCartCache"
maxElementsInMemory= 2500"
timeToLiveSeconds= 6000
!memoryStoreEvictionPolicy="LFU"/>
</ehcache>!
27
Example Configuration

28. 28
Testing Offheap Performance
28
• Public URL:
http://svn.terracotta.org/svn/forge/offHeap-test/
• Our Configuration: Six 4-core Xeon CPUs @ 2.93
GHz, 128 GB RAM, RHEL 5.1, Sun JDK 1.6.0_21
in 64 bit mode.
• Load the cache and then run 50 threads doing
90% Read (get() call) calls and 10 % Writes (put()
call)
• Run in Heap and Offheap, and compare GC
duration, latency, throughput.

29. 29
Offheap Perf Test: Garbage Collection Pauses
29

30. 30
Offheap Perf Test: Maximum Latency
30

31. 31
Offheap Perf Test: Mean Latency
31

32. Distributed
Ehcache
32

33. 3333
Distributed Applications
Consistency?

34. 3434
Distributed Ehcache using Terracotta Server
Array
§ Stateful Server Array
§ Synchronous TCP based connections to
clients
§ Highly Available using Master Slave
replication
§ Linear Scale Out Characteristics
§ Persistent
§ Various consistency optionsDatabase
Scale Out
App Server
Ehcache
Terracotta Server Array
BigMemory
App Server
Ehcache

35. 35
Terracotta Server Array
35
Terracotta
ServerArray
Stripe
Pure Java
cache server on
commodity HW
Transactional
updated mirror for
high availability
Stripe
Commodity Server
Application
Terracotta Driver
Stripe Stripe Stripe Stripe
Commodity Server
Application
Terracotta Driver
Commodity Server
Application
Terracotta Driver
TCP
§ Durability
§ Mirroring
§ Striping
§ Developer Console
§ Plug-in Monitoring
§ Operations Center
Commodity Server
Disk
Active
Server
Commodity Server
Disk
BigMemory
Mirror
BigMemory

36. 36
Example Configuration
Configuring Ehcache to use Terracotta
36
<ehcache>
!<terracottaConfig url="someserver:9510"/>
!<cache name= UserPreferencesCache
! !maxElementsInMemory="10000
! !timeToIdleSeconds= 300 />
!<cache name= ShoppingCartCache"
! !maxElementsInMemory= 25000"
! !timeToLiveSeconds= 6000 />!
!<terracotta />
</ehcache>!
Simply change two lines of configuration

37. 37
Tiered Storage
In-Memory
Tiered Data Storage
37
Terracotta Heap
Heap
Store
BigMemory
Off-Heap Store
2,000,000+
1,000,000
100,000
2
1,000
10,000+
Speed (TPS)
1,000s
Size (GB)
External Data Source
(e.g., Database)
Terracotta Offheap

38. 38
Example Configuration
Consistency options in TSA
38
Strongly
Consistent
Fully
Transactional
Eventually
Consistent
More Consistency More Performance
<cache name=”UserPreferencesCache"
maxElementsInMemory="10000"
timeToLiveSeconds="300”>
<terracotta consistency=”eventual"/>
</cache>
<cache name=”ShoppingCartCache"
maxElementsInMemory=”5000"
timeToIdleSeconds=”6000”>
<terracotta consistency=”strong"/>
</cache>
Example Configuration

39. 39
Terracotta Server Array
Scale with Data and Processing Needs
39
BigMemory
App Server
Ehcache
App Server
Ehcache
Traditional
3-Tier Architecture
Scale Up Scale Out
Elastic Scale
In the Cloud
App Server
Ehcache
Quartz
App Server
Ehcache
Quartz
Database
Database
Database
Database
App Server
Ehcache
App Server
Ehcache
BigMemory
Terracotta Server Array
BigMemory
Increase Data in Memory
Reduce Database Reliance

40. 40
Recent Developments
40
• JSR107
• Hadoop Integration: One way
(EhcacheOutputFormat)
• New Management and Monitoring

41. 4141
Thank You
Dhruv.Kumar@terracottatech.com