Rakuten Ichiba is the biggest online shopping mall in Japan and use GlassFish as core APIs of Rakuten Ichiba.
These slide decks show the exprience how to improve the performance of GlassFish at the peak time of Rakuten Ichiba's heavily-loaded sale.
2. About me
Name: Makito Hashiyama(@capyogu)
Age: 29
Role: Team manager / In charge of APIs for Rakuten Ichiba
Like: GlassFish, Tomcat, KVS(memcached, Coherence), GAE,
Android…
E-mail: makito.hashiyama@mail.rakuten.com,
hashiyaman@gmail.com
3. What is Rakuten?
E-commerce and Internet company based in Tokyo, Japan
B2B2C e-commerce platform
Head Office
E-Commerce
eBook
Travel
Other services & businesses
Rakuten Institute of Technology
Development center
Head Office / Regional Headquarters
4. How We Are Using GlassFish
Usage
One of core API service for Rakuten Ichiba
Require high availability
External
APIs
Environment
Use on production environment
GlassFish version 3.1.2.2
SOAP/
REST
Client side
Our API
GlassFish
Clusters
Oracle
Coherence
5. Benefits gleaned from Glassfish
Reference implementation of Java EE
Only GlassFish supported JAX-WS as standard in 2007
It was an advantage to evaluate new features earlier
Easy to manage with low cost
We need to manage huge cluster without stopping
Cost saving(Weblogic -> GlassFish)
6. What Worked with GlassFish
Community support(create patches)
https://java.net/jira/browse/GLASSFISH-5200
(If use jvmRoute JSESSIONID cookie is not secure even in
HTTPS)
https://java.net/jira/browse/GRIZZLY-1333
(NetworkAddressValidator will fail when passed property
substitution values)
We have contributed patched to GlassFish community.
7. Improvement to handle a huge traffic
Rakuten Super Sale
Biggest online sales in Japan
A lot of doorbuster deals(It causes a huge amount of traffic)
Performance bottleneck
External APIs called by our API were slow down
We needed to improve the system at the peek time
Client side
delay
Slow down
Our API
External
APIs
8. Improvement to handle a huge traffic
Worker Thread
Worker Thread
Request
Client
side
Task
Queue
Worker Thread
Worker Thread
Worker Thread
GlassFish
CPU load
was high
delay
delay
delay
delay
delay
External
APIs
9. Improvement to handle a huge traffic
Worker Thread
Request
Client
side
Task
Queue
Worker Thread
External
APIs
Worker Thread
GlassFish
(1)According to vmstat, ‘run queue’ was very high
(2)Decrease worker threads to keep ‘run queue’ low
(3)As a result, latency increased but throughput was improved
10. Improvement to handle a huge traffic
As a result…
Our API could process over 12,000 transactions / minute
The result showed the high reliability and availability of
GlassFish
11. Resolve issues & challenge to upper goal
Some issues
Instance down due to the full of task queue
Unknown exception on server.log
org.glassfish.flashlight.impl.client.ReflectiveClientInvoker
java.lang.reflect.InvocationTargetException
CAUSE: java.lang.NullPointerException
id=101
target=org.glassfish.web.admin.monitor.HttpServiceStatsProvider@193e1fc
method=public void org.glassfish.web.admin.monitor.HttpServiceStatsProvider.
connectionAcceptedEvent(java.lang.String,int,java.lang.String)
paramNames=[listenerName, connection, address]
probeIndices=[0, 1, 2]
useProbeArgs=true
hasComputedParams=false
Challenge to upper goal