2. About me
• Masahiro Ide
• Software Engineer of LINE server development
• Especially StickerShop backend server
• Joined: Oct, 2015
• Contribute to Armeria
• Our open sourced asynchronous RPC/REST library
• https://github.com/line/armeria
3. Agenda • Sticker Shop Server
• History
• Our Architecture
• Implementations
5. Sticker Shop … in 2015
One of the biggest monolithic Java application in LINE
• Purchasing, Serving, Ranking stickers in 1 app
• Many pains, troubles
• Takes more than 3 hours to restart app
6. Sticker Shop in 2018
Now shift to microservice way for scalability
• 1 million stickers
• 30 services
• Purchasing, Serving, Ranking stickers, etc.
• 70K requests/sec in peek time
7. StickerShop Architecture Overview
(Simplified)
Central Dogma – Configuration Repository Service
Talk server
ElasticSearch
Client
ReverseProxy
Thrift/HTTP2
Thrift/HTTP2
Redis
MySQL
Sticker capability
server
Shop server Search FE
Thrift/HTTP2
8. Implementation
• What we need is…
• Performance
• Prepared to faults
• Observability
• Stable feature rollout
9. Central Dogma
• Repository service for textual configurations
• JSON, YAML, XML …
• Highly available
• multi-master, eventually consistency
• Version controlled
• https://line.github.io/centraldogma/
10. Armeria - Our RPC layer
• Asynchronous RPC/REST library
• built on top of Java 8, Netty, HTTP/2, Thrift and gRPC
• Take care common functionality for microservice
1. Load balancing
2. Circuit Breaker / Retry / Throttling
3. Tracing (Zipkin) / Monitoring integration
• etc.
https://line.github.io/armeria/
11. 1. Client-side load balancing
• No proxy load balancers
• Proxy would be inefficient when
considering request heavy services
• CentralDogma as a service discovery
• All services register a name and
a list of machines in CentralDogma
• All clients monitor CentralDogma and
balance requests across machines
LB
Client
Server
Client
Server
Server
Proxy load balancing
12. 1. Client-side load balancing
• No proxy load balancers
• Proxy would be inefficient when
considering request heavy services
• CentralDogma as a service discovery
• All services register a name and
a list of machines in CentralDogma
• All clients monitor CentralDogma and
balance requests across machines
Central Dogma
Client
Server
Client
Server
Server
13. 2. Circuit Breaker - Avoid cascading failure
• Server die suddenly
• e.g. Hardware failure
• Upstream cannot proceed part
of requests, in worst case
• Blocks all of application thread
• LINE die
Service A
Service C
Service B
14. 2. Circuit Breaker - Avoid cascading failure
• Circuit Breaker pattern*
1. Monitors failures
2. Return error immediately without
RPC once the failures reach
a threshold
Service A
Service C
Service B
CircuitBreaker
CircuitBreaker
* https://martinfowler.com/bliki/CircuitBreaker.html
15. 3. Tracing
• Hard to point out the slowness/bug in microservice
• No one provides complete picture of system performance
Talk server
ElasticSearch
Reverse Proxy
Redis
MySQL
Shop server
Search FE
T=start
T=end
17. Rollout features using CentralDogma (CD)
• Rollout to production within
10 minutes of being changed
• Deploy a YAML file to all our
production servers via CD
• Record changes in CD
commit log
18. Rollout features using CentralDogma
• Rollout to production within
10 minutes of being changed
• Deploy a YAML file to all our
production servers via CD
• Record changes in CD
commit log
30%CacheV2 rollout:
Central
Dogma
Service
Service
Service
commit
YAMLPull & Reload config
Developer
19. Conclusion
• Our system is built on top of our OSSs
• Armeria, CentralDogma, and more
• … are still evolving
• Trouble may happen
• Make a well trouble-prepared system
• Let's do a smooth release
それでは2018年に戻ってきましょう。現在のstickershopは、この3つの数字で表せると思います。これは大体の数字ですが、2018年現在でおおよそ1 million productが実際のstickerとして皆さんの手元から相手のchatに届いています。そして、おおよそ3つの機能を提供するために30以上のmicroserviceによって作る、売る、送るという機能は作られてます。最後に、サービスとして処理するrequestの量ですが、peek of peekで話しますと大体70K req/sec程度が一番忙しい時にはあったりするという規模感になっています。では、それを実現するアーキテクチャはどのようになっているでしょうか?(次のページ)
この図はstickershopのアーキテクチャを表しています。先ほど30以上のサービスから構成されていると言いましたが、ここでは代表して数個のserviceが登場しています。青い線で書かれているように、LINE clientから送られてきたrequestはreverse proxyを通じてbackend serverに機能に応じてrequestを投げていき、storageまでアクセスされます。それぞれのサービスはお互いに通信しながらサービスを提供するので、大体メッシュ状のような形状になっていて、CentralDogmaと呼ばれるconfiugrationを一箇所で管理するserviceと繋がっています。そして、LINE ClientとserverはThrift over http2のrequestで通信が行われています。同様に、サーバ間の通信は大体Thrift over http2で行われています。
最後に紹介したいのは、どのようにfeatureのrolloutをしているかについてです。
All of our servers run a background thread called the “CentralDogma watcher” which constantly watches the CentralDogma_config.json copy on memory, reloading it when it changes. This allows CentralDogma to reload without interruption to the server.
最後に紹介したいのは、どのようにfeatureのrolloutをしているかについてです。
All of our servers run a background thread called the “CentralDogma watcher” which constantly watches the CentralDogma_config.json copy on memory, reloading it when it changes. This allows CentralDogma to reload without interruption to the server.