Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
LINE's messaging service
architecture underlying more than
200 million monthly active users,
and core techs
Yuto Kawamura
...
Speaker introduction
• Yuto Kawamura

• Senior software engineer

• Apache Kafka contributor

• Joined: Apr, 2015(2 years ...
About LINE
•Messaging service 

•More than 200 million active users1 in countries with top market
share like Japan, Taiwan...
Scale Metrics
Messages delivered 25 billion per day
Peak time Messages sent

(new year greetings)
420K per second
Total AP...
Messaging System
Architecture
LINE Apps
LEGY JP
LEGY DE
LEGY SG
Thrift RPC/HTTP
talk-server
Distributed Data Store
Distrib...
LEGY
• LINE Event Delivery Gateway

• API Gateway/Reverse Proxy

• Written in Erlang

• Deployed to many data centers all ...
talk-server
• Java based web application server

• Implements most of messaging functionality + some other
features

• Jav...
Datastore with Redis and
HBase
• LINE’s hybrid datastore =
Redis(in-memory DB, home-
brew clustering) +
HBase(persistent d...
When user sends a
message
LEGY
LEGY
talk-server
Storage
1. Find nearest LEGY
2. sendMessage(“Bob”, “Hello!”)
3. Proxy requ...
Communication between
internal systems
• Communication for queuing, transactional
updates:

• Query authentication/permiss...
Deeper look of some
core components
• Asynchronous HTTP/2 RPC/REST client/server library

• Built on top of Java8, Netty and RPC frameworks

• Open sourced: h...
DocService
• Quick API test call + Documentation
sb.serviceUnder("/docs", new DocServiceBuilder()
.exampleHttpHeaders(Http...
Other features
• Client side load balancer

• Retrieve server list from pluggable centralized
configuration directory and u...
Armeria
• Most of new services are built on top of Armeria

• and some old projects are/will be rewritten based on it

• S...
Apache Kafka
• A distributed streaming platform

• (narrow sense) A distributed persistent message queue
which supports Pu...
How it works
Producer
Brokers
Consumer
Topic
Topic
Consumer
Consumer
Producer
AuthEvent event = AuthEvent.newBuilder()
.se...
Consumer GroupA
Pub-Sub
Brokers
Consumer
Topic
Topic
Consumer
Consumer GroupB
Consumer
Consumer
Records[A, B, C…]
Records[...
Example: UserActivityEvent
Today’s Kafka usage
However…
• OSS usually isn’t a perfect solution

• We add/fix what we need

• Typically we need more than others, because o...
Sometimes we contribute
Sometimes we share
• Kafka Summit SF 2017

• One Day, One Data Hub, 100
Billion Messages: Kafka at
LINE

• https://youtu.b...
but we have more things to
do
• Performance improvement to enable true stability

• Kafka based reliable async task proces...
Summary
• LINE’s server architecture is keep evolving

• Almost everything are “Distributed”

• Software we use:

• Java/E...
End of presentation.
Any questions?
Prochain SlideShare
Chargement dans…5
×

LINE's messaging service architecture underlying more than 200 million monthly active users, and core techs

625 vues

Publié le

LINE Develoepr Meetup SF

Publié dans : Ingénierie
  • Soyez le premier à commenter

LINE's messaging service architecture underlying more than 200 million monthly active users, and core techs

  1. 1. LINE's messaging service architecture underlying more than 200 million monthly active users, and core techs Yuto Kawamura LINE Corp
  2. 2. Speaker introduction • Yuto Kawamura • Senior software engineer • Apache Kafka contributor • Joined: Apr, 2015(2 years 5 months)
  3. 3. About LINE •Messaging service •More than 200 million active users1 in countries with top market share like Japan, Taiwan, Thailand and Indonesia
 •Many family services •News •Music •LIVE (Video streaming) 
 1 As of June 2017. Sum of 4 countries: Japan, Taiwan, Thailand and Indonesia. 

  4. 4. Scale Metrics Messages delivered 25 billion per day Peak time Messages sent
 (new year greetings) 420K per second Total API requests per day 80 billion per day Number of event records published > 150 billion per day Accumulated analytics data 40PB
  5. 5. Messaging System Architecture LINE Apps LEGY JP LEGY DE LEGY SG Thrift RPC/HTTP talk-server Distributed Data Store Distributed async task processing
  6. 6. LEGY • LINE Event Delivery Gateway • API Gateway/Reverse Proxy • Written in Erlang • Deployed to many data centers all over the world • Features focused on needs of implementing a messaging service • Zero latency code hot swapping w/o closing client connections • Durability thanks to Erlang process and message passing • Single instance holds 100K ~ connection per instance => huge impact by single instance failure
  7. 7. talk-server • Java based web application server • Implements most of messaging functionality + some other features • Java8 + Spring + Thrift RPC + Tomcat8
  8. 8. Datastore with Redis and HBase • LINE’s hybrid datastore = Redis(in-memory DB, home- brew clustering) + HBase(persistent distributed key-value store) • Cascading failure handling • Async write in app • Async write from background task processor • Data correction batch Primary/ Backup talk-server Cache/ Primary Dual write
  9. 9. When user sends a message LEGY LEGY talk-server Storage 1. Find nearest LEGY 2. sendMessage(“Bob”, “Hello!”) 3. Proxy request 4. Write to storage talk-server X. fetchOps() 6. Proxy request 7. Read message 8. Return fetchOps() with message 5. Notify message arrival Alice Bob
  10. 10. Communication between internal systems • Communication for queuing, transactional updates: • Query authentication/permission • Synchronous updates • Communication for data synchronization, update notification: • Notify user’s relationship update • Synchronize data update with another service talk-server Auth Analytics Another Service HTTP/REST/RPC
  11. 11. Deeper look of some core components
  12. 12. • Asynchronous HTTP/2 RPC/REST client/server library • Built on top of Java8, Netty and RPC frameworks • Open sourced: https://github.com/line/armeria • Find more details: https://speakerdeck.com/trustin/ armeria-lines-next-generation-rpc-layer Armeria
  13. 13. DocService • Quick API test call + Documentation sb.serviceUnder("/docs", new DocServiceBuilder() .exampleHttpHeaders(HttpHeaders.of(AUTHORIZATION, "bearer b03c4fed1a")) .exampleRequest(new HelloService.hello_args("Armeria")) .build());
  14. 14. Other features • Client side load balancer • Retrieve server list from pluggable centralized configuration directory and use for load balancing • Structured logging • Supports logging request/response including payload of RPC • Supports external system as an output such as Apache Kafka
  15. 15. Armeria • Most of new services are built on top of Armeria • and some old projects are/will be rewritten based on it • Sophisticated yet minimum LINE’s “best practices” • PRs of line/armeria would be worth visiting for you • https://github.com/line/armeria/pulls • You will see how LINE developers are working for development
  16. 16. Apache Kafka • A distributed streaming platform • (narrow sense) A distributed persistent message queue which supports Pub-Sub model • Built-in load distribution • Built-in fail-over on both server(broker) and client
  17. 17. How it works Producer Brokers Consumer Topic Topic Consumer Consumer Producer AuthEvent event = AuthEvent.newBuilder() .setUserId(123) .setEventType(AuthEventType.REGISTER) .build(); producer.send(new ProducerRecord(“events", userId, event)); consumer = new KafkaConsumer("group.id" -> "group-A"); consumer.subscribe("events"); consumer.poll(100)… // => Record(key=123, value=...)
  18. 18. Consumer GroupA Pub-Sub Brokers Consumer Topic Topic Consumer Consumer GroupB Consumer Consumer Records[A, B, C…] Records[A, B, C…] • Multiple consumer “groups” can independently consume a single topic
  19. 19. Example: UserActivityEvent
  20. 20. Today’s Kafka usage
  21. 21. However… • OSS usually isn’t a perfect solution • We add/fix what we need • Typically we need more than others, because of our scale and challenging usages • https://issues.apache.org/jira/browse/KAFKA-4614 • https://issues.apache.org/jira/browse/KAFKA-4024
  22. 22. Sometimes we contribute
  23. 23. Sometimes we share • Kafka Summit SF 2017 • One Day, One Data Hub, 100 Billion Messages: Kafka at LINE • https://youtu.be/ X1zwbmLYPZg
  24. 24. but we have more things to do • Performance improvement to enable true stability • Kafka based reliable async task processing system • Multi-DC deployment
  25. 25. Summary • LINE’s server architecture is keep evolving • Almost everything are “Distributed” • Software we use: • Java/Earlang/Redis/HBase/Kafka/Armeria • We love OSS • we Use/Understand/Develop/Contribute • and it’s encouraged • 6 years since we started our service, still have many challenges • More stable and durable service architecture • Dealing with a lot of new feature developments • There are a lot of chances you make a great contribution to our systems and services, please come and talk to me!
  26. 26. End of presentation. Any questions?

×