1. Softwares
on Large Scale Internet Services
2013/07/18 at Tokyo Univ.
SATOSHI TAGOMORI (@tagomoris)
LINE Corp.

2. topics
About LINE Corp.
Service scales
Large is worse
Internet (Web) service architecture
Case studies on LINE Corp.
Q&A

3. TAGOMORI Satoshi (@tagomoris)
LINE corp.

4. tagomoris: 田籠 聡
-2005.03:
東京大学計数工学科システム情報工学コース
新研究室
2005.04-2006.03: 朝日ネット
2006.03-2010.06: NTTデータ先端技術
2010.08-:
ライブドア → NHN Japan → LINE

6. Service scales
small, middle, large and huge scale
Scale(1): Users, traffics, data amount, market share, ...
Scale(2): Servers, programmers, lines of code, ...

7. Service scales
small middle large huge
users ∼100k ∼1m ∼10m 100m∼
servers 1, 2 or 3 ... tens handreds
thousands
or more
ex. … …
livedoor blog
cookpad
Ameba, 楽天, ...
Google, Twitter
Facebook
Amazon, ...

8. More/less servers
Servers
less: computing power-- / management cost--
more: computing power++ / management cost++
Server types (complexity)
less: efficiency-- / management cost--
more: efficiency++ / management cost++

9. Large is worse:
more traffics needs more computing power
less server types more server types
less servers
more servers
small middle
huge large

10. Small scale internet (web) service
server-side architecture
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
Web Server + Application Server
(Apache + PHP, nginx + passenger(ruby), ...)
External API

11. Middle scale internet (web) service
server-side architecture
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
External API
Application Server
(Java, Perl, Ruby, ...)
Caches
(memory, disk)
Web Server (Apache, nginx, ...)

12. Large scale internet (web) service
server-side architecture
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
External API
Application Server
(Java, Perl, Ruby, ...)
Caches
(memory, disk)
CDN
Storage
Service
Web Server (Apache, nginx, ...)
Data
analysis

13. Huge scale internet (web) service
server-side architecture
Users (PC, smartphone, mobilephone, ...)
Distributed datastore
(GFS+BigTable, HDFS+HBase, ...)
Application server farm (IaaS, PaaS)
Web server cluster + CDN

14. Software engineering:
for large scale services
Performance: application, database, storage, cpu
Stability: network, high availability, monitoring, logging
Low costs: deployment, continuous testing
Incomes: response time, data analysis, visualization

15. Our scope in server side
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
External API
Application Server
(Java, Perl, Ruby, ...)
Caches
(memory, disk)
CDN
Storage
Service
Web Server (Apache, nginx, ...)
Data
analysis

16. Our scope in server side
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
External API
Application Server
(Java, Perl, Ruby, ...)
Caches
(memory, disk)
CDN
Storage
Service
Web Server (Apache, nginx, ...)
Data
analysis

17. BREAK.

18. CASE STUDIES OF
LINE CORP.

19. LINE secret sale:
traffic spike
Heavy traffic on sale time
transactions
remaining updates
image outbounds
SSD
Application tuning
transactions
cache updates

20. LINE manga:
content delivery
content (image) delivery
STF
distributed object
storage

21. livedoor blog:
heavy traffic
STF based image storage
+ cache
CDN
MySQL database cluster
Server Side Include (SSI)
Access analysis tools

22. system administration:
server setup, deployment, monitoring, ...
server setup / deployment:
automation(kickstart), testing(serverspec)
itempotent deployment(chef, puppet)
transactional operation
monitoring:
server/switch metrics
CDN API integration

23. Service monitoring:
Hadoop, stream processing, ...
Daily service metrics (hadoop+hive)
Pageviews, Unique users, ...
(Semi-)Real-time service metrics (fluentd+norikra)
Req/sec, Error rate, Response time percentile, ...
+ system integration
autoscaling, service status for end users, ...

24. micro level tunings
Log archive server
linux md raid-6 → hardware raid-1+0
I/O performance dramatically decreased
I/O schedular: CFQ → deadline
Stream processing node (ruby, fluentd)
Increasing memory usage (fragmentation)
Memory allocator: glibc malloc → jemalloc

25. Our scope in server side
Users (PC, smartphone, mobilephone, ...)
Database
(RDBMS, NoSQL)
External API
Application Server
(Java, Perl, Ruby, ...)
Caches
(memory, disk)
CDN
Storage
Service
Web Server (Apache, nginx, ...)
Data
analysis

26. Our scope of computer stack
PC/smartphone browser iOS/Android App
web servers / application servers
application code
language runtime object stores RDBMS / KVS
OS
server, NIC, memory, HDD/SSD, ...
network, firewall, load balancing
authentication(policies), audit
logger / watchdogs / cron / ...
HTML/CSS, Javascript ...

27. Softwares
on large scale internet services
All of available things!
linux, lvs, haproxy, apache, nginx, squid, varnish,
memcached, mysql, kyoto tycoon, mongodb,
cassandra, redis, riak, perl, ruby, php, python, java,
javascript, node.js, hadoop, hbase, hive, fluentd, ...
except for
global-scale distributed systems
processor design

28. ALL OPTIONS
ARE AVAILABLE
IN
SERVICE PROVIDERS.