Inside Zalo: Developing a mobile messenger for the audience of millions

1. Inside Zalo: Developing a mobile messenger for the
audience of millions
BY THANH DAO, VNG CORP.

2. About me
Career:
• VNG: Senior Manager - Head of Zalo & Zing
Mobile Product Group
• Past: Microsoft, Kofax
Contact:
• thanhdn@vng.com.vn
• http://me.zing.vn/thanhngocdao
• http://facebook.com/thanhdao

3. My Goals
• Talk about how we started a mobile first
product, growing the user base and the
learned lessons that lead us here.
• Give you a blueprint for how to scale your
mobile server system to millions.

4. Agenda
• Background of Zalo
• The user growth story
• Continuous development

5. Landscape before Zalo
5
Sticky, but...
• Don’t support mobile first
use-case and phonebook
graph.
• Not focus on enhance chat
and group chat
• My mom, sister don’t
contact me via SNS
Social Networks Traditional Messengers OTT IM
Early movers & viral, but…
• No profile, social features.
• Lack of fun features appeal
to the young users
• Can’t customize a feature
for the Vietnamese
community
• Low barrier to entry
Strong network effect on PC,
but...
• No social features
• Not fun enough
• Not mobile friendly

6. Model of Zalo
Mobile social messaging:
• Based on Mobile First & Phonebook Graph
• Combine Context Messaging & Connecting &
Circle Socializing
• Viral (i.e SMS/Zing/Facebook…)
• Sticky
• Mass market appeal

7. Target
18-25 26-30
Singles Core segment Core segment
Unmarried
couples
Potential to grow share ?
Potential to penetrate
Married
couples
X
Not target yet
X
Not target yet

8. Product
P
P
P
P
latform
roduct
1. Communication: Voice messaging & Call…
2. Social: Timeline, People Near By, Room…
3. Platform: Page, 3rd Apps, Games.
“Fun & Connection & Private”
Anchored by basic: Fast, Quality, Cheaproposition(Value)
lace (Channel)
1. App Stores: Itunes, Google Play, Nokia Store
2. Factory Preload on Nokia Asha, WP…
3. Side-load at retail shop

9. Marketing
• Position as an Expression/Lifestyle.
• Focus on messaging the brand benefit
– “Talk more to your loved ones”
• 360 Campaign launch at Online, Social, PR, TVC, OOH (Cinema,
University,…)

10. Agenda
• Background of Zalo
• The user growth story
• Continuous development

11. Agenda
• Background of Zalo
• The user growth story
• Continuous development

12. 0
5
10
15
20
25
Millions
Registered User Daily msg
User Discovery Q3/2012: Born
Product
• Zing & Phonebook mobile messenger
• voice, sticker, draw and people near by
Users feedback and data told us
• Good for dating purpose
• The draw & games, sticker are fun
• The concept of mixing up Zing &
phonebook graph is confusing, they want
a separated buddy list
• Sending speed is slow with 3G, drain the
battery lifecycle
• Nokia version is not available yet
So we
• Removed “Chat with Zing friends”
feature and focused on the Mobile First
use-cases
• Rewrote our own socket chat servers
instead of relaying via Zing Me Web Chat
servers

13. User Discovery Q4/2012: Totter
Product
• New version Nokia/Droid/IOS
• Group, Room
• Social Timeline
Users feedback and data told us
• Voice/text is fast
• UX is much better than the prev one
• Spam and privacy concerns
• Not enough friends so both retention and
engagement are problems
So we
• Improve notification services
• Improve picture quality
• Scale up advertising budget, Run 360
degree marketing campaign 0
5
10
15
20
25
Millions
Registered User Daily msg

14. User Discovery Q1/2013: Walk
Product
• Friend Circle
• Zalo Page (VIP)
• Friend invitation
• Support 18 countries
Users feedback and data told us
• Voice message is addictive
• Multi-picture upload is convenient
So we
• Run Friend Invite Campaign
• Developing Call.
• Work with 3RD party apps
0
5
10
15
20
25
Millions
Registered User Daily msg

15. Agenda
• Background of Zalo
• The user growth story
• Continuous development

16. Stats
• 2.3M registered users (40% Droid; 30% IOS;
30% Nokia).
• >500K new users join a month.
• 800K DAU
• 380K peak CCU
• 20M messages sent a day.
• 39 servers, 150M requests a day.

17. Server Platform
• CentOS
• Binary protocol, SSL, TCP/IP, UDP.
• C++ for core, java for other business services.
• Za-Router, long connection service map and load balancer.
• Za-DB: a key-value db storage similar to Amazon Dynamo/Google Big T.
• Za-Cache: memory caching system.
• Za-Session manager
• Za-CDN
• Za-Messaging/Signal Socket Server
• Za-Stream Download/Upload Server.
• Za-Voice/Video Media Proxy Server
• Za-Configuration center: synchronize config among services.
• Za-Monitor center + zabbix + nagios + cacti + dogslow: trend monitoring.
• Postgre SQL(PostGis): store lbs data, can handle >10k requests/sec with memory caching.
• Dynamic DNS
• HA Proxy: web load balancer(https login services).
• Squid: static content caching, small photo…
• Hadoop: data analyzing and mining
• Elastic search
• Scribe for log, backup…

18. ZAComm Router
ZA Socket Protocol
ZAComm #1
Worker
Worker Pool
Conn Manager
ZAComm #2
Worker Pool
Conn Manager
ZAComm #N
Worker Pool
Conn Manager
VIP
LBS
Room
Worker
Msg
Group
Share
Worker
Avatar
Photo
Cover
Worker
Contact
Privacy
Friend
MemCache
LBS
Persistent
Caching
Message Items
Persistent
Caching
Photo
Persistent
Caching
Background Job
Monitoring

19. Server Infrastructure
39 Servers:
• 6 relay storage servers
• 9 database servers
• 6 business application servers
• 3 proxy caching servers
• 4 other caching servers
• 4 backup servers
• 3 log and monitor/configuration servers
Capacity:
• 1M CCU
• I/O speed: 200K message/sec

20. Scalable Mobile Servers
• Scaling connection servers
– Each connection server handle a shard of users.
– Configuration center.
– Service map.
• Scaling database
– Distributed hash table design.
– Backup & recovery.
– No alter on large table, create new one.
– Soft deletion, Lazy write.
• N+1 design
– Build horizontally, never less than two of anything.
– No single point of failure.
• Think about monitor during design, not after.
• Identify where bottlenecks are in each tier
– Control Port(CPU).
– Data Port(Network/Memory/Cache/Storage IO…).

21. High Performance Mobile Client
• Speed:
– Different network type, different connecting strategy
• Use wi-fi as much as possible.
• Maximize bandwidth in fewer calls and group multiple
requests concurrently into a single multi-get request.
• Use bigger buffers.
– Decouple user transitions from data interactions:
• I.e Apply tricky like, silently uploading the image in
background during the user inputting description.
– Avoid large thumb:
• Send high-resolution thumb to high resolution devices and
low-resolution thumb to low resolution devices
– Cache more often, flush cache regularly
• Battery:
– HTTP versus TCP/IP reuse.
– Avoid constant polling(but be tricky when using
poll/push)…
– Download in bursts/chunks rather than continuously
streaming.
– Minimize aggressive behaviors
– Read/Write to cache then flushing a big mount of data
into device storage once.
– Use profiler tool to benchmark and measure your app
battery consume speed.

22. Operation
• Tracking user
feedbacks(CS, Store, FB, ZM, Noti5, Buzz…).
• Monitor everything:
– System performance(Server load/traffics, Memory, Network…).
– Business metrics.
– QoS statistics: Load-times; errors.
– Trending monitors.

23. Team & Process
• 27 developers: Server & Backend, Mobile Client, R&D framework.
• 2 system engineers
• 5 QA engineers
• 3 graphic designers: 1 artist & 2 graphic design
• 5 product managers, each person:
– Responsible for different group of features: acquisition, connection, communication, social…etc
– Has key business metric: conversion, retention, engagement, chat…and combined metrics.
• Scrum & continuous deployment process:
– Build isolation, enable testing single build without deploy.
– Auto Unit Test -> Integrating-> Staging -> Experiment -> Launch.

24. Questions?

25. Ready to embrace new
challenges?
We are hiring…
lepm@vng.com.vn
Thank you!