Presentation for Mongo Tokyo 2012. Talking about Animal Land, which uses MongoDB for the primary database.

1. The Case for using
MongoDB in Social Game
Masakazu Matsushita
Cyberagent, Inc.

2. About Me
• Masakazu Matsushita
• @matsukaz
• Cyberagent, Inc. - SocialApps Division
- Ameba Pico (2011/01∼)
- Animal Land (2011/03∼)
• DevLOVE Staff

3. Agenda
• About Animal Land
• System Overview
• Why MongoDB ?
• Considered Points
• Troubles
• Current Problems

4. About
Animal Land

5. Demo
http://apps.facebook.com/animal-land/

6. Development Period
• Started on 2011/03
• First meeting was held on 3.11
• Released on 2011/12/09

7. Team Member
• Producer 2
• Designer 1
• Flash Developer 3
• Engineer 4
+α

8. System
Overview

9. Using Cloud Services
• Amazon Web Services
- EC2 (Instance Store + EBS)
- S3
- CloudFront
- Route53
- Elastic Load Balancing
• Google Apps
• Kontagent

10. System Relations
iframe API Call Callback for
Payment API
Web
HTML
Server
JSON
Flash
Command
AMF Server
AMF : Actionscript Message Format

11. L m1.large
Servers XX
EBS
m2.2xlarge
EBS
ELB ELB
S3
Web L 3 Command L 4 CloudFront
nginx nginx Route 53
Tomcat Tomcat
バッチ L
mongos mongos
バッチ
MySQL
Shard 5 MySQL L memcached L 2
monitor L
MongoDB XX MySQL memcached
EBS munin
mongod
MySQL L nagios
MongoDB XX
MySQL admin L ビルド L
mongod EBS
nginx redmine
MongoDB XX MongoDB XX 3 Tomcat jenkins
mongod mongoc
EBS EBS mongos Maven SVN
EBS
Secondary

12. Middleware
• nginx 1.0.x
• Tomcat 7.0.x
• MongoDB 2.0.1
• MySQL 5.5.x
• memcached 1.4.x

13. Framework / Libraries
• Spring Framework、Spring MVC
• BlazeDS、Spring Flex
• Spring Data - MongoDB 1.0.0 M4
• mongo-java-driver 2.6.5
• Ehcache
• spymemcached
• RestFB
• MyBatis

14. Why MongoDB ?

15. • Used in Ameba Pico
http://apps.facebook.com/amebapico/

16. • I like MongoDB features !!
- ReplicaSet
- Auto-Sharding
- Can handle complex data structures
- Index, Advanced Queries
- Developed actively
- Quite easy to understand

17. • Fits in Animal Land requirements
- Complex data structures (City s grid
data, user/structure parameters, ...)
- Most processes run sequencially and
doesn t update same data at a time
- Maintenance-less
• Change data structures
dynamically
• Durability, Scalability

18. • Resolve MongoDB s weak points by
other way
- Some data are stored in other DB
• Payments history needs reliability,
so they are stored in MySQL
• Temporary data are stored in
memcached
- Don t think of using transactions

19. Considered
Points

20. Developing Application
• Define data without using transaction
- User data are defined in 1 document
and updated only once in block
User Data
{ facebookId : xxx,
Image
status : { lv : 10, coin : 9999, ... },
layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0... ,
structure : {
1¦1 : { id : xxxx, depth : 3, width : 3, aspect : 1, ... }, ...
},
inventory : [ { id : xxx, size : xxx, ... }, ... ],
neighbor : [ { id : xxx, newFlg : false, ... }, ... ],
animal : [ { id : xxx, color : 0, pos : 20¦20 , ... } ],
...
}

21. Developing Application
• Cutdown data size as possible
- City s grid data are stored in 1 field
(Expanded when application uses)
- Data in design phase（500 KB）
layerInfo : {
1¦1 : 0,
1¦2 : 1,
....
}
- Current data（50 KB）
layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0...

22. Developing Application
• Careful not to define too many fields
- It took more than 5 sec for find()
when data contained more than
20,000 fields (144x144 City s grid
data)
- Consider data size, and also for the
time to parse BSON

23. Developing Application
• Shard Key is decided in following
policy
- Don t use low-cardinality key
- Recent used data should be on
memory, and data not used should
not be
- Frequently read/write data should be
in each shard in nice balance

24. Developing Application
• Use targeted operation as possible
- Use Shard Key
- Use index for non Shard Key op
Operation Type
db.foo.find({ShardKey : 1}) Targeted
db.foo.find({ShardKey : 1, NonShardKey : 1}) Targeted
db.foo.find({NonShardKey : 1}) Global
db.foo.find() Global
db.foo.insert(<object>) Targeted
db.foo.update({ShardKey : 1}, <object>) Targeted
db.foo.remove({ShardKey : 1})
db.foo.update({NonShardKey : 1}, <object>) Global
db.foo.remove({NonShardKey : 1})

25. Developing Application
• Decrease update frequency
- Store master data in memory
- Queue mechanism in Flash
- User operations are processed in
block (once per 5 sec)
• Server processes sequencially
Processes
User op. Stored in
Sequencially
Queue
Queue
Command
Flash
Server
Send in Block
once per 5 sec

26. Developing Application
• Develop efficiently using O/R Mapper
- Spring Data - MongoDB and
Wrapper classes
@Autowired
protected MongoTemplate mongoTemplate;
public void insert(T entity) {
mongoTemplate.save(entity);
}
- Can use any serializable object
- Maybe easier to deal with than RDB
O/R Mapper

27. Developing Application
• Implements without expecting rollback
- Give up for some inconsistent data
- Careful not to inflict disadvantages
to user

28. Constructing Infrastructure
• Estimate same as other DBs
- Data size/user (50 KB)
- Expected user (DAU 70万)
- Update frequency
- Each servers max connections
- Number of servers, specs, costs
- Consider when to scale servers
according to user growth

29. Constructing Infrastructure
• Performance Verification
- Bandwith (EC2 is about 350Mbps)
- Verify MongoDB (MongoDB 2.0.1、
2Shard with Replica Set、enable
journaling)
• READ/WRITE performance
• Performance during migration
• Performance through Java Driver
- Performance through application

30. Constructing Infrastructure
• Prepare for troubles
- When mongod dies ...
• SECONDARY becomes PRIMARY ?
• Data synchronize when it restart ?
• Safe when SECONDARY dies ?
- When mongoc dies ...
• Nothing happens ?
- Succeed to restore from backup ?
→ no problem

31. Constructing Infrastructure
• ReplicaSet and Shard Construction
- Use large memory size
- Use high speed I/O disk
- Place mongoc independent from
mongod
- Use EBS to raise reliability (run only
as SECONDARY)
• Enable Journaling
• Set oplogsize as 10GB

32. Constructing Infrastructure
• Create index in advance
- Index blocks all operation ...
- Took about 2 min to create index for
200,000 data (each about 50KB)
- Create index during maintanance
time or in background for running
services

33. Constructing Infrastructure
• Connection Pool Tuning
※ connnection pool is for mongos
property description value
connectionsPerHost number of connections 100
threadsAllowedToBlockFor wait threads per
4
ConnectionMultiplier connection
- set the value with nginx worker and
Tomcat thread size in mind
- Careful with Out of Semaphores
errors. Above case would be
100 + (100 * 4) = 500

34. In Operation
• Check chunk balance routinely
- Move chunk manually if needed
• Careful when adding new Collections
- Primary Shard might be overloaded
because new collection data will be
placed there

35. Troubles

36. mongod & mongoc died..
• Caused by virtual server host for EC2
instance. Nothing to do with MongoDB
• There was NO impact for service !!
• Recovered just starting mongod and
mongoc processes

37. Current
Problems

38. Online Backup
• Difficult to backup for running services
• Backup while in maintenance mode
• Thinking of doing backup from
SECONDARY as follows, but need
some verification ...
1. set balancer off
2. write lock SECONDARY and backup
3. set balance on

39. Upgrade
• Upgrade is also difficult for the timing
• Need to do following steps
1. Stop Arbiter, upgrade, run
2. Stop SECONDARY, upgrade, run
3. Stop PRIMARY, upgrade, run

40. Adding Shards
• It s possible to add shards later on,
but have to consider it s performance
for balancing
- Pico is adding shards while in
maintenance mode for safety

41. Best Chunksize
• Migration frequently occurs, because
user data was too large and chunksize
was small (default 64MB)
• Need to adjust chunksize according to
it s data size
Maybe nice if you can set the chunksize
for each collections ...

42. Analyzing Data
• Hard to analyze user data, because we
handle the data in 1 document
• Creating index if necessary
- Tested MapReduce, but some
performance problems occured. Not
using it right now

43. Thank you !
Please enjoy Animal Land !!
http://apps.facebook.com/animal-land/