presentation of thesponcer session from RECRUIT

1. Experiences of
Rails site stabilized
operation
Hirotomo Oi
Chief Architect
Media Technology Lab., Recruit Co.,Ltd.

2. Agenda
• Our “Rails” websites
• Software
• httpd
• Server virtualization
• Tips

3. About Me
• Hirotomo Ōi
• Chief Architect of infrastructure of web.
• And “engekilife.com” tech guy.
• And...planning/developing new website.

4. About MTL
• MTL(Media Technology Lab.) is an
applied study institute to promote future
media communication, and to seek for a
sympathetic world for all of us.
• One of our focus:
Develop prototype web sites, and verify
responses and feedbacks through beta
releases.

5. MTL Websites

6. MTL Websites

7. MTL Websites

8. Software
• Httpd for Ruby Apps.
• Virtualization

9. Rails Version
• Use various versions.
• 1.2.x, 2.0.x, 2.1.x, 2.2.x, 2.3,x...
• We choose the newest version at
launch time of each site.

10. httpd
• ...Donʼt fix, now trying...
• Mongrel
• Thin
• Passenger
• Some sites were changed from
Mongrel.

11. Benchmark
Time per request(ms):
31.00
23.25
15.50
7.75
0
Mongrel Passenger

12. Ruby Version
• 1.8.6
• If apps didnʼt work, use another vers.
• Ruby Enterprise Edition
• for Passenger

13. Virtualization

14. Virtualization
+

15. Virtualize servers
• Two or more servers
can be running on one
Web,
hardware. App DB cache
(memory-
(CPU- (IO-bound)
bound)
• Generally, different bound)
resources are put
together is good
efficiency.

16. virtualize webservers
• We are virtualizing
web+app tier.
• Generally web site A
(ruby)
site B
(php)
site C
(perl)
apps are CPU-
bound.
• RoR apps need
memory.

17. How many VMs?
• Hardware
• 16GB memory
• 2x Quad Core Xeon
• VM Spec
• 1GB memory
• 10GB disk space
• 8 CPU cores
available

18. How many VMs?
• Hardware
• 16GB memory
• 2x Quad Core Xeon
• VM Spec 13
• 1GB memory VMs / Hard
• 10GB disk space
• 8 CPU cores
available

19. How many
instances?

20. How many
instances?
• 10 Mongrel instances / 1GB mem VM.
• 24 Mongrel instances / 2GB mem VM

21. Tips
• DB Tuning
• Log rotation
• Restart Mongrel

22. DB Tuning
• Developers often forget DB tuning when
launch the site.
• Developers use ActiveRecord, not
connect DB directly.

23. DB Tuning
• Developers often forget DB tuning when
launch the site.
• Developers use ActiveRecord, not
connect DB directly.
DO NOT FORGET DATABASE!!

24. Get Slow Query Log
• Step 1: know slow query.
• my.cnf
[mysqld]
log-slow-queries
long_query_time=1

25. EXPLAIN
• Step 2: Analysis of slow query
• Use “EXPLAIN”
mysql> EXPLAIN SELECT...Your Slow Query;

26. EXPLAIN
• Step 2: Analysis of slow query
• Use “EXPLAIN”
mysql> EXPLAIN SELECT...Your Slow Query;

27. Fix Slow Query
• Add Index
• Examine to change (ex:divide) the
query
• Donʼt use join...
• Donʼt use subquery...

28. ex)
mysql> show create table rubykaigi_demo
G
| rubykaigi_demo | CREATE TABLE
`rubykaigi_demo` (
`id` int(11) NOT NULL auto_increment,
`type` varchar(255) default NULL,
`bukken_id` int(11) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

29. 1) Know Slow Query
mysql> SELECT * FROM `rubykaigi_demo`
WHERE `type` = 'PlayguideTicketInfoShow'
and `bukken_id` = 435;
+-------+-------------------------+-----------+
| id | type | bukken_id |
+-------+-------------------------+-----------+
| 13139 | PlayguideTicketInfoShow | 435 |
+-------+-------------------------+-----------+
1 row in set (0.01 sec)

30. 2) EXPLAIN
mysql> EXPLAIN SELECT * FROM
`rubykaigi_demo` WHERE `type` =
'PlayguideTicketInfoShow' and `bukken_id`
= 435G
id: 1
select_type: SIMPLE
table: rubykaigi_demo
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 2299
Extra: Using where

31. 2) EXPLAIN
mysql> EXPLAIN SELECT * FROM
`rubykaigi_demo` WHERE `type` =
'PlayguideTicketInfoShow' and `bukken_id`
= 435G
id: 1
select_type: SIMPLE
table: rubykaigi_demo
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL Not Using Index
rows: 2299
Extra: Using where

32. 3) ADD INDEX
mysql> ALTER TABLE `rubykaigi_demo`
ADD INDEX( `type`, `bukken_id`);
Query OK, 1977 rows affected (0.18 sec)
Records: 1977 Duplicates: 0 Warnings: 0
or migration:
add_index "type_bukken", ["type",
"bukken_id"], :name => "type_bukken"

33. 4) RESULT
mysql> EXPLAIN SELECT * FROM
`rubykaigi_demo` WHERE `type` =
'PlayguideTicketInfoShow' and `bukken_id`
= 435G
id: 1
select_type: SIMPLE
table: rubykaigi_demo
type: ref
possible_keys: type
key: type
key_len: 773
ref: const,const
rows: 1
Extra: Using where; Using index

34. 4) RESULT
mysql> EXPLAIN SELECT * FROM
`rubykaigi_demo` WHERE `type` =
'PlayguideTicketInfoShow' and `bukken_id`
= 435G
id: 1
select_type: SIMPLE
table: rubykaigi_demo
type: ref
possible_keys: type
key: type
key_len: 773
ref: const,const Using Index
rows: 1
Extra: Using where; Using index

35. Log rotation
• “production.log” increase continuously.
• Itʼs necessary to rotate logs.
ex) logrotate.d conf file
(Your Rails App Log Dir)/*.log {
daily
missingok
rotate 30
compress
notifempty
copytruncate
create 0644 user group
}

36. Log rotation
• “production.log” increase continuously.
• Itʼs necessary to rotate logs.
ex) logrotate.d conf file
(Your Rails App Log Dir)/*.log {
daily
missingok
rotate 30
compress
notifempty
copytruncate Important!
create 0644 user group
}

37. Restart Mongrel
• Restart periodically for fix memory leak
of Mongrel processes.
• Using swap, not only the VM but all
VMs on same HW are slow down...

38. Restart Mongrel
• Cron
• Logrotate.d
• Monit

39. Agenda
• Our “Rails” websites
• Softwares
• httpd, virtualization,...
• Tips
• Donʼt forget DB Tuning
• Donʼt forget log rotation
• Restart Mongrel

40. ...one more thing

41. Comming Soon!!
http://mashupaward.jp/

42. Thank you
Please contact : http://mtl.recruit.co.jp/