Contenu connexe Similaire à 1台から500台までのMySQL運用(YAPC::Asia編) (20) Plus de Masahiro Nagano (20) 1台から500台までのMySQL運用(YAPC::Asia編)1. 1台から500台までの
MySQL運用
(YAPC::Asia2012編)
長野雅広 (kazeburo)
6. Recent CPAN modules
• Apache::BumpyLife
• Twiggy::Prefork
• Module::Install::ShareFile
• App::LoadWatcher
• File::RotateLogs
• Plack::Middleware::AxsLog
• Proclet
• DBIx::DSN::Resolver [new!]
7. Proclet
minimalistic supervisor
use Proclet::Declare;
env(
PLACK_ENV => 'development',
LM_COLOR => 1,
);
service('web', 'plackup -p 9413 app.psgi');
service('memcached', qw!/usr/bin/memcached -p 11211!);
service('worker', sub { MyWorker->run });
worker( worker => 2 );
run;
8. proclet
foreman for perl
% cat Procfile
worker: ./bin/worker
web: plackup web.psgi
% proclet start --color
19. 構成
PC mobile PC mobile
CMS APP CMS APP APP APP
Spam
STF Analyzer
Filter
Job Main Clustered
memcached
Queue DB DB
コア 画像 ログ スパム
20. Database Sharding
SELECT node
from blog
where
id = 941
App
① ②
Mapping Cluster Cluster Cluster Cluster Cluster
(Main DB) 1 2 3 4 5
すごく.. 古典的です..///
21. dbm dbm dbm dbm dbm
dbs dbs dbs dbs dbs dbs dbs dbs dbs dbs
dbm dbm dbm dbm dbm
dbs dbs 160cluster
dbs dbs dbs dbs dbs dbs dbs dbs
dbm dbm*1 dbs*2+
dbm dbm dbm dbm
dbs dbs
=> 500 servers
dbs dbs dbs dbs dbs dbs dbs dbs
dbm dbm dbm dbm dbm
dbs dbs dbs dbs dbs dbs dbs dbs dbs dbs
22. 歴史の積み重ね
• CPU: Pen4 ∼ Quad Core Xeon
• Memory: 2GB ∼ 16GB
• OS: FeeBSD・CentOS4.x∼5.x
• サーバによって異なるmy.cnf・
MyISAMのテーブル・Index不足
23. ,,、,、、,,,';i;'i,}、,、
ヾ、'i,';¦¦i !} 'i, ゙〃
゙、';¦i,! 'i i"i, 、__人_从_人__/し、_人_入
`、¦¦i ¦i i l¦, 、_)
',¦¦i }i ¦ ;,〃,, _) 集約だ∼っ!!
.}.¦¦¦¦ ¦ ! l-' 、ミ `)
,<.}¦¦¦ il/,-'liヾ;;ミ '́͡V^'^Y͡V^V͡W^Y͡
.{/゙'、}¦¦¦// .i¦ };;;ミ
Y,;- ー、 .i¦,];;彡
iil¦¦¦¦¦liill¦¦¦¦¦¦¦¦li!=H;;;ミミ
{ く;ァソ '';;,;'' ゙};;彡ミ
゙i [`''' ヾ. '' ¦¦^!,彡ミ _,,__
゙i } } ';;:;li, ゙iミミミ=三=-;;;;;;;;;''
,,,,---''''''} ̄ フハ, 二゙́ ,;/;;'_,;,7'' ,-''::;;;;;;;;;;;;;'',,=''
;;;;;;;;''''/_ / ¦ ¦ `ー--'́_,,,-',,r' `ヽ';;:;;;;;;;, '';;;-'''
''''' ,r' `V ヽニニニ二、-'{ 十 )__;;;;/
25. 集約化方法
m m
s s s s dbm
m m dbs dbs
s s s s
(1) 1台のサーバに複数のmysqld
(2) 1つのmysqldに複数のデータベース
26. 集約化方法 (1)
• Pros Cluster
1
Cluster
2
Cluster
3
Cluster
4
• レプリケーションを使いなが
ら、サービスの停止なく移行
可能
•
Cluster Cluster Cluster Cluster
Cons 1 2 3 4
• 元のサーバの性能差やデータ
mysqld
(3401)
mysqld
(3402)
mysqld
(3403)
mysqld
(3404)
量を考慮にいれたbuffer_pool
サイズの調整が必要 dbm
• 複数プロセスの管理
27. 集約化方法(2)
•
Cluster Cluster Cluster Cluster
Pros 1 2 3 4
• 細かい buffer_pool
の調整が必要なく、効率性も
高い
• プロセスが1つなので運用の Cluster Cluster Cluster Cluster
手間は少ない 1 2 3 4
• Cons
mysqld
• データベース名を変更するた
めのメンテナンス時間が必要 dbm
28. 採用
集約化方法(2)
•
Cluster Cluster Cluster Cluster
Pros 1 2 3 4
• 細かい buffer_pool
の調整が必要なく、効率性も
高い
• プロセスが1つなので運用の Cluster Cluster Cluster Cluster
手間は少ない 1 2 3 4
• Cons
mysqld
• データベース名を変更するた
めのメンテナンス時間が必要 dbm
30. 手順(前日まで)
Cluster Cluster Cluster Cluster
1 2 3 4
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
仮db mysqld
dbs
31. 手順(前日まで)
Cluster Cluster Cluster Cluster
1 2 3 4 レプリケーション
構築と
ALTER TABLE
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
仮db mysqld
dbs
32. 手順(前日まで)
Cluster Cluster Cluster Cluster
1 2 3 4 レプリケーション
構築と
ALTER TABLE
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
レプリケーション
仮db mysqld
dbs
33. 手順(当日)
Cluster Cluster Cluster Cluster
1 2 3 4
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
仮db mysqld
dbs
34. 手順(当日)
Cluster Cluster Cluster Cluster
1 2 3 4
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
仮db mysqld
dbs
35. 手順(当日)
Cluster Cluster Cluster Cluster
1 2 3 4
Cluster Cluster Cluster Cluster
1 2 3 4
Cluster Cluster Cluster Cluster mysqld
1 2 3 4
dbm
mysqld mysqld mysqld mysqld
(3401) (3402) (3403) (3404)
mysqldump cluster ¦
仮db mysqld
ssh dbm cat ¦ mysql ${n}_cluster
dbs
36. 手順(後日)
Cluster Cluster Cluster Cluster
1 2 3 4
mysqld
dbm
mysqld
仮db dbs
37. 手順(後日)
Cluster Cluster Cluster Cluster
1 2 3 4
mysqld
dbm
mysqld mysqld
仮db dbs dbs
39. えーマジ
手作業!?
自動化必須
手作業が許されるのは
小学生までだよねー
42. Matrix (表)
#cluster id #work2dbm
#from #to #id #work #dbm
1 1 1 10.94.18.30 10.94.14.30
51 1 2 10.94.18.31 10.94.14.31
57 1 3 10.94.18.32 10.94.14.32
65 1 4 10.94.18.33 10.94.14.33
2 2 5 10.94.18.34 10.94.14.34
39 2 6 10.94.19.30 10.94.15.30
77 2 7 10.94.19.31 10.94.15.31
91 2 8 10.94.19.32 10.94.15.32
3 3 9 10.94.19.33 10.94.15.33
29 3 10 10.94.19.34 10.94.15.34
85 3 11 10.94.14.35 10.94.16.35
4 4 12 10.94.14.36 10.94.16.36
52 4 13 10.94.14.37 10.94.16.37
78 4 14 10.94.14.38 10.94.16.38
43. Matrixからの設定生成
while(my $l=<DATA>){
my ($cluster, $node) = split /s+/, $l;
push @{$nodes{$node}}, $cluster;
}
for my $node ( sort { $a <=> $b } keys %nodes ) {
open( my $fh, '>', sprintf("my_%d.cnf",$node));
print $fh $tx->render('my.cnf.tx',
{ clusters => $nodes{$node} });
}
#from #to
__DATA__
1 1
51 1
2 2
45. Matrixを利用した自動化
set -e
CDIR=$(cd $(dirname $0) && pwd)
#id #work #dbm
SRV=$(cat <<"EOF"
1 10.94.18.30 10.94.14.30
2 10.94.18.31 10.94.14.31
3 ...
EOF
)
IPADDR=$(ifconfig eth1 |grep inet|
awk -F: '{print $2}'|awk '{print $1}')
CID=$(echo "$SRV" | grep $IPADDR | awk '{print $1}')
cp $CDIR/my_$CID.cnf /etc/my.cnf
for datadir in $(grep datadir /etc/my.cnf|sed 's/datadir=//')
do
/mysql_install_db --datadir=$datadir
done
46. atnodes を使った一括実行
$ cpanm SSH::Batch
$ atnodes ‘git clone git://./setup.git’ dbs02.[01-20].cluster
============== dbs02.01.cluster =============
..
============== dbs02.02.cluster =============
..
$ atnodes -c 4 ‘sh ./setup/setup.sh’ dbs02.[01-20].cluster
============== dbs02.01.cluster =============
..
============== dbs02.02.cluster =============
..
48. コマンド1つ!
$ atnodes -c 10 ‘sh ./setup/copy.sh’ dbs02.[01..20].cluster
============== dbs02.01.cluster =============
..
============== dbs02.02.cluster =============
..
51. Recreate Slave
Cluster Cluster Cluster Cluster
1 2 3 4
データ mysqld
複製 dbm
mysqld
dbs01
52. Recreate Slave
Cluster Cluster Cluster Cluster
1 2 3 4
データ mysqld
複製 dbm
mysqld mysqld
dbs02 dbs01
55. mysqldump (4.0.x)
“$ mysqldump --master-data”
blocks update query
MySQL 5.1.x MySQL 4.0.x
FLUSH LOCAL TABLES FLUSH TABLES WITH READ LOCK
FLUSH TABLES WITH READ LOCK BEGIN
SET SESSION TRANSACTION ISOLATION DB選択
LEVEL REPEATABLE READ テーブルからデータの読み込み
START TRANSACTION WITH
CONSISTENT SNAPSHOT COMMIT;
SHOW MASTER STATUS SHOW MASTER STATUS
UNLOCK TABLES UNLOCK TABLES;
DB選択
テーブルからデータの読み込み
57. MySQL mysql40dump
FLUSH TABLES WITH READ LOCK
SHOW MASTER/SLAVE STATUS
pipe my $rh, my $wh;
mysqldump
fork
open STDOUT, '>&', $wh
exec(mysqldump
--single-transaction)
BEGIN
while(<$rh>) { print }
pipe
UNLOCK TABLES
m!^CREATE TABLE! データ読み込み
COMMIT
wait($child) exit
print “CHANGE MASTER”
58. mysql40dump
$dbh->do('FLUSH TABLES WITH READ LOCK');
$dbh->select('SHOW MASTER STATUS');
pipe my $logrh, my $logwh;
my $pid = fork
if ( $pid == 0 ) { #子プロセス
open STDOUT, '>&', $logwh;
exec("mysqldump","--single-transaction","--
databases","...");
}
while(<$logrh>){
print;
if ( m!^CREATE DATABASE!) {
$dbh->do('UNLOCK TABLE');
}
}
59. mysql40dump
$ mysql40dump --master --repl
-- [2012-09-25T09:00:01] Done "FLUSH TABLES WITH READ LOCK"
-- [2012-09-25T09:00:01] [FROM MASTER STATUS] * CHANGE MASTER TO
MASTER_HOST='10.9.41.3', MASTER_USER='repl', MASTER_PASSWORD='xxx',
MASTER_LOG_FILE='mysql-bin.176', MASTER_LOG_POS=789372555;
-- [2012-09-25T09:00:01] [FROM SLAVE STATUS] CHANGE MASTER TO
MASTER_HOST='10.9.41.4', MASTER_USER='repl', MASTER_PASSWORD='xxx',
MASTER_LOG_FILE='mysql-bin.178', MASTER_LOG_POS=103928719;
-- [2012-09-25T09:00:01] [START] mysqldump --quick --add-locks --
extended-insert --single-transaction --databases tbl1 tbl2 tbl3
set FOREIGN_KEY_CHECKS=0; -- for mysql4.0
-- MySQL dump 9.11
...DUMP DATA...
CHANGE MASTER TO MASTER_HOST='10.9.41.3', MASTER_USER='repl',
MASTER_PASSWORD='xxx', MASTER_LOG_FILE='mysql-bin.176',
MASTER_LOG_POS=789372555;
START SLAVE;
60. mysql40dumpの利用例
• slave作成
# masterからslave
$ mysql40dump --master --repl | ssh slave ‘cat | mysql’
# slave から slave
$ mysql40dump --slave --repl | ssh slave2 ‘cat | mysql’
• 定期的なバックアップ実行
mysql40dump --master |
gzip > /backup/mysql-$(date +%Y%m%d%H%M).sql.gz
65. Uptime監視
OK∼ OK∼ OK∼
Nagios Nagios Nagios
5min 5min
t
DOWN UP
MySQL 4.0.xの再起動は速い
66. Uptime監視
> 5min < 5min > 5min
OK∼ NG!!1 OK∼
Nagios Nagios Nagios
show status like show status like show status like
'Uptime' 'Uptime' 'Uptime'
5min 5min
t
DOWN UP
72. Splogの痕跡
バイナリログの調査
% mysqlbinlog --start-datetime ‘2012-09-21 22:50’ --end-
datetime ‘2012-09-21 23:00’ mysql-bin.941 | perl -e
'while(<>){ chomp; next if m!^#!; if ( m{/*!*/;$} ) { $p .=
$_; print "$pn"; $p="" } else { $p .= $_." "} }'|perl -nle
'm!^(DELETE FROM|REPLACE INTO|INSERT INTO|UPDATE)s+([^ ]+)!i
&& print "$1 $2"' | sort | uniq -c | sort -nr | head
43 INSERT INTO entry
33 UPDATE entry
19 INSERT INTO tag2entry
18 UPDATE tag
17 INSERT INTO category
73. Splogの痕跡MySQL
バイナリログの調査 5.0∼
% mysqlbinlog --start-datetime ‘2012-09-21 22:50’ --end-
datetime ‘2012-09-21 23:00’ mysql-bin.941 | perl -e
'while(<>){ chomp; next if m!^#!; if ( m{/*!*/;$} ) { $p .=
$_; print "$pn"; $p="" } else { $p .= $_." "} }'|perl -nle
'm!^(DELETE FROM|REPLACE INTO|INSERT INTO|UPDATE)s+([^ ]+)!i
&& print "$1 $2"' | sort | uniq -c | sort -nr | head
43 INSERT INTO entry
33 UPDATE entry
19 INSERT INTO tag2entry
18 UPDATE tag
17 INSERT INTO category
78. HDD/SSD
Performance Test
HDD-RAID10 SSD-RAID1
Intel SSD 910 某Flush Device
50,000
43,117
37,500
25,000
14,153
12,500 9,440
751
0
4並列 2GB Random Read
Notes de l'éditeur \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n