Introduction to cloudforecast

1. Introduction to
CloudForecast
Yokohama.pm #6.x
@kazeburo

2. •
• @kazeburo
• 2010 5
• 2010 6
• Web

3. CloudForecast
•
•
• perl rrdtool sysadmin
(by tokuhirom)

4. •
•

5. (1)

6. (1)
• ping

7. (1)
• ping
• HTTP,SSH

8. (1)
• ping
• HTTP,SSH
• Disk

9. (2)
•
•
•
• Nagios
• monit

10. (1)

11. (1)
•

12. (1)
•
• CPU

13. (2)
•
•
•
• Cacti
• munin
• Ganglia
• CloudForecast

14. CloudForecast
•
• YAML
• Web
• Perl!

15. CPAN
• Plack + Starlet
• Gearman
• Parallel::Prefork
• DBI DBD::SQLite
• YAML::Syck
• Text::Xslate
• Data::Section::Simple
• local::lib

16. • RRDTool RRDs.pm
• SNMP.pm
• Shirahata(WAF)

17. CloudForecast 1
RRD
Radar Web
SQLite

18. CloudForecast 1
(1)
RRD
Radar Web
SQLite

19. CloudForecast 1
(1)
RRD
Radar Web
SQLite
(2)

20. CloudForecast 1
(1)
RRD
Radar Web
SQLite
(2) (3)

21. CloudForecast 1
(1)
RRD
Radar Web
SQLite
(2) (3)

22. CloudForecast 2
fetcher
worker
RRD
update
Radar Web
worker
SQLite

23. CloudForecast 2
Gearman
Worker
fetcher
worker
RRD
update
Radar Web
worker
SQLite

24. CloudForecast 2
Gearman
Worker
RRD
update
Radar Web
worker
SQLite

26. 1
# ubuntu
$ sudo apt-get install librrds-perl libsnmp-perl
# CentOS rrdtool EPEL
$ sudo yum install net-snmp-perl
$ sudo yum install rrdtool-perl

27. 2
$ git clone git://github.com/kazeburo/cloudforecast.git
$ cd cloudforecast
$ cpanm -l extlib --installdeps .
$ ls
Changes cloudforecast_web
Makefile.PL docs
README host_config
cf_devstarter htdocs
cf_fetcher_worker lib
cf_updater_worker server_list_sample.yaml
cloudforecast_radar t
cloudforecast_sample.yaml

28. #
$ cp cloudforecast_sample.yaml cloudforecast.yaml
#
$ cp server_list_sample.yaml server_list.yaml

29. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
servers:
- config: basic.yaml
hosts:
....

30. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
servers:
- config: basic.yaml
hosts:
....

31. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
servers:
- config: basic.yaml
hosts:
....

32. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
IP
servers:
- config: basic.yaml
hosts:
....

33. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
IP
servers:
- config: basic.yaml
hosts:
....

34. (server_list.yaml)
--- #LOCALHOST
servers:
- config: basic.yaml
hosts:
- 127.0.0.1 server1 my great localhost
--- #My Servers
IP
servers:
- config: basic.yaml
hosts:
....

35. (basic.yaml)
---
component_config:
resources:
- basic
- traffic:eth0
basic = eth0 CPU TCP

36. (basic.yaml)
---
component_config:
resources:
- basic
- traffic:eth0
basic = eth0 CPU TCP

37. (basic.yaml)
---
component_config:
resources:
- basic
- traffic:eth0
basic = eth0 CPU TCP

38. • Basic(CPU TCP )
• Traffic
• Disk Usage
• DiskIO Count
• Apache Status
• Nginx Status
• Squid
• Mysql
• Innodb

39. $ perldoc CloudForecast::Data::

40. $ perldoc CloudForecast::Data::

41. CloudForecast
http://blog.nomadscafe.jp/2010/07/cloudforecast-1.html

42. #
$ ./cloudforecast_radar -c cloudforecast.yaml
-l server_list.yaml
# Web
$ ./cloudforecast_web -p 5000 -c cloudforecast.yaml
-l server_list.yaml

43. Web ( )

44. Web ( )

45. http://blog.nomadscafe.jp/2010/07/cloudforecast-2.html

47. •

48. •
•

49. •
•
•

50. # site-lib
$ mkdir -p site-lib/CloudForecast/Data

51. Gearman Worker
gearman worker process
http://blog.nomadscafe.jp/2010/07/gearman-worker-process.html

52. % ./gearman-starter.pl --max-prcess 8 -s 127.0.0.1:7004
--scoreboard-dir /var/run/worker --port 7005 MyWorker
% telnet localhost 7005
Trying 127.0.0.1...
System: gearman_servers: 127.0.0.1:7004 class: MyWorker
BusyWorkers: 0
IdleWorkers: 8
--
pid Status Counter Comment
1630 . 0
1631 . 0
1632 . 0
1633 . 0
1634 . 0
1635 . 0
1636 . 0
1637 . 0

53. % ./gearman-starter.pl --max-prcess 8 -s 127.0.0.1:7004
--scoreboard-dir /var/run/worker --port 7005 MyWorker
% telnet localhost 7005
Trying 127.0.0.1...
System: gearman_servers: 127.0.0.1:7004 class: MyWorker
BusyWorkers: 0
IdleWorkers: 8
--
pid Status Counter Comment
1630 . 0
1631 . 0
1632 . 0
1633 . 0
1634 . 0
1635 . 0
1636 . 0
1637 . 0

54. •
•

55. package CloudForecast::Data::Gearmanstarter;
use CloudForecast::Data -base;
use IO::Socket::INET;
=pod
host_config)
resources:
- gearmanstarter[:port]
eg)
- gearmanstarter # 9000
- gearmanstarter:9005 # port
=cut

56. rrds map { [$_,'GAUGE'] } qw/busy idle/;
# rrds ‘key’, ‘TYPE’ or rrds [‘key’,‘TYPE’], []..

57. • COUNTER
•
•
• ) Traffic ( octet )
• GAUGE
•
•
• )

58. (1)
graphs 'status' => 'worker status';
# graphs ‘key1’ => ‘label’;
# graphs ‘key2’ => ‘the great graph’;

59. (2)
graphs 'status' => 'worker status';
..
__DATA__
@@ status
DEF:my1=<%RRD%>:busy:AVERAGE
DEF:my2=<%RRD%>:idle:AVERAGE
AREA:my1#00C000:Busy
GPRINT:my1:LAST:Cur: %4.1lf
GPRINT:my1:AVERAGE:Ave: %4.1lf
GPRINT:my1:MAX:Max: %4.1lf
GPRINT:my1:MIN:Min: %4.1lfc
STACK:my2#0000C0:Idle
GPRINT:my2:LAST:Cur: %4.1lf
GPRINT:my2:AVERAGE:Ave: %4.1lf
GPRINT:my2:MAX:Max: %4.1lf
GPRINT:my2:MIN:Min: %4.1lfc

60. (2)
graphs 'status' => 'worker status';
..
key
__DATA__
@@ status
DEF:my1=<%RRD%>:busy:AVERAGE
DEF:my2=<%RRD%>:idle:AVERAGE
AREA:my1#00C000:Busy
GPRINT:my1:LAST:Cur: %4.1lf
GPRINT:my1:AVERAGE:Ave: %4.1lf
GPRINT:my1:MAX:Max: %4.1lf
GPRINT:my1:MIN:Min: %4.1lfc
STACK:my2#0000C0:Idle
GPRINT:my2:LAST:Cur: %4.1lf
GPRINT:my2:AVERAGE:Ave: %4.1lf
GPRINT:my2:MAX:Max: %4.1lf
GPRINT:my2:MIN:Min: %4.1lfc

61. fetcher {
my $c = shift;
my $host = $c->address;
my $port = $c->args->[0] || 9000;
my $sock = IO::Socket::INET->new(
PeerAddr => $host,
PeerPort => $port,
Proto => 'tcp',
);
my $raw_stats;
$sock->sysread( $raw_stats, 8192 );
my ($busy, $idle);
foreach my $line ( split /[rn]+/, $raw_stats ) {
if ( $line =~ /^Busy.+: (d+)/ ) {
$busy = $1;
}
if ( $line =~ /^Idle.+: (d+)/ ) {
$idle = $1;
}
}
return [$busy,$idle];
};

62. fetcher {
my $c = shift;
my $host = $c->address;
my $port = $c->args->[0] || 9000;
my $sock = IO::Socket::INET->new(
PeerAddr => $host,
PeerPort => $port,
Proto => 'tcp',
);
my $raw_stats;
$sock->sysread( $raw_stats, 8192 );
my ($busy, $idle);
foreach my $line ( split /[rn]+/, $raw_stats ) {
if ( $line =~ /^Busy.+: (d+)/ ) {
$busy = $1;
}
if ( $line =~ /^Idle.+: (d+)/ ) {
$idle = $1;
}
}
return [$busy,$idle];
};

63. •

65. IDLE
BUSY

66. TODO/Plan
• Web
• RRD SQLite
•

67. • CloudForecast Perl
•
•