Practical resource monitoring with munin (English editon)

Munin User Group Japan http://munin.jp/
Masahito Zembutsu @zembutsu
September 8, 2012 OpenSource Conference 2012 Tokyo/Fall (#osc12tk)

“Practical Resource Monitoring with Munin - English Edition”

Nice to meet you. I’m @zembutsu.
Thank you for giving an opportunity of the presentation to me!

They are characters of Touhou Project, and "Please take it easy!!"(yukkuri site itte ne!) is one of the famous slang in Japan.
http://en.wikipedia.org/wiki/Touhou_Project

Today’s topic is…
Recipe for monitoring

I’m expecting that various burdens on
operation engineers reduce it by utilizing

Munin.

Why Am I Here? this is me

• Masahito ZEMBUTSU @zembutsu

– Solutions Engineer ( fiery zeal Otaku mind engineer )
• Working as a server infrastructure engineer.
• I want to provide relaxation and rest for theengineers.(Operation/Monitoring/Automation)
– Communities of an opensource and the cloud computing
• My website http://pocketstudio.jp/
– Experience http://opencloud.jp/ http://jaws-ug.jp/
• April 2000 - Support engineer of server hosting and the ISP
• May 2008 - Company internal network management and support
• November 2010 - Service development and upper escalation operation
Don’t mind the careful thing!
• July 2012 – Operation, Development, Research
at datacenter somewhere.

filleting mackerel
It resembles server operations.

DevOps!

―Don't forget. always, somewhere,
someone is fighting for you.
―As long as you remember her.
you are not alone.
Operation (Reference: “Puella Magi Madoka Magica” Episode 12 “My Very Best Friend” )

Monitoring

This is an image photograph of the data center that I’m working.

This Photo is under creative commons license by torkildr
http://www.flickr.com/photos/torkildr/3462606643/sizes/l/in/photostream/

A Dedicated Hosting Services
A HUMAN WORK
Shutdown Attack, An Unfamiliar Specifications,
Cloudcomputing’s Arrival in Japan, Shape of Server,
A Business That’s Changing, My Purest Heart for Our Customers.

Troubleshooting
DECISIVE BATTLE
The Phone That Never Stop Ringing, The Day a Datacenter Stood Still,
The Choice of Priority, In sickness unto shutdown, and…, Sales
Representative’s Invasion, Customer’s office the Throne of Souls, Tears.

You’re a loser only when you fail to try
We can (not) advance.
The Birth of Special task force, The Value of Miracles, At Least, Be Human.

If you are a server administrator,
you will have thought once.
Perhaps…

My Little Servers Can't Be This Heavy...
But, it may work with Munin
and a solution of the problem.

This is that I want to do a share today.
• I think that it is necessary to adopt resource
monitoring for an operative flow.

• As a result, it may reduce the burden on
administrators. I'm extremely happy. XD

• We need is the culture to leave the office on time!!
(Only as for me?)

Agenda
1. What is Munin?

2. Munin’s Architectre

3. How to use Munin

4. Practical trobleshootings!

5. MY VERY BEST MONITORING TOOL

I hope…
1. Let's obtain a weapon
called “resource monitoring” for us.
Wille zur Macht

2. We improve the efficiency of
our working (server and network operations).

“Let's find happiness together.”
(Reference: Kiichi Goto, Patlabor: The Movie, 1989) I guess everybody's happy, that's fine.

Munin User Group Japan
Munin Community in Japan

Munin.jp
• Munin User Group Japan
– http://munin.jp/

• Wiki
– http://munin.jp/wiki/

• Demo
– http://demo.munin.jp/

• How to join us
– http://munin.jp/mailing-list/

I dare to say, Munin is
resource monitoring tool.
Munin is a networked resource monitoring tool.
I dare to say!!

This is overview (frontpage) of Munin 2.0

When you click it any, a graph of a day / week / month / year are displays it

This zooming function is convenient. It is useful for me to make a report.

Vertical axis is a server, cross axle is metrics. The grouping function is characteristic.

The Melancholy of
Server Administrators
Monitoring is like a box of chocolate

By the Time we Realized It, It Had Already Begun.

• troubles - alert systems can’t detect it (increased)
– Mainly clientage for Social Networking Service
– When the threshold of the alert exceeds it, it is already late.

• demand of the clientage – rapidly response
– Because a loss per one second is wrong number of digits
than before.
– a loss of several hundred dollars / minutes :(

“There is something weird, will you check servers? :)”
Request from my customer of us
• Very difficult request...
– Clear cause identification often takes time.

• I want to do my best more!
– Yes!! I stir myself and go to work.
Administrators got exhausted…
– I want to aim at the service improvement, but this
thought is bad. Why? Let’s see the next slide.

An old network constitution.

One web server and one database server.
It’s very simple!

An old network constitution.

If it was a general
Web server, it was
such a constitution
BIND
to the utmost.

One web server and one database server.
It’s very simple!

On the other hand, at present…

This Just Can't Be Right!!

BIND

Number and the management objects of the server are increasing in comparison with the past.
Therefore support takes the time, and the degree of difficulty rises, too.

Why did this happen?
• On the changing environment
– Network
– Server
– Software
– Middleware
– Application
– etc

Be freed from CONSOLE
Ace Console: Fires of Liberation

most important thing, by troubleshooting
• Cause investigation work has top priority.
“When we act, it is a first thing to do condition to notice.
If there is a technique, anything cannot be settled. It
becomes necessary to notice before a technique. The
technical expert is in Japan no matter how much, but
cannot be readily settled. The reason is because it
does not notice.”
Soichiro Honda (2008) "akku baran” (candidness ) PHP inc, 10pp.
http://en.wikipedia.org/wiki/Soichiro_Honda

You sure that’s enough armor(tools)?
• “No problem. Everything’s fine.”
– ps
– top
– vmstat
– iostat
– free
– sar (sysstat) …etc
Really?

Situation has changed
Past Now (present day present time, hahaha!!)
• One or several servers • Plural servers in the same
• Apache, Sendmail, Perl network (we assume)
• PostgreSQL, MySQL • Conventional software +
nginx,Tomcat,ruby,PHP,Python,memcac
• Network appliance hed,Key-Value
(sometimes) Store,Hadoop,Cassandra,MongoDB…etc

• No scale • The need for scalability
• Upgrading is effective • Upgrading is not effective

I think that one of the answers to this problem
is resource monitoring using Munin.

The essence of Munin is
many resources visualization
I Know What Your Server Did Last Summer

MRTG has declined
Is This MRTG? No, This Is Munin.

We have lost a hero to our glorious and noble cause, but does this foreshadow our defeat?
No. It is a new beginning. Compared to Cloud Computing Federation the national resources
of Dedicated Server are less than one thirtieth of theirs. Despite this major difference,
how is it that we have been able to fight the fight for so long? It is because our goal in
this war is a righteous one. It’s been over fifty years since the elite of Cloud Computing,
consumed by greed took control of the Cloud Computing Federation. We want our freedom.
Never forget the times when the Federation has trampled us! We, the Principality of
Dedicated Server, have had a long and arduous struggle to achieve freedom for all
e n g i n e er s o f o u r g r e a t n e t w or k . Ou r f i g h t i s s a cr ed , ou r c a u s e d i v i n e .
My beloved brother, MRTG, was sacrificed. Why? The war is at a stalemate.

Comparing resource monitoring tool

http://www.munin-monitoring.org/ http://oss.oetiker.ch/rrdtool/ http://oss.oetiker.ch/mrtg/
I’m interested, too!

http://www.zabbix.com/

http://ganglia.info/
http://www.cacti.net/ http://www.nagios.org/

Comparing resource monitoring tool

http://www.munin-monitoring.org/ http://oss.oetiker.ch/rrdtool/ http://oss.oetiker.ch/mrtg/
snmpd

http://www.zabbix.com/

http://ganglia.info/
http://www.cacti.net/ http://www.nagios.org/
C

Comparerative table
Tool name Type Datastore Config Web interaface alerting
Resource
Munin monitoring RRDTool CUI
Reference only
Resouce
Cacti monitoring RRDTool & MySQL CUI/GUI
We are friends all the time...

MRTG
Resource
monitoring original CUI
Reference only
×
IT
Zabbix infrastructure
monitoring
MySQL, PostgreSQL, etc GUI

IT
Nagios infrastructure
monitoring
MySQL or PostgreSQL CUI/GUI

It is good points and bad points both.
I use Munin and a Nagios-based tool properly by my team.

What is Munin?
The Munin We Saw That Day

About Munin Be alert!

• http://munin-monitoring.org/

• Resource monitoring tool
– Munin can analyze resource trends
– “what just happened to kill our performance?”

• Plug and Play architecture
– It can monitor many items by default
Munin is a networked resource monitoring tool that can help analyze resource trends and
"what just happened to kill our performance?" problems. It is designed to be very plug and
play. A default installation provides a lot of graphs with almost no work.

Developers
Munin project github
• http://munin-monitoring.org/ • https://github.com/munin-monitoring

Documents / FAQ / Trac / Wiki Repository / tools / plugins

Progress in development
• Community based
– Github
• https://github.com/munin-monitoring
– Mailing list
• https://lists.sourceforge.net/lists/listinfo/munin-users
– IRC
• irc://irc.oftc.net/#munin

• Licence
– GNU Public License version2
– There is not commercial support

History
• 2002 - project began
– The original name is “LRRD”

• 2004 - Munin 1.0 released
– “munin-eye” name was changed to “munin-node”
– took long time, and daily improvement continued

• 2009 - Munin 1.4 released
– Perhaps I think that it is a version spreading most in 1.x.

• May 30, 2012 - Munin 2.0 (stable) released

Where is the Japanese information?
• NOT YET!
• Let’s make it together now!
– How about write something to wiki first?
• http://munin.jp/wiki/

“Is the number of the invitation to the Munin
“I’m sorry, user group ZERO case this week, too?
my applogies…” Hum? Do you have a mind to do?”

This Photo is under creative commons license
http://en.wikipedia.org/wiki/File:Odin_hrafnar.jpg

What obstacle
factors there are!!
Are you getting
wise with me?

Speaking munin-eye’s mind ( now, munin-node )

Summarize the points
• Munin is a resource monitoring tool. (GPL v2)

• Simple and powerful architecture.

• Munin frees us from a console. (effectiveness)

• Munin mean is “memory”.
You are never alone!
Munin always here for you 24x7x365

An Amazing Simple Munin
The Architecture Of Munin

User’s viewpoint

It is only simple structure.
The user refers to the data on the server.

Client Server Model
Munin Master And Munin Node

Let's watch a little more
detailed movement.

This is the data which we
referred to some time ago.

This is the work of the main Munin master, and a program is
executed by cron.
It thereby carry out the generation of the collection of data,
checking threshold, HTML files and graphs one by one.

This is “munin-node” agent. Munin
Master acquires data via plugins.

munin-update connects with munin-node.
Its port number of munin-node is TCP 4949.

Plugins are executed in munin-node, and program is a
script acquiring various data. Munin-update stores the
data which I acquired in RRDTool.
And, munin-limits checks the threshold.

And munin-graph and munin-html
generate a graph and HTML for the
material in data (.rrd) stored away by
RRDtool.

These flows are basic movement of Munin.
I think that it is really simple and cool!

Constitution of Munin
master ( SERVER ) munin-node ( CLIENT )
• Perl Libs • Perl Libs
– Munin::Common – Munin::Common
• munin-cron • munin-node
– munin-update – config: munin-node.conf
– munin-limits – Plugins
– munin-html • Tools
– munin-graph – munin-node-configure
• config: munin.conf – munin-cron

About data collection
• munin-node collect various data.
• Port 4949(TCP)
– Munin protocol
• LIST
• CONFIG
• FETCH
•
•
VERSION
QUIT
(T_T)4949
“4949” is onomatopoeia of Japanese "tearful face".

Data storage and graph generation are
work of RRDtool
• Data format is RRD (round robin database)
– /var/lib/munin/<hostname>/<plugin’s name>.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-idle-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-iowait-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-irq-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-nice-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-softirq-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-steal-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-system-d.rrd
-rw-r--r-- 1 munin munin 50612 10月 18 2010 localhost-cpu-user-d.rrd

• 50KByte/one RRD file
– More than 200KB/one plugin (MUST)
– 150 to 250 files/munin-node (total about 8 to 15MB/node)

generate graphs per plugin
Munin Master And Munin Node

Munin prepares for much plugins
• System resources
– CPU, memory, Load Average, disk, S.M.A.R.T…
• Network
– Traffic, SNMP, HTTP loadtime, TCP, UDP, ICMP…
• Applications, middleware
– Apache, Nginx, Sendmail, Postfix, MySQL,
PostgreSQL, MongoDB, memcached, PHP… etc

Ex) Load Average plugin
• /etc/munin/plugins/load
– “Load average” is five minutes average
– It’s a symbolic link
• Original is /usr/share/munin/plugin/load
– Simple shell script
echo -n "load.value "
cut -f2 -d' ' < /proc/loadavg
load .value 3.22

Environment
• Perl5
• OS
– Linux
• Source code ( version 2.0.6 )
• Binary Package
– Red Hat Enterprise Linux 系 ( EPEL )
– Debian
– openSUSE
– MacOS X
– Windows

Setting up flow
• Install Munin and Perl Libraries
• Change a config file ( munin.conf )
• Setting up munin-node ( munin-node.conf )
• Check its graphs

Case) Red Hat Enterprise Linux
• Use EPEL*1(testing repository) package or source
• procedure
– 1. enabling EPEL
– 2. “yum install munin”
– 3. configure munin.conf
– 4. turn on munin-node and setup
– 5. check
*1 Extra Packages for Enterprise Linux(EPEL) https://fedoraproject.org/wiki/EPEL

Case) Debian / Ubuntu
• Use apt (Debian PTS is testing) or Source
• Procedure
– 1. setting up Perl libraries (via apt-get)
– 2. install munin
– 3. configure munin.conf
– 4. turn on munin-node and setup
– 5. check

Basic of the setting
How To Configure

Config files
Munin Master munin-node
• /etc/munin/munin.conf • /etc/munin/munin-node.conf
– Host tree (targeting nodes) – Access control
– Graph strategy • Host (IP address)
• Cron or realtime generation • Network CIDR
– Paths – Node’s hostname
• RRD files – Port number
• logfiles • Default: TCP 4949 (T_T)
– Plugin’s option

[munin.conf] set target node
[GroupName;node1.pocketstudio.net]
address 127.0.0.1
use_node_name yes

[munin-node.conf] Access control
• allow ^127.0.0.1$
– Regular expression

• cidr_allow 192.0.2.0/24
– Not regular expression

• If you change files, then you must restart
munin-node!

Basic of the plugin
How To Configure plugin

Basic knowledge of Munin plugin
• Original files is here ( shell or perl scripts )
– /usr/share/munin/plugins/
• How to use
– To make symbolic link to /etc/munin/plugins
– configure munin-node.conf
– munin-node restart (MUST)
– Check graph and html

How to debug plugin
• /usr/sbin/munin-run <plugin-name>
– “--debug” shows more detail
– behavior is same as munin-node
– useful

• Command line tool ( I made )
– muninwalk & muninget ; perl script
https://github.com/zembutsu/muninwalk

Plugins catalog
How To Configure plugin

Apache
• Symbolic link
# ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/

• munin-node.conf
[apache_*]
env.url http://127.0.0.1:%d/server-status?auto
env.ports 80

• httpd.conf
ExtendedStatus On

<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>

MySQL
• Symbolic link
# ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins/

• munin-node.conf
[mysql*]
env.mysqlopts -u root -pPASSWORD
env.mysqladmn /usr/bin/mysqladmin

BIND
• Symbolic link
# ln -s /usr/share/munin/plugins/bind9_rndc /etc/munin/plugins/

• munin-node.conf
[bind9_rndc]
env.rndc /usr/sbin/rndc
env.querystats /var/named/chroot/var/named/data/named_stats.txt
user root

• named.conf
statistics-file "/var/named/data/named_stats.txt";

Plugins I made
Plugins Catalog

https://github.com/zembutsu/AWS-EstimateCharge
Amazon Web Services; it estimated charges realtime via API

Quantity of generation of the electric power company (TEPCO)

And electricity consumption rate (TEPCO).
Munin can make anything plug in if digitized.

How to make plugin
Take It Easy

Sample case; httping plugin
• http://www.vanheusden.com/httping/
• "httping" is a command-line tool which can check
response time of the Web server like a “ping”
command.
• If you set –S opsion, then you can check response
time and processing time.
$ httping -S http://210.239.46.254/
PING 210.239.46.254:80 (http://210.239.46.254/):
connected to 210.239.46.254:80 (380 bytes), seq=0 time=0.10+0.69=0.79 ms
Got signal 2
--- http://210.239.46.254/ ping statistics ---
4 connects, 4 ok, 0.00% failed

Plugin: httping_
#!/bin/sh
#
# Plugin to monitor HTTP response (httping)

#%# family=auto
#%# capabilities=autoconf

URL=${URL:-"http://localhost/"}
COUNT=${COUNT:-"5"}
httping_bin=$(which httping) This is substance of a httping plugin, and a
if [ "$1" = "autoconf" ]; then file itself is a simple shell script.
echo yes
exit 0 The contents are the definition about the
fi
Define graphing graph and commands to really acquire a value.
if [ "$1" = "config" ] ; then
echo "graph_args -r --lower-limit 0 "; A point is to acquire data, and therefore the
echo "graph_title http response $URL";
echo "graph_category httping"; plug in can make even what kind of language
echo "graph_info httping response time: $URL";
echo 'graph_vlabel msec' including perl and PHP.
echo "connect.label connect time"
echo "connect.draw AREA"
echo "connect.type GAUGE"
echo "processing.label processing time"
echo "processing.draw STACK"
echo "processing.type GAUGE"
exit Output format is “xxx.Value ***”
fi

# format for httpiing 1.5.3 http://www.vanheusden.com/httping/
$httping_bin -c $COUNT -G -S $URL | tr '+|=' ' ' | awk '{connect+=$9; processing+=$10} END{print "connect.value",connect/'$COUNT'"n""processing.value",processing/'$COUNT'}'

Config: httping_
• /etc/munin/plugin-conf.d/httping
[httping_localhost]
env.URL http://pocketstudio.jp/
env.COUNT 5

[httping_blog]
env.URL http://pocketstudio.jp/log3/
env.COUNT 5

[httping_node1]
env.URL http://node1.pocketstudio.net/
env.COUNT 5

• # ln -s /usr/share/munin/plugins/httping_ /etc/munin/plugins/httping_localhost

httping live demo
• http://demo.munin.jp/munin2/httping-day.html

It is a case having any problem neither for this server, There is much partial (processing time) of this server group
response time and processing time. blue.It takes the processing time by certain CMS.
On the other hand, I understand that the network is good.

Never say never.
• Agility is the pivot of the service (in my case)
– LOOKOUT, its cause solution of the trouble
• Hardware or Software or Network
– We need investigation
• where a problem happens promptly

Live Munin demo
• http://demo.munin.jp/
– Then let's observe the resource situation through
this demonstration site of Munin.
• Where is a bottleneck? or will be?
• Even if you do not log in to a server, I think that you can
refer to many resources.

This Just Can’t Be Right
A Real Troubleshooting

Case) identified unauthorized access
• By the Time we Realized It, It Had Already
Begun.
• situation
– 1. Error emails beguns to arrive to postmaster
– 2. There was not the alert with the monitoring tool
– 3. Therefore at first I checked a resource in Munin
– 4. I identified that CMS had vulnerability from the
situation and acted promptly.
I was able to perform the above-mentioned movement quickly in a short time by Munin.

How to find it.
Sendmail’s queue rised suddenly Load Average has no problem

I confirmed the time MySQL’s queries were
when traffic was strange rised suddenly, too

From the above-mentioned situation, I supposed illegal access for CMS. Actually, I understood
the attack for the specific URL when I investigated log of the time.
Identification and the action of the cause should have taken time more if I did not use Munin.

#5
My Bery Vest Monitoring tool

New Features of Munin 2.0
I think that Munin is truly wonderful

Munin 2.0 has new features!
• Better UI and CGI integration
– New look, Graph Zooming, FastCGI
• asynchronous I/O support
– Better performance
• Native SSH transport
– secure (port 22) & easy setup
• asynchronous proxy support
– async-server substitutes for munin-node
• And more…
– https://github.com/munin-
monitoring/munin/blob/devel/Announce-2.0

[RemoteNetwork;backend-DB]
addressssh://210.239.46.254/opt/munin/lib/munin-async-client --spooldir /var/opt/munin/spool/192.168.0.230 –spoolfetch
use_node_name yes

No munin, No Troubleshoot.
I'm Not Afraid of Anything Anymore

Munin changed support flow (my case)
• If I don’t use tools
– Troubleshooting is various command execute (sysstat) and
investigation of the log files.
– But, this method need long time and many human resources
need, and is bad for service.

• If I use Munin (now).
– Even if I do not log in, I can understand the situation.
– I can judge abnormality visually
• “I see the ending of this troubleshooting!”
– Agile Support
• Troubleshooting that has Plan-Do-Check-Action (PDCA) cycles.

In work of my dedicated server hosting
• I really depend on Munin
– Always, I setup Munin. Neat

– Munin is almost in several I cannot part with Munin
hundred servers which for my work.
I manage directly.
– I think that Munin is
indispensable to our
service quality You believe it!
improvement.
BAM
BAM!

Trobuleshoot PDCA

Law of
Cycles
Presage!!

Detecting problem
What are
Plan
and situation
these alerts?
For real?

Trobuleshoot PDCA

Law of
Cycles
Presage!!

Detecting problem
What are
Plan
and situation
these alerts?
For real?

Trobuleshoot PDCA Do
Law of Suppose a cause

Cycles OK, Munin. Please tell me
that trouble lies hidden in
wherever?
Presage!!
Fire!

Please stop!!

Detecting problem
What are
Plan
and situation
these alerts?
For real?

Trobuleshoot PDCA Do

wherever?
Presage!!
Fire!

I just talk about
what I just looked
in Munin!! Check Please stop!!
To check resources
remotely

Detecting problem
What are
Plan
and situation
these alerts?
For real?
Wow!

click-clack Trobuleshoot PDCA Do
click-clack


Action
wherever?
Presage!!
Fire!

Log in and
I just talk about
execute commands
what I just looked
in Munin!! Check Please stop!!
To check resources
remotely

You are never alone!
Munin always here for you
24 x 7 x 365
The Only Thing I Have Left To Guide Me

Munin’s overview
・Munin is the resource monitoring tool that
specialize to notice by the visualization.

・Simple architecture, and many plug-ins.

・Ths is most suitable for the system that
quick support is necessary in a short time.

Conclusion * This is my personal impression.

No munin, No Operation.
While there’s Munin, there’s hope.
MY VERY BEST MONITORING TOOL.
Thank you for MUNIN. Good-bye to MRTG.

I wish…
• I would appreciate you use Munin that
if you were interested in Munin by my
presentation.

• Tomorrow is another day. Up to you.

Squidn’t you use Munin?
(Shoudn’t)

Questions?
• Do you have a questionable point for munin?

I'm glad you asked.
Let's give the rights that the reward buys Opoona for you.
(but, here is wagon sale...)

References
• Munin
– http://munin-monitoring.org/

• Munin User Group Japan
– http://munin.jp/
– http://munin.jp/wiki/

• Website
– Waiting for Munin 2.0 – Introduction – Personal Workflow Blog
• http://blog.pwkf.org/post/2010/06/Waiting-for-Munin-2.0-Introduction

– /tags/2.0.0/ChangeLog – Munin – Trac
• http://munin-monitoring.org/browser/tags/2.0.0/ChangeLog

Please feedback me zem@pocketstudio.jp or @zembutsu ( twitter )
Thank you for your reading!

Practical resource monitoring with munin (English editon)

Recommended

Recommended

More Related Content

Similar to Practical resource monitoring with munin (English editon)

Similar to Practical resource monitoring with munin (English editon) (20)

More from Masahito Zembutsu

More from Masahito Zembutsu (20)

Recently uploaded

Recently uploaded (20)

Practical resource monitoring with munin (English editon)