Alfresco monitoring setups with Nagios and ELK stack using JMX and OOTB Support tools

1. Alfresco monitoring
setups with Nagios and
ELK stack using JMX and
OOTB Support tools
Cesar Capillas <cesar@zylk.net>

2. Learn. Connect. Collaborate.
About me
• I am an IT/Solution Architect at zylk.net, a content management and big
data consultancy in Bilbao (Spain). I worked with Alfresco-based solutions
during the last 8 years in large organisations. I am a member of the OOTB,
the author of a popular Alfresco Blog in Spain, and an active user in
Alfresco Community forums.
• https://www.linkedin.com/in/cesarcapillas

3. Learn. Connect. Collaborate.
Intro and motivations
• Nagios/Icinga is a nice tool for alerts and notifications but:
– It is not the best option for visualizations
– It is more oriented for a given service or host, not for a complete view
of a cluster, for example.
• We use ELK stack with Kibana for:
– JMX metrics in Alfresco Enterprise
– A basic solution for your Alfresco logs
– A support tools-like system performance and active sessions
dashboard, but in a persisted way, for Alfresco Community.

4. Learn. Connect. Collaborate.
Minimal setup
1
Tested on:
- Alfresco 201707GA
- Alfresco 5.0.25 EE / 5.1.3 EE
Tested on:
- ELK stack 5.6.3
Tested on:
- Nagios 3 / Icinga 1.x

5. Learn. Connect. Collaborate.
Nagios setup
 Nagios is an IT industry standard for infrastructure monitoring, with tons of
plugins.
 It allows to define thresholds for alerts and notifications (email, slack)
for your hosts and services. Combined with graphs (PNP, nagiosgraphs), it
may help to check and understand behaviour patterns of our Alfresco
installation.
 The starting point is to monitor your infrastructure parameters such as CPU,
Memory, Swap, Load, Disk usages, check_http direct service urls or
check_tcp used ports of your Alfresco installation.

6. Learn. Connect. Collaborate.
The next step - JMX (Alfresco EE)
• JMX provides monitoring information of many properties such as
NumActive (Connection Pool), TicketAllNonExpired, NumDocuments
(SOLR), Disk sizes for your contentstores and indices and many many
more….
• JMX bean objects in Alfresco may be browsed and inspected with the help
of jconsole, jvisualvm or jmxshell.
• Alfresco objects are only available in Alfresco EE, although basic java,
tomcat and system information may be used with Alfresco CE
https://www.zylk.net/en/web-2-0/blog/-/blogs/enabling-jmx-in-alfresco

7. Learn. Connect. Collaborate.
JMX in Nagios via check_jmx (Alfresco EE)
• I create a basic template (for both Alfresco Community and Enterprise) in
Docker and Vagrant for using Nagios / Icinga.
http://www.github.com/zylklab/alfresco-nagios
https://github.com/toniblyx/alfresco-nagios-and-icinga-plugin
$ git clone https://github.com/zylklab/alfresco-nagios
$ cd alfresco-nagios
$ vim Dockerfile
ENV ACS_HOST alf-ee.melmac.net
ENV ACS_ADDR 127.0.0.1
ENV JMX_USER monitorRole
ENV JMX_PASS change_asap
$ docker build -t zylklab/icingalf .
$ docker run -i -t zylklab/icingalf
Tested on:
- Alfresco 201707GA + OOTB Support Tools
Addon > 0.1
- Alfresco 5.0.25 EE
- Nagios/Icinga 3
- Docker version 1.12.6
- Vagrant 1.8.1

8. Learn. Connect. Collaborate.
OOTB Support Tools webscripts (Alfresco CE)
• OOTB Support tools provides JSON-based webscripts that may be used
for extracting useful information for a monitoring service
– System Performance
– Active Sessions
– SOLR
• First step was to create Nagios scripts for getting this info via curl
command, providing alerts and notifications for Alfresco Community.
http://www.github.com/zylklab/alfresco-nagios 2
https://github.com/OrderOfTheBee/ootbee-support-tools

9. Learn. Connect. Collaborate.
Nagios / Icinga checks (Alfresco CE)
http://www.github.com/zylklab/alfresco-nagios
Tested on:
- Alfresco 201707GA + OOTB Support Tools
Addon > 0.1
- Alfresco 5.0.25 EE
- Nagios/Icinga 3
- Docker version 1.12.6
- Vagrant 1.8.1

10. Learn. Connect. Collaborate.
ELK setup: Kibana dashboards
• Kibana is a nice tool for inspecting logs, to do searches and inspect
graphically a given metric. It is part of the well known ELK stack.
• ELK = Elastic Search + Logstash + Kibana
– Elastic Search is the index server.
– Kibana is an interface for searching and visualizing the indexed data
(logs, metrics...)
– Logstash is the tool for extracting, transforming and loading data in
Elastic Search.

11. Learn. Connect. Collaborate.
Logstash input JMX plugin (Alfresco EE)
• Logstash has a lot of plugins. One of them is the logstash-input-jmx
• The plugin allows to define an URL service for JMX, user credentials, a set
of JMX objects and a polling frequency. And without many
transformations, we can index a JSON object in Elastic Search with the
given JMX fields and values.
## logstash.conf  JMX objects defined in $LS_HOME/jmx/jmx.conf
input {
jmx{
type => "jmx"
path => "./jmx"
polling_frequency => 60
}
}

12. Learn. Connect. Collaborate.
System Performance via JMX in Kibana
https://www.zylk.net/en/web-2-0/blog/-/blogs/kibana-dashboard-for-monitoring-alfresco-jmx-metrics
JVM Heap
usage for
3 Alfresco
nodes in a
cluster

13. Learn. Connect. Collaborate.
Alfresco JMX (./jmx/jmx.conf)
{
"host" : "alfie",
"port" : 50500,
"url" : "service:jmx:rmi:///jndi/rmi://localhost:50500/alfresco/jmxrmi",
"username" : "monitorRole",
"password": "change_asap",
"alias": "alfresco",
"queries" : [
{
"object_name" : "java.lang:type=Memory",
"attributes" : [ "HeapMemoryUsage" ],
"object_alias" : "Heap_Memory"
},
{
"object_name" : "Alfresco:Name=SolrIndexes,Core=alfresco",
"attributes" : [ "NumDocuments" ],
"object_alias" : "Alfresco_Solr_Indexes"
},
{
"object_name" : "Alfresco:Name=ConnectionPool",
"attributes" : [ "MaxActive", "NumActive" ],
"object_alias" : "DB_Connection_Pool"
}
] }
JMX url
JMX objects
3

14. Learn. Connect. Collaborate.
Alfresco Logs (Kibana dashboard)
https://www.zylk.net/en/web-2-0/blog/-/blogs/simple-kibana-dashboard-for-monitoring-alfresco-logs
Number of logs
vs t.
Last Alfresco Startup
Last LDAP Sync
Number of logs by
level (with filters)
Last LDAP SyncLast LDAP Sync
Number of logs by
class (with filters)
Log console

15. Learn. Connect. Collaborate.
Alfresco Logs (logstash.conf)
##
## Tomcat catalina.out logs
##
input {
file {
type => "alfresco-log"
path => ["/opt/alfresco/tomcat/logs/catalina.out", "/opt/alfresco/tomcat/logs/catalina*out*"]
}
}
##
## Filters for alfresco-log
##
filter {
if [type] == "alfresco-log" {
# Match incoming log entries to fields logLevel, class and Msg
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logdate}s*%{LOGLEVEL:logLevel} %{NOTSPACE:class}s*%{GREEDYDATA:Msg}" ]
}
# Match logdate from timestamp
date {
match => ["logdate" , "yyyy-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
}
}
}
Logstash input file
Grok parsing for extracting loglevel,
class and msg
Valid for both Alfresco CE and EE

16. Learn. Connect. Collaborate.
System Performance dashboard in Kibana (Alfresco CE)
https://www.zylk.net/en/web-2-0/blog/-/blogs/kibana-dashboard-for-monitoring-alfresco-community-via-ootb-support-tools
Kibana allows to
build a
dashboard via
saved searches,
tables, metrics,
gauges or
visualizations
over your indices
Heap Memory
System /
Process Load
Threads
Support Tools
like System
Performance
(but in a
persisted way)

17. Learn. Connect. Collaborate.
System Performance (logstash.conf)
##
## OOTB Admin performance webscript
##
input {
http_poller {
id => "ootb-poller-performance"
type => "ootb-performance"
urls => {
alfie_perform => {
url => "http://localhost:8080/alfresco/service/ootbee/admin/admin-performance.json"
method => "GET"
headers => {
Accept => "application/json"
}
auth => {
user => "monitor"
password => "s3cret"
}
}
}
automatic_retries => 0
#schedule => { cron => "* * * * * UTC"}
schedule => { every => "60s" }
request_timeout => 8
tags => ootb
}
}
System
Performance
webscript
in JSON format
-
OOTB
Support Tools
Polling
frequency
logstash-input-http_poller
4

18. Learn. Connect. Collaborate.
More dashboards in Kibana
ACTIVE SESSIONS SOLR SUMMARY

19. Learn. Connect. Collaborate.
Summarizing Nagios setup
• Requirement for monitoring Alfresco CE:
– Nagios curl scripts with OOTB Support Tools addon.
– Alfresco Hackathon (Zaragoza - Beecon 2017) – Added SOLR page to OOTB
Support Tools (new webscript)
– Docker and vagrant resources for Nagios / Icinga (for Alfresco Community)
– Docker and vagrant resources for Nagios / Icinga (for Alfresco Enterprise) -
based on previous Nagios template of Toni de la Fuente.
http://www.github.com/zylklab/alfresco-nagios
https://github.com/toniblyx/alfresco-nagios-and-icinga-plugin

20. Learn. Connect. Collaborate.
Summarizing ELK setup
• We needed a better solution for graphs:
– Dashboard for System Performance, Active Sessions and SOLR based on
logstash-input-http_poller plugin using OOTB Support Tools webscripts (for
Alfresco CE)
– Other dashboards for Alfresco Logs based mainly from Docker ELK resources
by Miguel Rodriguez.
• For Alfresco EE:
– Dashboard for System Performance based on logstash-input-jmx plugin.
http://www.github.com/zylklab/alfresco-elk
https://github.com/miguel-rodriguez/Docker-ELK-Alfresco-Monitoring

21. Learn. Connect. Collaborate.
What’s next: Apache stack instead ELK stack
• Alternatives to ELK in Apache stack ?
− We may use logstash-output-solr_http for saving metrics and log
information in SOLR.
− We may use Banana port for SOLR of Kibana.
− We may use Apache Zeppelin for querying SQL in SOLR.
− Apache Nifi vs. Logstash ?
https://www.zylk.net/es/web-2-0/blog/-/blogs/more-on-monitoring-dashboards-for-
alfresco-using-solr-banana-and-apache-zeppelin

22. Alfresco monitoring
setups with Nagios and
ELK stack using JMX and
OOTB Support tools
http://www.github.com/zylklab/alfresco-nagios
http://www.github.com/zylklab/alfresco-elk
@zylknet