Cool and easy way to do logging and monitoring at EEA.

Cool and easy way to do
logging and monitoring at EEA.
How to track Plone with TraceView
and GrayLog using Docker
Plone Conference 2015 - Bucarestluca.pisani@abstract.it /

Who am I?
I work with Abstract since 2008,
implementing web solutions with Plone and
other Python frameworks.
Since July I'm working on Plone sites of the
European Environment Agency (EEA) in a
collaborative project between Abstract and
Eau de Web
Cool and easy way to do logging and monitoring at EEA.Luca Pisani - luca.pisani@abstract.it /

Infrastructure - www.eea.europa.eu
● 2 x frontends for high availability
on each frontend
○ Apache
○ Varnish
○ Memcached
● 5 x backends
on each backend
○ 6x RelStorage clients
○ 1x Zope asyns client
● PostgreSQL cluster
○ Cloud - OpenStack
○ 1 x DB master
○ 2 x DB replicas
● ~ 30k page views per day
● average: 30 editors per day
● average: 10 simultaneous
editors

Solution
Monitoring
● TraceView
● eea.controlpanel
Logging
● https://github.com/eea/eea.docker.graylog2
○ Graylog
● https://github.com/eea/eea.docker.logcentral
● https://github.com/eea/eea.graylogger

TraceView provides code-level performance monitoring for web
applications, as well as real user monitoring (RUM).
Is deployed by
Important features
● full stack monitoring
hardware level, db level, backend/application level and frontend/webserver level
● spend very small ammount of time to identify bottle-necks
● quick audits/overviews
Monitoring -

Monitoring - collective.traceview
Traceview times the full request from the browser through frontend servers to
application servers.
collective.traceview gives you insight into Zope/Plone internals and adds these
layers to Traceview:
● Zope HTTP Server
● Zope publisher
● ZODB
● Portal Transforms
● Outbound calls to e.g. webservices
● Portal Catalog searches
● Chameleon ZPT engine
It also adds tags to the HTML header and footer to instrument Traceview Real User
Monitoring (RUM), so you'll get metrics about user network connectivity and how long
time your site takes to render inside the browsers of the real users.

Monitoring -
Cool and easy way to do logging and monitoring at EEA.Luca Pisani luca.pisani@abstract.it /

Monitoring - eea.controlpanel
● quick overview of logged in users and their activity
○ who is active, not all users with a valid cookied, just real active users in that instant
https://github.com/collective/eea.controlpanel

● quick overview of logged in users and their activity
○ database activity (zmi undo tab) and logged in users in the same screen

● auto-refresh
● data is saved/kept via a python logger (very simple)
● safe portal upgrades
quickly check who is really online/active and if he is actually doing something
● verify unusual logged-in user activity

Logging -
Graylog is a fully integrated open source log
management platform for collecting,
indexing, and analyzing both structured and
unstructured data from almost any source.
● Open source
● Unified platform for
centralized logging
management
● Multi-tier architecture
enables scale-out of
ingestion, processing,
and storage
separately
● Pluggable architecture
and REST API for
runtime extensibility

Logging -
Any data you want
The Graylog Collector can forward data from any
log source. Graylog also supports flexible input
types including syslog, plaintext, and GELF, a
Graylog format that lets you pre-structure and
normalize messages before sending.
Parse unstructured data
Graylog “extractors” and Grok patterns help
identify valuable information contained in
log data.

Logging -
Real-time processing
Graylog “streams” apply rules in real time to
find specific message conditions or to
categorize data by software type, application
stack, location, or any other attribute that
makes finding answers faster and easier.
Dashboards and alerting
Any analysis or search result can be added as a
widget to dashboards. Keep all your important data
and trends in one place.
Alerts are highly configurable and can notify you
when specific conditions or events occur.

Logging - Installing
Graylog can be installed in many different ways
● Virtual Appliance
Production ready virtual appliances as OVA images (VMware,
VirtualBox) for OpenStack, Amazon EC2, and Docker.
● DEB and RPM packages
● Chef, Puppet, Ansible, Vagrant
● Manual Installation
use

Logging - concept
Docker is an awesome tool we can use
to deploy production systems.
The Dockerfile is a sort of installation
'recipe' that can be used to initialize a
system environment.
A Docker container is a fresh Linux
environment that bootstraps from a
host systems Linux kernel - this gives us
a lot of the benefits of the VM but
without the slow execution and bulk
often associated with VMs.

Logging - concept
To understand Docker, you need to know about three components:
● Docker images.
● Docker registries.
● Docker containers.
Docker images is a read-only template. For example, an image could contain an
Ubuntu operating system with Apache and your web application installed.
Docker registries hold images. These are public or private stores from which you
upload or download images.
Docker containers are similar to a directory. A Docker container holds everything
that is needed for an application to run.

Logging - concept
Why use Docker?
● Faster delivery of applications
Docker is perfect for helping you with the development lifecycle.
● Deploying and scaling more easily
Docker's portability and lightweight nature also make dynamically
managing workloads easy. You can use Docker to quickly scale up or
tear down applications and services.

Logging - eea.docker.graylog2
All-in-one, ready to run Graylog2 Docker image.
Dockerfile
Docker image
everything already setup

Logging - eea.graylogger
Zope/Plone plugin to send log to Graylog.

Logging - eea.graylogger
To install
https://github.com/eea/eea.graylogger/blob/master/buildouts/plone4/buildout.cfg
How it works
Based on graypy sends messages to Graylog in GELF (Graylog Extended Log Format).
It can sends messages using AMQP (Advanced Message Queuing Protocol) with
RabbitMQ.
Messages sends via UDP protocol

Logging - eea.docker.logcentral
docker-compose
Compose is a tool for defining and running multi-container
applications with Docker.
Features
● UDP and TCP
● LDAP integration
● GELF / RabbitMQ / Syslog
● easy testing/demos: https://github.com/eea/eea.
docker.logcentral/tree/master/demo
● works with any app not only Plone, it’s a generic
solution for logging

Questions

Thanks
Luca Pisani
luca.pisani@abstract.it
linkedin.com/in/lucapisani
github.com/jslucas

Cool and easy way to do logging and monitoring at EEA.

Cool and easy way to do logging and monitoring at EEA.

Recommandé

Recommandé

Contenu connexe

Dernier

Dernier (20)

En vedette

En vedette (20)

Cool and easy way to do logging and monitoring at EEA.