This document discusses logging in Perl applications. It recommends using structured logging by creating log event objects that contain contextual information like date, hostname, etc. as well as custom fields. It introduces Log::Message::Structured, which provides roles to easily create log event classes that can be stringified to JSON or other formats and passed to logging modules like Log::Dispatch. This structured logging approach packages log data as objects for rich formatting and transmission while avoiding reimplementing basic logging functionality.
2. Me
Damien 'dams' Krotkine
Paris.pm member
French Perl Monger vice-president
Perl Dancer developer
11 modules on CPAN
(pretty weak)
Author of Moderne Perl
(french book, go buy it)
14. When the app crashes
• Why did it crash ? ( reason, stack trace )
15. When the app crashes
• Why did it crash ? ( reason, stack trace )
• At crash time
16. When the app crashes
• Why did it crash ? ( reason, stack trace )
• At crash time
• What was the situation ?
17. When the app crashes
• Why did it crash ? ( reason, stack trace )
• At crash time
• What was the situation ?
• What was the data it was processing ?
20. When there are issues
• How bad is this failure ?
• Why did it fail to process this line ?
21. When there are issues
• How bad is this failure ?
• Why did it fail to process this line ?
• Is it only this line ?
22. When there are issues
• How bad is this failure ?
• Why did it fail to process this line ?
• Is it only this line ?
• Is it reproducible ?
23. When there are issues
• How bad is this failure ?
• Why did it fail to process this line ?
• Is it only this line ?
• Is it reproducible ?
• Does the rest fail ? (user, account)
25. When the app runs fine
• Information harvesting loop
26. When the app runs fine
• Information harvesting loop
• 1: get more info about the process run
27. When the app runs fine
• Information harvesting loop
• 1: get more info about the process run
• 2: go see my boss/colleague/...
28. When the app runs fine
• Information harvesting loop
• 1: get more info about the process run
• 2: go see my boss/colleague/...
• 3: (s)he asks something I don’t know
29. When the app runs fine
• Information harvesting loop
• 1: get more info about the process run
• 2: go see my boss/colleague/...
• 3: (s)he asks something I don’t know
• 4: go back to step 1
45. Exception Module
• Pick your own from CPAN
• I wrote mine anyway :)
• Dancer::Exception
• Extracted it as a standalone module
46. Exception Module
• Pick your own from CPAN
• I wrote mine anyway :)
• Dancer::Exception
• Extracted it as a standalone module
• Exceptions are objects with message
pattern, stack trace... raise(), throw(),
register(), introspection
47. Exception Module
• Pick your own from CPAN
• I wrote mine anyway :)
• Dancer::Exception
• Extracted it as a standalone module
• Exceptions are objects with message
pattern, stack trace... raise(), throw(),
register(), introspection
• Should end up on CPAN one day
48. foreach account {
try {
foreach user {
try {
foreach data line {
try {
process_line()
} catch {
log warning
};
}
} catch {
log important
};
}
} catch {
log important
};
}
log fatal
53. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
54. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
55. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
• log( in_file => "$level, '$data' is not good")
56. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
• log( in_file => "$level, '$data' is not good")
• my $logger = MyOwnLogger->new(write_in => 'file',
display_level => 'WARNING');
57. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
• log( in_file => "$level, '$data' is not good")
• my $logger = MyOwnLogger->new(write_in => 'file',
display_level => 'WARNING');
• $logger->log(warning =>"$Level, '$data' is not good");
58. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
• log( in_file => "$level, '$data' is not good")
• my $logger = MyOwnLogger->new(write_in => 'file',
display_level => 'WARNING');
• $logger->log(warning =>"$Level, '$data' is not good");
• Put that in a function, then a module, etc
59. Logging a line
• say "warning, data is not good"
• say "warning, data '$data' is not good"
• say "$level, data '$data' is not good"
• log( in_file => "$level, '$data' is not good")
• my $logger = MyOwnLogger->new(write_in => 'file',
display_level => 'WARNING');
• $logger->log(warning =>"$Level, '$data' is not good");
• Put that in a function, then a module, etc
62. The wheel...
• Home-made logging system : I've seen that
in every single company I've worked.
• Sometimes multiple home-made logging
system in a single company
63. The wheel...
• Home-made logging system : I've seen that
in every single company I've worked.
• Sometimes multiple home-made logging
system in a single company
• Sometimes multiple home-made logging
system in a single project
64. The wheel...
• Home-made logging system : I've seen that
in every single company I've worked.
• Sometimes multiple home-made logging
system in a single company
• Sometimes multiple home-made logging
system in a single project
• It's so easy to start with a print STDERR...
65. The wheel...
• Home-made logging system : I've seen that
in every single company I've worked.
• Sometimes multiple home-made logging
system in a single company
• Sometimes multiple home-made logging
system in a single project
• It's so easy to start with a print STDERR...
• Stop reinventing the wheel
68. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
69. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
• I choose Log::Dispatch
70. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
• I choose Log::Dispatch
• $log->log(level => 'info', message => 'Blah' );
71. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
• I choose Log::Dispatch
• $log->log(level => 'info', message => 'Blah' );
• Good ?
72. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
• I choose Log::Dispatch
• $log->log(level => 'info', message => 'Blah' );
• Good ?
• No, we are still nowhere !
73. Pick up a logger
• Many Log modules on CPAN
• Pick one you like
• I choose Log::Dispatch
• $log->log(level => 'info', message => 'Blah' );
• Good ?
• No, we are still nowhere !
• We want contextual information
78. Contextual information
• What do we need to log ?
• A log message - ok, but
• Hardware info (Mem, CPU, disk, host)
• Software info (time, user / account id...)
79. Contextual information
• What do we need to log ?
• A log message - ok, but
• Hardware info (Mem, CPU, disk, host)
• Software info (time, user / account id...)
• Message (filename, line nb, stacktrace...)
80. Contextual information
• What do we need to log ?
• A log message - ok, but
• Hardware info (Mem, CPU, disk, host)
• Software info (time, user / account id...)
• Message (filename, line nb, stacktrace...)
• Log line = mix contextual and specific info
81. Contextual information
• What do we need to log ?
• A log message - ok, but
• Hardware info (Mem, CPU, disk, host)
• Software info (time, user / account id...)
• Message (filename, line nb, stacktrace...)
• Log line = mix contextual and specific info
• Even with a great logger, need to build that
84. Richer logging
• How to pack these information together ?
• Use a key-value structure
85. Richer logging
• How to pack these information together ?
• Use a key-value structure
• Better: a class, fields are attributes
86. Richer logging
• How to pack these information together ?
• Use a key-value structure
• Better: a class, fields are attributes
• So we can benefit of defaults, lazyness, etc
87. Richer logging
• How to pack these information together ?
• Use a key-value structure
• Better: a class, fields are attributes
• So we can benefit of defaults, lazyness, etc
• One log line = one class instance
88. Richer logging
• How to pack these information together ?
• Use a key-value structure
• Better: a class, fields are attributes
• So we can benefit of defaults, lazyness, etc
• One log line = one class instance
• We need it as a string at the end
89. Richer logging
• How to pack these information together ?
• Use a key-value structure
• Better: a class, fields are attributes
• So we can benefit of defaults, lazyness, etc
• One log line = one class instance
• We need it as a string at the end
• So stringify to JSON,YAML...
93. Log::Message::Structured
• A set of roles, consume in MyEvent class
• 4 types of roles :
• L::M::S : basic setup + "" overloading
94. Log::Message::Structured
• A set of roles, consume in MyEvent class
• 4 types of roles :
• L::M::S : basic setup + "" overloading
• L::M::S::Components : provides ready to
use additional fields
95. Log::Message::Structured
• A set of roles, consume in MyEvent class
• 4 types of roles :
• L::M::S : basic setup + "" overloading
• L::M::S::Components : provides ready to
use additional fields
• L::M::S::Stringify : provides stringifiers
96. Log::Message::Structured
• A set of roles, consume in MyEvent class
• 4 types of roles :
• L::M::S : basic setup + "" overloading
• L::M::S::Components : provides ready to
use additional fields
• L::M::S::Stringify : provides stringifiers
• Your custom fields : additional attributes
97. package MyLogEvent; use Moose;
with qw/
Log::Message::Structured
Log::Message::Structured::Component::Date
Log::Message::Structured::Component::Hostname
Log::Message::Structured::Stringify::AsJSON
/;
has message => ( is => 'ro', required => 1);
has account_id => ( is => 'ro', required => 1);
has user_id => ( is => 'ro' );
98. package MyLogEvent; use Moose;
with qw/
Log::Message::Structured
Log::Message::Structured::Component::Date
Log::Message::Structured::Component::Hostname
Log::Message::Structured::Stringify::AsJSON
/;
has message => ( is => 'ro', required => 1);
has account_id => ( is => 'ro', required => 1);
has user_id => ( is => 'ro' );
...elsewhere...
99. package MyLogEvent; use Moose;
with qw/
Log::Message::Structured
Log::Message::Structured::Component::Date
Log::Message::Structured::Component::Hostname
Log::Message::Structured::Stringify::AsJSON
/;
has message => ( is => 'ro', required => 1);
has account_id => ( is => 'ro', required => 1);
has user_id => ( is => 'ro' );
...elsewhere...
use My::Log::Event;
$logger->log( warning => MyLogEvent->new(
account_id => 42,
user_id => 12,
message => "watch out! behind you!" ));
100. package MyLogEvent; use Moose;
with qw/
Log::Message::Structured
Log::Message::Structured::Component::Date
Log::Message::Structured::Component::Hostname
Log::Message::Structured::Stringify::AsJSON
/;
has message => ( is => 'ro', required => 1);
has account_id => ( is => 'ro', required => 1);
has user_id => ( is => 'ro' );
...elsewhere...
use My::Log::Event;
$logger->log( warning => MyLogEvent->new(
account_id => 42,
user_id => 12,
message => "watch out! behind you!" ));
That's sending the stringified JSON to Log::Dispatch
105. Let's shorten the code
In my application I have:
while (my $account_id = AccountIterator->next) {
while (my $user_id = UserIterator->next) {
while (my $data_line = DataIterator->next) {
... do stuff ...
106. Let's shorten the code
In my application I have:
while (my $account_id = AccountIterator->next) {
while (my $user_id = UserIterator->next) {
while (my $data_line = DataIterator->next) {
... do stuff ...
package MyLogEvent; use Moose;
with qw/Log::Message::Structured .... /;
use AccountIterator; use UserIterator;
has account_id => ( is => 'ro',
default => sub { AccountIterator->current } );
has user_id => ( is => 'ro',
default => sub { UserIterator->current } );
116. Exceptions...
Remember ?
try { ... } catch {
log_warning(exception => $_ )
};
• To be able to write that, you need in your
MyLogEvent class:
117. Exceptions...
Remember ?
try { ... } catch {
log_warning(exception => $_ )
};
• To be able to write that, you need in your
MyLogEvent class:
• these attributes : 'exception', 'stack_trace',
118. Exceptions...
Remember ?
try { ... } catch {
log_warning(exception => $_ )
};
• To be able to write that, you need in your
MyLogEvent class:
• these attributes : 'exception', 'stack_trace',
• with a BUILDARGS
119. Exceptions...
Remember ?
try { ... } catch {
log_warning(exception => $_ )
};
• To be able to write that, you need in your
MyLogEvent class:
• these attributes : 'exception', 'stack_trace',
• with a BUILDARGS
• that looks at the exception, and set the
'message' and the 'stack_trace' arguments
120. around BUILDARGS => sub {
my $orig = shift;
my $class = shift;
my $h = $class->$orig(@_);
my $e = $h->{exception}
or return $h;
if ( blessed($e) && $e->isa('My::Exception::Base') ) {
$h->{message} = $e->message;
$h->{shortmess} = $e->{_shortmess};
} else {
$h->{message} = $e
}
return $h;
};
121. around BUILDARGS => sub {
my $orig = shift;
my $class = shift;
my $h = $class->$orig(@_);
my $e = $h->{exception}
or return $h;
if ( blessed($e) && $e->isa('My::Exception::Base') ) {
$h->{message} = $e->message;
$h->{shortmess} = $e->{_shortmess};
} else {
$h->{message} = $e
}
return $h;
};
try { ... } catch {
log_warning(exception => $_ )
That would now
}; work !
127. Log::Message::Structured
• Does the job
• Please contribute
• we need more components
• we need more stringifiers
• compatibility with Moo
128. Log::Message::Structured
• Does the job
• Please contribute
• we need more components
• we need more stringifiers
• compatibility with Moo
• On CPAN / github
131. • We have all we need as JSON
• But that's not enough :(
132. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
133. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
134. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
• You need to do complex lookups / research
135. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
• You need to do complex lookups / research
• thousands of files, grep not good enough
136. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
• You need to do complex lookups / research
• thousands of files, grep not good enough
• => need a powerful search engine
137. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
• You need to do complex lookups / research
• thousands of files, grep not good enough
• => need a powerful search engine
• Also, what if you have different log sources
138. • We have all we need as JSON
• But that's not enough :(
• Your boss wants to look at the log
• => provide a simple interface
• You need to do complex lookups / research
• thousands of files, grep not good enough
• => need a powerful search engine
• Also, what if you have different log sources
• From DB events, Redis MQ, system logs ?
142. Web interfaces
• Two interfaces
• First one is easy to use
• Based on Dancer + MongoDB
143. Web interfaces
• Two interfaces
• First one is easy to use
• Based on Dancer + MongoDB
• Shows logs in natural manner, grouped by
job, hostname, and presented by time
144. Web interfaces
• Two interfaces
• First one is easy to use
• Based on Dancer + MongoDB
• Shows logs in natural manner, grouped by
job, hostname, and presented by time
• Second one is more for data mining
145. Web interfaces
• Two interfaces
• First one is easy to use
• Based on Dancer + MongoDB
• Shows logs in natural manner, grouped by
job, hostname, and presented by time
• Second one is more for data mining
• Elasticsearch: deep searching, filtering, etc
146. We need some kind of big stuff
syslog
Standard log files
Redis MQ logs $stuff Web interface
Other logs Search Engine
147. We need some kind of big stuff
syslog
Standard log files
Redis MQ logs $stuff Web interface
Other logs Search Engine
$stuff = Message::Passing
151. Message::Passing
• Written by Tomas Doran
• Logstash-like but in Perl and with style
• It's a daemon (AnyEvent based)
152. Message::Passing
• Written by Tomas Doran
• Logstash-like but in Perl and with style
• It's a daemon (AnyEvent based)
• basic concepts: inputs, filters, outputs
153. Message::Passing
• Written by Tomas Doran
• Logstash-like but in Perl and with style
• It's a daemon (AnyEvent based)
• basic concepts: inputs, filters, outputs
• run it as : command line, using the DSL,
using classes directly.
156. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
157. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
• Writing to log files
158. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
• Writing to log files
• Configure syslog-ng to output to a pipe
159. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
• Writing to log files
• Configure syslog-ng to output to a pipe
• Branch M::P on this named pipe
160. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
• Writing to log files
• Configure syslog-ng to output to a pipe
• Branch M::P on this named pipe
• Web interface: Dancer + MongoDB
161. Real Life
• My software runs on many machines
• Connected to one syslog-ng server
• Writing to log files
• Configure syslog-ng to output to a pipe
• Branch M::P on this named pipe
• Web interface: Dancer + MongoDB
• Search interface : Elasticsearch + plugin
169. Let's use Message::Passing
• step 1 : configure & run M::P daemon
• step 2 : ...
• step 3 : PROFIT
Remember the basic concepts?
inputs, filters, outputs
170. use Message::Passing::DSL; use Moose;
with 'Message::Passing::Role::Script';
sub build_chain { message_chain {
input filetail => (
class => 'FileTail',
filename => ‘/path/to/logs’,
output_to => 'cleanup_logs' );
filter cleanup_logs => (
class => '+My::Filter::DeJSON',
output_to => [ 'elasticsearch', 'mongodb' ] );
output elasticsearch_servers => (
class => 'ElasticSearch',
elasticsearch_servers => 'localhost:9200' );
output mongodb => (
class => 'MongoDB',
hostname => 'localhost',
database => 'reporting',
collection => 'logs',
indexes => [ # specify fields to index ] );}
}
__PACKAGE__->start;
171. My::Filter::DeJSON
package My::Filter::DeJSON;
use Moose;
sub consume {
my ( $self, $message ) = @_;
$message =~ m/(w+):JSON:(.*)/ or return;
my $structure_log = from_json($2);
$structure_log->{log_level} = $1;
foreach my $output_to ( @{ $self->output_to } ) {
$output_to->consume($structure_log);
}
}
172. Message::Passing::Output::MongoDB
use Moose; use MongoDB; use AnyEvent;
with qw/
Message::Passing::Role::Output
Message::Passing::Role::HasUsernameAndPassword
Message::Passing::Role::HasHostnameAndPort
/;
# ... hostname, port, user/pass attributes ...
has _db => (
is => 'ro', isa => 'MongoDB::Database', lazy => 1,
default => sub {
my $self = shift;
my $connection = MongoDB::Connection->new(
host => $self->hostname,
port => $self->port,
);
return $connection->get_database($self->database);
},
);
173. Message::Passing::Output::MongoDB
# ... _collection attribute
sub _build_logs_collection {
my ($self) = @_;
my $collection_name = $self->collection;
my $collection = $self->_db->$collection_name;
if ($self->_has_indexes) {
foreach my $index (@{$self->indexes}){
$collection->ensure_index(@$index);
}
}
return $collection;
}
174. Message::Passing::Output::MongoDB
sub consume {
my ($self, $data) = @_;
return unless $data;
my $date;
my $collection = $self->_collection;
$collection->insert($data)
or warn "Insertion failure: " . Dumper($data) . "n";
if ($self->verbose) {
$self->_inc_log_counter;
warn("Total " . $self->_log_counter
. " records inserted in MongoDBn");
}
}
On CPAN, by Bin Shu
177. Dancer web GUI
• Very simple Dancer application
• Could have used Dancer::Plugin::Mongo
178. Dancer web GUI
• Very simple Dancer application
• Could have used Dancer::Plugin::Mongo
• A screen to display log
179. Dancer web GUI
• Very simple Dancer application
• Could have used Dancer::Plugin::Mongo
• A screen to display log
• A screen to search (filter) logs
180. Dancer web GUI
• Very simple Dancer application
• Could have used Dancer::Plugin::Mongo
• A screen to display log
• A screen to search (filter) logs
• Result page always show logs ordered by time
185. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
186. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
187. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
• a mapreduce to display asearch form, => /result
188. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
• a mapreduce to display asearch form, => /result
• get /result :
189. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
• a mapreduce to display asearch form, => /result
• get /result :
• get search params from url, issue a find request
190. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
• a mapreduce to display asearch form, => /result
• get /result :
• get search params from url, issue a find request
• display in a table
191. Dancer GUI routes
• get / :
• a mongoDB group request per host, command, => /result
• get /search :
• a mapreduce to display asearch form, => /result
• get /result :
• get search params from url, issue a find request
• display in a table
• clicking on a result brings up a JS detailed result popup
197. Elasticsearch
• Installation : trivial
• Configuration : none
• Web GUI : elasticsearch-head
• Code ? there is no code to show :)
198. Elasticsearch
• Installation : trivial
• Configuration : none
• Web GUI : elasticsearch-head
• Code ? there is no code to show :)
• Usage : perform advanced search on massive backlog