WSO2 BAM 2.0 provide a rich set of tool for aggregation, analyzing and presentation for large scale data sets and any monitoring scenario can be easily modeled according to the BAM architecture. We selected WSO2 BAM as the backbone of our logging architecture with a Log4JAppender to send LogEvents to bam.

1. MT-Logging with WSO2 BAM
Amani Soysa
Senior Software Engineer

2. How it Works??
Logs are published to BAM via Log4JAppender
Published Logs are Stored in Cassandra (per tenant per day)
When the day is over Logs are published to HDFS file system
daily using BAM analyzers
To view Logs :-
Application Logs – Log events on web applications/services filter
by the application name
System Logs – View the system log of each product {With
filtering categories}
Log Viewers get logs from cassandra <<real time log>> and
HDFS file system <<Archive logs>>

4. Log Publisher - Log4JAppender

5. Setting Up LogEventAppender
log4j.rootLogger=INFO, LOGEVENT
# LOGEVENT is set to be a LogEventAppender using a PatternLayout to send logs to LOGEVENT
log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppender
log4j.appender.LOGEVENT.url=tcp://localhost:7611
log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout
log4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktrace
log4j.appender.LOGEVENT.userName=admin
log4j.appender.LOGEVENT.password=admin
log4j.appender.LOGEVENT.processingLimit=1000
log4j.appender.LOGEVENT.maxTolerableConsecutiveFailure=20

6. Stream Definition
streamId = dataPublisher.defineStream("{" + "'name':'log" + "." + tenantId +
"."
+ severName + "." + currDateStr + "'," + " 'version':'1.0.0',"
+ " 'nickName': 'Logs'," + " 'description': 'Logging Event',"
+ " 'metaData':[" + " {'name':'clientType','type':'STRING'}" + " ],"
+ " 'payloadData':[" + " {'name':'tenantID','type':'STRING'},"
+ " {'name':'serverName','type':'STRING'},"
+ " {'name':'appName','type':'STRING'},"
+ " {'name':'logTime','type':'LONG'},"
+ " {'name':'priority','type':'STRING'},"
+ " {'name':'message','type':'STRING'},"
+ " {'name':'logger','type':'STRING'},"
+ " {'name':'ip','type':'STRING'},"
+ " {'name':'instance','type':'STRING'},"
+ " {'name':'stacktrace','type':'STRING'}" + " ]" + "}");

7. Log Analyzer

8. Logging-Config.xml

Basic Information
<isDataFromCassandra>true</isDataFromCassandra>

Information Regarding Cassandra
<cassandraHost>localhost:9160</cassandraHost>
<userName>admin</userName>
<password>admin</password>

Information Regarding HDFS Server
<archivedHost>hdfs://localhost:9000</archivedHost>
<archivedHDFSPath>/stratos/logs</archivedHDFSPath>

9. Logging Summarizer
Archive logs daily at a given time and store in hdfs file system

10. summarizer-config.xml

Basic Information
<isDataFromCassandra>true</isDataFromCassandra>
<deleteColFamily>false</deleteColFamily>

Information Regarding Cassandra
<cassandraHost>localhost:9160</cassandraHost>
<userName>admin</userName>
<password>admin</password>

Information Regarding BAM
<publisherURL>localhost:7611</publisherURL>
<userName>admin</userName>
<password>admin</password>
<cronExpression>0 0 1 ? * * *</cronExpression>

Information Regarding HDFS Server
<archivedHost>hdfs://localhost:9000</archivedHost>
<archivedLogLocation>/stratos/logs</archivedLogLocation>

11. Advantages

Asynchronous and None Blocking Data
publishing

Receives and Stores Log Events Cassandra
Cluster which is high scalable and a big Data
Repository

Rich tools set for analytics

Can be shared with CEP for real time Log
Event analysis.

Can provide Logging tool boxes and
dashboards for system administrators using
WSO2 BAM

12. Log Viewer

Get Logs From Cassandra - <<Latest>>

Get Archived Logs From HDFS File system

14. Log Viewer – Archived Logs

15. Log Viewer – Application Logs