Understanding the state of your web application using Apache Kafka, Spark

A global consulting and IT solutions leader

Understanding
the State of your Web Application
using Apache Kafka, Spark

What if webapps ….
… have feelings?

java.lang.NullPointerException
at com.insideout.riley.anger.doAngry(Riley.java:123)
at com.insideout.riley.joy.doHappy(Riley.java:456)
at com.insideout.riley.sadness.doCry(Riley.java:789)
at com.insideout.riley.disgust.doEwwwww(Riley.java:143)
at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888)
at com.insideout.riley.doPuberty(Riley.java:58)
15/11/09 20:57:42 DEBUG
Invalid character in customer name: ‘s@dness’
15/11/09 20:54:56 INFO
Successfully saved customer
record: customer.name=’Joy’

Exception in thread "main" java.lang.
OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at java.io.BufferedReader.readLine(Unknown Source)
at com.insideout.riley.nightmare .overtime(ScaryClown.java:13)
at com.insideout.riley.nightmare .deadline(ScaryClown.java:666)
at com.insideout.riley .main(ScaryClown.java:1)

Log Data Tell You Everything/Anything/Nothing
● Who tried to login to your app?
● Was the login attempt successful?
● Which pages the user visited?
● What did the user do?
● Did ze make any purchases?
● Did the customer encountered any problems
while using your app?
● What were they doing at your website when
‘Tamang Panahon’ was ongoing at Philippine Arena?

More Users = More Logs = TMI !!!
How to make sense of it all???

Data Analyst!
All Your Data!
NOW!

Real-Time Insights Are More:
● Relevant
● Actionable

LOG
DATA
REAL-TIME
INSIGHTS
???

Logback Appender
● Natural integration to your webapp
● Take advantage of logback framework (i.e. specific appenders per module,
log level, etc.)
● Easier if you use structured data for your kafka messages (i.e. JSON, XML,
etc.)
● Couple of OSS appenders available, but doesn’t seem to have an “official
logback version”

Apache Kafka
● Publish-Subscribe Messaging
● Fast, Scalable, Durable
● Distributed by Design
● Well Integrated to
Hadoop Ecosystem
● … makes you yell ‘Hooray!’

Analytics Platform
● Apache Spark Streaming (micro-batch)
● Apache Storm (real-time stream processing)

Summary
● Application logging is important! Tells the story of your webapp.
● Not all useful data are stored in your database.
● Listening to your webapp will give you better insights to your business.
● Data is becoming the hottest commodity right now. The faster you are able to
utilize it, the more edge you will have.

Understanding the state of your web application using Apache Kafka, Spark

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Understanding the state of your web application using Apache Kafka, Spark

Similaire à Understanding the state of your web application using Apache Kafka, Spark (20)

Plus de Exist

Plus de Exist (20)

Dernier

Dernier (20)

Understanding the state of your web application using Apache Kafka, Spark