2. What is BrandsEye?
• We monitor online conversation (Twitter,
Facebook, G+, websites, blogs, etc.) about brands
(e.g.ABSA, Gautrain) and derive insights from the
data
• We process between 1m and 3m tweets and other
brand mentions per day
• Relevancy, sentiment analysis, country and language
and other variables all automated and crowd
sourced
• Clients use our web app to see the results and
explore the data
4. BrandsEye in June 2011
• Single Java application (ear file) for everything
• Mention collection, mention processing & client web app
• All accessing MySQL directly using Hibernate & JDBC
• Single MySQL server at Rackspace
• Separate database per client account
• 2 app servers at Rackspace (Apache & JBoss)
• Client web app load balanced
• Single instance for mention collection and processing
• Appropriate architecture when BE started (2006)
• It was website mentions and not tweets back then and 20/day was a lot!
• Quick to get to market
5. Problems in 2011
• Fragile
• Any problem with mention processing or MySQL would stop mention
collection and lead to missed tweets (and grumpy clients)
• Re-deploying the app (e.g. for client web app change) would interrupt
mention collection and processing
• Hard to recover from mention processing problems as mentions were
not stored prior to processing
• Any change to any part of the application risked breaking some other
part of it (no tests)
• Hard to change MySQL schema as all parts of the application needed to
be changed and manually tested
• If our MySQL server dies we are down for a while and will lose data
• Slow
• Everything used MySQL and our server (16 cores, 72G RAM) was taking
strain
• Poor latency - long time for a mention to appear in a client account
6. Solution & Challenges
• Separate out and decouple mention collection,
mention processing, the client app and the database
• Had to keep the business running at the same time
• Small team (1.5 - 4 developers) and limited budget
so big bang new architecture not an option
=> Incremental approach which is still in progress!
The major components and technologies are described
on the following slides with details on what worked and
what didn’t work for us
7. Simplified Architecture
Chicken
Mention Store
Analytics
Public API
Account Mention Data
PostgreSQL master+slave
Beef
Feedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEye
Javascript app
JSON
HTTP
Pork
Mention
Processing
Pipeline
Redis
Mash
Account Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravy
The BrandsEye
Crowd
MySQL
Mentions
AMQP
9. Mention Flow
Feedproxy
Mention Collector RabbitMQ
Pork
Mention
Processing
Pipeline
Chicken
Mention Store
Analytics
Public API
Account Mention Data
PostgreSQL master+slave
Gravy
The BrandsEye
Crowd
Rater (starving student)
10. Feedproxy (Mention Collector)
• Java app deployed on 2 virtual servers
• Collects mentions from many different sources using Redis sorted sets
for efficient polling and de-duping
• Buffers mentions in a MongoDB capped collection as JSON messages
• Writes them to a RabbitMQ queue on a remote server for processing
• Can replay mentions from a point in the past to recover from processing
failures
• Has a web UI to display status and stats
11. Feedproxy (2)
• Redis
• Redis is a semi-persistent key/value store with sets, lists etc.
• Perfect for this application
• Uses clever fork trick with copy-on-write to save to disk periodically
• The data fits in memory easily and its not terribly bad if we lose the
most recent 2 minutes or so
• Lightning fast and uses hardly any CPU
• As easy as using in memory data structures but the app continues where
it left off after a re-deploy
• Have to watch out for “leaks” in Redis data structures
• Clustered version not available yet but you can do replication
12. Feedproxy (3)
• MongoDB
• JSON data store with indexing, querying and so on
• Uses memory mapped files for everything and relies on the OS
• Has capped collections (ring buffer) with fixed size
• Capped collection uses a lot of IO once it fills up even though we only
use one index
• Gets “swapped out” so occasional retrieval of old mentions takes a long
time - Mongo not so good on a machine doing other stuff as well
• “Expensive” way of buffering our mentions
• Have written a replacement but its not in production yet
13. Feedproxy (4)
• RabbitMQ
• RabbitMQ is a message broker
• You setup exchanges which distribute messages to queues for
consumers to process
• Initially used it to buffer mentions on the mention collector server and a
RabbitMQ shovel to get those to the RabbitMQ instance on the
processing machine
• The shovel got “stuck” sometimes (every couple of weeks)
• Rabbit’s memory usage climbs linearly with the number of messages in
its queues regardless of queue durability settings + it stops accepting
messages when a “high watermark” of memory usage is reached
• Cannot “replay” mentions from a point in time in the past
• Not good as a durable store for messages
14. Chicken API & Mention Store
Chicken
Mention Store
Analytics
Public API
Account Mention Data
PostgreSQL master+slave
Beef
Feedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEye
Javascript app
JSON
HTTP
Pork
Mention
Processing
Pipeline
Redis
Mash
Account Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravy
The BrandsEye
Crowd
MySQL
Mentions
AMQP
15. Chicken API & Mention Store
• Provides a REST API to access mentions and
analytics
• Written using Grails, a Groovy+Java Ruby on Rails clone for Spring stack
• The only app with access to the account mention databases
• Translates our high level mention filter language into SQL
• Has good set of functional tests
• BrandsEye customers can use the API to build their own apps
• Supports multiple different mention stores with a single API
• We are busy transitioning from a single MySQL server to several
PostgreSQL clusters
• Keeps stats in Redis
• Uses Apache SOLR for full text search (likely to be replaced with
PostgreSQL)
• Stateless (will be load balanced soon)
16. Chicken API & Mention Store
• Online documentation
• We use semantic versioning
• Major: Increment on breaking API changes
• Minor: Increment when new functionality added
• Patch: Increment for bug fixes
The Book of Chicken
Welcome to the BrandsEye API, v1.17.10
https://api.brandseye.com/rest/accounts/BESC27AA/mentions
?filter=Published inthelast month and Language isnt 'en'
select id, title, extract ... from mention
join ... where published_date >= ? and
language <> ?
Can also do groupby etc.
17. Why PostgreSQL?
• Synchronous replication since 9.1
• Transaction on master only commits when slave has received the log
records
• Keeps pair of servers exactly in sync
• Replication done over dedicated gigabit network link
• Read-only queries can go to master or slave, writes only to master
• We wrote an app (running on 3rd machine) to monitor the pair and
promote the slave / disable replication as needed (uses Hetzner failover
IP addresses to make the dead machine inaccessible)
• MySQL now also has something similar but it all seems a bit ropey
• Other reasons
• PostgreSQL has good full text search and arrays
• MySQL query planner doesn’t handle some simple subqueries +
inexplicably fails to use indexes for others
• We need to stick with RDBMS as we do lots of ad-hoc queries
18. Why Grails?
• Quick to build apps
• Groovy has syntax similar to Ruby and Python but runs on the JVM and
integrates seamlessly with Java code
• Most Java code is also valid Groovy code so great for a team with Java
background
• Grails is very much like Rails but using familiar Java stuff (Spring,
Hibernate etc.)
• Lots of plugins and they are easy to write
• Performance is available
• Need something fast? Just write that little bit in Java
• We won’t have to switch stacks (e.g. from Ruby to Java) at some point
for performance reasons
19. Mash Account Meta Data
Chicken
Mention Store
Analytics
Public API
Account Mention Data
PostgreSQL master+slave
Beef
Feedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEye
Javascript app
JSON
HTTP
Pork
Mention
Processing
Pipeline
Redis
Mash
Account Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravy
The BrandsEye
Crowd
MySQL
Mentions
AMQP
20. Mash Account Meta Data
• Provides a REST API for account meta data
• Grails app using MySQL (moving to PostgreSQL)
• Brands, search phrases, processing rules etc.
• Notifies client applications of changes via RabbitMQ topic exchange
• Client apps typically cache the data until it changes or a timeout expires
• Most client apps use a Java library which handles the caching and maps
the JSON to a data model
• Simple way to distribute the information to many de-coupled apps
21. Pork Mention Processor
Chicken
Mention Store
Analytics
Public API
Account Mention Data
PostgreSQL master+slave
Beef
Feedproxy
Mention Collector
Redis Mongo
RabbitMQ
Account Mention Data
PostgreSQL master+slave
BrandsEye
Javascript app
JSON
HTTP
Pork
Mention
Processing
Pipeline
Redis
Mash
Account Meta
Data
PgSQL
3rd party apps
AMQP
JDBC
MentionsGravy
The BrandsEye
Crowd
MySQL
Mentions
AMQP
22. Pork Mention Processor
• Consumes mentions (JSON messages) from
RabbitMQ queues
• Grails app with basic UI for monitoring
• Annotates mentions with extra information (relevancy, sentiment,
country, language etc.) using machine learning and other techniques
• Applies automated processing rules
• Sends & receives mentions from the BrandsEye crowd
• Writes mentions to Chicken (Mention Store)
• ACK mention if all good, otherwise NACK and re-process
• Lots of batching and shared models for performance and rate limiting
reasons
• Groovy closures and other features result in compact maintainable code
• Small amount of performance centric code in Java + machine learning
libraries
• Can process 1m+ mentions/hour, mostly waiting for Chicken
• Keeps stats in Redis
23. RabbitMQ
• Good
• ACK/NACK model is very convenient for development
• Can limit number of un-ACKed messages allowed to control app
memory usage
• Exchange types and routing keys allow for flexible setup
• Easy to duplicate messages on the fly (e.g. for debugging in live)
• Nice admin console
• Limitations
• Cannot cluster queues so even for clustered rabbit losing a machine
loses everything in its queues
• Memory usage climbs rapidly if you aren’t consuming messages
24. Beef Client Web App
• Grails + Javascript using Backbone, Handlebars etc.
• Communicates with Chicken using the same API we offer to clients
• Maintains brands, phrases etc. by communicating with Mash
• All REST using JSON
• Makes it possible for us
to refactor all of the
backend apps (e.g.
change database
schemas) so long as we
keep the API the same
25. Monitoring
• We use Wormly for
server and app alerts
• Each app has a
simple web console
that can be checked
for “ERROR”
• Logs are aggregated
using Graylog2
26.
27. SCM and Packaging
• We use Git & Bitbucket
• Same as Github but much cheaper for a small team with many repos
• Master must always be deployable, we don’t use branches much
• Apps are packaged as executable war files
• Servlet container (Jetty) embedded in war
• java -jar crackling.war and it will come up listening on its port
• Built on the target machine (no CI yet ...)
• Apache or Nginx in front
• Good documentation
• Each app has a README.md describing its purpose, how to build and
run/publish it, API endpoints, dependencies (and how to install them)
etc.
28. Hetzner & AWS vs Rackspace
• Price
• Rackspace is at least 6x more expensive that Hetzner for a similar
(supposedly better quality) machine
• Rackspace cloud servers are also much more expensive than Amazon
servers (6x or more) and they charge full price even when your server
isn’t running
• So we are moving towards “more machines that can fail” instead of “a
few really reliable machines”
• Other factors
• Hetzner give failover IPs that you can change with an API for
implementing HA stuff
• Servers have 2 NICs so you can create private nets for replication
• Traffic is free (great for backups) up to 10000G/month
• We need physical hardware for at least the database servers for
performance
29. Tech Summary
• Redis
• Fast solid software, lots of use cases for data sets that fit in memory
• Watch out for leaks
• MongoDB
• Capped collections slower than you expect
• Really wants to be the only thing installed on the server
• MySQL
• Not very good at optimizing queries or using indexes, dodgy replication
• PostgreSQL
• Synchronous replication is cool
• Advanced query optimizer
• RabbitMQ
• Great for short term routing of messages, watch out for memory usage
30. Conclusion
• We can now easily scale BrandsEye to handle any
number of clients and volume of mentions
• All of this is still in progress
• We have lots of other little apps not described
here interacting using the same tech
• We have done a lot of work with chef for our new
servers but its not handling everything yet
