Configuring elasticsearch for performance and scale

1. Configuring Elasticsearch For Performance and Scale Based on the knowledge gained after attending elasticsearch webinar on 30th September 2014 Prepared By: Bharvi Narayan Dixit Software Engineer, Orkash Services Pvt. Ltd.

2. Contents  The Elasticsearch Open Source Model  The Popularity of Elasticsearch  Insights across The Guardian  Ophan - The real time analytics tool  Datadog’s Elasticsearch Story  How Datadog’s event dashboards look like  Elasticsearch use @ Captora  Captora dashboard and it’s architecture  Webinar Poll for type of infrastructures used for elasticsearch

3. The Elasticsearch Open Source Model

4. The Popularity of Elasticsearch 10M downloads in 2 years and counting..

5. Insights across the Guardian • A large portion of The Guardian’s business relies on Elasticsearch to understand how their content is being consumed. • Before Ophan, guardian used a traditional analytics package which had a four-hour lag and that is too with so many restrictions. • ~40M documents is processed per day and 360M documents can be easily queried. • Real-Time traffic analysis of each content, which enables the organization to see the audience engagement. • Easy scaling the cluster (Adding more capacity) whenever there is any stress on elasticsearch because of any new feature.

6. Ophan - The real time analytics tool created by the Guardian based on elasticsearch

7. Datadog’s Elasticsearch Story • Elasticsearch is used as Datadog’s primary data store for events/logs. • Before elasticsearch Postgres was being used. • Event data is always structured with flexibility of adding/removing fields as needed. • Hundreds of millions of full-text events across 12+ indices. • ~10M documents/day. Doubling the volume every 4-5 months.

8. First version of elasticsearch cluster in Datadog • One node per AZ (availability zone) handling HTTP and data. • One large index storing all events from all time. • Writing to a pool of all nodes in the cluster. • Worked well for 1-1.5 years.

9. Faster and more scalable cluster • Split cluster into head and data nodes. • Head nodes act as a load balancer, accepting the HTTP requests. • Data nodes just interact with head and data nodes. • Use a rolling index with one month of event data each.

10. What Datadog’s engineers learned?? • Give some planning time to sizing before setting on data format. – With a bit of planning, they could have avoided migrating to a rolling index later on. – But you can’t plan for everything, so architect deployments, with migration in mind. • Monitor your elasticsearch cluster from the beginning. • Creating tooling around backup and restore should almost be in your first deployment

11. How Datadog’s event dashboards look like..

13. How Datadog’s event dashboards look like.. Provides ability to write comments over events by mentioning peers.

15. Elasticsearch use @ Captora • Captora is the first marketing cloud solution to automatically expand and optimize the marketing campaigns to engage and convert thousands of new future buyers. • It provides an approach of Adaptive Marketing, market discovery, engagement, and convert new buyers by intelligently and automatically scaling content-driven campaigns across multiple channels (search, advertising, and social). • Read more at http://www.captora.com/technology/

16. Elasticsearch use @ Captora @captora Elasticsearch is primarily used for • Indexing all textual data (i.e. crawled multi-channel content streams, user generated documents etc.) • Power the textual search, rankings, and relevant calculation of the content recommendation engine. • Power the user portal search of the content stream. Elasticsearch stats @captora • Mostly semi-structured data (i.e. web-pages, white-papers, meta data of videos from YouTube, LinkedIn updates, blogs, Tweets etc.) • ~200M documents, ~300GB of data. • Partitioned across ~1200 indices, 2300 shards, with replication factor of 4. • 6 EC2 nodes (c3.2xlarge, provisioned SSD), two AWS availability zones, ELB balanced. • Index rate: 10 to 500 requests/Sec. • Query rate: 100 to 2000 requests/Sec.

17. Captora’s Dashboard

18. Captora’s Architecture

19. Poll Time (Based on the votes by webinar attendees)

20. Thank You