Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

DataXDay - Building a Real Time Analytics API at Scale

27 vues

Publié le


This talk will cover how we redesigned our analytics API from the ground up to serve metrics in near real time from billions of events per day. We'll go through the tools we considered for the job to how we actually implemented our solution, starting from the datastore up to the whole data pipeline and its API, leveraging Golang, Kubernetes, GCP and Citus.

Sylvain Friquet - Algolia
https://dataxday.fr/

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

DataXDay - Building a Real Time Analytics API at Scale

  1. 1. Building a Real Time Analytics API at Scale DataXDay, May 17th 2018 Sylvain Friquet @sylvainfriquet Software Engineer
  2. 2. Algolia: Search as a Service As-you-type Speed Results in milliseconds at every keystrokes. Relevance Finding the best content for every intent. User Experience Delightfully engaging, impressively intuitive. @DataXDay
  3. 3. Algolia: Search as a Service @DataXDay
  4. 4. Algolia by the numbers 16 Regions 55 Data centers Offices Regions 40B Searches /mo 150B API calls /mo 2012 Founded 200 Employees 4500 Customers $74M Funding
  5. 5. Algolia Search Analytics @DataXDay@DataXDay
  6. 6. Where we started > 4 years old project > ElasticSearch > Self Hosted > 500M to 40B searches/month > Upgrading ES Cluster too tedious @DataXDay
  7. 7. What we wanted > Sub second API response time > Low latency > Large retention > Billions of events per day > Scale with us > Hosted solution @DataXDay
  8. 8. The big picture @DataXDay
  9. 9. Datastore we considered > BigQuery > RedShift > Citus Google BigQuery @DataXDay
  10. 10. Citus > Postgres Extension > Distributed > Multi Tenant > Near real time analytics > Scale Out @DataXDay
  11. 11. Sub-second analytics query > Ingesting raw events > Rolling them up > Sub-second API queries @DataXDay
  12. 12. Rollup > Aggregation > Coarse grain analysis > Pre determined queries @DataXDay
  13. 13. Ingesting raw events > Simple schema CREATE TABLE queries ( app_id text, timestamp timestamp, query text, user_id text, created_at timestamp default now() ); @DataXDay
  14. 14. Ingesting raw events > Simple schema > Shard by tenant CREATE TABLE queries ( app_id text, timestamp timestamp, query text, user_id text, created_at timestamp default now() ); SELECT create_distributed_table('queries', 'app_id'); @DataXDay
  15. 15. Ingesting raw events > Simple schema > Shard by tenant > Batch insert (parallel COPY) > Up to 7M rows/s CREATE TABLE queries ( app_id text, timestamp timestamp, query text, user_id text, created_at timestamp default now() ); SELECT create_distributed_table('queries', 'app_id'); @DataXDay
  16. 16. Rollup table CREATE TABLE rollups_5min ( timestamp timestamp, app_id text, query_count bigint, user_count HLL, top_queries JSONB ); > Aggregate metrics per time window @DataXDay
  17. 17. Rollup table CREATE TABLE rollups_5min ( timestamp timestamp, app_id text, query_count bigint, user_count HLL, top_queries JSONB ); > Aggregate metrics per time window > TOPN and HLL extension @DataXDay
  18. 18. Rollup table CREATE TABLE rollups_5min ( timestamp timestamp, app_id text, query_count bigint, user_count HLL, top_queries JSONB ); SELECT create_distributed_table('rollups_5min', 'app_id'); > Aggregate metrics per time window > TOPN and HLL extension > Collocated with raw event tables @DataXDay
  19. 19. Rollup query > Periodic rollup INSERT INTO rollups_5min SELECT date_trunc('seconds', …) AS minute, app_id, count(*) AS query_count, hll_add_agg(hll_hash_bigint(user_id)) AS user_count, topn_add_agg(query) AS top_queries FROM queries WHERE created_at >= $1 AND created_at <= $2 GROUP BY app_id, minute @DataXDay
  20. 20. Rollup query INSERT INTO rollups_5min SELECT date_trunc('seconds', …) AS minute, app_id, count(*) AS query_count, hll_add_agg(hll_hash_bigint(user_id)) AS user_count, topn_add_agg(query) AS top_queries FROM queries WHERE created_at >= $1 AND created_at <= $2 GROUP BY app_id, minute > Periodic rollup > Concurrently executed across workers @DataXDay
  21. 21. Rollup query INSERT INTO rollups_5min SELECT date_trunc('seconds', …) AS minute, app_id, count(*) AS query_count, hll_add_agg(hll_hash_bigint(user_id)) AS user_count, topn_add_agg(query) AS top_queries FROM queries WHERE created_at >= $1 AND created_at <= $2 GROUP BY app_id, minute > Periodic rollup > Concurrently executed across workers > Out of order events @DataXDay
  22. 22. Rollup query INSERT INTO rollups_5min SELECT date_trunc('seconds', …) AS minute, app_id, count(*) AS query_count, hll_add_agg(hll_hash_bigint(user_id)) AS user_count, topn_add_agg(query) AS top_queries FROM queries WHERE created_at >= $1 AND created_at <= $2 GROUP BY app_id, minute ON CONFLICT (app_id, minute) DO UPDATE SET query_count = query + EXCLUDED.query_count, user_count = user_count + EXCLUDED.user_count, top_queries = top_queries + EXCLUDED.top_queries; > Periodic rollup > Concurrently executed across workers > Out of order events > Incremental or idempotent @DataXDay
  23. 23. Rolling up the rollups > Further aggregate 5min rollups into 1day rollups > 200x to 50,000x compression ratio in our case @DataXDay
  24. 24. API queries Sample queries > Count SELECT sum(query_count) FROM … rollups_1day UNION ALL rollups_5min … WHERE … > Distinct Approx Count SELECT hll_cardinality(sum(user_count))::bigint FROM ... > TopN SELECT (topn(topn_union_agg(top_queries), 10)).* FROM ... @DataXDay
  25. 25. Some numbers > every 5min: ~20s to rollup 20M rows > 64 shards, 48 vCPUs, 366G RAM > API latency p99 < 800ms, p95 < 500ms @DataXDay
  26. 26. Conclusion > Rollup approach working at scale > Citus becoming the foundation for several new products (Click Analytics...) @DataXDay
  27. 27. Thank you
  28. 28. The video of this presentation will be soon available at dataxday.fr Thanks to our sponsors Stay tuned by following @DataXDay

×