UV logic using redis bitmap

929 vues

Publié le

UV is basic statistics.
It's the document to improve the logic of UV using Redis Bitmap functions and hashmap.

Publié dans : Données & analyses
4 commentaires
3 j’aime
Aucun téléchargement
Nombre de vues
Sur SlideShare
Issues des intégrations
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

UV logic using redis bitmap

  1. 1. UV Logic using Redis Bitmap Jooyong Oh awesome2828@gmail.com
  2. 2. What we want? ● Periodical UV ○ daily ○ weekly ○ monthly ○ total ● UV by dimension ○ country ○ service
  3. 3. before logic ● traditional logic ○ make and maintain daily UV list ○ make weekly/monthly UV list using daily UV Daily Logs Daily UV Daily UV Daily UV Daily UV Weekly UV Monthly UV Total UV UV by country UV by service
  4. 4. before logic ● size is too large !!! ○ if daily UV is 50 million, and store (Date, User-ID, country, service), we need about 55GB ● cannot support real-time UV !!! ● problem is... ○ “maintain” daily UV list ○ aggregate from “whole” daily logs
  5. 5. new logic ● Concept : ○ use Redis HashSet & Bitmap ○ maintain whole user HashSet ○ maintain daily bitmap ○ calcurate weekly, monthly, total UV with Bit operation
  6. 6. new logic Conceptual diagram
  7. 7. new logic Diagram with real command
  8. 8. new logic ● When we need UV for a day, we can get it with BITCOUNT command > BITCOUNT UV_{YYYYMMDD} ● If we need for UV for n-day, we can get it with BITOP, BITCOUNT command > BITOP OR dest UV_{YYYYMMDD1} UV_{YYYYMMDD2} … UV_ {YYYYMMDDn} > BITCOUNT dest
  9. 9. new logic ● The Bitmap name “UV_{YYYYMMDD}” can be extended to “UV_service_{YYYYMMDD}” and “UV_country_ {YYYYMMDD}” ● When we need UV for ServiceA and CountryA, we can get it with BITOP command > BITOP AND dest UV_serviceA_{YYYYMMDD} UV_countryA_ {YYYYMMDD} > BITCOUNT dest
  10. 10. Benefit of the new logic ● We can get UV for elastic period (last 3day, 4day or temporary period) ● We can get UV in real-time ● Time complexity of Redis Bitmap operation is just O(1) ● It need very small memory for UV (1.5GB when total user is 1 billion.
  11. 11. Future work ● Redis Bitmap can cover to 2^32 (about 4 billion) ● If user count increased, this logic need to be improved (shard would be one candidate)
  12. 12. Jooyong Oh awesome2828@gmail.com End of Slide