Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Kicking ass with redis

44 609 vues

Publié le

Publié dans : Technologie
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Shameless plug: just yesterday, I published a Ruby gem for bit-wise operations over sparse bitmaps in Redis using natural syntax that lets you store millions of users without wasting precious memory:

    Memory consumption depends on how the set bits are spread out but for 1 bit it uses just around 20kb, for 8 bits from 30kb to 160kb (using the default bitmap chunk size).

    users[128_000_000] = true

    Uses 20kb as opposed to 122MB.

    Btw. syntax is also pretty nice:

    result = (active_users & premium_users & (google_search | ~google_adwords)

    (The above will be optimized down to just three BITOPs).

    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • @dvirsky Thank you for your reply! What is the best practice if you want to sort by strings? For example, want to list your users in alphabetical order?
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • @SzabolcsCsermk If you use a ZSET to index strings, you can't do range queries on it, because the hash values are not lexically ordered as the original strings.

    but if you use a ZSET to index numeric value, say a user's age. So your ZSET looks like {userId=>age, userId=>age, ...} you can use ZRANGE to get the the top N youngest users, or ZRANGEBYSCORE to get all users within an age group.
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • What do you mean about 'Ranges with ZRANGE, ZRANGEBYSCORE on numeric values only'?

    Could you show an example?
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Kicking ass with redis

  1. 1. Kicking Ass WithRedis for real world problemsDvir Volk, Chief Architect, Everything.me (@dvirsky)
  2. 2. O HAI! I CAN HAS REDIS?Extremely Quick introduction to Redis● Key => Data Structure server● In memory, with persistence● Extremely fast and versatile● Rapidly growing (Instagr.am, Craigslist, Youporn ....)● Open Source, awesome community● Used as the primary data source in Everything.me: ○ Relational Data ○ Queueing ○ Caching ○ Machine Learning ○ Text Processing and search ○ Geo Stuff
  3. 3. Key => { Data Structures } "Im a Plain Text String!" Strings/Blobs/Bitmaps Key1 Val1 Hash Tables (objects!) Key2 Val 2 Key C B B A C Linked Lists A B C D Sets Sorted Sets A: 0.1 B: 0.3 C: 500 D: 500
  4. 4. Redis is like Lego for Data● Yes, It can be used as a simple KV store.● But to really Use it, you need to think of it as a tool set.● You have a nail - redis is a hammer building toolkit.● That can make almost any kind of hammer.● Learning how to efficiently model your problem is the Zen of Redis.● Here are a few examples...
  5. 5. Pattern 1: Simple, Fast, Object StoreOur problem:● Very fast object store that scales up well.● High write throughput.● Atomic manipulation of object members.Possible use cases:● Online user data (session, game state)● Social Feed● Shopping Cart● Anything, really...
  6. 6. Storing users as HASHes email john@domain.com name John users:1 Password aebc65feae8b id 1 email Jane@domain.com name Jane users:2 Password aebc65ab117b id 2
  7. 7. Redis Pattern 1● Each object is saved as a HASH.● Hash objects are { key=> string/number }● No JSON & friends serialization overhead.● Complex members and relations are stored as separate HASHes.● Atomic set / increment / getset members.● Use INCR for centralized incremental ids.● Load objects with HGETALL / HMGET
  8. 8. Objects as Hashesclass User(RedisObject): > INCR users:id def __init__(email, name, password): (integer) 1 self.email = email self.name = name > HMSET "users:1" self.password = password "email" "user@domain.com" self.id = self.createId() "name" "John" "password" "1234"user = User(user@domain.com, John, OK1234) > HGETALL "users:1" { "email": "user@domain.com", ... }user.save()
  9. 9. Performance with growing data
  10. 10. Pattern 2: Object IndexingThe problem:● We want to index the objects we saved by various criteria.● We want to rank and sort them quickly.● We want to be able to update an index quickly.Use cases:● Tagging● Real-Time score tables● Social Feed Views
  11. 11. Indexing with Sorted Sets k:users:email email john@domain.com user:1 => 1789708973 name Johnusers:1 score 300 user:2 => 2361572523 id 1 .... email Jane@domain.com k:users:score name Jane user:2 => 250users:2 score 250 user:1 => 300 id 2 user:3 => 300
  12. 12. Redis Pattern● Indexes are sorted sets (ZSETs)● Access by value O(1), by score O(log(N)). plus ranges.● Sorted Sets map { value => score (double) }● So we map { objectId => score }● For numerical members, the value is the score● For string members, the score is a hash of the string.● Fetching is done with ZRANGEBYSCORE● Ranges with ZRANGE / ZRANGEBYSCORE on numeric values only (or very short strings)● Deleting is done with ZREM● Intersecting keys is possible with ZINTERSTORE● Each class objects have a special sorted set for ids.
  13. 13. Automatic Keys for objectsclass User(RedisObject): > ZADD k:users:email 238927659283691 "1" 1 _keySpec = KeySpec( UnorderedKey(email), > ZADD k:users:name 9283498696113 "1" UnorderedKey(name), 1 OrderedNumericalKey(points) > ZADD k:users:points 300 "1" ) .... 1 > ZREVRANGE k:users:points 0 20 withscores#creating the users - now with points 1) "1"user = User(user@domain.com, John, 2) "300"1234, points = 300) > ZRANGEBYSCORE k:users:email 238927659283691 238927659283691#saving auto-indexes 1) "1"user.save() redis> HGETALL users:1 { .. }#range query on rankusers = User.getByRank(0,20)#get by nameusers = User.get(name = John)
  14. 14. Pattern 3: Unique Value CounterThe problem:● We want an efficient way to measure cardinality of a set of objects over time.● We may want it in real time.● We dont want huge overhead.Use Cases:● Daily/Monthly Unique users● Split by OS / country / whatever● Real Time online users counter
  15. 15. Bitmaps to the rescue
  16. 16. Redis Pattern● Redis strings can be treated as bitmaps.● We keep a bitmap for each time slot.● We use BITSET offset=<object id>● the size of a bitmap is max_id/8 bytes● Cardinality per slot with BITCOUNT (2.6)● Fast bitwise operations - OR / AND / XOR between time slots with BITOP● Aggregate and save results periodically.● Requires sequential object ids - or mapping of (see incremental ids)
  17. 17. Counter API (with redis internals)counter = BitmapCounter(uniques, timeResolutions=(RES_DAY,))#sampling current userscounter.add(userId)> BITSET uniques:day:1339891200 <userId> 1#Getting the unique user count for todaycounter.getCount(time.time())> BITCOUNT uniques:day:1339891200 #Getting the the weekly unique users in the past weektimePoints = [now() - 86400*i for i in xrange(7, 0, -1)]counter.aggregateCounts(timePoints, counter.OP_TOTAL)> BITOP OR tmp_key uniques:day:1339891200 uniques:day:1339804800 ....> BITCOUNT tmp_key   
  18. 18. Pattern 4: Geo resolvingThe Problem:● Resolve lat,lon to real locations● Find locations of a certain class (restaurants) near me● IP2Location searchUse Cases:● Find a users City, ZIP code, Country, etc.● Find the users location by IP
  19. 19. A bit about geohashing● Converts (lat,lon) into a single 64 bit hash (and back)● The closer points are, their common prefix is generally bigger.● Trimming more lower bits describes a larger bounding box.● example: ○ Tel Aviv (32.0667, 34.7667) => 14326455945304181035 ○ Netanya (32.3336, 34.8578) => 14326502174498709381● We can use geohash as scores in sorted sets.● There are drawbacks such as special cases near lat/lon 0.
  20. 20. Redis Pattern● Lets index cities in a sorted set: ○ { cityId => geohash(lat,lon) }● We convert the users {lat,lon} into a goehash too.● Using ZRANGEBYSCORE we find the N larger and N smaller elements in the set: ○ ZRANGEBYSCORE <user_hash> +inf 0 8 ○ ZREVRANGEBYSCORE <user_hash> -inf 0 8● We use the scores as lat,lons again to find distance.● We find the closest city, and load it.● We can save bounding rects for more precision.● The same can be done for ZIP codes, venues, etc.● IP Ranges are indexed on a sorted set, too.
  21. 21. Other interesting use cases● Distributed Queue ○ Workers use blocking pop (BLPOP) on a list. ○ Whenever someone pushes a task to the list (RPUSH) it will be popped by exactly one worker.● Push notifications / IM ○ Use redis PubSub objects as messaging channels between users. ○ Combine with WebSocket to push messages to Web Browsers, a-la googletalk.● Machine learning ○ Use redis sorted sets as a fast storage for feature vectors, frequency counts, probabilities, etc. ○ Intersecting sorted sets can yield SUM(scores) - think log(P(a)) + log (P(b))
  22. 22. Get the sourcesImplementations of most of the examples in thisslideshow:https://github.com/EverythingMe/kickass-redisGeo resolving library:http://github.com/doat/geodisGet redis at http://redis.io