About Twitter's distribute db sharding framework, and Data Access Layer from iMobile, and a Discuss

1. gizzard, DAL and more
@tangfl
2011.2.22

2. gizzard 1
• sharding framework
• extract from FlockDB
• written in Java/Scala

3. gizzard 2
• middle ware :
• like mysql-proxy ?
• any backends
• like mytrigger ?
• adapter ?

4. gizzard 3
• rules: forwarding table
• maps key ranges to partitions
• replication tree

5. gizzard 4
• eventually consistent
• idempotent : f(f(n)) = f(n)
• commutative : f(a) + f(b) = f(b) + f(a)
• retry queue

6. gizzard 5
• gracefully failures
• read : route to remain ones
• write : make high priority to remain
replicas if any in a shard are unavailable
• retry queue

7. gizzard 6
• gracefully migrations
• replica tree
• WriteOnly shard

8. gizzard 7
• summary
• nothing new
• Get work done

9. DAL 1
• Data Access Layer
• from iMobile
• by longker, simaliu and fulin
• Not only sharding

10. DAL 2
• data table, hash by pri id, less index
• index tables, hash by any “where field” on
demand
• front php devs : I don’t care !

11. DAL 3
• insert / update / delete : auto update index
table
• query : analyze sql, choose best index table

12. DAL 4
• build-in cache : php don’t care about cache
anymore

13. DAL 5
• any backend
• treat Lucene just like mysql : select field
from table where condition
• no replica
• backend do the job

14. DAL 6
• summary
• nothing new
• Get work done

15. more
• Discuss
• fans_all list in redis
• vary from 0 to 500w
• single hash at most 5000
• auto resharding
• async

16. Thanks
• @tangfl http://t.sina.com.cn
• http://blog.fulin.org