On the Papers of Giants

Notes de l'éditeur

1. \n
2. \n
3. \n
4. \n
5. \n
6. \n
7. \n
8. \n
9. \n
10. The problem is, databases are complicated, and I’m just not that smart. But here I am, talking to you about databases, so I must have fixed that somehow.\n
11. My solution was to start a user group called ChicagoDB. Every month we read an academic paper, and then discuss it as a group. I saw a problem, and found a solution for it.\n
12. \n
13. \n
14. The problem I’m talking about is fixating on Solutions instead of problems. We tend to see this a lot when new gems are released. \n
15. \n
16. \n
17. We’re talking about the core of your application, it’s data. This is the heart of your app, and shouldn’t be left to ‘just picking something cool’\n
18. I think most of us would call ourselves responsible programmers. Part of that is having at least a basic understanding of the tools and technologies you use and recommend. Knowledge that extends the bullet points of a website or the readme of a gem.\n
19. Particular systems were designed with different goals in mind. Do those goals match yours?\n
20. What kind of data model is it? Does it only offer a simple key value store, or does it offer a richer model, maybe it’s a graph database?\n\nWhat kind of querying?\nCan I only do a primary key lookup?\nWhat about secondary indexes?\nWhat about adhoc querying for something like BI reporting? Embedding documents in Mongo makes this difficult, does it still match up to my problem space well, or can I live with that?\n\nThe more of these questions you can ask upfront, the better of your life will be.\n
21. While I was on the train going to the airport to fly over here, I saw this sign, and I think it sums it up pretty well. Either way you are investing time, you might as well gain as much knowledge as you can along the way.\n\nInvestigate before you invest, true for stock portfolios and databases.\n
22. With all of that said, let’s do a little bit of investigation.\n
23. \n
24. published in 2007.\nOutlined the building blocks of a distributed system\nconsistent hashing, vector clocks, gossip protocols, hinted handoff were all described, although none of these things were new\n
25. Riak, developed by Basho, is an open source implementation of the Dynamo system\n
26. \n
27. \n
28. \n
29. Eventually this setup will fail. Something will happen to your master database and your site will effectively go down. At this point, I think it would be pretty common for people to start looking at a master-master setup, maybe some sort of sharding. But there are other solutions that live outside of the relational world that can help.\n
30. At the heart of consistent hashing is the hash function. In our over simplified version, it only returns values between 1 and 100. Riak has a 160-bit integer space.\n\n\n
31. \n
32. Each available node in the cluster is pseudo-randomly mapped onto the ring.\n
33. \n
34. \n
35. \n
36. \n
37. \n
38. or in other words, we now have a system with extremely high, transparent, availability\n
39. In order to help distribute the load evenly, we break the ring into various sections, and assign each section a vnode. Then, those vnodes are mapped back to a real server in the cluster.\n
40. We all know that nothing comes for free, except for drinks last night, so what exactly have we given up?\n
41. \n
42. We’ve all been raised on acid.\n\nAtomicity - Modifications must follow an all or nothing rule. Meaning, they either complete or do not complete, but no data is left in some sort of between state\nConsistency - Nothing in your transaction will violate the rules of the database. Integrity. More importantly, the database goes from one consistent state to the next.\nIsolation - Each transaction operates independently of every other transaction. Other transactions cannot see the\nDurability - Once the database says that data is committed there is no opportunity for that to be undone. Things like database restart, kill-9ing the process, shouldn’t cause any data lose \n\nThe properties of ACID are always desirable, they’re just not always possible as transactional growth increases\n
43. Eric Brewer theorized this in 2000, and was published in 2002\n\nThe client perceives that a set of operations has occurred all at once.\nAll nodes are available for reading and writing data.\nOperations will complete, even if individual components are unavailable.\n\nAt any given point in time, we can only have two of these.\n
44. Just so we’re all clear. A partition is anytime a node can’t communicate with the rest of the cluster. Whether that be from hardware failure, or high latency\n
45. Since we’ve already determined that a node can die, we can go ahead and circle ‘P’. And really, the CAP theorem needs to be updated to reflect this, any distributed system will need tolerance for failures, to not have it is just not acceptable.\n
46. AP system, high availability, but weak consistency guarantee\n
47. Strong consistency. Low availability, requests only return true once a quorum is met.\n\nWhat would our system look like if we wanted to switch it to CP?\n
48. In order to guarantee us some strong consistency, we would need to do a blocking write. \n
49. BASE is the opposite of ACID, get it?\n\nThe System seems to be working all the time\nIt’s not consistent all the time\nbecomes consistent at some point in time\nWhere ACID is pessimistic and forces consistency at the end of every operation, BASE is optimistic and accepts that the database consistency will be in a state of flux.\n
50. \n
51. \n
52. \n
53. \n
54. A vector clock is an algorithm for generating partial ordering of events, and can be used to detect causality violations. Each time a node updates the value for a specific key, it increments its own part of the counter.\n
55. \n
56. \n
57. If merging doesn’t fit your domain model, the other option is to actually bubble this up the application layer, and having that figure it out.\n
58. Hinted handoffs are a way of dealing with node failure and recovery\n
59. \n
60. Like school girls, the nodes like to gossip.\n\nEach node randomly communicates with the other nodes in the cluster about it’s view of the state of the cluster.\n
61. By doing a little investigation we’ve now not only about consistent hashing, vector clocks, gossip protocols, read repairs, but we’ve also learned a bit about the problems they solve, and what we’re giving up in return. In other words, we’re winning.\n
62. \n
63. \n
64. \n