Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

No REST - Architecting Real-time Bulk Async APIs

405 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2eapFFq.

Michael Uzquiano talks about how to scale API to accept many items. He examines how to evolve the Evolution of ReST over HTTP to transactional, asynchronous bulk operations. He covers job descriptors, workers, the job queue and scaling workers across an API cluster elastically. He also talks about polling methods for job completion including HTTP long polling and WebSockets. Filmed at qconnewyork.com.

Michael Uzquiano is Founder and CTO of CloudCMS and Alpaca.js Committer.

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

No REST - Architecting Real-time Bulk Async APIs

  1. 1. No REST Real-Time Bulk Asynchronous APIs
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ rest-evolution-async-operations
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. About me • Michael Uzquiano • Founder @ Cloud CMS • First time dad, not getting much sleep
  5. 5. Git
  6. 6. keep it simple
  7. 7. Building Blocks • Services • createNode() • queryNodes() • readNode(id)
 • Object Methods • update() • del() • transform(mimetype)
  8. 8. Building Blocks • Services • createNode() • queryNodes() • readNode(id)
 • Object Methods • update() • del() • transform(mimetype) POST POST GET PUT DELETE POST
  9. 9. Adding 3 Nodes branch
 .createNode({ “title”: “title1” })
 .createNode({ “title”: “title2” })
 .createNode({ “title”: “title3” });

  10. 10. Latency makes life tough branch
 .createNode({ “title”: “title1” })
 .createNode({ “title”: “title2” })
 .createNode({ “title”: “title3” });
 
 100 ms 100 ms 100 ms
  11. 11. Life is not simple branch
 .readRepository(“repository1”)
 .readBranch(“master”)
 .queryNodes({
 “category”: “my_category”
 }).each(function) {
 this.discount = 0.2;
 this.update();
 });
 

  12. 12. Problem: Latency branch
 .readRepository(“repository1”)
 .readBranch(“master”)
 .queryNodes({
 “category”: “my_category”
 }).each(function) {
 this.discount = 0.2;
 this.update();
 });
 100 ms 100 ms 100 ms 100 ms?
  13. 13. Problem: Payload Size branch
 .readRepository(“repository1”)
 .readBranch(“master”)
 .queryNodes({
 “category”: “my_category”
 }).each(function) {
 this.discount = 0.2;
 this.update();
 });

  14. 14. Problem: No Transactions App Server Transaction API Server Transaction Op Log Update #2 Update #1 Update #3 Update #1 Update #2
  15. 15. A Way Out? helper.updateNodes({
 “repositoryId”: repositoryId,
 “branchId”: branchId,
 “query”: {
 “category”: “my_category”
 },
 “properties”: {
 “discount”: 0.2
 }
 });

  16. 16. A Way Out? helper.updateNodes({
 “repositoryId”: repositoryId,
 “branchId”: branchId,
 “query”: {
 “category”: “my_category”
 },
 “properties”: {
 “discount”: 0.2
 }
 });
 100 ms } } query upsert
  17. 17. Simplicity
  18. 18. Dev
  19. 19. Dev
  20. 20. Staging / Test Dev
  21. 21. Production Staging / Test Dev
  22. 22. The hard problems • Reduce Payload Size • Multiple Calls -> Single Call • Transactions (Rollback / Commit)
  23. 23. reducing payload
  24. 24. RFC 5789 (HTTP Patch) • Modify an existing HTTP resource • Send a “set of instructions” to modify resource • Atomic • Idempotent
  25. 25. Using HTTP PUT PUT /repositories/{repositoryId}/branches/ {branchId}/nodes/{nodeId}
 
 {
 “title”: “My Title”,
 “description”: “My Description”,
 “discount”: 0.2
 }
  26. 26. Using HTTP PATCH PATCH /repositories/{repositoryId}/branches/ {branchId}/nodes/{nodeId}
 
 {
 “discount”: 0.2
 }
  27. 27. Using HTTP PATCH PATCH /repositories/{repositoryId}/branches/ {branchId}/nodes/{nodeId}
 
 {
 “discount”: 0.2
 } {
 “title”: “My Title”,
 “description”: “My Description”
 } Original JSON
  28. 28. Using HTTP PATCH PATCH /repositories/{repositoryId}/branches/ {branchId}/nodes/{nodeId}
 
 {
 “discount”: 0.2
 } {
 “title”: “My Title”,
 “description”: “My Description”,
 “discount”: 0.2
 } Modified JSON
  29. 29. RFC 6902 - JSON Patch • Sequence of Operations to modify a JSON resource • Operations • add, remove, replace, move, copy, test
  30. 30. JSON Patch Example {
 “title”: “Hello World”,
 “category”: “my_category”
 } Original JSON
  31. 31. JSON Patch Example Original JSON 
 JSON Patch {
 “title”: “Hello World”,
 “category”: “my_category”,
 “discount”: 0.2
 }
 
 
 
 [{
 “op”: “add”,
 “path”: “discount”,
 “value”: 0.2
 }]
  32. 32. HTTP Patch and JSON Patch PATCH /repositories/{repositoryId}/branches/ {branchId}/nodes/{nodeId}
 
 [{
 “op”: “add”,
 “path”: “discount”,
 “value”: 0.2
 }]
  33. 33. Path-based Changes • add • remove • replace • move • copy • test
  34. 34. multiple calls
  35. 35. Multiple Resources
  36. 36. Creates
  37. 37. Deletes X X
  38. 38. Updates X X
  39. 39. Moves X X
  40. 40. Operations Queue • create • update • remove • move • copy
  41. 41. Partial Writes? Lack of rollback or commit
  42. 42. Changesets
  43. 43. Let’s Say… my:book my:chapter my:chapter my:page my:page my:page my:page my:page
  44. 44. Create #1 my:book my:chapter my:page my:page
  45. 45. Create #2 my:book my:chapter my:page my:page my:page
  46. 46. Create #3 my:book my:chapter my:chapter my:page my:page my:page my:page
  47. 47. Create #4 my:book my:chapter my:chapter my:page my:page my:page my:page my:page
  48. 48. Changeset #1 my:chapter my:page my:page C1
  49. 49. Changeset #2 my:chapter my:page my:page my:page C1 C2
  50. 50. Changeset #3 my:chapter my:page my:page my:page my:chapter my:page C1 C2 C3
  51. 51. Changeset #4 my:chapter my:page my:page my:page my:chapter my:page my:page C1 C2 C3 C4
  52. 52. Lack of Transaction Boundaries C0 C1 C2 C3 C4 C5
  53. 53. Transaction Boundaries C0 C1 C2 C3 C4 C5 Read/Write Lock? Accumulation vs commit
  54. 54. transactions
  55. 55. Branches Long Running Transactions
  56. 56. Branches C0 C1’ C5C1 C2 C2’ C3’ C4’ Fork Merge
  57. 57. Merge • Detect collisions (various strategies) • Automatically merge where possible (JSON Patch with merge algorithm) • JSON Schema validation • Graph validation • Transactional write (Hazelcast shout out)
  58. 58. Create a Transaction var t = cloudcms.createTransaction(branch);
  59. 59. Create a Node var t = cloudcms.createTransaction(branch); t.create({
 “title”: “Hello World”
 });
  60. 60. Commit the Transaction var t = cloudcms.createTransaction(branch); t.create({
 “title”: “Hello World”
 }); t.commit(function(results) {
 console.log(“Result: “ + results.success);
 });
  61. 61. Client Side: Build Transaction API ServerApp Server
  62. 62. Client Side: Build Transaction API ServerApp Server Create #1
  63. 63. Client Side: Build Transaction API ServerApp Server Create #1 Update #2
  64. 64. Client Side: Build Transaction API ServerApp Server Create #1 Update #2 Delete #3
  65. 65. Transaction Client Side: Call commit() App Server API Server Transaction Op Log Create #1 Update #2 Delete #3 Create #1 Update #2 Delete #3
  66. 66. Transaction Executes App Server API Server Transaction Op Log Create #1 Update #2 Delete #3
  67. 67. Client polls for completion App Server API Server Transaction Op Log Results Create #1 Update #2 Delete #3
  68. 68. • Lower Latency • Reduced Payload Size • No Partial Writes / Reads Pros
  69. 69. • Reuse / repeatability • Custom “loader” code Cons
  70. 70. packages
  71. 71. Packages Package up objects, binaries and manifest (operations log) on client Send over to API and say “deal with this”
  72. 72. Create Package API ServerApp Server Create #1
  73. 73. API ServerApp Server Create #1 Update #2 Create Package
  74. 74. API ServerApp Server Create #1 Update #2 Delete #3 Create Package
  75. 75. Transaction App Server API Server Package into Archive
  76. 76. Upload Archive to Server App Server API Server Transaction
  77. 77. Server Imports the Archive App Server API Server Work Items Create #1 Update #2 Delete #3
  78. 78. Polls for completion App Server API Server Transaction Op Log Results Create #1 Update #2 Delete #3
  79. 79. Create a Package var p = cloudcms.createPackage();
  80. 80. Add Nodes var p = cloudcms.createPackage(); var node1 = p.addNode({ “title”: “title1” });
 var node2 = p.addNode({ “title”: “title2” });
  81. 81. Add Binary Attachments var p = cloudcms.createPackager(); var node1 = p.addNode({ “title”: “title1” });
 var node2 = p.addNode({ “title”: “title2” });
 
 var stream = fs.createReadStream(filePath);
 var attachment1 = p.addAttachment(node2._doc, “default”, stream);
  82. 82. Create the Archive var p = cloudcms.createPackager(); var node1 = p.addNode({ “title”: “title1” });
 var node2 = p.addNode({ “title”: “title2” });
 
 var stream = fs.createReadStream(filePath);
 var attachment1 = p.addAttachment(node2._doc, “default”, stream); var archive = p.package(“org.cloudcms”, “demo”, “1.0”);
  83. 83. Upload the Archive vault.uploadArchive(archive, function(results) {
 console.log(“Success: “ + results.success);
 });
  84. 84. Import the Archive vault.uploadArchive(archive, function(results) {
 console.log(“Success: “ + results.success);
 }); branch.importArchive(“org.cloudcms”, “demo”, “1.0”, function(results) {
 console.log(“Success: “ + results.success);
 });
  85. 85. Production Staging / Test Dev
  86. 86. The hard problems • Reduce Payload Size • Multiple Calls • Transactions
  87. 87. Michael Uzquiano
 @uzquiano https://www.cloudcms.com
  88. 88. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/rest- evolution-async-operations

×