Bulk APIを使った高速並列データロード
•
7 likes•3,260 views
Bulk API allows for parallel loading of large amounts of data into Salesforce faster than sequential loading. It does this by splitting the data into separate jobs that are processed concurrently by multiple threads. This reduces the total load time compared to loading all of the data with a single thread. In a demo, 1 million records were loaded in 41 minutes using Bulk API, 24,300 records per second, much faster than sequential loading would have been.
Recommended
Recommended
More Related Content
Viewers also liked
Similar to Bulk APIを使った高速並列データロード
Similar to Bulk APIを使った高速並列データロード (20)
More from Salesforce Developers Japan
More from Salesforce Developers Japan (20)
Recently uploaded
Recently uploaded (20)
Bulk APIを使った高速並列データロード
- 1. Salesforce API シリーズ Bulk APIを使った高速並列データロード
- 2. Safe Harbor Safe harbor statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of intellectual property and other litigation, risks associated with possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce. com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-Q for the most recent fiscal quarter ended July 31, 2012. This documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements. #salesforcedevjp
- 3. #salesforcedevjp Speakers Mitsuhiro Okamoto Developer Program Manager @mitsuhiro
- 4. 䝋䞊䝅䝱䝹䜰䜹䜴䞁䝖䜢䝣䜷䝻䞊䛟䛰䛥䛔 @salesforcedevjp / #salesforcedevjp Salesforce Developers Japan +Salesforce Developers Japan Developer Force Japan Developer Force Group #salesforcedevjp 本Webinarは録画されています 録画ビデオはYoutube及びWebinar 詳細ページにて公開されます (参加登録URLと同じです)
- 5. ㉁ၥ䛜䛒䜛ሙྜ䛻䛿? § ㉁ၥ䛿᭱ᚋ䜎䛷ᚅ䛴ᚲせ䛿䛒䜚䜎䛫䜣! – Question䛾ᶵ⬟䜢䛳䛶㉁ၥ䜢䛧䛶ୗ䛥䛔䚹 ྍ⬟䛺ሙྜ䛿ู䛾䝇䝍䝑䝣䛜㉁ၥ䛻䛴䛔䛶ゎ⟅䛧䜎䛩䚹 䛩䛠䛻ゎ⟅ฟ᮶䛺䛔ሙྜ䛿䚸ᚋ䜋䛹䝇䝢䞊䜹䞊䛜ゎ⟅䛧䜎䛩 § 䝷䜲䝤Q&A䜢᭱ᚋ䛻⾜䛔䜎䛩 – 㛫䛾チ䛩㝈䜚䚸Webinar䛾᭱ᚋ䛻Q & A䛾㛫䜢タ䛡䜎䛩 § 㛤Ⓨ⪅䝣䜷䞊䝷䝮䜢ά⏝䛧䜎䛧䜗䛖 – 䜘䜚䛯䛟䛥䜣䛾㉁ၥ䛜䛒䜛ሙྜ䛿?䛣䛱䜙䜢䛤⏝ୗ䛥䛔 http://developer.salesforce.com/jp/forums #salesforcedevjp
- 8. 䝕䞊䝍䝻䞊䝗䛾䝇䝹䞊䝥䝑䝖 25,000,000 20,000,000 15,000,000 10,000,000 5,000,000 - #salesforcedevjp レコード件数 / 1時間 OK なかなか 素晴らしい
- 9. ୪⾜ฎ⌮
- 14. 䜒䛳䛸䜒᭱㐺䛺୪⾜ฎ⌮ 並列 順次 #salesforcedevjp 2,000㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 時間
- 15. ᭱㐺䛥䜜䛶䛔䛺䛔୪⾜ฎ⌮ 並列 順次 #salesforcedevjp 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 2,000㻌䝺䝁䞊䝗 Time
- 16. 䝻䝑䜽, እ, 䝖䝸䜺, 㛵㐃, …䛺䛹䛺䛹 並列 順次 #salesforcedevjp 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 500㻌䝺䝁䞊䝗 2,000㻌䝺䝁䞊䝗 Time 䝇䝹䞊䝥䝑䝖 䜢㜼ᐖ
- 17. 䝕䞊䝍䝻䞊䝗䜿䞊䝇䝇䝍䝕䜱 § Salesforce Bulk API 䜢ᐇ㝿䛻ヨ䛧䛶䜏䜛 § 㡰ḟ䝕䞊䝍ฎ⌮㻌䛸㻌୪ิ䝕䞊䝍ฎ⌮㻌䜢ẚ䜉䜛 § ୪ิᗘ㻌ཬ䜃㻌䝇䝹䞊䝥䝑䝖䜢⤫ィ䛩䜛 § 䝇䝹䞊䝥䝑䝖㜼ᐖせᅉ䜢≉ᐃ䛧ᅇ㑊䛩䜛 § ᭱䛾䝇䝹䞊䝥䝑䝖䜢㐩ᡂ #salesforcedevjp
- 18. ๓ᥦ᮲௳
- 19. Salesforce Bulk API § 㠀ྠᮇ㻌䛾䝕䞊䝍䝻䞊䝗 § 㔞䝕䞊䝍䝉䝑䝖䛻᭱㐺 § REST API § ᵝ䚻䛺䝒䞊䝹䛷⏝䛥䜜䛶䛔䜛 § ௵ព䛾䝥䝻䜾䝷䝭䞁䜾ゝㄒ㻌(Java➼)䛛䜙 䜃ฟ䛩䛣䛸䛜䛷䛝䜛 #salesforcedevjp
- 20. Force.com Web Service Connector (WSC) § Salesforce API䜢⡆༢䛻䛖䛯䜑䛾Java䝒䞊䝹䜻䝑䝖 – Web Services/SOAP – Asynchronous/BULK § ౫Ꮡ䝷䜲䝤䝷䝸䛜↓䛟䝅䞁䝥䝹䛻⏝ྍ⬟ #salesforcedevjp
- 22. Bulk API䛾䝻䞊䝗䛻䜎䛴䜟䜛䜻䞊䝽䞊䝗 … ealize, nvestigate, lan ᢕᥱ䚸ㄪᰝ䚸ィ⏬
- 23. 䜿䞊䝇䝇䝍䝕䜱
- 24. 㡰ḟ䝕䞊䝍䝻䞊䝗
- 25. 㡰ḟ䝻䞊䝗: ᐃ䝥䝷䞁 Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread #salesforcedevjp 時間 • 単一ジョブ • 100䝞䝑䝏 • 10,000䝺䝁䞊䝗 / 䝞䝑䝏 • 䝖䞊䝍䝹100䝺䝁䞊䝗
- 29. Demo 㡰ḟ䝻䞊䝗
- 30. 㡰ḟ䝻䞊䝗⤖ᯝ ྠᐇ⾜䝰䞊䝗 㡰ḟ ฎ⌮䝺䝁䞊䝗ᩘ 1,000,000 ኻᩋ䝺䝁䞊䝗ᩘ 0 ᐇ⾜㛫 41 minutes 䝇䝹䞊䝥䝑䝖 24,300 䝺䝁䞊䝗 / ศ ᖹ⾜ฎ⌮⋡ 0.92 ၥ㢟Ⅼ ୪⾜ฎ⌮⋡䛿ᚲ䛪1௨ୗ䛸䛺䛳䛶䛧䜎䛖 ゎỴ⟇ 䝇䝹䞊䝥䝑䝖ྥୖ䛾Ⅽ䛻୪⾜䝻䞊䝗䜢᳨ウ䛩䜛 #salesforcedevjp
- 31. ୪⾜ฎ⌮ vs ༢୍䝆䝵䝤䛾䝇䝹䞊䝥䝑䝖 350000 分 1300000 / ) 数250000 ドー200000 コレ150000 (ト100000 ップ50000 ール0 ス並列処理 #salesforcedevjp 㡰ḟᐇ⾜ • ୪⾜ᛶ䛻ၥ㢟᭷䜚 㡰ḟฎ⌮ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
- 32. ୪⾜䝕䞊䝍䝻䞊䝗
- 33. ୪⾜ㄞ䜏㎸䜏: ᐃ䝥䝷䞁 Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread #salesforcedevjp • 複数ジョブ • 100䝞䝑䝏 • 10,000 䝺䝁䞊䝗㻌/䝞䝑䝏 • 䝖䞊䝍䝹100䝺䝁䞊䝗 時間
- 35. ௨ୗ䛻ὀ┠ § 䝻䝑䜽䛿୪ิᐇ⾜䛻ከ䛺ᙳ㡪䜢䛘䜛 – ฎ⌮⬟ຊ䜢↓㥏䛻䛧䛶䛧䜎䛖 – 䝇䝹䞊䝥䝑䝖䛾పୗ – 䜶䝷䞊 § ᐇ⾜䛿䛩䜉䛶䜢ゎỴ䛩䜛䜟䛡䛷䛿䛺䛔 #salesforcedevjp
- 36. 䝕䝰 ୪ิฎ⌮ v1
- 37. ୪ิ䝻䞊䝗㻌v1 䜎䛸䜑 ྠᐇ⾜䝰䞊䝗 ୪ิ ฎ⌮䝺䝁䞊䝗ᩘ 404200 ኻᩋ䝺䝁䞊䝗ᩘ 198600 ᐇ⾜㛫 10ศ 䝇䝹䞊䝥䝑䝖 25,000䝺䝁䞊䝗 / ศ ᖹ⾜ฎ⌮⋡ 15.79 ၥ㢟Ⅼ 䝻䝑䜽እ䛜Ⓨ⏕䚹㻌䝃䞊䝞䛿ฎ⌮䜢ศᩓ䛥䛫䛶⾜䛳䛯䛜䚸䝇䝹䞊䝥䝑䝖䛿ୖ䛜䜙䛺䛛䛳䛯 ゎỴ⟇ 㡰ḟ䝰䞊䝗䛷ືస䛩䜛䛛䚸䝻䝑䜽䜢⟶⌮䛩䜛 #salesforcedevjp
- 38. ୪⾜ฎ⌮v1 vs 㡰ḟฎ⌮ : 䝇䝹䞊䝥䝑䝖 350000 分 1300000 / ) 数250000 ドー200000 コレ150000 (トッ100000 プー50000 ルス0 並列処理 #salesforcedevjp ୪⾜ᐇ⾜䚷v1 • 㧗䛔୪⾜ฎ⌮⋡ • 䝻䝑䜽䛻䜘䜛䝇䝹䞊䝥䝑䝖䛜పୗ 㡰ḟฎ⌮ ୪⾜ฎ⌮ v1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
- 39. ᭱㐺!! § ᭱㐺䛺䝕䞊䝍䝻䞊䝗䛾䛯䜑䛾䜻䞊䝽䞊䝗䛂㻌 䛃 § ealize – ᢕᥱ – 䝻䝑䜽䛿୪⾜ᛶ䛸䝇䝹䞊䝥䝑䝖䜢㜼ᐖ § nvestigate – ㄪᰝ – ఱ䛜䝻䝑䜽䛾ཎᅉ䛛 § lan – ィ⏬ – 䝻䝑䜽䜢䝁䞁䝖䝻䞊䝹䛩䜛 #salesforcedevjp
- 41. ୪ิฎ⌮v2 : ⤖ᯝ ྠᐇ⾜䝰䞊䝗 ୪⾜ ฎ⌮䝺䝁䞊䝗ᩘ 100䝺䝁䞊䝗 ኻᩋ䝺䝁䞊䝗ᩘ 0 ᐇ⾜㛫 2ศ36⛊ 䝇䝹䞊䝥䝑䝖 400,000䝺䝁䞊䝗 / ศ ᖹ⾜ฎ⌮⋡ 19 ၥ㢟Ⅼ 䛺䛧 ゎỴ⟇ n/a #salesforcedevjp
- 42. ୪⾜ฎ⌮v2 vs 㡰ḟฎ⌮ : 䝇䝹䞊䝥䝑䝖 350000 分 300000 1/ ) 250000 数ド200000 ーコ150000 レ(ト100000 ップ50000 ール0 ス並行処理 #salesforcedevjp ୪⾜ฎ⌮ v2 • 㧗䛔୪⾜ᛶ • 㧗䝇䝹䞊䝥䝑䝖 㡰ḟฎ⌮ ୪⾜ฎ⌮ v1 ୪⾜ฎ⌮ v2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
- 43. 䝻䝑䜽䛾⟶⌮ … ὀព䛧䛺䛡䜜䜀䛺䜙䛺䛔䝫䜲䞁䝖 § ᚑ㛵ಀ § ཧ↷㛵ಀ § ✚䜏ୖ䛢㞟ィ㡯┠ § 䝖䝸䜺 § 䝽䞊䜽䝣䝻䞊䝹䞊䝹 § 䜾䝹䞊䝥䝯䞁䝞䞊䝅䝑䝥䝻䝑䜽 #salesforcedevjp
- 44. 䝻䝑䜽䛿⟶⌮ฟ᮶䜛 § 䝻䝑䜽䛾ཎᅉ䛸䛺䜛せᅉ䜢㝖䛩䜛 – 䝖䝸䜺䚸䝽䞊䜽䝣䝻䞊䜢୍ⓗ䛻Ṇ䛩䜛 – 䝣䜯䜲䝹ෆ䛾䝕䞊䝍䜢䝋䞊䝖䛩䜛 #salesforcedevjp
- 45. Parallel load: Sample results ྠᐇ⾜䝰䞊䝗 Parallel ฎ⌮䝺䝁䞊䝗ᩘ 1 million ኻᩋ䝺䝁䞊䝗ᩘ 0 ᐇ⾜㛫 4 minutes ᡤせ㛫 1 hour 䝇䝹䞊䝥䝑䝖 250,000 records per minute ᖹ⾜ฎ⌮⋡ 16.5 ၥ㢟Ⅼ Minimal overhead due to locks ゎỴ⟇ Remove all unnecessary locks #salesforcedevjp
- 46. 䝇䝻䝑䝖ไᚚ/ ୪⾜䝕䞊䝍䝻䞊䝗
- 47. 䝇䝻䝑䝖ไᚚ䛷䛾䝕䞊䝍䝻䞊䝗 § 䝇䝹䞊䝥䝑䝖䛸ᖹ⾜ไ䜢⪃៖䛧䛯䝇䝻䝑䝖䝸䞁䜾 – ୪⾜ฎ⌮ཬ䜃䝻䞊䝕䜱䞁䜾 – 㠀ྠᮇฎ⌮䛾ඃඛ㡰䛡 § 」㞧䛺䝆䝵䝤䛻䛚䛡䜛㜼ᐖせᅉ䜢⟶⌮䛩䜛 – 䝕䞊䝍䛾䝍䜲䝭䞁䜾 – 」ᩘ䛾䝻䝑䜽 #salesforcedevjp
- 48. 䝇䝻䝑䝖ไᚚ䝴䞊䝇䜿䞊䝇1:ᐃ Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread #salesforcedevjp 時間 • 100 ジョブ • 1 䝞䝑䝏/䝆䝵䝤 • 10,000 䝺䝁䞊䝗/䝞䝑䝏 • 100䝺䝁䞊䝗
- 49. 䝇䝻䝑䝖ไᚚ vs 㡰ḟฎ⌮ : 䝇䝹䞊䝥䝑䝖 350000 300000 250000 200000 150000 100000 50000 0 #salesforcedevjp 䝇䝻䝑䝖ไᚚ v1 • ప䛔୪ิᛶ • ప䛔䝇䝹䞊䝥䝑䝖 㡰ḟ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Degree of Parallelism Throughput Records/Min ୪⾜ฎ⌮ v1 ୪⾜ฎ⌮ v2 䝇䝻䝑䝖ไᚚ v1
- 50. Controlled Feed Use Case 2: Expected plan Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread Thread #salesforcedevjp Time • 10 ジョブ • 10 䝞䝑䝏/䝆䝵䝤 • 10,000 䝺䝁䞊䝗/䝞䝑䝏 • 100䝖䞊䝍䝹䝺䝁䞊䝗
- 51. 䝇䝻䝑䝖ไᚚ vs 㡰ḟฎ⌮ : 䝇䝹䞊䝥䝑䝖 350000 分 300000 1/ 250000 ) 数ド200000 ーコ150000 レ(ト100000 ップ50000 ール0 ス並列処理 #salesforcedevjp 䝇䝻䝑䝖ไᚚᐇ⾜ • ୪⾜ᛶ䜢ᢚ䛘䜛 • ᮇᚅ㏻䜚䛾䝇䝹䞊䝥䝑䝖 㡰ḟ Parallel 1 ୪⾜ฎ⌮ v2 䝇䝻䝑䝖ไᚚ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
- 52. 䜎䛸䜑 § ୪ิ䝕䞊䝍䝻䞊䝗䛾䛯䜑䛾䜻䞊䝽䞊䝗䛂㻌 䛃 § ealize – ᢕᥱ – 䝻䝑䜽䛿୪⾜ᛶ䛸䝇䝹䞊䝥䝑䝖䜢㜼ᐖ § nvestigate – ㄪᰝ – ఱ䛜䝻䝑䜽䛾ཎᅉ䛛 § lan – ィ⏬ – 䝻䝑䜽䜢䝁䞁䝖䝻䞊䝹䛩䜛 #salesforcedevjp
- 53. Q & A #salesforcedevjp Mitsuhiro Okamoto Developer Program Manager @mitsuhiro
- 54. Thank you