EMF Compare 2Scaling to MillionsMikaël Barbero        26 March 2013
http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.htmlEMF Compare 2.0 Released mid-se pt...
http://www.flickr.com/photos/9781230@N06/2270505202Goal: Performance Should re late to number of differenc     es, not to t...
Brand New UI   Better integration with Team/Compare
http://www.flickr.com/photos/viggum/2417430161/DemoBecause it worths a thousand words
How does it works?   in a Nutshell
Match
Two Strategies
Two Strategies            Technical ID  XMI:id, resource URI + URI fragment,               your own...
Two Strategies            Technical ID                    Content based  XMI:id, resource URI + URI fragment,             ...
http://www.flickr.com/photos/brianjmatis/3209203494/Differencing
RationalizationLongest Common Subsequencehttp://www.youtube.com/watch?v=RUckZMzqUcw
Dependencies               http://www.flickr.com/photos/qilin/2262821682/
Package 1                     Addition of ‘Package 1’ within ‘something’State machine 1                              requi...
Equivalences               http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg
Class 1                         Setting ‘Property 1’ to reference ‘Associtation 1’     Property 1                         ...
http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/       ...
Match                  DifferenceSummary                           Requirement                                   Equivalen...
How to scale to Millions?                            http://www.flickr.com/photos/somemixedstuff/2403249501/
A graph of EObjectsA «logical» EMF model
A                                  C                                      DA graph of EObjects           B                ...
A                                    C                                        DA graph of EObjects             B          ...
A   B       CA graph of Resources                   D                               E                       G             ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
AA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
B     GA   B       C                                 To be LoadedG           E          D         To be Unloaded          ...
B     GA   B       C                                 To be Loaded                                  AG           E         ...
B     GA   B       C                                 To be Loaded                                  AG           E         ...
GA   B       C                                 To be Loaded                                  AG           E          D    ...
GA   B       C                                 To be Loaded                                  AG           E          D    ...
GA   B       C                                 To be Loaded                                  AG           E          D    ...
G     CA   B       C                                 To be Loaded                                  AG           E         ...
G     CA   B       C                                 To be Loaded                                  A     BG           E   ...
G     CA   B       C                                 To be Loaded                                  A     BG           E   ...
G     CA   B       C                                 To be Loaded                                  BG           E         ...
G     CA   B       C                                 To be Loaded                                  BG           E         ...
G     CA   B       C                                 To be Loaded                                  BG           E         ...
G     CA   B       C                                 To be Loaded                                  BG           E         ...
A   B       C                                 To be Loaded                                  BG           E          D     ...
A   B       C                                 To be Loaded                                  BG           E          D     ...
A   B       C                                 To be Loaded                                  BG           E          D     ...
E     DA   B       C                                 To be Loaded                                  BG           E         ...
E     DA   B       C                                 To be Loaded                                  B     C       GG       ...
E     DA   B       C                                 To be Loaded                                  B     C       GG       ...
DA   B       C                                 To be Loaded                                  GG           E          D    ...
DA   B       C                                 To be Loaded                                  GG           E          D    ...
DA   B       C                                 To be Loaded                                  GG           E          D    ...
DA   B       C                                 To be Loaded                                  GG           E          D    ...
D     FA   B       C                                 To be Loaded                                  G     EG           E   ...
D     FA   B       C                                 To be Loaded                                  G     EG           E   ...
FA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
FA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
FA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
FA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
FA   B       C                                 To be LoadedG           E          D         To be Unloaded                ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be Loaded                                   FG           E          D    ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
A   B       C                                 To be LoadedG           E          D         To be Unloaded                 ...
Why concurrent loading?The HDD is the bottleneck, right?
Scheduling is crucialBetween loading and unloading threads toavoid high-memory consumption (our goal)
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
Summary
Summary  Build the graph of      resources
Summary  Build the graph of      resources     Concurrent     resolve all
Summary                         Find which  Build the graph of                       resources has      resources         ...
Summary                          Find which  Build the graph of                        resources has      resources       ...
Summary                          Find which  Build the graph of                       Compare resource                    ...
Summary                          Find which  Build the graph of                       Compare resource                    ...
Kill two pigs with one (angry) bird                                      http://extremehdwallpapers.com/wp-content/uploads...
http://www.flickr.com/photos/roberts87/2646193801/                                                    Lightweight   How lig...
http://www.flickr.com/photos/themonnie/2495892146/                                                    Fast   How fast?
Number of fragments (resources)              Number of EObjects                   Disk Usage (XMI files size)10000        ...
The                               promised millions        Number of fragments (resources)              Number of EObjects...
Small        Medium         Large                                          120                                           9...
Small        Medium          Large                                          120                                           ...
600                                       450                      Max Heap (MiB)Memory Usage                             ...
http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411   http://www.flickr.com/photos/exploratorium/...
Model Resolving        Scope Narrowing    Matching                    Differencing           Post-Process                 ...
Index Cross-Resources Links                              http://www.flickr.com/photos/manchesterlibrary/5425247171/
Eclipse Papyrus support                          http://www.flickr.com/photos/manchesterlibrary/5425247171/
Porting Eclipse Papyrus support
CA     B           C                                           To be LoadedG                 E           D            To b...
http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg                                       Generic Content ...
http://cdn.memegenerator.net/instances/400x/20451927.jpg                                                           Merge W...
http://www.space.com/17376-sharing-the-light-of-two-suns.html                                                             ...
What you should remember    New UI       Blazing Fast and         Lightweight              More to come ;)
http://www.eclipse.org/emf/compare/http://www.slideshare.net/mikaelbarbero/
Prochain SlideShare
Chargement dans…5
×

EMF Compare 2.0: Scaling to Millions (updated)

5 017 vues

Publié le

Slides from the Eclipse Con North America 2013 presentation

Publié dans : Technologie
0 commentaire
0 j’aime
Statistiques
Remarques
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Aucun téléchargement
Vues
Nombre de vues
5 017
Sur SlideShare
0
Issues des intégrations
0
Intégrations
3 414
Actions
Partages
0
Téléchargements
29
Commentaires
0
J’aime
0
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

EMF Compare 2.0: Scaling to Millions (updated)

  1. 1. EMF Compare 2Scaling to MillionsMikaël Barbero 26 March 2013
  2. 2. http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.htmlEMF Compare 2.0 Released mid-se ptember 2012 Sponsore d by Ericsson under Polarsys and Modeling Platform Workin g Group (MPWG)
  3. 3. http://www.flickr.com/photos/9781230@N06/2270505202Goal: Performance Should re late to number of differenc es, not to the size of the modelBut also simpler API, better understanding of differences for end-users...
  4. 4. Brand New UI Better integration with Team/Compare
  5. 5. http://www.flickr.com/photos/viggum/2417430161/DemoBecause it worths a thousand words
  6. 6. How does it works? in a Nutshell
  7. 7. Match
  8. 8. Two Strategies
  9. 9. Two Strategies Technical ID XMI:id, resource URI + URI fragment, your own...
  10. 10. Two Strategies Technical ID Content based XMI:id, resource URI + URI fragment, Heuristics, String distance your own...
  11. 11. http://www.flickr.com/photos/brianjmatis/3209203494/Differencing
  12. 12. RationalizationLongest Common Subsequencehttp://www.youtube.com/watch?v=RUckZMzqUcw
  13. 13. Dependencies http://www.flickr.com/photos/qilin/2262821682/
  14. 14. Package 1 Addition of ‘Package 1’ within ‘something’State machine 1 requires the merge of Addition of ‘State Machine 1’ within ‘Package 1’ Requirements between differences
  15. 15. Equivalences http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg
  16. 16. Class 1 Setting ‘Property 1’ to reference ‘Associtation 1’ Property 1 the merge is equivalent to ends ass. Association 1 Setting ‘Association 1’ to reference ‘Property 1’‘ass.’ and ‘ends’ are opposite references Equivalence between differences
  17. 17. http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/ Conflicts
  18. 18. Match DifferenceSummary Requirement Equivalence Conflict
  19. 19. How to scale to Millions? http://www.flickr.com/photos/somemixedstuff/2403249501/
  20. 20. A graph of EObjectsA «logical» EMF model
  21. 21. A C DA graph of EObjects B GIn Resources E F
  22. 22. A C DA graph of EObjects B GCross-resources links E F
  23. 23. A B CA graph of Resources D E G F
  24. 24. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  25. 25. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  26. 26. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  27. 27. A B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  28. 28. AA B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  29. 29. A B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  30. 30. A B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  31. 31. A B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  32. 32. B GA B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  33. 33. B GA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  34. 34. B GA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  35. 35. GA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  36. 36. GA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  37. 37. GA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  38. 38. G CA B C To be Loaded AG E D To be Unloaded A Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  39. 39. G CA B C To be Loaded A BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  40. 40. G CA B C To be Loaded A BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  41. 41. G CA B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  42. 42. G CA B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  43. 43. G CA B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  44. 44. G CA B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  45. 45. A B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  46. 46. A B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  47. 47. A B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  48. 48. E DA B C To be Loaded BG E D To be Unloaded A B Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  49. 49. E DA B C To be Loaded B C GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  50. 50. E DA B C To be Loaded B C GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  51. 51. DA B C To be Loaded GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  52. 52. DA B C To be Loaded GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  53. 53. DA B C To be Loaded GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  54. 54. DA B C To be Loaded GG E D To be Unloaded A B C G Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  55. 55. D FA B C To be Loaded G EG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  56. 56. D FA B C To be Loaded G EG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  57. 57. FA B C To be LoadedG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  58. 58. FA B C To be LoadedG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  59. 59. FA B C To be LoadedG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  60. 60. FA B C To be LoadedG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  61. 61. FA B C To be LoadedG E D To be Unloaded A B C G E Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  62. 62. A B C To be LoadedG E D To be Unloaded A B C G E D Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  63. 63. A B C To be LoadedG E D To be Unloaded A B C G E D Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  64. 64. A B C To be LoadedG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  65. 65. A B C To be LoadedG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  66. 66. A B C To be LoadedG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  67. 67. A B C To be Loaded FG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  68. 68. A B C To be LoadedG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  69. 69. A B C To be LoadedG E D To be Unloaded A B C G E D F Already browsed F Loading Thread Unloading Thread Concurrent Resolve All Multi-Threaded
  70. 70. Why concurrent loading?The HDD is the bottleneck, right?
  71. 71. Scheduling is crucialBetween loading and unloading threads toavoid high-memory consumption (our goal)
  72. 72. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  73. 73. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  74. 74. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  75. 75. Summary
  76. 76. Summary Build the graph of resources
  77. 77. Summary Build the graph of resources Concurrent resolve all
  78. 78. Summary Find which Build the graph of resources has resources changed Concurrent resolve all
  79. 79. Summary Find which Build the graph of resources has resources changed Concurrent Binary comparison resolve all of resources
  80. 80. Summary Find which Build the graph of Compare resource resources has resources with differences changed Concurrent Binary comparison resolve all of resources
  81. 81. Summary Find which Build the graph of Compare resource resources has resources with differences changed Match, Diff, etc. Concurrent Binary comparison without resolving resolve all of resources proxies
  82. 82. Kill two pigs with one (angry) bird http://extremehdwallpapers.com/wp-content/uploads/2012/08/angry-birds-wallpaper-hd-.jpg
  83. 83. http://www.flickr.com/photos/roberts87/2646193801/ Lightweight How light?
  84. 84. http://www.flickr.com/photos/themonnie/2495892146/ Fast How fast?
  85. 85. Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
  86. 86. The promised millions Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
  87. 87. Small Medium Large 120 90CPU Usage Time (sec) 60Some changes in 1 resource 30 1 10 100 number of changes
  88. 88. Small Medium Large 120 90CPU Usage Time (sec) 601 change in some fragments 30 1 5 10 20 number of changed fragments
  89. 89. 600 450 Max Heap (MiB)Memory Usage 300Max Heap (-Xmx512m) 150 0 Small Medium Large
  90. 90. http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4680503314/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4689346802/in/set-72157624060408411 What’s Next?
  91. 91. Model Resolving Scope Narrowing Matching Differencing Post-Process 100 % 75 %Time distribution 50 % 25 % Small 0 % Nominal Large
  92. 92. Index Cross-Resources Links http://www.flickr.com/photos/manchesterlibrary/5425247171/
  93. 93. Eclipse Papyrus support http://www.flickr.com/photos/manchesterlibrary/5425247171/
  94. 94. Porting Eclipse Papyrus support
  95. 95. CA B C To be LoadedG E D To be Unloaded Already browsed F Loading Thread Unloading Thread Protect the Integrity of the Model When committing, updating...
  96. 96. http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg Generic Content Based Match Engine Decently Fast
  97. 97. http://cdn.memegenerator.net/instances/400x/20451927.jpg Merge Wizard Help to Resolve Conflict
  98. 98. http://www.space.com/17376-sharing-the-light-of-two-suns.html EMF Compare 2.1 With Kepler (Eclipse 4.3), June 2013
  99. 99. What you should remember New UI Blazing Fast and Lightweight More to come ;)
  100. 100. http://www.eclipse.org/emf/compare/http://www.slideshare.net/mikaelbarbero/

×