Le téléchargement de votre SlideShare est en cours. ×
0
EMF Compare 2Scaling to MillionsMikaël Barbero        24 October 2012
http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.htmlEMF Compare 2.0• Released mid-sept...
Brand New UI   Better integration with Team/Compare
http://www.flickr.com/photos/viggum/2417430161/DemoBecause it worths a thousand words
How does it works?   in a Nutshell
Match
http://www.flickr.com/photos/brianjmatis/3209203494/Differencing
Dependencies               http://www.flickr.com/photos/qilin/2262821682/
Equivalences               http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg
http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/       ...
Merge        http://www.flickr.com/photos/brendanlynch/5013468408/
Match                  DifferenceSummary                           Requirement                                   Equivalen...
How to scale to Millions?                            http://www.flickr.com/photos/somemixedstuff/2403249501/
A graph of EObjectsA «logical» EMF model
A                                  C                                      DA graph of EObjects           B                ...
A                                    C                                        DA graph of EObjects             B          ...
A   B       CA graph of Resources                   D                               E                       G             ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
AG       B        C    D                        To be loaded    F                        Loaded    E                   Unl...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
A           B       C        Ancestor                                                    D                                ...
Summary
Summary  Build the graph of      resources
Summary  Build the graph of      resources  Smart resolve all
Summary                         Find which  Build the graph of                       resources has      resources         ...
Summary                          Find which  Build the graph of                        resources has      resources       ...
Summary                          Find which  Build the graph of                       Compare resource                    ...
Summary                          Find which  Build the graph of                       Compare resource                    ...
Kill two pigs with one (angry) bird                                      http://extremehdwallpapers.com/wp-content/uploads...
http://www.flickr.com/photos/roberts87/2646193801/                                                    Lightweight   How lig...
http://www.flickr.com/photos/themonnie/2495892146/                                                    Fast   How fast?
Number of fragments (resources)              Number of EObjects                   Disk Usage (XMI files size)10000        ...
The                               promised millions        Number of fragments (resources)              Number of EObjects...
Small        Medium         Large                                          120                                           9...
Small        Medium          Large                                          120                                           ...
600                                       450                      Max Heap (MiB)Memory Usage                             ...
http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411   http://www.flickr.com/photos/exploratorium/...
Model Resolving        Scope Narrowing    Matching                    Differencing           Post-Process                 ...
Index Cross-Resources Links                              http://www.flickr.com/photos/manchesterlibrary/5425247171/
Porting Eclipse Papyrus support                                  http://www.flickr.com/photos/manchesterlibrary/5425247171/
A           G       B                   C               D                                     To be loaded               F...
A           G       B                   C               D                                     To be loaded               F...
http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg                                       Generic Content ...
http://cdn.memegenerator.net/instances/400x/20451927.jpg                                                           Merge W...
http://www.space.com/17376-sharing-the-light-of-two-suns.html                                                             ...
What you should remember    New UI       Blazing Fast and         Lightweight              More to come ;)
http://www.eclipse.org/emf/compare/http://www.slideshare.net/mikaelbarbero/
Prochain SlideShare
Chargement dans... 5
×

EMFCompare 2.0: Scaling to Millions

812

Published on

When it started in 2007, EMFCompare 1.x was designed to compare models that could fit entirely in memory. Since then, EMF has been used to design bigger and bigger models, to the point that they can sometimes barely fit entirely in a laptop's memory. EMFCompare 1.x is irrelevant to compare such big models because its comparison engine needs to handle 2 or 3 versions (three-way diff) of the models under comparison.

To be able to work with such large models, models are often split in multiple resources to form a set of strongly connected components in a way that a single component can fit entirely in memory. Yet EMFCompare 1.x cannot handle strategies adapted to these models such as not loading the entire model in memory or loading it piece after piece.

EMFCompare 2 is a rewrite from scratch with scalability in minds. It now has a smart scope feature to leverage the above strategies. It only loads the fragments susceptible to have changed and then compares only these parts. This way, EMFCompare 2 is able to compare models with millions of elements in a number of steps proportional to the number of differences.

During this talk, we will introduce you to the new framework and how we now are able to scale to millions. This will be shown with a lot of demo in support. We will also show you the brand new user interface that has been revamped to scale along with the new engine.

0 commentaires
0 mentions J'aime
Statistiques
Remarques
  • Soyez le premier à commenter

  • Be the first to like this

Aucun téléchargement
Vues
Total des vues
812
Sur Slideshare
0
À partir des ajouts
0
Nombre d'ajouts
0
Actions
Partages
0
Téléchargements
21
Commentaires
0
J'aime
0
Ajouts 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • identity == id currently\ncontent based (heuristics) still to come\n
  • detect differences between matched elements\n\nComputer History Museum The difference engine\nan automatic, mechanical calculator designed to tabulate polynomial functions) based upon the designs of Charles Babbage\n
  • between differences => what has to be merged when the current diff is asked to be\n
  • what differences has to be marked as merge if one is asked to be\neOpposite\n
  • detecting conflict between differences when doing a three-way comparison (SCM)\n
  • automatically (accept all changes) resolve conflicts... \n
  • \n
  • \n
  • \n
  • \n
  • references between EObject that cross the boundaries of resources\n
  • let’s try to compare three resource set\n
  • \n
  • \n
  • this is the basic idea to reduce memory consumption AND cpu\nonly few of the resources have to be really loaded and compared\nbut still, no idea of the graph of resources before the comparison\nif we load resource set one by one to detect graph of resources, still take a lot of time and memory\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • this is the basic idea to reduce memory consumption AND cpu\nonly few of the resources have to be really loaded and compared\nbut still, no idea of the graph of resources before the comparison\nif we load resource set one by one to detect graph of resources, still take a lot of time and memory\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • logarithmic scale, see, each sample is an order of magnitude bigger than the previous one.\ntwo metrics to see how we scale: several modifications in 1 fragments, 1 modifications in several fragments. \n
  • \n
  • \n
  • 2GB with Medium in 1.3\n
  • \n
  • \n
  • \n
  • par 1.3\n
  • \n
  • A (fast) Generic one, you can plug in your own already\n
  • and see the differences required to be merged when merging one\n
  • milestones before, the performance presented here are from the master\n
  • \n
  • \n
  • Transcript of "EMFCompare 2.0: Scaling to Millions"

    1. 1. EMF Compare 2Scaling to MillionsMikaël Barbero 24 October 2012
    2. 2. http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.htmlEMF Compare 2.0• Released mid-september 2012• Sponsored by Ericsson under Polarsys and Modeling Platform Working Group (MPWG)• Goal: performance should relate to number of differences, not to the size of the model • But also simpler API, better understanding of differences for end- users...
    3. 3. Brand New UI Better integration with Team/Compare
    4. 4. http://www.flickr.com/photos/viggum/2417430161/DemoBecause it worths a thousand words
    5. 5. How does it works? in a Nutshell
    6. 6. Match
    7. 7. http://www.flickr.com/photos/brianjmatis/3209203494/Differencing
    8. 8. Dependencies http://www.flickr.com/photos/qilin/2262821682/
    9. 9. Equivalences http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg
    10. 10. http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/ Conflicts
    11. 11. Merge http://www.flickr.com/photos/brendanlynch/5013468408/
    12. 12. Match DifferenceSummary Requirement Equivalence Conflict
    13. 13. How to scale to Millions? http://www.flickr.com/photos/somemixedstuff/2403249501/
    14. 14. A graph of EObjectsA «logical» EMF model
    15. 15. A C DA graph of EObjects B GIn Resources E F
    16. 16. A C DA graph of EObjects B GCross-resources links E F
    17. 17. A B CA graph of Resources D E G F
    18. 18. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    19. 19. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    20. 20. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    21. 21. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    22. 22. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    23. 23. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    24. 24. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    25. 25. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    26. 26. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    27. 27. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    28. 28. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    29. 29. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    30. 30. AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
    31. 31. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    32. 32. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    33. 33. A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
    34. 34. Summary
    35. 35. Summary Build the graph of resources
    36. 36. Summary Build the graph of resources Smart resolve all
    37. 37. Summary Find which Build the graph of resources has resources changed Smart resolve all
    38. 38. Summary Find which Build the graph of resources has resources changed Binary comparison Smart resolve all of resources
    39. 39. Summary Find which Build the graph of Compare resource resources has resources with differences changed Binary comparison Smart resolve all of resources
    40. 40. Summary Find which Build the graph of Compare resource resources has resources with differences changed Match, Diff, etc. Binary comparison Smart resolve all without resolving of resources proxies
    41. 41. Kill two pigs with one (angry) bird http://extremehdwallpapers.com/wp-content/uploads/2012/08/angry-birds-wallpaper-hd-.jpg
    42. 42. http://www.flickr.com/photos/roberts87/2646193801/ Lightweight How light?
    43. 43. http://www.flickr.com/photos/themonnie/2495892146/ Fast How fast?
    44. 44. Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
    45. 45. The promised millions Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
    46. 46. Small Medium Large 120 90CPU Usage Time (sec) 60Some changes in 1 resource 30 1 10 100 number of changes
    47. 47. Small Medium Large 120 90CPU Usage Time (sec) 601 change in some fragments 30 1 5 10 20 number of changed fragments
    48. 48. 600 450 Max Heap (MiB)Memory Usage 300Max Heap (-Xmx512m) 150 0 Small Medium Large
    49. 49. http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4680503314/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4689346802/in/set-72157624060408411 What’s Next?
    50. 50. Model Resolving Scope Narrowing Matching Differencing Post-Process 100 % 75 %Time distribution 50 % 25 % Small 0 % Nominal Large
    51. 51. Index Cross-Resources Links http://www.flickr.com/photos/manchesterlibrary/5425247171/
    52. 52. Porting Eclipse Papyrus support http://www.flickr.com/photos/manchesterlibrary/5425247171/
    53. 53. A G B C D To be loaded F Loaded E UnloadedProtect the Integrity of the Model When committing, updating...
    54. 54. A G B C D To be loaded F Loaded E UnloadedProtect the Integrity of the Model When committing, updating...
    55. 55. http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg Generic Content Based Match Engine Decently Fast
    56. 56. http://cdn.memegenerator.net/instances/400x/20451927.jpg Merge Wizard Help to Resolve Conflict
    57. 57. http://www.space.com/17376-sharing-the-light-of-two-suns.html EMF Compare 2.1 With Kepler (Eclipse 4.3), June 2013
    58. 58. What you should remember New UI Blazing Fast and Lightweight More to come ;)
    59. 59. http://www.eclipse.org/emf/compare/http://www.slideshare.net/mikaelbarbero/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×