• Partagez
  • E-mail
  • Intégrer
  • J'aime
  • Télécharger
  • Contenu privé
EMFCompare 2.0: Scaling to Millions
 

EMFCompare 2.0: Scaling to Millions

on

  • 920 vues

When it started in 2007, EMFCompare 1.x was designed to compare models that could fit entirely in memory. Since then, EMF has been used to design bigger and bigger models, to the point that they can ...

When it started in 2007, EMFCompare 1.x was designed to compare models that could fit entirely in memory. Since then, EMF has been used to design bigger and bigger models, to the point that they can sometimes barely fit entirely in a laptop's memory. EMFCompare 1.x is irrelevant to compare such big models because its comparison engine needs to handle 2 or 3 versions (three-way diff) of the models under comparison.

To be able to work with such large models, models are often split in multiple resources to form a set of strongly connected components in a way that a single component can fit entirely in memory. Yet EMFCompare 1.x cannot handle strategies adapted to these models such as not loading the entire model in memory or loading it piece after piece.

EMFCompare 2 is a rewrite from scratch with scalability in minds. It now has a smart scope feature to leverage the above strategies. It only loads the fragments susceptible to have changed and then compares only these parts. This way, EMFCompare 2 is able to compare models with millions of elements in a number of steps proportional to the number of differences.

During this talk, we will introduce you to the new framework and how we now are able to scale to millions. This will be shown with a lot of demo in support. We will also show you the brand new user interface that has been revamped to scale along with the new engine.

Statistiques

Vues

Total des vues
920
Vues sur SlideShare
914
Vues externes
6

Actions

J'aime
0
Téléchargements
20
Commentaires
0

2 Ajouts 6

https://twitter.com 5
https://si0.twimg.com 1

Accessibilité

Catégories

Détails de l'import

Uploaded via as Apple Keynote

Droits d'utilisation

© Tous droits réservés

Report content

Signalé comme inapproprié Signaler comme inapproprié
Signaler comme inapproprié

Indiquez la raison pour laquelle vous avez signalé cette présentation comme n'étant pas appropriée.

Annuler
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Votre message apparaîtra ici
    Processing...
Poster un commentaire
Modifier votre commentaire
  • \n
  • \n
  • \n
  • \n
  • \n
  • identity == id currently\ncontent based (heuristics) still to come\n
  • detect differences between matched elements\n\nComputer History Museum The difference engine\nan automatic, mechanical calculator designed to tabulate polynomial functions) based upon the designs of Charles Babbage\n
  • between differences => what has to be merged when the current diff is asked to be\n
  • what differences has to be marked as merge if one is asked to be\neOpposite\n
  • detecting conflict between differences when doing a three-way comparison (SCM)\n
  • automatically (accept all changes) resolve conflicts... \n
  • \n
  • \n
  • \n
  • \n
  • references between EObject that cross the boundaries of resources\n
  • let’s try to compare three resource set\n
  • \n
  • \n
  • this is the basic idea to reduce memory consumption AND cpu\nonly few of the resources have to be really loaded and compared\nbut still, no idea of the graph of resources before the comparison\nif we load resource set one by one to detect graph of resources, still take a lot of time and memory\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • this is the basic idea to reduce memory consumption AND cpu\nonly few of the resources have to be really loaded and compared\nbut still, no idea of the graph of resources before the comparison\nif we load resource set one by one to detect graph of resources, still take a lot of time and memory\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • logarithmic scale, see, each sample is an order of magnitude bigger than the previous one.\ntwo metrics to see how we scale: several modifications in 1 fragments, 1 modifications in several fragments. \n
  • \n
  • \n
  • 2GB with Medium in 1.3\n
  • \n
  • \n
  • \n
  • from 1.3\n
  • \n
  • A (fast) Generic one, you can plug in your own already\n
  • and see the differences required to be merged when merging one\n
  • milestones before, the performance presented here are from the master\n
  • \n
  • \n

EMFCompare 2.0: Scaling to Millions EMFCompare 2.0: Scaling to Millions Presentation Transcript

  • EMF Compare 2Scaling to MillionsMikaël Barbero 24 October 2012
  • http://idealistpropaganda.blogspot.fr/2010/04/imaginary-foundations-make-difference.htmlEMF Compare 2.0• Released mid-september 2012• Sponsored by Ericsson under Polarsys and Modeling Platform Working Group (MPWG)• Goal: performance should relate to number of differences, not to the size of the model • But also simpler API, better understanding of differences for end- users...
  • Brand New UI Better integration with Team/Compare
  • http://www.flickr.com/photos/viggum/2417430161/DemoBecause it worths a thousand words
  • How does it works? in a Nutshell
  • Match
  • http://www.flickr.com/photos/brianjmatis/3209203494/Differencing
  • Dependencies http://www.flickr.com/photos/qilin/2262821682/
  • Equivalences http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg
  • http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/ Conflicts
  • Merge http://www.flickr.com/photos/brendanlynch/5013468408/
  • Match DifferenceSummary Requirement Equivalence Conflict
  • How to scale to Millions? http://www.flickr.com/photos/somemixedstuff/2403249501/
  • A graph of EObjectsA «logical» EMF model
  • A C DA graph of EObjects B GIn Resources E F
  • A C DA graph of EObjects B GCross-resources links E F
  • A B CA graph of Resources D E G F
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • AG B C D To be loaded F Loaded E Unloaded Smart Resolve All Multi-Threaded
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • A B C Ancestor D E GA B C Left (local) F D EG A B C F D E G F Right (remote)
  • Summary
  • Summary Build the graph of resources
  • Summary Build the graph of resources Smart resolve all
  • Summary Find which Build the graph of resources has resources changed Smart resolve all
  • Summary Find which Build the graph of resources has resources changed Binary comparison Smart resolve all of resources
  • Summary Find which Build the graph of Compare resource resources has resources with differences changed Binary comparison Smart resolve all of resources
  • Summary Find which Build the graph of Compare resource resources has resources with differences changed Match, Diff, etc. Binary comparison Smart resolve all without resolving of resources proxies
  • Kill two pigs with one (angry) bird http://extremehdwallpapers.com/wp-content/uploads/2012/08/angry-birds-wallpaper-hd-.jpg
  • http://www.flickr.com/photos/roberts87/2646193801/ Lightweight How light?
  • http://www.flickr.com/photos/themonnie/2495892146/ Fast How fast?
  • Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
  • The promised millions Number of fragments (resources) Number of EObjects Disk Usage (XMI files size)10000 10000K 1000M1000 1000K 100M 100 100K 10M 10 10K Small Medium Large Small Medium Large Small Medium Large Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)
  • Small Medium Large 120 90CPU Usage Time (sec) 60Some changes in 1 resource 30 1 10 100 number of changes
  • Small Medium Large 120 90CPU Usage Time (sec) 601 change in some fragments 30 1 5 10 20 number of changed fragments
  • 600 450 Max Heap (MiB)Memory Usage 300Max Heap (-Xmx512m) 150 0 Small Medium Large
  • http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4680503314/in/set-72157624060408411 http://www.flickr.com/photos/exploratorium/4689346802/in/set-72157624060408411 What’s Next?
  • Model Resolving Scope Narrowing Matching Differencing Post-Process 100 % 75 %Time distribution 50 % 25 % Small 0 % Nominal Large
  • Index Cross-Resources Links http://www.flickr.com/photos/manchesterlibrary/5425247171/
  • Porting Eclipse Papyrus support http://www.flickr.com/photos/manchesterlibrary/5425247171/
  • A G B C D To be loaded F Loaded E UnloadedProtect the Integrity of the Model When committing, updating...
  • A G B C D To be loaded F Loaded E UnloadedProtect the Integrity of the Model When committing, updating...
  • http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg Generic Content Based Match Engine Decently Fast
  • http://cdn.memegenerator.net/instances/400x/20451927.jpg Merge Wizard Help to Resolve Conflict
  • http://www.space.com/17376-sharing-the-light-of-two-suns.html EMF Compare 2.1 With Kepler (Eclipse 4.3), June 2013
  • What you should remember New UI Blazing Fast and Lightweight More to come ;)
  • http://www.eclipse.org/emf/compare/http://www.slideshare.net/mikaelbarbero/