Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

1 004 vues

Publié le

Slides of my presentation at International Conference on Model Driven Engineering Languages and Systems. To be presented during the session on Friday 11:00-12:30

Publié dans : Sciences
  • Login to see the comments

An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem

  1. 1. Flickr/BenNuttall An Empirical Study on the Maturity of the Eclipse Modeling Ecosystem Javier L. Cánovas Izquierdo, Valerio Cosentino, Jordi Cabot flickr/Kansasphoto
  2. 2. Motivation flickr/airpix
  3. 3. Eclipse Ecosystem
  4. 4. Eclipse Ecosystem
  5. 5. Eclipse Ecosystem vs.
  6. 6. Maturity Model ISO 25000 ISO 9126 OPM3 CMMI PRINCE Technology Readiness Level Polarsys Maturity Model McCabe Ayalew et al. …
  7. 7. Maturity Model ISO 25000 ISO 9126 OPM3 CMMI PRINCE Technology Readiness Level Polarsys Maturity Model McCabe Ayalew et al. …
  8. 8. Maturity Model DIMENSION ATTRIBUTE QUESTION Metric ID INF. Ecosystem Activity Developer Activity eco_avg_commits_developer ↗ SCM Activity eco_num_commits ↗ eco_num_contributors ↗ eco_avg_commits_month ↗ eco_avg_commits_week ↗ eco_avg_commits_last_year ↗ Diversity Developer Activity eco_ratio_outsiders ↗ eco_ratio_commits_top_committers ↘ eco_ratio_casuals ↗ Product Analysability Code Size prod_lines_code ↘ File Types prod_num_extensions ↘ Code Complexity prod_class_complexity ↘ prod_functions_complexity ↘ prod_file_complexity ↘ Changeability Code Smells prod_code_smells ↘ File Types (see metrics defined in analysability attribute) - Code Complexity (see metrics defined in analysability attribute) - Reliability Number of Issues prod_open_issues ↘ Code Complexity (see metrics defined in analysability attribute) - Reusability Code Documentation prod_comment_density ↗ Technical Debt prod_technical_debt ↘
  9. 9. Research Questions flickr/Despeja
  10. 10. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 Is the incubation status a valid indicator of the maturity of a modeling project? RQ3
  11. 11. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 Is the incubation status a valid indicator of the maturity of a modeling project? RQ3 PROJECT TYPE PROJECTSTATUS
  12. 12. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 Is the incubation status a valid indicator of the maturity of a modeling project? RQ3
  13. 13. Building the Dataset & Calculating Metrics flickr/Clement127
  14. 14. Dataset Construction Metric Calculation
  15. 15. Building the dataset
  16. 16. Building the dataset
  17. 17. Building the dataset INPUT DISCARDED OUTPUT PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS WEB SCRAPING 216 1,710 FILTERING 216 1,710 49 1,361 167 349 CLONING 167 349 22 28 145 321 IMPORT 145 321 0 0 145 321 SQL-BASED 145 321 0 0 145 321 SONARQUBE-BASED 145 321 9 47 136 274 FINAL 136 274
  18. 18. Building the dataset INPUT DISCARDED OUTPUT PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS WEB SCRAPING 216 1,710 FILTERING 216 1,710 49 1,361 167 349 CLONING 167 349 22 28 145 321 IMPORT 145 321 0 0 145 321 SQL-BASED 145 321 0 0 145 321 SONARQUBE-BASED 145 321 9 47 136 274 FINAL 136 274 Inactive repositories Companion repositories eclipse.org gerrit
  19. 19. Building the dataset INPUT DISCARDED OUTPUT PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS WEB SCRAPING 216 1,710 FILTERING 216 1,710 49 1,361 167 349 CLONING 167 349 22 28 145 321 IMPORT 145 321 0 0 145 321 SQL-BASED 145 321 0 0 145 321 SONARQUBE-BASED 145 321 9 47 136 274 FINAL 136 274 DECEMBER 2016 M T W T F S S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
  20. 20. Building the dataset INPUT DISCARDED OUTPUT PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS WEB SCRAPING 216 1,710 FILTERING 216 1,710 49 1,361 167 349 CLONING 167 349 22 28 145 321 IMPORT 145 321 0 0 145 321 SQL-BASED 145 321 0 0 145 321 SONARQUBE-BASED 145 321 9 47 136 274 FINAL 136 274 master Gitana https://github.com/ SOM-Research/gitana
  21. 21. Calculating metrics INPUT DISCARDED OUTPUT PROJECTS REPOS PROJECTS REPOS PROJECTS REPOS WEB SCRAPING 216 1,710 FILTERING 216 1,710 49 1,361 167 349 CLONING 167 349 22 28 145 321 IMPORT 145 321 0 0 145 321 SQL-BASED 145 321 0 0 145 321 SONARQUBE-BASED 145 321 9 47 136 274 FINAL 136 274
  22. 22. Dataset descriptive statistics
  23. 23. Answering our Research Questions flickr/AftabUzzaman
  24. 24. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects RQ2 Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3
  25. 25. PROJECT TYPE PROJECTSTATUS RQ1 Are modeling projects as mature as non- modeling projects?RQ1 M ¬Mvs.
  26. 26. PROJECT TYPE PROJECTSTATUS RQ1 Are modeling projects as mature as non- modeling projects?RQ1 M ¬Mvs. Our mission: Assess whether the distributions of a given metric are different… Student’s t-test Behrens-Fisher test Mann-Whitney-Wilcoxon for modeling vs. non-modeling
  27. 27. RQ1 ECOSYSTEM METRICS METRIC DIFF. INF. eco_avg_commits_developer  ↗ eco_num_commits  ↗ eco_num_contributors *** ↗ eco_avg_commits_month  ↗ eco_avg_commits_week  ↗ eco_avg_commits_last_year  ↗ eco_ratio_outsiders  ↗ eco_ratio_commits_top_committers  ↘ eco_ratio_casuals ** ↗ PRODUCT METRICS prod_lines_code - ↘ prod_num_extensions ** ↘ prod_class_complexity  ↘ prod_functions_complexity  ↘ prod_file_complexity  ↘ prod_code_smells  ↘ prod_open_issues  ↘ prod_comment_density *** ↗ prod_technical_debt * ↘
  28. 28. RQ1 ECOSYSTEM METRICS METRIC DIFF. INF. eco_avg_commits_developer  ↗ eco_num_commits  ↗ eco_num_contributors *** ↗ eco_avg_commits_month  ↗ eco_avg_commits_week  ↗ eco_avg_commits_last_year  ↗ eco_ratio_outsiders  ↗ eco_ratio_commits_top_committers  ↘ eco_ratio_casuals ** ↗ PRODUCT METRICS prod_lines_code - ↘ prod_num_extensions ** ↘ prod_class_complexity  ↘ prod_functions_complexity  ↘ prod_file_complexity  ↘ prod_code_smells  ↘ prod_open_issues  ↘ prod_comment_density *** ↗ prod_technical_debt * ↘
  29. 29. RQ1 ECOSYSTEM METRICS METRIC DIFF. INF. eco_avg_commits_developer  ↗ eco_num_commits  ↗ eco_num_contributors *** ↗ eco_avg_commits_month  ↗ eco_avg_commits_week  ↗ eco_avg_commits_last_year  ↗ eco_ratio_outsiders  ↗ eco_ratio_commits_top_committers  ↘ eco_ratio_casuals ** ↗ PRODUCT METRICS prod_lines_code - ↘ prod_num_extensions ** ↘ prod_class_complexity  ↘ prod_functions_complexity  ↘ prod_file_complexity  ↘ prod_code_smells  ↘ prod_open_issues  ↘ prod_comment_density *** ↗ prod_technical_debt * ↘
  30. 30. RQ1 Are modeling projects as mature as non-modeling projects? RQ1 ECOSYSTEM METRICS METRIC DIFF. INF. eco_avg_commits_developer  ↗ eco_num_commits  ↗ eco_num_contributors *** ↗ eco_avg_commits_month  ↗ eco_avg_commits_week  ↗ eco_avg_commits_last_year  ↗ eco_ratio_outsiders  ↗ eco_ratio_commits_top_committers  ↘ eco_ratio_casuals ** ↗ PRODUCT METRICS prod_lines_code - ↘ prod_num_extensions ** ↘ prod_class_complexity  ↘ prod_functions_complexity  ↘ prod_file_complexity  ↘ prod_code_smells  ↘ prod_open_issues  ↘ prod_comment_density *** ↗ prod_technical_debt * ↘ Some metrics present significant differences in their distribution Modeling projects achieve lower maturity levels than non-modeling ones but that the overall difference is small.
  31. 31. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3
  32. 32. PROJECT TYPE PROJECTSTATUS RQ2 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 MI M¬I ¬MI ¬M¬Ivs. vs.
  33. 33. PROJECT TYPE PROJECTSTATUS RQ2 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 MI M¬I ¬MI ¬M¬Ivs. vs. Our mission: Assess whether the distributions of a given metric are different… ANOVA T for modeling vs. non-modeling & incubation vs. non-incubation ~
  34. 34. RQ2 ECOSYSTEM METRICS METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF. eco_avg_commits_developer   ↗ eco_num_commits   ↗ eco_num_contributors  ** ↗ eco_avg_commits_month   ↗ eco_avg_commits_week   ↗ eco_avg_commits_last_year   ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers   ↘ eco_ratio_casuals  * ↗ PRODUCT METRICS prod_lines_code ***  ↘ prod_num_extensions ***  ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells ***  ↘ prod_open_issues ***  ↘ prod_comment_density *** *** ↗ prod_technical_debt *  ↘
  35. 35. RQ2 ECOSYSTEM METRICS METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF. eco_avg_commits_developer   ↗ eco_num_commits   ↗ eco_num_contributors  ** ↗ eco_avg_commits_month   ↗ eco_avg_commits_week   ↗ eco_avg_commits_last_year   ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers   ↘ eco_ratio_casuals  * ↗ PRODUCT METRICS prod_lines_code ***  ↘ prod_num_extensions ***  ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells ***  ↘ prod_open_issues ***  ↘ prod_comment_density *** *** ↗ prod_technical_debt *  ↘
  36. 36. RQ2 ECOSYSTEM METRICS METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF. eco_avg_commits_developer   ↗ eco_num_commits   ↗ eco_num_contributors  ** ↗ eco_avg_commits_month   ↗ eco_avg_commits_week   ↗ eco_avg_commits_last_year   ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers   ↘ eco_ratio_casuals  * ↗ PRODUCT METRICS prod_lines_code ***  ↘ prod_num_extensions ***  ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells ***  ↘ prod_open_issues ***  ↘ prod_comment_density *** *** ↗ prod_technical_debt *  ↘
  37. 37. RQ2 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects? RQ2 Numerous significant differences in product metrics between modeling and non-modeling projects when they are incubation ones Differences fade down once the projects leave the incubation status The incubation status of a project affects the maturity comparison between modeling and non-modeling projects ECOSYSTEM METRICS METRIC MI vs. ¬MI M¬I vs. ¬M¬I INF. eco_avg_commits_developer   ↗ eco_num_commits   ↗ eco_num_contributors  ** ↗ eco_avg_commits_month   ↗ eco_avg_commits_week   ↗ eco_avg_commits_last_year   ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers   ↘ eco_ratio_casuals  * ↗ PRODUCT METRICS prod_lines_code ***  ↘ prod_num_extensions ***  ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells ***  ↘ prod_open_issues ***  ↘ prod_comment_density *** *** ↗ prod_technical_debt *  ↘
  38. 38. Research Questions Are modeling projects as mature as non-modeling projects?RQ1 Is the incubation status of a project an important factor when comparing modeling and non-modeling projects RQ2 Is the incubation status a valid indicator of the maturity of a modeling project? RQ3
  39. 39. PROJECT TYPE PROJECTSTATUS Project type and status Is the incubation status a valid indicator of the maturity of a modeling project?RQ3 MI M¬I vs. Our mission: Assess whether the distributions of a given metric are different… ANOVA T for modeling vs. non-modeling & incubation vs. non-incubation ~
  40. 40. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘
  41. 41. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘
  42. 42. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘ Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3 No remarkable differences in the metrics for modeling projects according to its incubation status
  43. 43. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘ Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3 No remarkable differences in the metrics for modeling projects according to its incubation status
  44. 44. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘ Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3 No remarkable differences in the metrics for modeling projects according to its incubation status
  45. 45. RQ3 ECOSYSTEM METRICS METRIC M¬I vs. MI ¬M¬I vs. ¬MI INF. eco_avg_commits_developer  * ↗ eco_num_commits  ** ↗ eco_num_contributors ** *** ↗ eco_avg_commits_month  *** ↗ eco_avg_commits_week  ** ↗ eco_avg_commits_last_year  *** ↗ eco_ratio_outsiders - - ↗ eco_ratio_commits_top_committers  *** ↘ eco_ratio_casuals  *** ↗ PRODUCT METRICS prod_lines_code  *** ↘ prod_num_extensions  *** ↘ prod_class_complexity - - ↘ prod_functions_complexity - - ↘ prod_file_complexity - - ↘ prod_code_smells  *** ↘ prod_open_issues  *** ↘ prod_comment_density   ↗ prod_technical_debt   ↘ Is the incubation status a valid indicator of the maturity of a modeling Project? RQ3 No remarkable differences in the metrics for modeling projects according to its incubation status The incubation status makes a difference only for non-modeling projects
  46. 46. Extensions flickr/LinusBohman
  47. 47. Extensions Beyond a Git-centric Approach Collaboration as a key Maturity Dimension Collaboration Beyond the Code Why not consider other supporting tools? (e.g., forums, instant messages, etc.) Pay attention to the community as one of the cornerstones in Open Source Let’s combine previous extensions and analyze collaboration networks
  48. 48. Beyond a Git-centric Approach
  49. 49. Extensions Beyond a Git-centric Approach Collaboration as a key Maturity Dimension Collaboration Beyond the Code Why not consider other supporting tools? (e.g., forums, instant messages, etc.) Pay attention to the community as one of the cornerstones in Open Source Let’s combine previous extensions and analyze collaboration networks
  50. 50. Collaboration as a Key Maturity Dimension
  51. 51. Extensions Beyond a Git-centric Approach Collaboration as a key Maturity Dimension Collaboration Beyond the Code Why not consider other supporting tools? (e.g., forums, instant messages, etc.) Pay attention to the community as one of the cornerstones in Open Source Let’s combine previous extensions and analyze collaboration networks
  52. 52. Collaboration Beyond the Code JDT Papyrus
  53. 53. flickr/woodleywonderworks
  54. 54. Conclusion and Further Work • First studies on the maturity of modeling tools • Comparison of over a hundred of modeling and non-modeling Eclipse projects • There are differences, though less than expected • Extensions to increase the scope of our maturity model What we have shown Much more research needs to be done to really understand the technical and social factors that make both families of projects different
  55. 55. wikimedia
  56. 56. Maturity vs. Adoption flickr/crwr Vallecillo, A (2014). On the Industrial Adoption of Model Driven Engineering, Is you Company ready for MDE? . International Journal of Information Systems and Software Engineering for Big Companies (IJISEBC), Vol. 1, Num. 1, pp. 52-68
  57. 57. Maturity vs. Adoption flickr/crwr …are they related? Vallecillo, A (2014). On the Industrial Adoption of Model Driven Engineering, Is you Company ready for MDE? . International Journal of Information Systems and Software Engineering for Big Companies (IJISEBC), Vol. 1, Num. 1, pp. 52-68
  58. 58. Why modeling has not succeeded yet? flickr/TimNorris
  59. 59. Thanks! Replication Package http://hdl.handle.net/20.500.12004/1/C/MODELS/2017/218 Javier L. Cánovas Izquierdo jcanovasi@uoc.edu @jlcanovas Valerio Cosentino vcosentino@uoc.edu Jordi Cabot jordi.cabot@icrea.cat @softmodeling Modeling projects achieve lower maturity levels than non-modeling ones but that the overall difference is small. RQ1 The incubation status of a project affects the maturity comparison between modeling and non-modeling projects RQ2 The incubation status makes a difference only for non-modeling projectsRQ3 Take-home messages:

×