Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Experience in conducting 19 secondary (survey) studies in Software Engineering

1 391 vues

Publié le

Experience in conducting 19 secondary (survey) studies in Software Engineering
By: Dr. Vahid Garousi
Invited (online) talk for the University of Adelaide, Australia
May 8, 2020

Publié dans : Technologie
  • Soyez le premier à commenter

Experience in conducting 19 secondary (survey) studies in Software Engineering

  1. 1. 1Vahid Garousi Experience in conducting 19 secondary (survey) studies in Software Engineering Dr. Vahid Garousi Associate Professor (Senior Lecturer) of Software Engineering Queen’s University Belfast, UK v.garousi@qub.ac.uk www.vgarousi.com @vgarousi Invited (online) talk for the University of Adelaide, Australia May 8, 2020 Based on studies conducted between 2009-2020: • 11 Systematic Literature Mapping (SLM) studies • 4 Systematic Literature Reviews (SLR) • 3 Multivocal Literature Reviews (MLR) • 1 Grey Literature Review (GLR)
  2. 2. 2Vahid Garousi About me Education: PhD, Carleton University, Ottawa, Canada, 2006 MSc, University of Waterloo, Canada, 2003 BSc, Sharif University of Technology, Tehran, Iran, 2000 Work experience: Associate Professor, Queen’s University Belfast, UK, 2019- Associate Professor, Wageningen University, Netherlands, 2017-2019 Associate Professor, Hacettepe University, Ankara, Turkey, 2015-2017 Associate Professor, University of Calgary, Canada, 2006-2014 Senior Consultant, Bahar Software Engineering Consulting, since 2008- (Working closely with software companies since 2001) Software Engineer, Corsha Software Corp., Montreal, Canada, 1997-2001
  3. 3. 3Vahid Garousi A big “thank you” to my co-authors (since 2003)…  I have been fortunate to have the chance to collaborate with these great researchers / practitioners since 2003
  4. 4. 4Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  What is a secondary study?  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  It all depends on their RQs (should be as insightful as possible)  How do we conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction In each discussion, we will review a few working examples (papers)
  5. 5. 5Vahid Garousi Terminology: What is a Secondary study?  Secondary (survey) study: A study of (primary research) studies, or using data from primary studies
  6. 6. 6Vahid Garousi Terminology: What is a Review Study?
  7. 7. 7Vahid Garousi Secondary studies in other fields  … where they were adopted from, to software engineering Adopted to
  8. 8. 8Vahid Garousi SLRs in Social Sciences  There are many books…
  9. 9. 9Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  Secondary (survey) studies and their history  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  …  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  It all depends on their RQs (should be as insightful as possible)  How do we conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction
  10. 10. 10Vahid Garousi Different types of secondary studies  "A picture is worth a thousand words" is conducted by SLM (SM) SLR MLRMLM Papers in formal (academic) literature of includes of Mapping is conducted by Synthesis of evidence GLM GLR Sources in grey literature of includes is conducted by of includes includes includes includes SM/SLM: Systematic (literature) mapping (classification) SLR: Systematic literature review GLM: Grey literature mapping GLR: Grey literature review MLM: Multivocal literature mapping MLR: Multivocal literature review Method of analysis Sources under study (Primary studies) Secondary studies Technical online videos White papers Blog posts Types of secondary studies Tertiary studies STRSTM <<abstract>> Tertiary study <<abstract>> Secondary study Types of tertiary studies Mapping Synthesis ofof is conducted by is a study of a set of is a study of a set of Systematic Tertiary Review Systematic Tertiary Mapping Collected data or artifacts under analysis Source code Software processes Software models Software engineers, teams and organizations Other data / artifacts study discuss / share experience about <<abstract>> Data or artifacts under analysis Software tools Looks overwhelming… Let’s look at it, piece by piece Hint: Right click, then “Z”, then pan around
  11. 11. 11Vahid Garousi How do SLRs and SLMs differ?  A Systematic Literature Review (SLR) :  Synthesizes research evidence published in a given area  A Systematic Literature Mapping (SLM /SM):  Provides a classification (mapping) of papers published in a given area  A SLM study often requires less effort compared to SLRs, while providing a more coarse-grained overview  A SLM is usually a first step towards a more in-depth SLR (first classify the papers, then synthesize their findings together)  All systematic literature studies should be: unbiased and repeatable  Summary:  SLR = Synthesis of research evidence in a given area  SLM (SM) = Classification (mapping) of papers in a given area
  12. 12. 12Vahid Garousi How do SLRs and SLMs differ? Examples…
  13. 13. 13Vahid Garousi How do SLRs and SLMs differ? By their RQs  Different focus. SLRs synthesize evidence and have more depth. SLM SLR
  14. 14. 14Vahid Garousi MLRs and GLRs: If we want to (and we should!) synthesize knowledge /evidence from industry in our SE research  Multivocal Literature Reviews (MLR): both industrial (grey) and academic literature  Grey Literature Reviews (GLR): reviewing only industrial (grey) literature
  15. 15. 15Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  Secondary (survey) studies and their history  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  …  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  The raised RQs will determine the value and impact of the studies (RQs should be as insightful as possible)  How do we conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction
  16. 16. 16Vahid Garousi Values provided by secondary studies: Academic impacts/benefit  Helping the researchers (authors) themselves for determining what research topic to work on …  E.g., a few my PhD students in the past
  17. 17. 17Vahid Garousi Values provided by secondary studies: Academic impact  Helping other researchers in their works. Can be measured by citations  Some of my top cited papers are secondary studies: Sorted
  18. 18. 18Vahid Garousi Values provided by secondary studies: Industrial impact
  19. 19. 19Vahid Garousi Importance of raising insightful Research Questions  The raised RQs will determine the value and impact of the studies  RQs should be as “insightful” as possible  Not just “convenient” RQs (for data which is “easy” to extract! But may not provide much real value) Real industry needs led to RQs
  20. 20. 20Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  Secondary (survey) studies and their history  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  …  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  It all depends on their RQs (should be as insightful as possible)  How to conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction
  21. 21. 21Vahid Garousi How to conduct secondary studies  Three guideline papers have been published:  For conducting SLRs, 2004  For conducting SLMs, 2015  For conducting MLRs, 2019 Each present certain processes (with similarities, but also differences), e.g., for SLMs:
  22. 22. 22Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  Secondary (survey) studies and their history  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  …  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  It all depends on their RQs (should be as insightful as possible)  How to conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction
  23. 23. 23Vahid Garousi Conducting secondary studies is VERY effort-intensive, and error-prone 0 10 20 30 40 50 60 70 1 6 11 16 21 26 31 36 41 46 51 56 61 66 Dataextractiontime(inminutes) Papers' extraction order in each researcher's assigned pool (time horizon) PhD student 1 PhD student 2 Senior Researcher 1 Senior Researcher 2  See the empirical data below for data extraction efforts of one SLR  We thus need to find ways to make the work effective and efficient PhD student 1 PhD student 2 Senior Researcher 1 Senior Researcher 2 Total Number of papers assigned to extract 108 91 58 55 312 Total extraction time (in hours) 54.0 33.7 7.8 16.5 112.0 Average extraction time per paper (minutes) 30.0 22.2 8.1 18.0 21.5
  24. 24. 24Vahid Garousi Guidelines for effective and efficient data extraction  Even a paper on that topic! Process for efficient data extraction
  25. 25. 25Vahid Garousi Outline: Experience in conducting secondary (survey) studies in Software Engineering  Secondary (survey) studies and their history  What are the different types of secondary studies?  Systematic Literature Mapping (SLM) studies  Systematic Literature Reviews (SLR)  Multivocal Literature Reviews (MLR)  Grey Literature Reviews (GLR)  How do they differ?  Why should (do) we conduct secondary studies?  What values do they provide? Their academic impact and industry impact  It all depends on their RQs (should be as insightful as possible)  How do we conduct secondary studies  General process for conducing secondary studies  Efforts spent on in conducting secondary studies  Experience-based guidelines for effective and efficient data extraction Questions / Answers

×