Is Empirical Enough?
•
1 j'aime•503 vues
Is Empirical Enough? On contributions in software engineering research. Keynote at the CESSER-IP workshop
Recommandé
Contenu connexe
Tendances
Tendances (19)
Similaire à Is Empirical Enough?
Similaire à Is Empirical Enough? (20)
Dernier
Dernier (20)
Is Empirical Enough?
- 1. Is Empirical Enough? On contributions in software engineering research PER RUNESON, KEYNOTE CESSER-IP@ICSE’19 @SOFTENGRESGRP
- 2. CESSER-IP CFP • Researchers believe that practitioners are looking for quick fixes to their problems. • Practitioners have a view that case studies in research do not represent the complexities of real projects and have doubts in the results produced by research. • Hence, empirical studies are necessary to ensure the relevance and applicability of software engineering research.
- 3. How to conduct research? • Empirical Research Methods – Experiments – Case studies – Surveys – Systematic Literature reviews
- 4. Contents lists available at ScienceDirect Information and Software Technology journal homepage: www.elsevier.com/locate/infsof CERSE - Catalog for empirical research in software engineering: A Systematic mapping study Jefferson Seide Molléri ⁎ , Kai Petersen, Emilia Mendes BTH - Blekinge Tekniska Högskola Sweden A R T I C L E I N F O Keywords: Empirical research Empirical methods Mapping study A B S T R A C T Context Empirical research in software engineering contributes towards developing scientific knowledge in this field, which in turn is relevant to inform decision-making in industry. A number of empirical studies have been carried out to date in software engineering, and the need for guidelines for conducting and evaluating such research has been stressed. Objective: The main goal of this mapping study is to identify and summarize the body of knowledge on research guidelines, assessment instruments and knowledge organization systems on how to conduct and evaluate empirical research in software engineering. Method: A systematic mapping study employing manual search and snowballing techniques was carried out to identify the suitable papers. To build up the catalog, we extracted and categorized information provided by the identified papers. Results: The mapping study comprises a list of 341 methodological papers, classified according to research methods, research phases covered, and type of instrument provided. Later, we derived a brief explanatory review of the instruments provided for each of the research methods. Conclusion: We provide: an aggregated body of knowledge on the state of the art relating to guidelines, assessment instruments and knowledge organization systems for carrying out empirical software engineering research; an exemplary usage scenario that can be used to guide those carrying out such studies is also provided. 341 methodological papers
- 5. What are valid research contributions? ICSE’19 Soundness: rigorous application of appropriate research methods Significance: contributions are novel, original, and important, with respect to the existing body of knowledge Verifiability: includes sufficient information to support independent verification or replication of the paper’s contributions Presentation: writing meets the high standards of ICSE. ©2017 Artwork designed by Loogart.com 41st International Conference on Software Engineering May 25–31, 2019 | Montréal, QC, Canada Program Chairs Tevfik Bultan, Univ. of California, Santa Barbara, USA Jon Whittle, Monash University, Australia Program Board Sven Apel, University of Passau, Germany Andrew Begel, Microsoft Research, USA Antonia Bertolino, CNR-ISTI, Italy Eric Bodden, Paderborn University, Germany Yuriy Brun, University of Massachusetts Amherst, USA Margaret Burnett, Oregon State University, USA Jordi Cabot, ICREA – UOC, Spain Cristian Cadar, Imperial College London, UK Marsha Chechik, University of Toronto, Canada Jane Cleland-Huang, University of Notre Dame, USA Daniela Damian, University of Victoria, Canada Laura Dillon, Michigan State University, USA Bernd Fischer, Stellenbosch University, South Africa Alessandro Garcia, PUC-Rio, Brazil Dimitra Giannakopoulou, NASA Ames Res. Center, USA Sunghun Kim, HK Univ. of Sci. and Tech., Hong Kong Andrew J. Ko, University of Washington, USA Claire Le Goues, Carnegie Mellon University, USA David Lo, Singapore Management University, Singapore Darko Marinov, Univ. of Illinois Urbana-Champaign, USA Mira Mezini, TU Darmstadt, Germany Richard Paige, University of York, UK Corina Pasareanu, NASA Ames Res. Center, USA Lori Pollock, University of Delaware, USA Michael Pradel, TU Darmstadt, Germany Abhik Roychoudhury, NUS, Singapore Julia Rubin, University of British Columbia, Canada Koushik Sen, University of California, Berkeley USA Eleni Stroulia, University of Alberta, Canada Lin Tan, University of Waterloo, Canada Frank Tip, Northeastern University, USA Chao Wang, University of Southern California, USA Dongmei Zhang, Microsoft Research, China Andrea Zisman, The Open University, UK Program Committee Jonathan Aldrich, Carnegie Mellon University, USA Aldeida Aleti, Monash University, Australia Dalal Alrajeh, Imperial College London, UK Samik Basu, Iowa State University, USA Benoit Baudry, KTH Royal Inst. of Tech., Sweden Gabriele Bavota, USI, Switzerland Nelly Bencomo, Aston University, UK Ayse Bener, Ryerson University, Canada Domenico Bianculli, Univ. of Luxembourg, Luxembourg Christian Bird, Microsoft Research, USA Kelly Blincoe, University of Auckland, New Zealand Barbora Buhnova, Masaryk University, Czech Republic Marcel Böhme, Monash University, Australia Maria Christakis, MPI-SWS, Germany Siobhán Clarke, Trinity College Dublin, Ireland James Clause, University of Delaware, USA Rob DeLine, Microsoft Research, USA Danny Dig, Oregon State University, USA Yvonne Dittrich, IT University of Copenhagen, Denmark Emelie Engstrom, Lund University, Sweden Hakan Erdogmus, Carnegie Mellon University, USA Robert Feldt, Chalmers University of Technology, Sweden Maria Angela Ferrario, Lancaster University, UK Antonio Filieri, Imperial College London, UK Thomas Fritz, University of Zurich, Switzerland Diego Garbervetsky, Univ. of Buenos Aires, Argentina Jaco Geldenhuys, University of Stellenbosch, South Africa Milos Gligoric, University of Texas at Austin, USA Michael W. Godfrey, University of Waterloo, Canada Alessandra Gorla, IMDEA Software Institute, Spain Alex Groce, Northern Arizona University, USA John Grundy, Monash University, Australia Lars Grunske, Humboldt-Universität zu Berlin, Germany Paul Grünbacher, JKU Linz, Austria Arie Gurfinkel, University of Waterloo, Canada William G.J. Halfond, Univ. of Southern California, USA Tracy Hall, Brunel University, UK Sylvain Hallé, Univ. du Québec à Chicoutimi, Canada Dan Hao, Peking University, China Mark Harman, Facebook and UCL, UK Rachel Harrison, University of Oxford, UK Emily Hill, Drew University, USA Rashina Hoda, The University of Auckland, New Zealand Reid Holmes, University of British Columbia, Canada Jennifer Horkoff, University of Gothenburg, Sweden Jeff Huang, Texas A&M University, USA James Jones, University of California, Irvine, USA Sarfraz Khurshid, University of Texas at Austin, USA Moonzoo Kim, KAIST, South Korea Dimitris Kolovos, University of York, UK Call for Technical Papers ICSE is the premier forum for presenting and discussing the most recent and significant technical research contributions in the field of Software Engineering. We invite high quality submissions of technical research papers describing original and unpublished results of software engineering research. We welcome submissions addressing topics across the full spectrum of Software Engineering. If a submission is accepted, at least one author of the paper is required to attend the conference and present the paper in person. In addition, the authors of accepted papers will be invited to submit artifacts related to the paper; these will be evaluated by the Artifact Evaluation Committee. Evaluation Criteria Each paper submitted to the Technical Track will be evaluated based on the following criteria: • Soundness: How well the paper’s contributions are supported by rigorous application of appropriate research methods. • Significance: The extent to which the paper’s contributions are novel, original, and important, with respect to the existing body of knowledge. • Verifiability: Whether the paper includes sufficient information to support independent verification or replication of the paper’s claimed contributions. • Presentation: Whether the paper’s quality of writing meets the high standards of ICSE, including clear descriptions and explanations, adequate use of the English language, absence of major ambiguity, clearly readable figures and tables, and adherence to the formatting instructions provided below. Submission Instructions A Technical Track submission must not exceed 10 pages, including all text, figures, tables, and appendices; two additional pages containing only references are permitted. It must conform to the IEEE Conference Proceedings Formatting Guidelines. The submission must also comply with the ACM plagiarism policy and procedures. In particular, it must not have been published elsewhere and must not be under review elsewhere while under review for ICSE. The submission must also comply with the IEEE Policy on Authorship. For more instruction about submission, please visit the conference website. Lastly, the ICSE 2019 Technical Track will employ a double-blind review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to follow the double-blind review process. In particular, the authors’ names must be omitted from the submission and references to their prior work should be in the third person. Further advice, guidance and explanation about the double-blind review process can be found on the conference website. Submissions to the Technical Track that meet the above requirements can be made via the submission site (https://easychair.org/conferences/?conf=icse2019). Patricia Lago, Vrije Universiteit Amsterdam, Netherlands Wei Le, Iowa State University, USA Emmanuel Letier, University College London, UK Grace Lewis, Carnegie Mellon SEI, USA Antónia Lopes, University of Lisbon, Portugal Sam Malek, University of California, Irvine, USA Shahar Maoz, Tel Aviv University, Israel Wes Masri, American University of Beirut, Lebanon Na Meng, Virginia Tech, USA Marija Mikic, Google, USA Raffaela Mirandola, Politecnico di Milano, Italy Henry Muccini, University of L'Aquila, Italy Sarah Nadi, University of Alberta, Canada Nachiappan Nagappan, Microsoft Research, USA Shiva Nejati, University of Luxembourg, Luxembourg Tien Nguyen, University of Texas at Dallas, USA Liliana Pasquale, Univ. College Dublin & Lero, Ireland Marco Pistoia, IBM Research, USA Adam Porter, University of Maryland, USA Paul Ralph, University of Auckland, New Zealand Cindy Rubio-Gonzalez, Univ. of California, Davis, USA Caitlin Sadowski, Google, USA Anita Sarma, Oregon State University, USA Federica Sarro, University College London, UK Ina Schaefer, TU Braunschweig, Germany Carolyn Seaman, University of Maryland, USA Alexander Serebrenik, Eindhoven Univ. of Tech., Netherlands Jocelyn Simmonds, University of Chile, Chile Kathryn Stolee, North Carolina State University, USA Zhendong Su, University of California, Davis, USA Gabriele Taentzer, Universität Marburg, Germany Christoph Treude, The University of Adelaide, Australia Burak Turhan, Brunel University, UK Daniel Varro, McGill University, Canada Bogdan Vasilescu, Carnegie Mellon University, USA Helene Waeselynck, LAAS-CNRS, France Westley Weimer, University of Michigan, USA Jim Whitehead, University of California, Santa Cruz, USA Xin Xia, Monash University, Australia Tao Xie, University of Illinois at Urbana-Champaign, USA Yingfei Xiong, Peking University, China Tuba Yavuz, University of Florida, USA Cemal Yilmaz, Sabancı University, Turkey Xiangyu Zhang, Purdue University, USA Minghui Zhou, Peking University, China Ying Zou, Queen's University, Canada Marcelo d'Amorim, Federal Univ. of Pernambuco, Brazil Cleidson de Souza, Federal Univ. of Pará Belém, Brazil Arie van Deursen, Delft Univ. of Technology, Netherlands André van der Hoek, University of California, Irvine, USA IMPORTANT DATES Submission: Aug. 24, 2018 Response Period: Nov. 12-14, 2018 Notification: Dec. 12, 2018 Camera Ready: Feb. 15, 2019 2019.icse-conferences.org/track/icse-2019-Technical-Papers
- 6. What? Examples from ICSE 2018 (best papers) • Spatio-Temporal Context Reduction: A Pointer-Analysis-Based Static Approach for Detecting Use-After-Free Vulnerabilities • Identifying Design Problems in the Source Code: A Grounded Theory • Static Automated Program Repair for Heap Properties • Automated Localization for Unreproducible Builds • Large-Scale Analysis of Framework-Specific Exceptions in Android Apps • Generalized Data Structure Synthesis • Traceability in the Wild: Automatically Augmenting Incomplete Trace links • Towards Optimal Concolic Testing
- 7. Empirical contributions vs design contributions CC BY 2.0 Tim Reckman @ Flickr
- 8. What is design? • Physical artifact design • Software system design • Solution approach to a Software Engineering problem
- 9. •A little bit of knowledge theory
- 10. Research paradigms 1. The formal sciences, such as philosophy and mathematics. 2. The explanatory sciences, such as the natural sciences and major sections of the social sciences. 3. The design sciences, such as the engineering sciences, medical science and modern psychotherapy. J. E. van Aken. Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules. Journal of Management Studies, 41(2):219–246, 2004.
- 11. Empiricism in research paradigms 1. The formal sciences – ”empirically void” 2. The explanatory sciences – observation studies 3. The design sciences – evaluation studies What paradigm does software engineering belong to?
- 12. Software engineering paradigms • Explanatory science – How is SE practice wrt X? – Which test/review/requirements method is ’best’? • Design science – How can SE practice be improved wrt X? – Which test/review/requirements method should I use? van Aken Organization theory Management theory Simon Science of the ’natural’ Science of the ’artificial’
- 13. Design science goals • produce prescriptive knowledge for professionals in a discipline • share empirical insights gained from investigations of such prescriptions applied in context
- 14. Design and Science Alëna Iouguina
- 15. Design Science – my view Problem instance SolutionEvaluationProblem understanding Solution design Theoretical foundations Theoretical contributions Problem Solution
- 16. Design Science – Hevner’s view ence researchers in the various engineering fields, architecture, the arts, and other design-oriented communities. Figure 1. Design Science Research Cycles Knowledge BaseDesign Science Research Build Design Artifacts & Processes Evaluate Design Cycle Application Domain • People • Organizational Systems • Technical Systems • Problems & Opportunities Relevance Cycle • Requirements • Field Testing Rigor Cycle •Grounding •Additions to KB Foundations •Scientific Theories & Methods • Experience & Expertise • Meta-Artifacts (Design Products & Design Processes) Environment
- 17. Design Science – Wieringa’s extension improvement, and to design a treatment that aims to change the real world in the direction of stakeholder goals. Attempting to answer a knowledge question, by contrast, requires us to identify the questions and unit of study, and define measurements that will provide the quantitative or qualitative data by which we can answer these questions. Not only should we do different things to solve improvement problems or an- swer knowledge questions, the results of these efforts are also evaluated IS design science Know- ledge base Environ- ment Know- ledge question investigat ion Improve- ment problem solving Goals, budget Know- ledge artifacts Fig. 2. Framework for design science
- 18. Design Science Contributions TheoryPractice Problem elements/ constructs Solution domain Technological Rule Solution instance(s) Problem instance(s) Empirical validation Abstraction Problem Characteri- zation Problem domain Instantiation Design elements/ constructs Solution design Theoretical contributions • Technological rules • Constructs Practical contributions • Problems • Solutions
- 20. Design Science as a Lens for SE Design science research approach Prescriptive knowledge Research context Trade-offs Research methods
- 21. • ESEM’17 Using a Visual Abstract as a Lens for Communicating and Promoting Design Science Research in Software Engineering Margaret-Anne Storey University of Victoria BC, Canada mstorey@uvic.ca Emelie Engstr¨om Lund University Sweden emelie.engstrom@cs.lth.se Martin H¨ost Lund University Sweden martin.host@cs.lth.se Per Runeson Lund University Sweden per.runeson@cs.lth.se Elizabeth Bjarnason Lund University Sweden elizabeth@cs.lth.se Abstract—Empirical software engineering research aims to generate prescriptive knowledge that can help software en- gineers improve their work and overcome their challenges, but deriving these insights from real-world problems can be challenging. In this paper, we promote design science as an effective way to produce and communicate prescriptive knowledge. We propose using a visual abstract template to communicate design science contributions and highlight the main problem/solution constructs of this area of research, as well as to present the validity aspects of design knowledge. Our conceptualization of design science is derived from existing not accepted due to unclear benefits or disagreements about what constitutes a valuable “software engineering research contribution”. Excellent research contributions also go unno- ticed by practitioners that could benefit from those insights. Many software engineering research papers do not ex- plicitly describe the class of problems addressed, nor do they sufficiently describe the relevance to practitioners. They tend to focus on problem solving instances but do not sufficiently reason about how the knowledge can be generalized [6]. We feel that design science should be promoted as it is a powerful way to frame prescriptive software engineering ESEM´17 https://doi.org/10.1109/ESEM.2017.28
- 23. Prescriptive knowledge for professionals Technological rules [van Aken] To achieve <effect/change> in <situation/context> use <solution/intervention> Prescriptive knowledge
- 24. Defect detection Static Dynamic Inspection Usage-based Inspection Branch Testing Structural Testing Informal Code Inspection Req Coverage Testing Experiment 3 Experiment 1,2 Functional Testing Goal Approach Instance Technique Step-wise abstraction Equivalence Partitioning Juristo Hetzel, Basili Empir Software Eng (2014) 19:1781–1808 DOI 10.1007/s10664-013-9262-z Variation factors in the design and analysis of replicated controlled experiments Three (dis)similar studies on inspections versus unit testing Per Runeson · Andreas Stefik · Anneliese Andrews Published online: 18 August 2013 © Springer Science+Business Media New York 2013 Abstract In formal experiments on software engineering, the number of factors that may impact an outcome is very high. Some factors are controlled and change by design, while others are are either unforeseen or due to chance. This paper aims to explore how context factors change in a series of formal experiments and to identify implications for experimentation and replication practices to enable learning from experimentation. We analyze three experiments on code inspections and structural unit testing. The first two experiments use the same experimental design and instrumentation (replication), while the third, conducted by different researchers, replaces the programs and adapts defect detection methods accordingly (reproduc- tion). Experimental procedures and location also differ between the experiments. Contrary to expectations, there are significant differences between the original experiment and the replication, as well as compared to the reproduction. Some of the differences are due to factors other than the ones designed to vary between experiments, indicating the sensitivity to context factors in software engineering experimentation. In aggregate, the analysis indicates that reducing the complexity of software engineering experiments should be considered by researchers who want to obtain reliable and repeatable empirical measures. Communicated by: Natalia Juristo P. Runeson ( ) Lund University, Lund, Sweden e-mail: per.runeson@cs.lth.se A. Stefik University of Nevada, Las Vegas, NV, USA e-mail: stefika@gmail.com A. Andrews University of Denver, Denver, CO, USA e-mail: andrews@cs.du.edu Author's personal copy Levels of abstraction Prescriptive knowledge http://dx.doi.org/10.1007/s10664-013-9262-z
- 25. Research context The proper place to study elephants is the jungle, not the zoo The proper place to study bacteria is the laboratory, not the jungle CC BY 2.0 paul bica @ flickr Research context K.-J. Stol and B. Fitzgerald. The ABC of software engineering research. ACM Trans. Softw. Eng. Methodol., 27(3):11:1–11:51, 2018.
- 26. Choise of research context K.-J. Stol and B. Fitzgerald. The ABC of software engineering research. ACM Trans. Softw. Eng. Methodol., 27(3):11:1–11:51, 2018. The ABC of Soware Engineering Research • 1:11 Experimental Simulations Laboratory Experiments Judgment Studies Sample Studies Formal Theory Computer Simulations Field Experiments Field Studies Less obtrusive research More obtrusive research Increasingly more universal contexts and systems Increasingly more specific contexts and systems Quadrant II Contrived settings Quadrant I Natural settings Quadrant IV Non-empirical settings Quadrant III Neutral settings Maximum potential for generalizability over Actors Maximum potential for realism ofContext Maximum potential for precision ofmeasurement ofactors’ Behavior A C B Fig. 1. The ABC framework: eight research strategies as categories of research methods for soware engineering (Adapted from Runkel and McGrath [170]). adrants I to IV represent dierent research seings. Research context
- 27. Research context Case study [Yin]: an empirical enquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. Research context
- 28. Trade-off between research contributions Research methods Rigor Solutions Relevance Trade- offs
- 29. Assessing Design Science Research • Relevance – does it adress a real problem, and is the solution useful? • Rigor – what actions are taken to ensure problem/solution are valid? • Novelty – which contribution is new? Trade- offs
- 30. (CC BY-NC-ND 2.0) ericalaspada@flickr
- 31. Design science methods? Problem formulation Problem/ issue Industry Academia Study state of the art Candidate solution Static validation Validation in academia Dynamic validation Release solution T. Gorschek, P. Garre, S. Larsson, and C. Wohlin. A model for technology transfer in practice. IEEE Software, 23(6):88–95, 2006. Problem instance SolutionEvaluationProblem understanding Solution design Theoretical foundations Theoretical contributions Knowledge level Instance level Research methods
- 32. Conclusion • Empirical studies are means to progress and ensure the relevance of software engineering research contributions • The ultimate contributions is the theoretical and practical design knowledge • The design science frame helps assessing and communicating the contributions in/between industry and academia
- 33. Thanks • Margaret-Anne Storey • Emelie Engström • Elizabeth Bjarnason • Martin Höst • Maria Teresa Baldassarre CC BY-NC 2.0 Future Telecom @ Flickr Contact per.runeson@cs.lth.se @softengresgrp