Research design for Evaluation of Strongly Sustainability Business Model Ontology

Research Design for the
Evaluation of the Strongly Sustainable
Business Model Ontology
Prepared Prior to Commencing Field Work for a Master Thesis at

PUBLIC VERSION

Toronto Antony Upward
Ontario Student # 211135423
Canada email: aupward@yorku.ca / +1 416 576 2542

MES Thesis: Strongly Evaluation Research Design December 19, 2011
Sustainable Business Model Ontology 1 Version 2.1

Research Design for the Evaluation of the Strongly
Sustainable Business Model Ontology

Table of Contents

List of Figures 4
List of Tables 5
Abbreviations 7
1. Introduction 8
1.1 Research Purpose 8
1.2 Document Purpose 9
1.3 Commentary and Possible Contribution 10
1.4 Document Structure 11
2. Research Design – Overview 12
2.1 Introduction 12
2.2 Design Science – A Definition 12
2.3 The Design of the SSBMO Research Project 13
2.4 The Scope of the SSBMO Research Project 15
2.5 The Framework for Conducting Design Science Research 16
2.6 The Framework for Conducting the SSBMO Research 17
2.7 The Research Cycle for Conducting Design Science Research 18
2.8 Process of Inquiry for Conducting the SSBMO Research 19
2.9 Summarizing the SSBMO Build Research Activities and Outputs (D1-4) 21
2.10 Summarizing the SSBMO Evaluate Research Activities and Outputs (E1-3) 24
3. Evaluation in Ontology, Design Science and Systems Research 25
3.1 Introduction 25
3.2 Reviewing the Literature 26
3.3 Evaluation: A Definition 26
3.4 Purpose / Objective of Evaluation 28
3.5 Process of Evaluation 32
3.6 Research Outputs Requiring Evaluation 39
3.7 Evaluation Metrics 39
3.8 Techniques for Capturing Values of the Metrics 51
4. Choosing the Approach for Evaluation of the SSBMO 65
4.1 The Current State of the Art 65
4.2 Overall Evaluation Research Design Goal 65
4.3 Detailed Evaluation Research Design Goals 66
4.4 Constraints on the Choices of Evaluation Research Design 66
4.5 Process for Creating Evaluation Research Design 67


5. Evaluation Research Design – Overview 68
5.1 Introduction 68
5.2 Evaluation Design Framework 68
5.3 Unit of Analysis 70
5.4 Evaluation Process Context 70
5.5 Research Outputs to Be Evaluated 71
5.6 Sources and Validity of Comparator Knowledge 72
5.7 Chosen Metrics 79
5.8 Chosen Techniques for Gathering Valid Values of the Metrics 82
5.9 Evaluation Stage Activity Groups – the Overall Evaluation Process 86
5.10 Chosen General Techniques Increasingly Likelihood of Collecting Valid
Metric Values 94
5.11 Summary of Evaluation Research Design 96
6. Evaluation Activity Group 1 – Comparative Analysis (E1) 97
6.1 Introduction to Comparative Analysis 97
6.2 Comparative Analysis Using the CATWOE (K1) Knowledge Source (E1a) 97
6.3 Comparative Analysis Using the B-Lab Impact Assessment v3 (K2)
Knowledge Source (E1b) 100
6.4 Comparative Analysis of The Timberland Company (K3) Knowledge Source
(E1c) 104
6.5 Updating Ontology Design Based on Evaluation Results 109
7. Evaluation Activity Group 2 – Third Party Review (E2) 110
7.1 Gathering Informal Feedback (E2a) 110
7.2 Introduction to Formal Expert Interviews (E2b) 112
7.3 Execution Protocol for Formal Expert Interviews (E2b-1 thru 7) 116
8. Evaluation Activity Group 3 – Case Studies (E3) 126
8.1 Introduction to Case Studies 126
8.2 Execution Protocol for Case Studies (E3) 130
8.3 Additional Objectives of Case Study Work 140
9. Updating Ontology Design Based on Evaluation Results (D4) 141
10. Concluding on Overall Research Results 141
11. Bibliography 142
12. Appendix E2: Third Party Reviewer Names (CONFIDENTIAL) 145
12.1 Informal Third Party Review (E2a) 145
12.2 Formal Third Party Review (E2b) 146
13. Appendix E3: Case Study Names (CONFIDENTIAL) 147
14. Appendix: Human Participants Research Protocol and Risk
Assessment 148
14.1 Ethics Approval 148
14.2 Informed Consent Form 149


List of Figures
Figure 1: Information Systems Research Framework 16
Figure 2: Research Framework for the SSBMO Research Project 17
Figure 3: Design Science Research Cycle 18
Figure 4: Overall Process of Inquiry for the SSBMO 19
Figure 5: Generic Steps for Evaluating a Designed Artefact 34
Figure 6: Soft Design Methodology (SDM) 37
Figure 7: Number of the 83 Metrics Mentioned By How Many of the 17 Sources 42
Figure 8: Number of the 17 Sources Mentioning How Many of the 83 Metrics 43
Figure 9: Prescription vs. Description in the Build and Use of the SSBMO 74
Figure 10: Relationship of Knowledge Sources for the Build and Evaluation of the SSBMO
(K0 thru K6) 79
Figure 11: Summary of Knowledge Sources, Techniques and Metrics used in Comparative
Analysis Evaluation Activity Group (E1) 89
Figure 12: Summary of Knowledge Sources, Techniques and Metrics used in 3rd Party
Review Evaluation Activity Group (E2) 91
Figure 13: Summary of Knowledge Sources, Techniques and Metrics used in Case Study
Evaluation Activity Group (E3) 93


List of Tables
Table 1: Guidelines for Undertaking High Quality Design Science Research 14
Table 2: Scope of the SSBMO Research Project 15
Table 3: Summary of the SSBMO Build Research Activities and Outputs (D1-4) 21
Table 4: Recommended Evaluation Steps and the Research Stage When They Should be
Undertaken 33
Table 5: Framework for Designing Design Science Research 36
Table 6: Types of Evaluation in Soft Design Science 38
Table 7: Types and Counts of Metrics Literature Consulted 41
Table 8: Metrics Designed to Give Feedback on the Context of an Evaluation 44
Table 9: Metrics Designed to Give Generic Feedback on the Utility of the Artefact Being
Evaluated 45
Table 10: Metrics Designed to Give Feedback on the Completeness Aspect of the Utility of
the Artefact Being Evaluated 46
Table 11: Metrics Designed to Give Feedback on the Quality Aspect of the Utility of the
Artefact Being Evaluated 47
Table 12: Metrics Designed to Give Feedback on the Beauty Aspect of the Utility of the
Artefact Being Evaluated 48
Table 13: Definitions of Evaluation Metrics 50
Table 14: Characteristics of Evaluation Techniques and Metric Validity 52
Table 15: Summary of Evaluation Techniques 56
Table 16: Expectation / Desirability Matrix 58
Table 17: Summary of Artificial Intelligence Ontology Evaluation Techniques / Metrics 59
Table 18: Evaluation Technique Groupings 60
Table 19: Techniques Employed to Evaluate BMO 61
Table 20: Techniques and Metrics Employed to Evaluation Innovation Ontology 63
Table 21: Techniques Employed to Gather Valid Values of Certain Metrics 64
Table 22: SSBMO Evaluation Design Framework 69
Table 23: SSBMO Expectation / Desirability Matrix 71
Table 24: Source of Comparator Knowledge for the Evaluation of the SSBMO 78
Table 25: Evaluation Metrics for each SSBMO Research Objective and Research Output
Component 80
Table 26: Definition of Selected Metrics for Evaluation of SSBMO Utility 82
Table 27: Evaluation Techniques, Required Comparator Knowledge Sources and Design
Artefacts for each SSBMO Research Output Component 85
Table 28: SSBMO Evaluation Activity Groups 87
Table 29: Identification of Risks and Application of Mitigation Techniques in All Evaluation
Activities 95
Table 30: Summary of Evaluation Research Design 96
Table 31: Metrics Whose Values Are Being Gathered by E1a CATWOE Comparative
Analysis Evaluation Activity 98
Table 32: Evaluation Activity Sub-Group E1a – Questions and Metrics 100


Table 33: Metrics Whose Values Are Being Gathered by E1b B-Labs Impact Assessment
Survey v3 Comparative Analysis Evaluation Activity 102
Table 34: Scoring Scheme for Comparison of B-Labs Impact Assessment Survey (K2), to
Osterwalder’s BMO (K0-PF) and the SSBMO (K0-SS) 103
Table 35: Scheme for Interpreting Values of Metrics from Scores in Evaluation Activity Sub-
Group E1b 104
Table 36: Metrics Whose Values Are Being Gathered by E1c The Timberland Company
Comparative Analysis Evaluation Activity 106
Table 37: Metric Values to Result from Comparing CATWOE (K1) Knowledge Source and
The Timberland Company Business Model Described Using SSBMO (K3-BM) 107
Table 38: Metric Values to Result from Comparing Public Knowledge of The Timberland
Company (K3) and the SSBMO constructs and model (K0-SS, K0-PF) 108
rd
Table 39: Metrics Whose Values Are Being Gathered by E2a Informal 3 Party Expert
Review Evaluation Activity 111
Table 40: Comparison Knowledge Source and Design Artefact to be used in Evaluation
Activity E2b Formal Expert Interviews 114
Table 41: Metrics Whose Values Are Being Gathered by E2b Formal Expert Interviews
Evaluation Activity (Overall) 116
Table 42: Items to Validate Demographics of Interviewees (E2b) 119
Evaluation Activity (Fit of K4 to SSBMO) 120
Table 44: Items to Assess Fit of Expert Knowledge of Business Models (K4) to SSBMO
(K0-PF, K0-SS) 121
Evaluation Activity (Fit of K5 to SSBMO Example Instantiation K4-BM) 122
Table 46: Assess Expert Knowledge of Operating Firms (K5) to SSBMO Example
Instantiation (K3-BM) 125
Table 47: Items to Close Expert Feedback Interviews 125
Table 48: Metrics Whose Values Are Being Gathered by E3 Case Studies Evaluation
Activity 130
Table 49: Metric Values Resulting from Comparing CATWOE (K1) Knowledge Source and
Case Business Model Described Using SSBMO (K6-BM1 thru 3) 132
Table 50: Items to Gather Demographics of Case Firm Employees (E3) 135
Table 51: Items to Gather Case Information (K6-E1 thru 3) 136
Table 52: Metric Values Resulting from Comparing Case Employee Knowledge Source (K6-
E1 thru 3) and Case Business Model Described Using SSBMO (K6-BM1 thru 3) 137
Table 53: Items to Assess Fit of Case Firm Employee Knowledge (K6-E1 thru 3) to Specific
Example of SSBMO Instantiation (K6-BM1 thru 3) 139
Table 54: Items to Close Case Firm Employee Feedback Interviews 140
Table 55: Informal Third Party Reviewer List and Demographics (Confidential) 145
Table 56: Formal Third Party Reviewer List and Demographics (Confidential) 146
Table 57: Case Firm List and Employee Demographics (Confidential) 147


Abbreviations
AI Artificial Intelligence
BMO Osterwalder Business Model Ontology (Osterwalder, 2004)
MIS Management Information Systems
SDM Soft Design Methdology
SSBMO Strongly Sustainable Business Model Ontology. The output from this
design science research project
SSM Soft Systems Methodology or Soft Systems Method


Research Design for the Evaluation of the Strongly
Sustainable Business Model Ontology

1. Introduction

1.1 Research Purpose
The working title1 of this thesis is

An Outline of a Normative Ontology for Strongly Sustainable Business Models: An
Exploration of a Proposition Using a Design Science Approach including a Comparative
Case Study of Firms Seeking to Improve Their Sustainability

The business problem this design science research project is looking to solve is:

Increasing the quality (reliability, consistency, effectiveness) of strongly sustainable business
models and the efficiency of business model designers who create them.2

This project is seeking to solve a problem in the environment at large by creating something
useful. Hence this is an applied research project.

1
A possible improved title is: Towards an Ontology for Strongly Sustainable Business Models: A Design Science
Exploration
2
Osterwalder noted in a recent presentation that “in entrepreneurship [unlike in car design] we still rely on real-life
crash tests [through the creation of news firms with new business models] which leads to costly failures” This
means the sustainability of those businesses is low (even in conventional profit-first terms), hence risk is high for
business model designers and the stakeholders of the firms instantiating those business models. Overall the design
process is inefficient and ineffective in the use of existing knowledge of how to design better business models and
the communication of the design. As a result the failure rate of new businesses is high (Osterwalder, 2011b,slide 19
[minute 3.00-3.30]).
Aligned with this, Bullinger, in her review of the design science literature, states “the value of an information system
design theory [and implicitly instantiations of artefacts using that theory] lies in the reduction of uncertainty by
limiting the system features and development activities to a manageable set. Thus, reliability of development as
well as likelihood of success could be increased […]” (Bullinger, 2008, p.222).
This also aligned well with my own experience from consulting projects over the past 20 years.


The purpose of this research3 is:

To explore, using design4 and systems methodologies, whether a pragmatic descriptive tool
can be built to improve the application of the science of design5 to strongly sustainable
business models.

Hence, my overall research question is:

Is it possible to design a useful normative ontology of a business model that can be used to
describe a firm’s strongly sustainable business model design?

As examples, this includes answering questions such as:
• How difficult is it for a business model designer to describe a strongly sustainable
business model using Osterwalder’s BMO?
• What is a list of constructs and relationships required to describe strongly sustainable
business models that are missing, incomplete and surplus in Osterwalder’s BMO?
• Is it easier, for business model designers who, based on their world-views, have different
conceptions of success (from maximizing short term monetary profit to balancing
achieving defined environmental, social and economic objectives), to use the SSBMO to
describe a their chosen business’s business model?
See sections 6 thru 8 for more questions this research will attempt to answer.

1.2 Document Purpose
This document builds on the Research proposals submitted earlier6. Its primary function is to
describe in detail the research design I propose to execute during the evaluation phase of this
design science research project.
My extensive review of the relevant literature strongly suggests that the ability of a researcher to
undertake the evaluation stage of a design science research project rigorously and achieve a high
quality of results is strongly dependent on the level of preparation of the researcher. i.e. does the
researcher consciously understand and can justify, based on legitimate precedent:
• Their world-view / biases,
• The objectives of the research

3
Version 4.1. of my research proposal suggested the purpose of this research was two fold:
1. How a normative business model ontology can describe instances of firms’ strongly sustainable business
models, and
2. What are the perceptions and reactions of managers, in firms’ attempting various levels of attempt to
improve their sustainability, to the validity and utility of the ontology
Is this new statement of purpose an improvement?
4
i.e. Design as a scientific research method
5
i.e. The (art, craft and) science of how to do (good) design
6
Version 3.2 of my Research Proposal was submitted following my MES II-III exam (June 8, 2011) and Version 4.1
was submitted following extensive discussions with David Johnston (August 8, 2011) and was subsequently
reviewed by Martin Bunch and Rod MacRea (September 2011)


• The objectives evaluation stage
• The process to be used to undertake the evaluation of the designed artefact
• The rationale for this process
• How the results of the evaluation will be used to meet the objectives of the research

1.3 Commentary and Possible Contribution
Before describing the structure of this document I find it necessary to comment on both the on
process of creating this document and the nature of this document.
Uncovering and analysing the literature has turned out to be a time consuming process since the
relevant literature comes from multiple fields. Further it has been come clear the literature on
how to best undertake design science evaluations is far from comprehensive and far from
prescriptive (and perhaps can never be).
The work on this documented started in September 2011 and consumed the majority of my time
until mid/late December 2011.
As this work proceeded I made three observations:
1. I had not encountered in the works reviewed as complete an analysis of the literature
related to ontology engineering, design science and systems thinking evaluation theory
and practice.
2. My overall research design appears to be novel in several respects.
My research design is based on my understanding of the nature of the research topic
(sustainability of human organizations) and hence the attributes of the research
methodology required to generate legitimate knowledge. This has led me to integrate
ontology engineering, design science and systems thinking into a single research design7.
3. It is intellectually challenging (and hence enjoyable) to construct a rigorous and high
quality evaluation process.
This leads me to ask: Is there a contribution to the design science field based which can be
derived from my:
1. Analysis and integration of the ontology engineering, design science and soft systems
methodology (SSM) literature on evaluation (section 3 of this document – ~40 pages,
plus several analysis working documents and spreadsheets)?
2. The process by which I constructed the design of the evaluation stage of this project?
(section 4 and portions of section 5 - ~10 pages)
3. The design of the research for undertaking the evaluation stage of this project? (sections 5
thru 8 - ~60 pages)
Feedback on the potential for a publishable contribution is requested.

7
See the most recent version of the presentation “Design science, systems thinking and the creation of ontologies” ,
for details. These details will be included in the final thesis. Presentation included with this document.


1.4 Document Structure
I will begin by presenting an updated overview of my overall research design, based on
additional insights gained since August 2011 (see Section 2 – Research Design – Overview).
Section 2 will be used, along with my recent presentation “Design science, systems thinking and
the creation of ontologies” (included with this document) to prepare the final research design
section of my thesis.
Next I will review the literature related to the process, metrics and techniques for the evaluation
design science research outputs (such as ontologies) (Section 3 – Evaluation in Ontology Design
Science Research). This then serves as a basis for the rest of the document, as follows:
• Section 4 – Choosing the Approach for Evaluation of the SSBMO describes how I chose
(how I designed) the evaluation approach based on the literature reviewed in section 3.
• Section 5 thru 8 describes the details of the evaluation research design as follows:
– Section 5 provides an overview, including identifying the unit of analysis, describing
the overall process and discussing issues of research quality.
– Section 6 thru 8 describe the three evaluation activities selected: comparative
analysis, third-party review and case studies.
• Section 9 and 10 conclude by providing a link to the research tasks which remain once
the evaluation is completed.


2. Research Design – Overview

2.1 Introduction
This section presents an updated overview of my overall research design, based on additional
insights gained since August 2011. This material will be used, along with the earlier documents,
to prepare the final research design section of my thesis.

2.2 Design Science – A Definition
Using design as a formal framework for conducting research is relatively new, and its use within
the social sciences and management sciences, such as information systems, is probably less than
25 years old8.
Hevner et. al. published what is now considered to be a seminal article9 about design science
research in the information systems field in MIS Quarterly in 2004. Based on their summary of
the field these authors present the following definitions:
Design science […] creates and evaluates […] artefacts intended to solve indentified
organizational problems (Hevner et al., 2004, p.77)
Design is both a process (set of activities) and a product (artefact) – a verb and a noun.
It describes the world as acted upon (processes) and the world as sensed (artefacts). This
Platonic view of design supports a problem solving paradigm that continuously shifts
perspective between design processes and designed artefacts for the same complex
problem. The design process is a sequence of expert activities that produces an
innovative product (i.e., the designed artefact). The evaluation of the artefact then
provides feedback information and a better understanding of the problem in order to
improve both the quality of the product and the design process. This build-and-evaluate
loop is typically iterated a number of times before the final design artefact is generated.
During this creative process, the design-science researcher must be cognizant of evolving
both the design process and the design artefact as part of the research. (Hevner et al.,
2004, p.78)
Hevner et. al. were writing the above from the perspective of the MIS management science sub-
discipline. Writing from the Innovation management science sub-discipline, Bullinger states, in
her 2008 design science PhD10 that developed an ontology for management of the innovation
process in small and medium businesses, that:
“Design science researches strive to solve problems by an action-oriented approach, in
order to find an viable artefact”, i.e. a solution to a problem (Bullinger, 2008, p.216)

8
Final thesis will include appropriate citations for this statement.
9
Google Scholar reports 2612 citations as of November 24, 2011
10
Bullinger’s PhD thesis has been published as a monograph without change. See bibliographic entry.


2.3 The Design of the SSBMO Research Project
As a result of newness of this approach to research Al-Debei noted in his 2010 design science
PhD11, which developed an ontology for designing innovative mobile data services, that “the
scheme to construct design artefacts in information systems design-science research is still very
broad” (Al-debei, 2010, p.35).
From a disciplinary perspective, following Bullinger, Al-Debei and Osterwalder (Osterwalder,
2004), I consider ontology, such as the SSBMO, to be primarily an Information Systems artefact.
Citing others, Hevner et. al. describe the creation of an artefact to solve a problem in a specific
context as an experiment “posing a question to nature”… “existing knowledge is used where
appropriate; however, often the requisite knowledge is nonexistent”, hence “reliance of creativity
and trial-and-error search are characteristic of such research efforts”, i.e. abduction (Hevner et
al., 2004, p.81).
This makes it challenging to design this type of research in detail before undertaking the research
activities, but no less important than as for natural science. Much can be learned from the
comparison of planned activities vs. activities deemed required by the researcher in the moment.
To provide assistance to researchers Hevner et. al. have proposed guidelines for designing
information systems design science research projects (Hevner et al., 2004, pp.82-90).
Researchers such as Al-Debei and Bullinger, used these guidelines to ensure the appropriateness
of their overall approach of their work (Al-debei, 2010, p.42; SeeBullinger, 2008, p.225 &
p.232)12.
The following table presents Hevner et. al.’s guidelines (first two columns) and describes how
my research design is applying each (third column).
Application of Guidelines to SSBMO Research
Guideline Description
Design
1. Design as Design-science research must produce an Resulting artefacts are the ontology for strongly
an Artefact innovative purposeful (viable) artefact in sustainable business model design. The ontology
the form of a construct, a model, a consists of constructs related in a model.
method, and/or an instantiation. Instantiations of the ontology are created to aid in
the evaluation of the validity and utility of the
constructs and model.
2. Problem The objective of design-science research The business problem addressed by the solution is
Relevance is to develop an innovative purposeful increasing the quality (reliability, consistency,
artefact for a specified problem domain effectiveness) of strongly sustainable business
models and the efficiency of business model
designers who create them.
This has the additional benefit of reducing the risks
to business model designers and users.

11
Al-Debei lists 13 works published or pending based on his PhD, including scholarly journal articles, conference
papers and book chapters.
12
Osterwalder was working on his PhD before these guidelines had been published.


Application of Guidelines to SSBMO Research
Guideline Description
Design
3. Design Because the artefact is purposeful, it must Evaluation takes place using a set of metrics
Evaluation yield utility (including quality, and gathered via a number of techniques. See sections 3
efficacy) for the specified problem, i.e. a thru 8 of this document.
design artefact must be rigorously
demonstrated via well-executed
evaluation methods.
4. Research Since the artefact must be innovative, Contributions are expected from
Contributions novelty is crucial (solving a heretofore 1. The capture of novel key concepts and their
unsolved problem, or solving a known relationships that organization’s should consider
problem in a more effective or efficient when attempting to be strongly sustainable
manner), effective design-science 2. A novel tool which practitioners can use to more
research must provide clear and verifiable efficiently and effectively design organizations
contributions in the areas of the design strongly sustainable business models.
artefact, design foundations, and/or
design methodologies.
5. Research The artefact itself must be rigorously Guidelines from the field of Design and Information
Rigor defined, formally represented, coherent, Sciences are followed during the build and
and internally consistent. Design-science evaluation of the ontology, e.g. the Ontology is
research relies upon the application of formally represented using the Entity Relationship
rigorous methods in terms of construction Modelling formalism.
and evaluation of the artefact.
6. Design as a The process by which it is created, and Specific guidelines for ontology design and the
Search often the artefact itself, incorporates or general science of design are applied for the first
Process enables a search process whereby a time to the domain of strongly sustainable business
problem space is constructed and a models.
mechanism posed or enacted to find an
effective solution. The search for an
effective artefact requires utilizing
available means to reach desired ends and
satisfy laws in the environment.
7. Design-science research must be Results of research are presented to industrial
Communicati presented effectively both to technology- partners as well as to the research community.
on of oriented as well as management-oriented This is helped by one of the overarching purposes
Research audiences. of an ontology – establishing a shared language to
support understanding and problem solving

Table 1: Guidelines for Undertaking High Quality Design Science Research
Derived from (Hevner et al., 2004, pp.82-90)

2.3.1 Inclusion of Systems Thinking in the Research Design
The basis for the design of this research project is the research framework (process, methods,
techniques, tools) adapted from design science to the strongly sustainable business model


ontology domain. However, I have also chosen to include significant systems thinking elements
in my research design.
The inclusion of systems thinking elements is an extension to the “pure” design science ontology
build/evaluate approach adopted by Osterwalder, Bullinger and Al-Debei. This is required
because of:
1. The inherently holonic nature of the domain a strongly sustainable business model is
attempting to describe and
2. Because I believe that you can’t effectively research systems using linear non-systemic
methods. 13

2.4 The Scope of the SSBMO Research Project
The following table has been adapted from the important14 article by March and Smith (March &
Smith, 1995, p.255) and updated to include an additional design output commonly accepted by
design science researchers (e.g. Vaishnavi & Kuechler, 2009, p.6).
Design Science Research Activities
D. E. T. J.
Build Evaluate Theorize Justify
Output (Develop / (Validate)
Elements Design)
1. Constructs
Design Science Research

2. Models
Output

3. Instantiations

4. Method 15

5. Better Theories

Table 2: Scope of the SSBMO Research Project

The areas of the table shaded in green is the scope of my thesis (boxes D1-4 and boxes E1-3),
based on what might be practically accomplished within the scope of a masters thesis, given I am
anchoring my ontology upon Osterwalder’s existing ontology.

13
See Version 4.1 of my research proposal and the recent presentation “Design science, systems thinking and the
creation of ontologies”. These two working documents will be used to justify and describe my integration of
systems thinking elements into my research design. This will be included in my thesis.
14
Google Scholar report 1122 citations as of November 24, 2011
15
As noted in sub-section 1.3, working on this document, reviewing the design science, ontology engineering and
systems literature has highlighted that the approach documented here for ontology evaluation may have some novel
aspects, and may also hence be a contribution.


2.5 The Framework for Conducting Design Science Research
In the same article in which Hevner et. al. provide guidelines for designing information systems
design science research, the authors also provide a process perspective on the information
systems research. They refer to this process perspective as the “Information Systems Research
Framework”, and use to illustrate how information systems research may include both the
descriptive and design science approaches.

Environment Relevance Research Rigor Knowledge Base
People Foundations (What)
• Roles 2a Develop / Build • Theories
• Capabilities • Theories • Frameworks
• Characteristics • Artefacts • Instruments
• Constructs
Organizations • Models
• Strategies assess • Methods
Iterative

refine
• Structure & Culture 1b. Applicable • Instantiations
• Processes 1a. Problem Design Knowledge • Etc.
Process
Technology Methodologies (How)
• Infrastructure 2b. Justify / Evaluate • Data Analysis Techniques
• Applications • Analytical • Formalisms
• Communications • Case Study • Measures
Architecture • Experimental • Validation Criteria
• Development Capabilities • Field Study • Etc.
• Simulation

3a. Application 3b. Additions to
to solve problem knowledge base

Figure 1: Information Systems Research Framework
(Hevner et al., 2004, Figure 2 p.80)

The sequence in which the research process unfolds is 1a. defining the problem, 1b. determine
the applicable knowledge, and then iteratively (possibly in one project, or over multiple projects
over time), 2a. develop / build theories or artefacts, and 2b. justify the theories or evaluate the
artefacts. This is then followed by 3a. application of the research output to solve problems in the
environment and/or 3b. additions to the knowledge base.
Hevner et. al. note that the knowledge base from which applicable knowledge is drawn comes
from many “reference disciplines” which “provide foundational theories, frameworks,
instruments, constructs, models, methods, and instantiations used in the develop/build phase of a
research study, and methodologies to provide guidelines in the justify/evaluate phase”. They go
on to state that “rigor is achieved by appropriately applying existing foundations and
methodologies” to each research acitvity (Hevner et al., 2004, p.76).


Al-Debei, summarizing the advice from several scholars, notes that such foundational theories
are normally originated outside the information systems field [and that] such theories could be
useful as they may suggest helpful approaches to information design problems” (Al-debei, 2010,
p.35).

2.6 The Framework for Conducting the SSBMO Research
To help set the overall framework for my research design, I have followed Bullinger’s lead
(Bullinger, 2008, p.231), and adapted Hevner et. al.’s framework to this research project.

Environment Relevance Research Rigor Knowledge Base
People Philosophical
• Executives, Entrepreneurs, • Critical pragmatism
D. Build
Investors, Business
• Strongly Epistemological
Architects, Consultants Sustainable • Systems
Business Model • Information
Organizations Ontology
artefact: • Design
• Strategy, operations and
innovation planning and 1. Constructs Disciplinary Frames*
2. Model
decision making groups 3. Method • Natural science
4. Instantiation P2. Applicable • Ecological: sociology,
Technology P1. Problem economics & management
Knowledge
• Communication support • Organization (Innovation,
assess

Iterative
refine

• Generative (Abduction) Strategy, OM/IS)
support
Quality (reliability, Design
consistency, Methods
• Evaluative (Decision effectiveness) Process (D1-4)
• Data collection, analysis
Making) support and efficiency of
creation of strongly E. Evaluate design and evaluation
sustainable E1: Comparative techniques
business models
E2: Third-Party Tools / Techniques /
E3: Case Study Formalisms
• Literature Review
• Entity Relationship
Modelling
• Interviews

C1. Application C2. Additions to
to solve problem knowledge base

Figure 2: Research Framework for the SSBMO Research Project


Using Figure 2, the conceptual sequence of my research project is as follows:
1. The problem is understood (P1) and the applicable knowledge identified (P2) (These
activities will be collectively referred to as “Preparation”).
2. Iteratively Build (D1-4) and Evaluate (E1-3) the SSBMO artefact. On occasion this will
include iterations of preparation activities.
3. Communication the results of the research (C1, C2) (These activities will be collectively
referred to as “Communication”).

2.7 The Research Cycle for Conducting Design Science Research
Kuechler and Vaishnavi provide the following “research cycle” which suggests how this
framework can be turned into a process for inquiry (aka a task oriented project plan), showing
prototypical, but more specific tasks for the researcher to undertake (Kuechler & Vaishnavi,
2008, p.493)

Figure 3: Design Science Research Cycle
(Kuechler & Vaishnavi, 2008)

The process steps (activities) in the research cycle aligns well with the overall research process
described by Hevner et. al. in their framework. Specifically, the Prepare (P) activities maps to
“awareness of problem”, the Build” (D) maps to “suggestion and development, Evaluate (E)
maps to “evaluation, and Communicate (C) maps to “conclusion”.


2.8 Process of Inquiry for Conducting the SSBMO Research
Other recent ontology development research projects have successfully used this “research
cycle” to develop their own research project plans (Al-debei, 2010, p.21). Other researchers
have used research cycles from the ontology engineering field which are highly similar
(Bullinger, 2008,"1.1 Ontology Engineering" pp.199-215, p.207, p.232)16.
Inspired by other researchers, and using both Hevner et. al.’s research framework and Kuechler
and Vaishnavi’s research cycle, I developed my overall process of inquiry / project plan,
integrating design and systems thinking17, for this research project. See figure below.

P p
P. r a Literature Review
e r
- e

B
D1: First D2: Second D3: Third Iteration of Build D4: Forth
u Iteration of Iteration of Iteration of
D. i Build 1 Build 2 3 Build 4
l
d

E1: Comparative E2b: 3rd Party
E Analysis Review: Expert
v
a
Interviews
E. l
u
a E2a: 3rd Party E3: Formal
t
e
Review: Informal Evaluation:
Events Case Study

Design Working Papers #1..n
C Write-up Lit. Review, Finalize
o
m Design, E1, E2, E3 & Write-up:
m Original Revised Research Design Design &
u Proposal Proposal Case Study
C. n
i
c Research Logs and Reflection Diary / Logs
a
t Possible Articles for Publication &
e
Other Communication

20 31
20 27

28
16

30
12
11

12
ay
ay

y

ov

b
ly

Fe
l
Ju
M

M
Ju

N

Today: Develop Detailed Evaluation
Research Design

Figure 4: Overall Process of Inquiry for the SSBMO

16
Osterwalder had completed his PhD before this research cycle had been developed.
17
This aspect was introduced in Version 4.1 of my Research Proposal (August 8, 2011), and has been subsequently
elaborated in my recent presentation “Design science, systems thinking and the creation of ontologies”. These
materials will be included in the final methodology section of my thesis.


The process enquiry consists of four related “swim lanes” of activities. These swim lanes
correspond to the conceptual sequence of activities from the research framework Prepare (P),
Build (D), Evaluation (E) and Communicate (C).
The details and justification for the activities in the in the Prepare, Build and Communicate swim
lanes will be described in the final thesis document. Describing and justify the details of the
activities in the Evaluation swim lane is the purpose of this document.
Within this swim lane structure the practical sequence of my research project is as follows:
1. The original project proposal is written and approved (MES II-III exam May 27, 2011)
2. The templates to capture my research logs, reflections, and research diary are established
and start to be used.
3. To understand the problem and the applicable knowledge, the “key theoretical frames”
(K0), which will be required to build and evaluate the SSBMO the literature review work
is started.
4. Using the initial output from the literature review a first version of the SSBMO artefact
(constructs and model) is built (D1). The build uses an iterative systems thinking
approach examining the function, structure, process and context of business models18.
5. Details of the SSBMO artefact are captured in the initial version of the “Design Working
Papers”.
6. Based on learning from the first version of the SSBMO further literature review work is
undertaken and a second version of the SSBMO artefact is built (D2) and described in the
Design Working Papers. Again an iterative systems thinking approach is applied to the
build activity.
7. Based on learning from the second version of the SSBMO further literature review work
is undertaken and an initial third version of the SSBMO artefact is built (D3 is started) and
described in the Design Working Papers. Again an iterative systems thinking approach is
applied to the build activity.
8. Based on the accumulated learnings a revised project proposal is prepared, reviewed and
approved (August 8, 2011).
9. The detailed research design of the evaluation activities is determined and documented
(this document).
10.The Comparative Analysis and Informal Third Party Review Evaluation activities are
undertaken (E1, E2a) and written up.
11.Based on the learning from evaluation activities E1 and E2a as well as the learning from
the initial work on the third version of the SSBMO and the third version of the SSBMO
artefact is finalized (D3 is completed) and described in the Design Working Papers.
12.Using the completed third version of the SSBMO the Formal Third Party Review and
Case Study evaluation activities are undertaken (E2b and E3) and written up.

18
See Version 4.1 of my Research Proposal for details.


13.Based on the learning from evaluation activities E2b and E3 the fourth and final version
of the SSBMO is built (D4) and described in the final project write-up.
14.The final write up of all aspects of the project is then undertaken (the thesis document),
and submitted for review and approval.
15.Activities to communicate the results of the thesis work to practitioners and academics are
then undertaken.

2.9 Summarizing the SSBMO Build Research Activities and Outputs (D1-4)
The following table summarizes all the above and describes the goals, outcomes and metrics for
the build activities of this project. This table is complemented by Table 30, sub-section 5.11
which summarizes the evaluation research design, the development of which is the topic of the
remainder of this document.
D. Build Research Activity
Output
Elements Goals Outcomes Methods
1. Constructs Identify the Ontology will contain D1-D4:Literature Review using
relevant issues descriptions of the entities, Osterwalder’s PhD as a anchor
for strongly and the contextual systems / along with numerous other
sustainable groupings important to sources of key disciplinary
business models describing strongly knowledge
sustainable business models
D1-D4: Secondary Data
2. Models Describe the Ontology will contain Gathering:
Research Output

“logic” of a descriptions of the
strongly relationships between the • Knowledge gained from
sustainable entities, systems and groups Course Work
firm’s business and hence the “logic” of the • Informal discussions
model business model with former colleagues
and students
3. Instantiations With limited The ontology (the constructs
explanation, and the model of their •Attending relevant
have a manager relationships) will be practitioner events
in a firm be able expressed diagrammatically D1-D4: “Science” of Design19
to understand and be presentable to
D1-D4: Systems thinking
the ontology managers
techniques20

Table 3: Summary of the SSBMO Build Research Activities and Outputs (D1-4)

19
i.e. There is a body of knowledge about what constitutes the good design of some thing (physical or conceptual),
and the processes, tools, techniques used to create such a design. This is the “science” of design. Contrast this to
the overall methodological approach for this thesis using the processes, tools and techniques of design to undertake
scientific research (design science).
20
See section 2.9.3 below for introduction to these techniques. More detail will be provided in final thesis.


The details behind these choices of activities were introduced in Version 4.1 of my Research
Proposal (August 8, 2011), and have subsequently been elaborated in my recent presentation
“Design science, systems thinking and the creation of ontologies”. This detail will be included
in the final methodology section of my thesis.
However, in reviewing the literature on design science artefact evaluation a number of points
relevant to both evaluation, but applicable to the build activities emerged. These points are
described in the following two sub-sections.

2.9.1 Evaluation as Part of Preparation Activities
Citing March and Smith’s 1995 work (1995, pp.260-261), Osterwalder (2004, pp.127-129)
suggests that the first level of evaluation is the strength of the researchers argument highlighting
the similarities and differences between previous work, i.e. the key theoretical frames (K0), and
the ontology’s design (constructs, models, instantiations) – i.e. evaluation actually starts during
prepare / literature review.
Brank in his 2005 review of ontology evaluation methods from an ontology engineering
perspective suggest there are four broad categories of evaluation methods, of which the third
aligns with March and Smiths recommendation “comparing the ontology to some authoritative
data source” (Brank, Grobelnik, & Mladenić, 2005).
This was also the approach explicitly taken by Al-Debei. During the initial build of his ontology
he “evaluated against the existing body of business model literature and his ontology Design
Quality Evaluation Framework” (Al-debei, 2010, p.21).
Hence, as shown in the SSBMO process of inquiry (sub-section 2.8, Figure 4), the extant
literature will be used throughout all four iterations of the build activity, to continuously evaluate
the SSBMO against “authoritative data sources”, such as the BMO and applicable literature
drawn from .natural and social sciences21, 22.
The importance to evaluation of existing research supports one of the primary reasons for
selecting Osterwalder’s BMO as the basis for the design of the SSBMO. The BMO has
considerable strength because of the evidence to support the BMO’s validity, and the apparent
lack of any directly comparable work 23.

21
Collectively I am referring to this body of applicability literature as the key theoretical frames for the SSBMO,
and labelling this as K0.
22
This body of literature is summarized in my recent presentation “Design science, systems thinking and the
creation of ontologies” and will be described in detail in my thesis literature review.)
23
Both Al-Debei and Bullinger review a range of business model (and related, e.g. enterprise) ontologies (Al-debei,
2010, pp.67-97; Bullinger, 2008, pp.133-171); Further with over 170,000 copies of the popular work derived from
his PhD sold, a iPad app and a busy speaking and consulting business, Osterwalder’s work has considerable
practitioner validation (Osterwalder, Pigneur, & Clark, 2009; Osterwalder, 2011a; Smith, Osterwalder, Business
Model Foundary, & Hortis - Le Studio, 2011)


2.9.2 Evaluation as Part of Build Activities
Hevner et. al. recommends that during the build activities “the artefact itself must be rigorously
defined [and] formally represented” (Hevner et al., 2004, p.81,table 1 p.82).
Based on his review of the literature of ontology engineering and design science Al-Debei
identifies six criteria for evaluating design artefacts (Al-debei, 2010, pp.43-46). The first
criteria, clarity, matches Hevner et. al.’s recommendation, and hence must be undertaken during
build activities.
Al-Debei, summarizing the literature on clarity, states that
An ontology needs to successfully and objectively communicate the intended meaning of
defined terms. Defined terms are concepts describing the domain, which will most likely
be nouns (i.e. objects [constructs]), or verbs (i.e. relationships [a model]). Creating a list
of these terms is important, as well as documenting their definitions in natural language
[impacts human understanding / communication]. (Al-debei, 2010, p.43)
Al-Debei goes on to provide a summary of items which lead, if they occur in the designed
ontology, to a reduction of clarity:
1. Construct overload: two or more ontological constructs map to one modelling (i.e.
grammatical) construct.
2. Construct redundancy: two or more modelling constructs map to one ontological
construct.
3. Construct excess: an existing modelling construct does not map to any existing
ontological construct.
4. Construct deficit: an existing ontological construct does not map to any existing
modelling construct. (Al-debei, 2010, p.44)
My selection of the Entity Relationship Modelling formalism, coupled with my extensive
documentation of the SSBMO in the Design Working Papers during the build activities will help
to meet these recommendations and maximize the clarity of the ontology24.

2.9.3 Systems Thinking Evaluation Techniques Relevant to the Build Activities
While I will leave the description of the systems thinking elements of the build activities until
my final thesis, I do want to describe some elements of systems thinking, related to evaluation,
which need to be considered during the build activities25. The systems thinking elements of the
evaluation activities are described in the sections below.

24
See Working Paper #1 along with SSBMO summary entity relationship diagram v1.021 (July 17, 2011) for
examples of this documentation. A summary of Working Paper #1 and the summary entity relationship diagram are
included in this Prezi Presentation. This presentation has been successfully used in evaluation activity E2a Informal
Third Party Review and a number of other occasions when I have needed to communicate the content of the
SSBMO’s design.
25
A familiarity with Soft Systems Methdology is assumed in this document. An overview to SSM will be provided
in the thesis. ( For a good summary see Jackson, 2000, pp.246-270)


Firstly, considering Baskerville et. al.’s innovative and recent work which proposed a
methodology which integrates Soft Systems Methodology (SSM) and Design Science
(Baskerville, Pries-Heje, & Venable, 2009). These authors suggest how design science
evaluation activities are highly similar to SSM comparison activities and serve the same purpose:
determining whether the designed artefact (which in the case of SSM are root definitions and a
model) are fit for purpose.
"The search for the design solutions [SSM stage 3 root definitions and stage 4 modelling,
design science build activities] and the evaluation of the design solution are activities
that take place in the abstract world of design thinking. Artefact construction and its
evaluation are activities that take place in the real world of the social systems into which
the artifact becomes situated.”
This suggests that during build activities SSM techniques used to test completeness and
adequacy of root definitions and models would be useful. One of the key techniques of SSM in
this regard is the application of CATWOE framework26.
Specifically, I will use CATWOE to help determine whether each build iteration27 is complete
(i.e. during build activities D1, D2, D3 & D4).
Secondly, considering Ledington and Ledington’s 1999 paper (Ledington & Ledington, 1999).
These authors offer a number of critiques and one suggestion for a techniques to improve the
outcomes of the SSM comparison processes (, p.336). To be effective in a design science
context these techniques would need to be employed during the build phase.
Ledington and Ledington’s insight is that it is natural for designers to (need to) have high
expectations that the artefact they are building will be highly desirable and highly important to
solving the problem at hand. Similarly, it is natural for users to (want to) have high expectations
that the artefact that is being built will be highly desirable and highly important to solving their
problem. This mental frame of reference is (implicitly) shared / co-created by both the designers
and users. It acts as a context (a bias) to all subsequent evaluation activities. Thus, these authors
suggest, it is critical in the build activities to surface this context to help the designers to consider
alternatives and the users to specify their evaluation criteria (Ledington & Ledington, 1999,
p.336). This technique is discussed in more detail in sub-section 3.8.

2.10 Summarizing the SSBMO Evaluate Research Activities and Outputs (E1-3)
The explanation and justification for the four activities identified in the “Evaluate” swim lane of
the process of inquiry is the topic of the rest of this document.
A summary table for the evaluation activities, equivalent to Table 3, sub-section 2.9 which
summarizes the build activity, can be found in Table 30, sub-section 5.11.

26
The CATWOE framework consists of technique and knowledge. The technique aids the researching in
confirming that root definitions and models include all relevant: Customers, Actors, Transformation Processes,
Weltanschauung (World-Views), Owners and Environmental Constraints.
27
See Research Proposal v4.1 p.12


3. Evaluation in Ontology, Design Science and Systems Research

3.1 Introduction
The proximity of Design Science and Ontology Engineering is shown by the common notion of
conceptualization, construct and model. “The main difference is the inclusion of solutions
[italics in the original] in Design Science which is not relevant to modeling an ontology”
(Bullinger, 2008, p.221, footnote #245). Further, both approaches explicitly require the
evaluation of an artefact.
The proximity of Design Science and Soft Systems research is shown in three key ways:
1. The common desire to produce an improvement in the world. Design Science attempts
this via the build/evaluation of an artefact. Soft Systems attempts this via planned action
based on understanding a problem, modelling the problem and potential solutions, and
comparison of these with the real world problem at hand (Baskerville et al., 2009).
2. The high degree of alignment of the notions of constructs, models and instantiations in
Design Science with the notions of root definitions, models and solution construction in
Soft Systems research (Baskerville et al., 2009).
3. The common idea of evaluation to test utility of the of constructs, models and
instantiations in design artefacts and the comparison of root definitions and model to the
“real-world” problem, and the subsequent comparison of the implemented solution with
reduction in the “real-world” problem (Baskerville et al., 2009).
These proximities strongly support the use of these three approaches, design science, ontology
engineering and soft systems, in my research28.
Although there are clear commonalities between Design Science and Ontology Engineering, and
between Design Science and Soft Systems research, it is only recently that the literature in these
three traditions has started to be integrated. However, no work was located which attempted to
integrate Ontology Engineering with Soft Systems methods.
I hope, by reviewing the body of work from these three previously largely isolated fields, to offer
the following contributions:
1. Offer a (the?) first attempt at a comprehensive review of the literature in these three
diverse fields on their approach to evaluation to provide better guidance to other
researchers in designing high quality and rigorous systems oriented design research
focused on ontology evaluation (see section 3).
2. Offer a (the?) first version, based on this literature review, of a process to help myself and
other researchers choose their evaluation research design (see section 4)

28
The explanation and justification for this assertion as far as the evaluation activities in my research is included in
this document. An introductory explanation and justification of the application of these approaches to the prepare,
build and evaluate activities has been outlined the presentation “design science, systems thinking and ontologies”
and will be fully documented in my thesis.


3. Use this review to attempt to improve the quality and rigor of my own research design
and hence the utility of the SSBMO artefact over comparable artefacts produced by other
researchers (see sections 5 thru 8).

3.2 Reviewing the Literature
With these objectives in mind this section reviews the literature related to the evaluation of:
• Ontologies (from Ontology Engineering),
• Design science artefacts (from Design Science)
• Ontologies that have been produced through design science research, and
• Root definitions and models (From Soft Systems Methodology Research).
The next section (section 4) describes how I choose (how I designed) the evaluation approach for
the SSBMO based on this review of the literature. Subsequent sections (5 thru 8) provide the
details of the chosen evaluation research design for the SSBMO.
This section is organized as follows:
1. Definitions of and broad perspectives on evaluation are reviewed
2. Views on the purposes of the evaluation process are presented
3. Views on the processes which can be used to evaluate artefacts are discussed
4. The research outputs (artefacts) which require evaluation are described
5. The metrics which have been used to measure artefacts are explained, and finally
6. The techniques (methods) used to capture values for the metrics are considered29.

3.3 Evaluation: A Definition
Cleven et. al. (Cleven et al., 2009), propose a “general framework” to the design of the
evaluation of design science artefacts. However, they helpfully begin with a review of
antecedents. This begins with a historical review of the role of evaluation within every-day life
and more specifically within design science research.
Cleven et. al. note that it is hard to define evaluation. However, synthesizing the definitions of
evaluation presented, leads me to the following working definition:

Evaluation is the process of determining the worth, merit, significance and opportunities for
the improvement of artefacts through the objective and systematic collection of information.
Evaluations are the outcomes of that process.

29
It is only through the capturing of values for the metrics that enable the researcher (and others) to judge whether
or not an artefact has met its design goals or not.


In providing additional substance to this definition Cleven et al observe that:
• Evaluation may be “quantitative or qualitative, or a mix of these” and that evaluation “is
strongly although not always sharply distinct from explanation”.
• Accomplishing evaluation is hard work due to challenges with objectivity, comparability
and tractability and the complexity inherent in resolving these challenges in a cost and
time effective manner.
• These complexities require an “adequate framework […to] support a structure proceeding
of the evaluation intentions”.
• A variety of different evaluation techniques have been developed from a range of
disciplines: Information Systems, Business Administration, Sociology, Computer
Science.
• Many researchers have “deplored the absence of appropriate evaluation methods”, and
that there has been “uncontrolled growth of new methods developed by eager
practitioners and researchers” and that many of these “are lacking a theoretical
foundation and an empirical validation of their utility”. This is particularly observed for
artefacts, such as ontologies, created within the Information Systems field.
Hevner & Chatterjee ((2010, pp.109-111) align with my working definition of evaluation and
these observations on the process of evaluation, characterizing evaluation as a “rather difficult
and complex” process, refering to the “art” of evaluation (p.111).
As noted earlier, Baskerville et. al. have proposed an integration of Soft Systems Methodology
and Design Science (Baskerville et al., 2009). In this paper these authors note how design
science evaluation activities are highly similar to SSM comparison activities serving the same
purpose, and this also observation fits well with my working definition above.
Within this idea that soft systems comparison and design science evaluation are the same,
Ledington and Ledington’s 1999 paper introduce a number of useful observations and
innovations related to the problems of comparison, which will be discussed in detail below. For
now I note that Ledington and Ledington align with the design science researchers’ concerns
noted above about the poor state of knowledge and practice about evaluation. They state
“comparison is problematic both in theory and practice” and go on to call some existing soft
systems researchers comparison practices “nonsense”, adding “clearly, knowledge about
comparison is inadequate and creates practical difficulties in both transferring the approach to
others” (Ledington & Ledington, 1999).


3.4 Purpose / Objective of Evaluation

3.4.1 Overview
Despite the concerns described above, overall the literature on the purpose of evaluation appears
clear and consistent, albeit recent. Summarizing this literature it would seem that:

The purpose of evaluation within design science research is to provide feedback on the utility
of the designed artefact which is then used to prepare a final revised artefact.

Utility is a very broad concept. The research output whose utility should be assessed and the
metrics that might measure utility will be discussed in detail in the sub-sections 3.4 and 3.5, after
discussing the process of evaluation in sub-section 3.3. However, in summary the literature
suggests that utility includes three basic ideas:
1. Completeness
2. Quality
3. Beauty

3.4.2 Supporting Summary of the Literature
The following summary of the literature is provided to support the above definition of the
purpose of evaluation in design science and ontology engineering research, and the definition of
comparison in soft systems research.

The Design Science and Ontology Engineering Perspectives
In his 2004 PhD Osterwalder observed that “of all the authors that presented different business
model frameworks only [one] has written about some kind of evaluation having [been] applied”
and this was informal, via the use of the framework in consulting work. “None of the authors
has set up any hypothesis and tested them in a field setting” (Osterwalder, 2004, p.142).
Osterwalder suggests that the purpose of evaluation is to compare the designed artefact with the
“initial goals of the research” (i.e. the problem which the design is trying to solve) (Osterwalder,
2004, p.127).
Writing at the same time as Osterwalder, Hevner et. al. (2004, p.82, table 1 p.82) agree with
Osterwalders position, stating that since the designed artefact is purposeful, it must yield utility
(including quality, and efficacy) for the specified problem, i.e. a design artefact must be
rigorously demonstrated via well-executed evaluation methods.
Evaluation is a crucial component of the research process. The business environment
establishes the requirements upon which the evaluation of the artefact is based.
A design artefact is complete and effective when it satisfies the requirements and
constraints of the problem it was meant to solve (Hevner et al., 2004, p.85).


Going further than Osterwalder, Hevner et. al also describe how the output of evaluation is used:
“the evaluation of the artefact then provides feedback information and a better understanding of
the problem in order to improve both the quality of the product and the design process” (Hevner
et al., 2004, p.78). i.e. the purpose of evaluation is the “identification of weaknesses in the […]
artefact and the need to refine and reassess” (Hevner et al., 2004, p.80).
In 2005 Ontology engineering researcher Brank et. al. concurred with this perspective:
“Ontology evaluation is the problem of assessing a given ontology from the point of view of a
particular criterion of application, typically in order to determine which of several ontologies [i.e.
competing design choices] would best suit a particular purpose” (Brank et al., 2005).
Writing four years after Osterwalder and Hevner, Bullinger (2008, p.211) observed that “at
present, the field of ontology evaluation is only emerging”. She goes on to cite Hevner et. al.
(2004) noting that “it is typical for the field that the establishment of a working solution as well
as the characterization of the environments in which this artefact works are of primary concern,
even if it cannot (yet) be explained completely why the solution works” (Bullinger, 2008, p.216).
Summarizing Osterwalder and March & Smith ((March & Smith, 1995; Osterwalder, 2004)
Bullinger states the “objective of the evaluation process is to determine the degree of correctness
of the ontology” (Bullinger, 2008, p.213) and that “evaluation means determining whether the
design artefacts produced are effective, i.e. achieve their purpose, provide value, and / or product
adverse or unwanted side-effects” (p.220) and that artefacts “should be evaluated before (re)use
by one or more applications and users” (p.211).
Concluding on her review of the design science literature Bullinger states that “objective of the
evaluation process is to determine the degree of correctness of the ontology” (p.213) and goes to
say that the ultimate evaluation of a designed artefact is its viability and utility in solving the
problem which the designed artefact was intended to solve (p.216).
More recently still, Cleven et. al. (Cleven et al., 2009), propose a framework for design science
artefact evaluation. This framework is helpful in providing a more granular view of the purpose
of evaluation. In this framework they suggest that there are four “closely interweaved”
“functions” of evaluation:
1. Acquisition of Knowledge to help put “management decisions on a rational basis”.
2. Gaining Control over the utility of the designed artefact.
3. Development, based on the learning from the knowledge and control functions, enables
the improvement of the artefact through dialog between designer and users.
4. Legitimization of the artefact based on either the process of its design (e.g. ex-anti
traceability of antecedents and processes of construction), or the fulfilment of suitable
metrics (e.g. ex-post).
A year later, citing a range of antecedents, Al-Debei states in his PhD thesis that “it is important
that ontologies are of a good quality, in order that they serve their intended purposes and be
shared as well as reused [in] different applications” (Al-debei, 2010, p.17).
Finally, in a recent book chapter focused specifically on evaluation of MIS artefacts in design
science, Hevner & Chatterjee state “the designed […] artefact is a social-technical entity that
exists within an environment (business and/or social) which lays out the requirements for its


evaluation” (Hevner & Chatterjee, 2010, p.109). Hevner & Chatterjee go on to describe three
purposes of evaluation ((pp.110-111):
1. Promotional – provide evidence to uses of the utility of an artefact to increase the
artefact’s subsequent use.
2. Scholarly – to uncover the (disciplinary) principles related to the artefact and its use.
3. Practical – to provide evidence to designers of the efficacy or efficiency of the content
of, and methods or techniques used to build and evaluate the artefact.

The Soft Systems Perspective
As noted earlier, Baskerville et. al. have proposed an integration of Soft Systems Methodology
and Design Science (Baskerville et al., 2009). In this paper these authors note how design
science evaluation activities are highly similar to soft systems methodology comparison
activities serving the same purpose, determining whether the designed artefact (which in the case
of SSM are root definitions and a model) is fit for purpose.
Overall the purpose of these comparison activities, which occurs in two places in the soft
systems method (see Figure 6, next sub-section for details), is to ensure the people trying to solve
the problem reflexively / dialectically reflect on as many aspects as possible of their problem,
their understandings of it, their proposed solutions, and the fit of the real-world to those solutions
(Jackson, 2000, p.254; Ledington & Ledington, 1999).
The intent, through this systemic approach to comparison, is to increase the likelihood, through
dialog and single and double loop learning, of uncovering solutions which dissolve the problem
for as many of the participants (stakeholders) as possible. SSM solutions ought therefore to be
significantly better aligned with more of the participants’ world views and hence needs. Such
results are believed by many to create solutions superior to those arrived at through imposition or
negotiation.

3.4.3 Implications for the SSBMO
Considering the literature above from a practical perspective, there are implications for the
analysis of any interviews used to gather feedback within a design science research project.
Specifically, the idea that the purpose of evaluation is feedback to improve the designed artefact
calls into question both the need for and purpose of a “code book, a typical qualitative research
technique, as well as the nature of the “coding” process required following any interviews used
to gather feedback.
In descriptive science research, such as those taking a grounded theory approach, the code book
and the coding process are critical in uncovering the evidence gathered from interviews to justify
a theory, and to provide a documented chain of evidence from the evidence to the theory.
However, in design science it appears that the use of the evidence gathered during interviews is
different. Instead of codes the researcher is looking for feedback (positive and negative) on all
aspects of the utility of the designed artefact. Each element of feedback is then used in the
subsequent iteration of the build activities, specifically:
• Positive feedback is recorded to support the existing design choices


• Negative feedback is analysed to identify changes to the design which would improve its
utility30.
Clearly as the number of elements of feedback increases from multiple interviews the need for
some kind of grouping to aid the researcher in the analysis of negative feedback may be required.
While these groups may superficially appear to fulfill a similar purpose to descriptive science
codes, they do not.
However as with descriptive science interviews it is important for the researcher to undertake
cross-case / meta-analysis. Not all interviewees will provide the same feedback. Some of the
feedback will likely be contradictory, based on the interviewees different world-views. Such
feedback is a rich source of insight on artefact utility for the researcher. The researcher will need
to synthesize and justify appropriate responses in the revised design to such contradictory
feedback.
In synthesising contradictory feedback the researcher needs to be aware that the underlying
world-view differences may only be resolved through the researcher undergoing “double-loop”
learning. This means in the subsequent iteration of the build activity, the researcher must not
only consider revising existing constructs and relationships in the ontology (“single-loop”
learning), but consider revising assumptions made in the selection of these elements and
choosing wholly new assumptions and elements (“double-loop” learning).
This is explored in detail in sub-sections 3.8 and applied in the evaluation activities which use
interviews for gathering feedback (Section 7, Third Party Review – E2, and section 8, Case
Studies – E3)

3.4.4 Assumptions for the SSBMO
Artificial intelligence (AI) ontology engineering makes a distinction between processes of
evaluation, validation, verification, assessment and testing (Bullinger, 2008, p.211 footnote
#240).
El Debei suggests there are two types evaluation activities: verification and validation:
1. Verification mainly refers to technical activities that ensure the syntactic correctness and
cleanness of an ontology
2. Validation refers to semantic correctness; that is the process of ensuring that an ontology
corresponds to the phenomenon that it is supposed to represent.
(Al-debei, 2010, p.61)
The differences are, according to these authors, because for AI and highly formalized
(computerized) ontologies many of these processes can be undertaken by a computer.
In this project, and this document, I will take these processes as synonyms of evaluation.

30
See the next sub-sections for a discussion of the process of how the feedback is judged positive or negative.


Research design for Evaluation of Strongly Sustainability Business Model Ontology

Research design for Evaluation of Strongly Sustainability Business Model Ontology

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à Research design for Evaluation of Strongly Sustainability Business Model Ontology

Similaire à Research design for Evaluation of Strongly Sustainability Business Model Ontology (20)

Plus de Antony Upward

Plus de Antony Upward (20)

Dernier

Dernier (20)

Research design for Evaluation of Strongly Sustainability Business Model Ontology