1. Computer-Aided Qualitative Research Europe
7 & 8 Oct 2010, Lisbon
For more information about our events, please visit:
http://www.merlien.org

2. 3rd European workshop on COMPUTER-AIDED QUALITATIVE RESEARCH 2010
Quality in Qualitative Research:
The role of the software’s in Quality
Assurance
SÍLVIA SILVA
SARA RAMOS
ISCTE-IUL

3. OVERVIEW
Main points of our presentation:
• Aims
• Background
• Literature review and results
• Quality and CAQDAS
Additionally :
- Distribution of Some Examples
- Distribution of List of Main References

4. AIM
To present a review of quality criteria in
Qualitative Research and explore and
conclude about the potential roles that
software packages may play in quality
assurance.

5. Specific goals:
(1) Identify the criteria used
(2) Identify the most common/consensual
criteria;
(3) Link quality to CAQDAS
(4) Contribute for debate and consensus (?)

6. Background
• In the last thirty years several authors
proposed criteria for considering when
approaching the issue of Quality in Qualitative
Research (QR).
• It is somehow recognized that we are far from
having shared quality assumptions

7. Background: Examples on well-known
criteria presented in books
• Guba and Lincoln (1985)
Trustworthiness :
• Transferability ;
• Credibility ;
• Dependability;
• Confirmability
• Bauer & Gaskell (2000)
Accountability:
• Confidence
• Relevance
• Flick (2007/9)
• Transparency,
• Documentation
• Writing

8. Research accountability (Gaskell & Bauer, 2000)
• Confidence (c) Triangulation and
(results represent the Reflexivity (c)
reality studied) Transparency and
• Relevance (r) procedural clarity (c)
(utility and importance) Corpus construction
(c,r)
Thick description (c,r)
Local surprise (r)
Communicative
validation (r)

9. Background: Why to focus on Quality
• The relevance of quality criteria: epistemological issues and
practical issues
• A well-written description of the rigour in research analysis
should convince readers that the study findings are credible
and trustworthy (Belgrave et al., 2002).
Moreover:
• QR Internal needs (development and proliferation)
(Flick,2007)
• QR external challenges (e.g. competition: publication;
funding; teaching and curriculum planning) (Flick,2007)

10. The big question:
One size fits all?

11. Literature Review: Search Approach (I)
• Covering last 20 years
• Databases:
- Psycharticles;
- ABI/INFORM;
- ISI
- Google academics
• Search in the Abstracts and Titles

12. Literature Review: Search Approach (II)
• Papers focusing Quality Issues/Criteria in Qualitative
Research
• Keywords used:
Quality+ Qualitative Research/Methods;
Validity+ Qualitative Research/Methods;
Rigor/our + Qualitative Research/Methods;
Trustworthiness +Qualitative Research/Methods;
all the above with Computer Programs, Software and
CAQDAS
• Results: > 100 papers ; >30 specifically focusing quality
as the main topic

13. Literature Review: Main Characteristics
• Huge progressive increase of papers about
Quality Issues in QR or QS in the last 20 years
• Journals: Either general (Methods in General;
Methods in QR); either in specific fields
(mainly: health; business; education;
communication)

14. Literature Review: Main Characteristics
Therefore:
Papers focusing QR Quality in specific research
fields VS generic criteria and
recommendations
• Papers departing from validity & reliability and
framing them in the QR VS papers against that
assumptions

15. Literature Review: Main Findings
Main types of papers focus:
• Criteria used by Journals and Referees
• Specific Criteria for specific QR
• Generic Criteria for all QR
• Techniques for assuring quality: Audit;
triangulation; …
• Quality and CAQDAS
Most cited Authors:
• Guba and Lincoln (1985)

16. LR Results – Journals and Referees
criteria
• Papers focusing specific Criteria used by
Journals and Referees to sustain Qualitative
Papers Review
Examples:
• Academy of Management Journal Editors (2002,
2004 & 2009)
• Savall et al. (2008)
• Crescentini & Mainard (2009)

17. Academy of Management Journal 2009
Editors: Tips for Writing Qualitative Papers
1. Make sure your paper includes “the basics”:
• Discuss why this research is needed
• Are you building a new theory or elaborating existing theory?
• Why did you choose this context and this “unit of analysis”?
• How did you get from your data to your findings?
2. Show data in a smart fashion
3. Think about using/organizing figures
4. Think about telling a story
5. Consider “modeling” someone whose style you like who
consistently publishes qualitative

18. 10 (generic) Criteria used by Reviewers in an European
Management Journal
Savall et al. (2008)
• Rigor
• Formulation
• Coherency
• Originality
• Relevance
• Explication
• Positioning
• Contribution
• Rationale
• Delimitation

19. LR Results – Specific Criteria
• Papers focusing specific Criteria for specific
Qualitative Approaches: epistemological
adequate criteria
• For instance: adequate to the the type of
analysis: Content Analysis (e.g., Lombard et
al., 2002); Grounded Theory (e.g., Chiovitti &
Piran, 2002; Elliot et al., 2005); Discourse
Analysis (e.g., Nixon, 2007 )

20. LR Results – Specific Criteria
• For instance:
Characterizing the Philosophy and Politics of
Quality in QR. Distinguishing: review of
quality indicators attached to
- Foundational;
- Quasi-Foundational;
- Non-Foundational QR
(Amis & Silk, 2008 )

21. LR Results - Generic Criteria or
Recomendations I
• Papers focusing specific Criteria for
generic/transversal Qualitative Approaches:
criteria that applies to all approaches

22. LR Results - Generic Criteria or
Recomendations II
Authors Criteria
Akkerman et al. •Visibility •AUDITING
(2008) •Compreehensibility
•Acceptability
Morrow (2005) •Social validity Guidelines for Writing QR
•Subjectivity and Reflexivity
•Adequacy of data
•Adequacy of interpretation
Shank & •Investigative depth
Villela(2004) •Interpretative adequacy
•Illuminative fertility
•Participatory accountability
Whitemore et al., •Primary Criteria (for all QR) Primary criteria: Credibility;
(2001) •Secondary Criteria Authenticity; Integrity and
•Tecnhiques Criticality
Techniques: Design
consideration; Data generating;
Analytic; Presentation

23. Meyrick (2008): Good Qualitative
Research

24. Rolfe (2006)
“the commonly perceived quantitative–qualitative
dichotomy is in fact a continuum which requires a
continuum of quality criteria, or to recognize that
each study is individual and unique, and that the
task of producing frameworks and predetermined
criteria for assessing the quality of research studies
is futile.”
“individual judgments of individual studies”

25. Quality Objects
• Theory, Method and • Adequate
Epistemological methodological
Coherence approach considering
the analytic grounding
• Design and Report of
the Research General:
• Data Collection • Transparency
• Data Analysis • Reflexivity

26. LR Results – Quality and CAQDAS
• Papers focusing the role of CAQDAS on
assuring Rigor and Quality

27. LR Results – Quality and CAQDAS
Authors Focus
Lu & Shulman (2008) CAQDAS and Rigour and Flexibility and the use of CAT (Coding
Analysis Toolkit)
Rambaree (2007) Rigour in Qualitative Social Research: The Use of a CAQDAS
Atlas.ti example
Sin (2007) CAQDAS for achieving Transparency
Illustration with NVivo
Sinkovics et al. (2008) CAQDAS for achieving trustworthiness of QR in Business
Research
Westphal (2000) N4 and Trustworthiness:
Searching for negative evidence – Easier to Find Inconsistency
Linking Data and conclusions and theory
Conducting Coding Checks
Audit Trails and Conducting Audits
Detailed reporting

28. What Software Do (Gibbs, Lewins & Silver, 2005;
Lewins & Silver,2009)
• Structure work (access and organization of all project elements)
• 'Closeness to data' interactivity (quick access to source data
files)
• Explore data
• Code and Retrieve Functionality
• Project Management and Data Organisation
• Search and interrogating the database
• Writing tools - Memos, comments and annotations,
• Output - Reports to view a hard copy or export to another
package.

29. Advantages (Gibbs, Lewins & Silver, 2005; Lewins &
Silver,2009)
• Organised and controllable data set
• Support for coding
• Searching Text and codes
• Support for comparative analysis
• Models, networks and diagrams
• Interface with quantitative data

30. Qualitative Softwares – Possible
contributions for quality (I)
•Data and Coding:
More Easy to Be sure that all data are analysed (assure that
participants perspectives are covered) and code and retrieve
approach more complete and rigorous
Reporting:
More adequate support for reporting results and allowing
simple and complex results

31. Qualitative Softwares – Possible
contributions for quality (II)
•Good support for assisting a systematic analysis approach:
allows and simplifies the use of “equivalent” procedures to all
the data
•TRANSPARENCE and Reflexivity :
Allowing a more easy access to the data analysis (e.g., Codes;
Quotations; Memos); methodological analytic decisions, and
reflexions (Memos, Comments)
•Facilitation of TRIANGULATION and AUDIT:
Simplifies the combination of different sources of data (or
other type of triangulation); team work; …

32. Qualitative Softwares – Possible
contributions for quality (III)
Besides:
allows the researchers in assuring quality when
following most of the specific recommendations
But:
it can also be misused and give “overconfidence”

33. Conclusions (I)
• There is not a single criteria solution adequate
for all “one best way”
• Quality issues cover both theoretical issues
and technical problems
• There is an ethical obligation to demonstrate
“Rigor” and Integrity of research
• Quality Reflexion is needed but we are still far
from consensus: We suggest some debate on
this

34. Conclusions (II)
Anyway:
• Debate and establishing “criteria” about
Quality of QR contributes for a bigger
awareness of methodological decisions during
QR (Seale, 1999)
• CAQDAS may play an important role but it
always depend on it is used

35. Conclusions (III)
Myth 94:
qualitative researchers will agree
about
validity
Sparkes (2001)

36. Our Future work on this subject
• First Paper in Preparation
• Project about the Quality of Qualitative
Research and CAQDAS(without funding at the
moment) in the beginning; looking forward
possible cooperation/network for proposal to
be submitted to the Portuguese Science
Foundation

37. References
A list with the Main References will be
distributed

38. Computer-Aided Qualitative Research Europe
7 & 8 Oct 2010, Lisbon
For more information about our events, please visit:
http://www.merlien.org