Financial auditors routinely search internal as well as public knowledge bases as part of the auditing process. Efficient search strategies are crucial for knowledge workers in general and for auditors in particular. Modern search technology quickly evolves; and features beyond keyword search like facetted search or visual overview of knowledge bases like graph visualisations emerge. It is therefore desirable for auditors to learn about new innovations and to explore and experiment with such technologies. In this paper, we present a reflection intervention concept that intends to nudge auditors to reflect on their search behaviour and to trigger informal learning in terms of by trying out new or less frequently used search features. The reflection intervention concept has been tested in a focus group with six auditors using a mockup. Foremost, the discussion centred on the timing of reflection interventions and how to raise mo-tivation to achieve a change in search behaviour.

1. www.moving-project.eu
TraininG towards a society of data-saVvy inforMation
prOfessionals to enable open leadership INnovation
Angela Fessl, Viktoria Pammer, Michael Wiese & Stefan Thalmann
Improving Search Strategies of Auditors – A Focus
Group on Reflection Interventions
ARTEL 2017, EC-TEL 2017 Tallin

2. www.moving-project.eu
3 of x
Introduction
• Finding relevant information is necessary for most professions
• Setting: auditors performing audits
• Finding the best and up-to-date information is very crucial
• Auditors use features and platforms to find relevant changes in law and
regulation
• Outdated information can have serious negative effects
• Key Assumption: Auditors benefit from reflective search and
data analysis practice
• Reflective practice as “best practice” to knowledge work
• Reflective practice as highly relevant in the face of new and changing
technologies in the workplace

3. www.moving-project.eu
4 of x
• Challenge 1: Embed reflection in the work process
• Reflection interventions should not disrupt the users‘ workflow
• Reflection interventions need to be well connected to work
• Challenge 2: Change routinized search behaviour
• Search behaviour is routinized and „works“
• Search strategies used might not be optimal
Research Goal

4. www.moving-project.eu
5 of x
Research Method
User-Centered Design
• Informal discussions with key stakeholders
• Reflection Intervention – Prototype Mock-Up
• Focus Group with target end users
• Analysis
• Implications for Design
• Finetuning of Research Question

5. www.moving-project.eu
6 of x
Reflection Intervention
• Visualise feature usage
• Presents the last five used features
• Goal: raise awareness of feature usage and
search behaviour
• Compare
• Shows what features others use mostly /
search strategies of others
• Goal: Be inspired by search strategies of
others
• Guidance:
• Asks reflective questions related to
information search and use (reflection
prompts)
• Points to un-used features
• Goal: reflect on and improve search
behaviour

6. www.moving-project.eu
7 of x
Focus Group
• Company A: Financial auditors operating in 140 countries with
145.000 professionals
• Participants: Six auditors of the innovation group
• Focus Group – Impulses
• Comparison: What is it you would like to know about how others search?
• Different reflection prompts
• Timing: When (at what time) and where (on which device) to present them

7. www.moving-project.eu
8 of x
Results
• Comparison
• Company‘s point of view:
• Search performance and search time are crucial
• The quicker, the better as time is money
• Auditors‘ point of view
• Number of click’s to the desired results and number of search terms used
• Combination of several performance indicators
• Comparison with whom
• Top performers or persons in the same team
• Types of prompts
• Interaction with prompts - Auditors prefer “YES/NO” questions to text
input (time reasons)
• Decouple reflection from working activity
• e.g. on mobile devices on the way home

8. www.moving-project.eu
9 of x
Discussion & Conclusion
• Social comparison
• What are meaningful data about search behaviour?
• What type of social comparison is better to raise the intrinsic motivation:
• Competitive comparison (e.g. gamification)
• Informative comparison about other‘s search strategy
• Timing of reflection
• Reflection interventions must not disrupt the user‘s workflow
• Which type of intervention stimulates reflective learning more effective:
• always visible
• visible from time to time
• decoupled from work (e.g. at the bus stop)
• user defined timing (e.g. every Friday afternoon at 3p.m.)

9. www.moving-project.eu
10 of x
Discussion & Conclusion
Challenge to motivate auditors to
change current practice or platform
Show users‘ a clear benefit: „What
is in for me?“
Social comparison is potentially
helpful, but which performance
indicator is useful
Open issues
Privacy and protection need to
be carefully considered
Do not give the users‘ the
feeling to be observed
• Challenges
• Reflection interventions might not be accepted in all work-related settings
• Motivate auditors to change current work practice and/or search platform.
• Privacy and protection need to be carefully considered (click data)
• Do not give the users‘ the feeling to be observed

10. www.moving-project.eu
11 of x
Thank you for your attention!
Any questions?
Project consortium and funding agency
MOVING is funded by the EU Horizon 2020 Programme under the project number INSO-4-2015: 693092