Targeted follow-up meetings in general practice are important and missed often, because of both patient and general practitioners (GPs) related reasons. In this paper, we present a proof-of-concept interactive visualization dashboard that provides GPs with a powerful, yet easy to use method to identify those patients in need of follow-up. We applied a user centered, rapid prototyping methodology with 12 information visualization students and 15 GPs. We evaluated the final design using the evaluation framework by O’Leary et al., as well as a System Usability Scale questionnaire. Results indicate that there is indeed a need for a follow-up tool and that a dashboard is a right kind of tool. Our proof-of-concept shows useful insights into patient records and can indeed help GPs recognize patients in need of follow-up. The major strengths of the design are the ease with which GPs can query patient records using interactive visualizations, such as parallel coordinates, and the ability to check if the number of patients diagnosed with certain diseases differs from the amount predicted in evidence-based guidelines.

1. DESIGNAND EVALUATIONOF AN
INTERACTIVE PROOF-OF-CONCEPT
DASHBOARD FOR GENERAL
PRACTITIONERS
Robin De Croon,Joris Klerkx, Erik Duval
robin.decroon@cs.kuleuven.be
IEEE International Conference on Healthcare Informatics 2015, Dallas

2. Patient follow-up
Patients
• forget follow-up meetings
• do not considera follow-up necessary
• do not want tospend additionalmoney
• have the impression treatment is not working
General practitioners
• hard to find patientsin need of follow-up
• are toobusy to accommodateaprompt visit
• do not have enoughtime
Wednesday, October 21, 2015 2

3. How to improve follow-up quality?
Patient oriented
• Stimulate patient empowerment
General practitionerdriven
• Provide better tools to augment workflow
Wednesday, October 21, 2015 3

4. Wednesday, October 21, 2015 4
{ =, <, >, >=, <=, <>} | value | {and, or} | value

5. Research question
“How do we help general practitioners
identify patients in need of follow-up
using interactive visualizations?”
Wednesday, October 21, 2015 5

6. Research question
“How do we help general practitioners
identify patients in need of follow-up
using interactive visualizations?”
Wednesday, October 21, 2015 6

7. Research question
“How do we help general practitioners
identify patients in need of follow-up
using interactive visualizations?”
Wednesday, October 21, 2015 7

8. Research question
“How do we help general practitioners
identify patients in need of follow-up
using interactive visualizations?”
Wednesday, October 21, 2015 8

9. User centered, rapid prototyping
Wednesday, October 21, 2015 9
Initial
design
Prototype 2
- Content
- Missingdata
Result
Final Design
- Updated table
- Consistency
- bugs
Prototype 3
- Internal changes
Academic Center
General Practice
3 GPs 12 infovis students 9 GPs
design phase usability phase evaluation phase

10. User centered, rapid prototyping
Wednesday, October 21, 2015 10
Initial
design
Prototype 2
- Content
- Missingdata
Result
Final Design
- Updated table
- Consistency
- bugs
Prototype 3
- Internal changes
Academic Center
General Practice
3 GPs 12 infovis students 9 GPs
design phase usability phase evaluation phase

11. Visualization concepts
• Overview first, zoom and filter, then details-on-demand
• Data ink ratio
• Stevens’ model
Wednesday, October 21, 2015 11
Ben Shneiderman, The Eyes Have It: A Task by Data Type Taxonomy for Information
Visualizations. In Proceedings of the IEEE Symposium on Visual Languages, pages 336-
343, Washington. IEEE Computer Society Press, 1996
Edward Tufte. The Visual Display of Quantitative Information. 1983
S. Stevens, “On the theory of scales of measurement.” Science
(New York, N.Y.), vol. 103, no. 2684, pp. 677–680, 1946

12. Demographics
Wednesday, October 21, 2015 12
http://www.cdc.gov/nchs/data/databriefs/db81.htm

13. Population pyramid Most common diseases Religion
Highest degree Main caregiver
#days since last visit income systolic blood pressur diabolic blood pressure weight sugar age
Picture Name Condition Gender Age Degree Main caregiver

14. Parallel coordinates
Wednesday, October 21, 2015 14
https://syntagmatic.github.io/parallel-coordinates/

15. User centered, rapid prototyping
Wednesday, October 21, 2015 15
Initial
design
Prototype 2
- Content
- Missingdata
Result
Final Design
- Updated table
- Consistency
- bugs
Prototype 3
- Internal changes
Academic Center
General Practice
3 GPs 12 infovis students 9 GPs
design phase usability phase evaluation phase

16. AcademicCenter for General Practice
What
• 3 general practitioners+2 electronic medical recordexperts
• already perform audits on general practices
• Perceived usefulness
Result
• Too much demographics
• Medication (groups)more important
• primary and secondary condition
• Deal with noise in data
• outliers, missing elements
Wednesday, October 21, 2015 16

18. User centered, rapid prototyping
Wednesday, October 21, 2015 18
Initial
design
Prototype 2
- Content
- Missingdata
Result
Final Design
- Updated table
- Consistency
- bugs
Prototype 3
- Internal changes
Academic Center
General Practice
3 GPs 12 infovis students 9 GPs
design phase usability phase evaluation phase

19. Usability iteration
What
• 12 information visualization students
• 10 tasks
• time-to-task & error-rate & perceiveddifficulty
• ± 60 minutes
• Questionnaires
• initial& closing&System UsabilityScale
Wednesday, October 21, 2015 19

20. Results usability evaluation
• System UsabilityScore: 68
• 7/10 tasks < 60s
VisualUpdates:
• Visual update
• Consistentnavigation
• Patient table
Wednesday, October 21, 2015 20

22. User centered, rapid prototyping
Wednesday, October 21, 2015 22
Initial
design
Prototype 2
- Content
- Missingdata
Result
Final Design
- Updated table
- Consistency
- bugs
Prototype 3
- Internal changes
Academic Center
General Practice
3 GPs 12 infovis students 9 GPs
design phase usability phase evaluation phase

23. Final evaluation
• 9 general practitioners
• 7 male + 2 females
• ConcurrentThinkAloud protocol
• short introduction, free interaction, 15 ~ 20 minutes
C. Lewis, Using the"thinking Aloud" Method inCognitiveInterfaceDesign, (ibm resea ed., ser. Research
report. Yorktown Heights, NY: IBMT.J. Watson ResearchCenter
• SystemUsability Scale
Brooke, John. "SUS-Aquickand dirty usability scale."Usability evaluation inindustry 189.194 (1996): 4-7
• Questionsfrom O’Leary et al. à likert questions
O’Leary, P., Carroll, N., & Richardson, I. (2014). The Practitioner’s Perspective on Clinical Pathway
Support Systems. In 2014 IEEE International Conference onHealthcare Informatics (pp. 194–201)
• SWOTanalysis
Wednesday, October 21, 2015 23

24. System Usability Scale
Wednesday, October 21, 2015
74
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores
mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123

25. System Usability Scale
Wednesday, October 21, 2015 25
1 2 3 4 5
Learn a lot of things
Confident using the system
Very cumbersome to use
Learn to use this system very quickly
Too much inconsistency
Functions well integrated
Need the support of a technical person
Easy to use
Unnecessarily complex
Like to use this system frequently

26. Likert questions
Wednesday, October 21, 2015 26
1
1
2
3
5
1
3
6
4
2
2
3
3
5
6
3
3
4
2
5
2
4
2
0% 20% 40% 60% 80% 100%
Need for follow-up tool
Follow best practice
TrainGPs
Correct level of detail
Recognize follow-up patients
Reducing mistakes
Useful feedback
Right kind of tool
totally disagree disagree neutral agree totally agree
O’Leary, P., Carroll, N., & Richardson, I. (2014). The Practitioner’s Perspective on Clinical Pathway Support
Systems. In 2014 IEEE International Conference onHealthcare Informatics (pp. 194–201)

27. SWOT analysis
272
2
3
4
3
4
5
2
3
4
4
2
2
3
3
4
5
Control system
Privacy
Too little time
Averages can be dangerous
Can be improved with patient collected data
Triggers self-reflection
Ideal for research
Not needed often
Not clear which content too show
Not much structured data available
Map uses too much screenestate
No pseudo code needed
Improves team communication
Visual overview
Augment work
Check with guidelines
Easeto select patients

28. Technologies
Wednesday, October 21, 2015 28

29. Limitations
• Sample size
• total 15 GPsà exploratory study
• Semi realisticdata
• anonymous& assess perceived usefulness
• Requirements gathering
• user centered, rapid prototyping
• Evaluation setting
• limited time à perceived usefulness
Wednesday, October 21, 2015 29

30. Conclusion
• Powerful, yet easy to use
• Reduces burden to analyze patient records
• Recognize patients in need of follow-up
• Query patient with visual filters
• Needs to adapt to specific use cases
• Biggest opportunity is in research
Wednesday, October 21, 2015 30
http://cdn.makeuseof.com/wp-content/uploads/2012/12/3D-Man-Presenting-Intro-Image.jpg?a53b57

31. Looking for collaborations!
• Make the dashboard available in general practice
• evaluate impact on collaboration
• evaluate if self-reflectionistriggered
• Continue working with academic GPs
Wednesday, October 21, 2015 31

32. Acknowledgements
Wednesday, October 21, 2015 32
Research grand: IWT 120896
Access to their products and expertise
For their experienced feedback
+ All participants!

33. Thank you!
Wednesday, October 21, 2015 33
http://2.bp.blogspot.com/-gZjNR3XVULs/T_ZOVgE-5lI/AAAAAAAAAg8/6YVmd5Q064o/s1600/questions11.jpg
robin.decroon@cs.kuleuven.be

34. information seeking mantra
34
Ben Shneiderman, The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations.
In Proceedings of the IEEE Symposium on Visual Languages, pages 336-343, Washington. IEEE
Computer Society Press, 1996.

35. data ink ratio
Wednesday, October 21, 2015 35
Edward Tufte. The Visual Display of Quantitative Information. 1983

36. target user
Target Audience
Wednesday, October 21, 2015 36
• General Practitioners (in Belgium)
Independentof:
• Experience & age
• Individual or group practice
• ICT-knowledge
• Current medical software
http://marketingyoucanuse.com/wp-content/uploads/2010/12/HittingTarget.jpg

37. Related work: Eventflow
Wednesday, October 21, 2015 37
Monroe, M., Lan, R., Plaisant, C., Shneiderman, B. (May 2013) Temporal
Event Sequence Simplification In IEEE Trans. Visualization and Computer
Graphics, 19, 12 (2013), 2227-36. HCIL-2013-11

38. AS-IS
• Audit byAcademicCenter forGeneral Practice
• perceived as control
• Limited tools available:
Wednesday, October 21, 2015 38
query window from Medidoc