Book Paid Powai Call Girls Mumbai 𖠋 9930245274 𖠋Low Budget Full Independent H...
Computer Gesture Recognition in Aphasia Therapy: Findings from the GReAT project
1. Computer Gesture Recognition in
Aphasia Therapy:
findings from The GReAT Project
Abi Roper and Naomi Cocks
Division of Language and Communication Science
City University London
2. Today
•Gesture Therapy
•Existing Computer Therapies
•The GReAT Project
•Designing and Refining a Computer Gesture
Therapy - Gest
•Gest Demonstration
•Delivering a Computer Therapy
•Gest Pilot Study
•Outcomes and Implications
3. Previous Studies of Gesture Therapy
To compensate for speech (e.g. Daumuller &
Goldenberg, 2010)
To facilitate speech (e.g. Boo & Rose, 2011; Marangolo
et al, 2010; Rose & Douglas, 2008)
Findings
• Even people with severe aphasia can improve
gesture production
• Treatments with a gesture component can
enhance naming
4. But …
Gains are often very modest
Gains may relate to the intensity of therapy
(Caute, 2012)
5. Computer Therapy for Aphasia
• Sentence Shaper – Speech production
Exercises sentence level
• “The program is most likely to be effective for people
who are able to produce some speech, and whose
executive function and/or memory allows them to
utilize a software program that requires executing
certain actions in order (for example, turning a sound
recorder on, speaking then turning the recorder off)
and self-monitoring (playing back their utterances in
order to correct and expand them).”
6. SWORD Spoken Word Production Exercises.
Developed for Apraxia of Speech.
Software Design and Useability Strengths
• Mouse only control: All participants reported that they were
able to use the mouse / touchpad (following instruction) to
navigate the programme.
• AP1’s wife: “It’s about making it simple. Easy to read.”
• All participants responded that the programme never once
crashed.
• AOS 29’s Husband: “I could do me back garden!” Reports
being happy that his wife was doing something
independently.
• AP2’s wife: [whilst her husband was using the therapy] “I got a
lot of gardening done!”
7. Computer Therapy for Aphasia
Software Design and Usability Limitations
Mouse Access: AOS 29 reported that it took some time
to learn to use the mouse but she got used to using
it. Reported that a touch screen would have made
the laptop a lot easier to use.
SLT Comments: “Some patients needed prompting to
remind them that they need to click. Some patients
timed out before providing a response and were
awarded a fail. Is it possible to give even longer than
25 seconds?”
8. The GReAT Project
What’s different and novel about the GReAT
project?
• Use of Gesture Recognition Technology
(Gesture Recognition in Aphasia Therapy)
• Software Design and Useability Focus
• Participatory Design Process
9. Project Aims
• To develop an affordable, computer-
based technology that can be used in
therapy at home to help people with
severe aphasia to gesture.
• To evaluate the efficacy of the
technology within a pilot therapy study
10. Project Structure
• Phase 1: Designing a prototype gesture
therapy using participatory design
methods.
• Phase 2: Testing and piloting the
prototype
11. Project Team
Human Computer Interaction Design & Language and Communication Science
Stephanie Wilson Sam Muscroft Julia Galliers Jane Marshall
Naomi Cocks Tim Pring Abi Roper
12. Phase 1
• Designing a prototype gesture
therapy using participatory
design methods.
14. Consultants
•Role: to test and feedback about relevant technology.
•Person Specifications:
–Expressive aphasia language difficulties.
–Able to attend university once or twice a month for
participatory design sessions.
•Recruited through in house clinic and through links with
the Stroke Association Communication Support
Co-ordinators.
•Employed by City University London as Casual Staff
members.
15. Methods: Participatory Design
Sessions
•Participatory design – engaging end users in design
process
•Sessions explored offline gesture therapy, computer
gesture recognition, interaction within 3D worlds and
computer interfaces.
•Consultants took part in 9 sessions each
•Project team involved in each session
- 1 HCID Researcher
- 1 HCID Developer
- 1 Speech and Language Therapist Researcher
- 2 or 3 Consultants
16. Session Structure
1. Introduction to scheduled
activities
2. Round table gesture activity
3. Demonstration of Technology
4. Trial use of technology by one
consultant - followed by
interview at computer
5. Tea break
6. Trial use of technology by
remaining consultant(s)
18. What did we learn from the
Sessions?
1. Consistency
2. Simplicity
3. Pace
4. Reliability
5. Rewards
6. Individual Differences
7. Potential of ‘gaming’.
19. Key Features of GeST
Separate keyboard Gesture recognition
← →
OK
Gestures presented in
isolation & in context 3D worlds
22. Tool Design
•User sees gesture demonstrated twice
•Is invited to copy the gesture
•User monitors their
production attempt
on screen
•Receives reward and feedback for each
correctly recognised gesture
24. Using the Therapy at home
•Pilot study (Coming up next)
•How does this work at home?
Key differences between lab and
home –
User practising independently,
User intending to practise daily.
User practising in non-lab
conditions.
25. Things to consider when setting
up
•Lighting conditions
•Safety and permanence
(negotiate!)
•User comfort and access
26. Things to consider when training
•Develop the user’s confidence in the system.
(Be confident yourself)
Demonstrate:
1. Allow user to observe entirely
2. Allow user to observe and operate
interaction buttons
3. Allow user to operate alone but with
support as needed (confidence)
27. Things to consider when training
•Reinforce how to switch the computer
on and off several times.
•Make an appointment to come back in
one week to review.
•At review appointment, observe and re-
train difficult procedures.
28. Testing GeST
•Pilot study
•Participants received GeST for a total of 6
weeks.
•First three weeks supported by a
therapist with hour long session each
week.
•Second three weeks no support from
therapist.
30. Questions
• Will practice with Gest improve participants’ production of
gestures &/or spoken words?
• Will improvements be specific to items that feature in the
programme?
• Will gains occur when Gest is used without ongoing
therapist support?
• Will gains be maintained after Gest is withdrawn?
• What are participants’ views about Gest?
• What are carers’ views about Gest?
(where relevant)
• Is Gest easy and enjoyable to use?
31. Participants
• 9 people with severe aphasia
– Consent to take part
– Fluent pre-stroke users of English
– Naming score <20%
– Able to recognise pictures
– No known dementia or other cognitive impairment
32. Consent
Screening
Phase 1 with weekly
Tests (1)
visits from therapist
3 Weeks
Practice
Phase 2 with no weekly
Tests (2)
visits from therapist
3 Weeks
Practice
Tests (3)
3 weeks
no tool
Total time commitment: about 14 weeks Tests (4)
33. Practice Phases
• Each last 3 weeks
• Each practise 15 gestures with the tool
• Phase 1: Weekly visits from therapist
• Phase 2: Initial but no weekly visits
34. Tests
• 60 items
– Gesture from picture
– Name from picture What is the
name of this?
How would
you gesture
Items: this?
30 practised with Gest
15 familiarised only
15 controls
35. Scoring Gestures
• Gesture tests are filmed
• 4 Scoring videos created
• Each video contains 60 gestures in random
order:
– 15 from test 1
– 15 from test 2
– 15 from test 3
– 15 from test 4
36. Scoring Gestures
• Scores
– Recognition Score
– Rating Score
• Scorers are ‘blind’ to the time of assessment
38. Statistical Analysis
• Time x Group interaction
• Planned comparisons:
– Items that were treated with Supported Gest
changed over time, other groups did not
– Supported Gest items improved significantly
following practice (between time 1 and time 2),
but not in the other phases
– Recognition of Supported Gest items was
significantly improved against baseline even at
time 4
39. Interim Conclusions
• Repeated testing without intervention (Control
items) did not improve gesture production
• Independent practice with Gest and familiarisation
did not improve gesture production.
• Using Gest with therapist support improved gesture
production. However:
– Gains were modest
– Differences between the groups were very small.
40. Usage Logs
• Record
– Number of sessions
– Length of sessions
– Levels of programme accessed
– Number of gestures recognised
41. More Interim Conclusions
• Benefits from Supported Gest may reflect
– Therapist input (‘The Abi Factor’)
– Usage
43. More Conclusions
• Using Gest did not facilitate naming of the
gesture targets
• This was despite the repeated inclusion of
spoken names in the programme
45. Independence of Use
• ‘She uses it all on her own, I don’t know how to
operate it’
• The first session I stayed with L, after that I’ve
helped only if she’s found something
particularly frustrating’
• All comment that the participant initiated use
of Gest
46. Enjoyment
• All say that the participant enjoyed Gest
• ‘he likes it when they clapped’
• ‘some of the gestures are particularly fitting
and she enjoyed rainbow’
47. Views about Technology
• ‘I was a technophobe and when they said
‘computer’ I thought it was going to cause
problems. I thought I wouldn’t understand
and he wouldn’t understand it. But it’s so
‘easy’
48. Reservations
• Carry over to real life (1 carer):
• ‘while she works on it here (points to
computer) it doesn’t necessarily translate’
• She wanted a hankie last night and didn’t
make a gesture’
49. Conclusions
• Will practice with Gest improve participants’ production of
gestures?
Yes
• Will improvements be specific to items that feature in the
programme?
Yes
• Will gains occur when Gest is used without ongoing therapist
support?
No
• Will gains be maintained after Gest is withdrawn?
Yes
• Will practice with Gest improve naming of target items
No
50. Conclusions
• What are participants’ views about Gest?
Very Positive
• What are carers’ views about Gest?
Very Positive
• Is Gest easy and enjoyable to use?
Yes
51. Acknowledgements
The Research Councils UK Digital
Economy Programme
The Stroke Association
Consultants and their families
Participants and their families
All our wonderful students who helped to score data
Thank You
GReAT@city.ac.uk
www.soi.city.ac.uk/great
52. Find Out More
Website: www.soi.city.ac.uk/great
Facebook: www.facebook.com/aphasiatech
Vimeo: www.vimeo.com/aphasiatech
Email: GReAT@city.ac.uk
Editor's Notes
To develop any sort of skill requires repeated, regular practice. In one to one therapy we can offer short, intensive bursts of practice but not always enough to effect the desired changes. Computer supported therapy, offers an opportunity to supplement this approach by providing regular and structured exercises that a person with aphasia can carry out as often as they wish and at their own convenience.
Reminding, mapping and consistency – Language impairments present in aphasia can affect a users ability to retain sequences and instructions about how to navigate through tasks. The simpler and more transparent the relationship between the interface and the action being carried out – the better. Similarly the more consistent this action is throughout the entire navigation – the more successful a user can be.Simplicity: The presence of too much visual and or textual information posed a real challenge for our consultants so in our final prototype we have worked hard to keep such things to a minimum. Text instructions consist of single words and spoken instructions are usually accompanied with a video of the person delivering the instruction.Controlling pace – One might assume that the pace of activities should be slow to enable users with aphasia time to process information and details. Within design sessions, we found that some activities needed to be extended to give Consultants time to recognise the results of their interactions. However, as users became more familiar with the demands of the task they sought to skip forward to continue with activities at a faster pace.Reliability – in early comparisons of gesture recognition technology, we found users became less motivated when a technology was less reliable at recognizing gestures (as was the case in the use of the wiimote for this task). Achieving some level of success was key to remaining motivated.Rewards - In early tasks, a correct gesture was acknowledged by the computer moving on to the next activity. As we developed the tasks we found that consultants responded positively to receiving more explicit acknowledgement of their success. So, we added a round of applause in response to an accurate gesture production.As with any group of computer users, the GReAT consultants displayed many individual differences and a diversity of responses to the technologies. IT became apparent that the ability to tailor aspects such as pace and content can help to engage and motivate individual users depending on their level of expertise.Gaming - The positive effects of playing games and achieving a result became very evident in our sessions. In typical computer therapy, exercises are presented in a format very alike that which might be carried out in person. With the introduction of the 3d environment and a point scoring mechanism, we are beginning to explore the additional opportunities that technology-based therapy can offer.
Trains a total of 30 naturalistic gestures chosen for their ability to be used with one hand, their distinctiveness and their functionality bed, food, car with some exceptions (rainbow, spider) (Limitations)
As I said at the start, to develop skills we need to train regularly so users need the opportunity to practise as often as possible.
