This document discusses parallel user interface rendering (PUIR) to improve accessibility for custom widgets. PUIR defines the user interface abstractly and allows concurrent rendering across modalities like visual and speech. It introduces an abstract UI engine that handles interactions and dispatches events to rendering agents. Rendering agents provide modality-specific presentations synchronized with each other. PUIR also supports equivalent rendering of "cosmetic widgets" that have custom presentations but the same interaction semantics as core widgets. The approach aims to make custom widgets accessible independently of applications or specific uses.

1. Parallel User Interface Rendering
~
Accessibility for Custom Widgets
Kris Van Hees & Jan Engelen
Katholieke Universiteit Leuven
Department of Electrical Engineering
ESAT - SCD - DocArch
Kris Van Hees & Jan Engelen 1/16

2. Agenda
●
Introduction
●
Parallel User Interface Rendering
●
Widgets
●
Cosmetic widgets
●
Conclusions
●
Future...
Kris Van Hees & Jan Engelen 2/16

3. Introduction
●
GUIs in all aspects of life: computers, MP3 players,
home appliances, mobile phones, ...
●
Inherent visual interaction model poses a complication
for users with disabilities (especially blindness)
●
Mixing graphical toolkits within a single environment
complicates matters even more (Linux: Qt, GTK,
Athena, …)
●
Custom widgets: often not supported in assistive
technology without fancy scripting (if available)
Kris Van Hees & Jan Engelen 3/16

4. Introduction (cont...)
●
Abstracting the user interface offers advantages:
– Unified user interaction semantics across toolkits and
modalities
– Retaining flexibility in rendering for multiple modalities
– Assistive technology as 1st generation rendering
●
Parallel User Interface Rendering (PUIR) provides for
equivalent concurrent renderings of the UI
Kris Van Hees & Jan Engelen 4/16

5. Parallel User Interface Rendering
●
User interface defined in abstract form (XML file)
– Effectively defines data and how to present data
●
Abstract UI description interpreted by an AUI engine
– Handles user interaction semantics
●
UI presentation rendered by modality-specific agent(s)
●
Rendering agent depends on system specific toolkits
●
Rendering agents provide presentations concurrently
– Provide both visual and non-visual rendering
Kris Van Hees & Jan Engelen 5/16

6. PUIR: The AUI engine
●
Interprets an abstract UI description (XML file)
●
Implements user interaction semantics
●
Encapsulates data in the UI
●
Provides focus management
●
Dispatches events to the application and rendering agents
●
Can be called from application and rendering agents
●
Has no knowledge of implementation details of
application and rendering agents
Kris Van Hees & Jan Engelen 6/16

7. PUIR: GUI rendering agent
●
Java Swing based in-process rendering agent
●
Complex changes to bypass user interaction in widgets
●
Reduced to provide merely the presentation of widgets
●
Some widgets operate slightly differently
●
Pointer device UI interaction interpreted in the rendering
agent:
– Necessary because position is tied to GUI context
– Semantic operations mapped to AUI operations
Kris Van Hees & Jan Engelen 7/16

8. PUIR: Non-visual rendering agent
●
Remote rendering agent
●
Basic speech output presentation as proof-of-concept
●
Proxy objects for transparency and caching
●
Equivalent with GUI rendering agent:
– Receives the same events
– Synchronized with other rendering agents
Kris Van Hees & Jan Engelen 8/16

9. PUIR: Input processing
●
Keyboard:
– Textual input
– Navigation, text selection, editing, …
– Processed directly by AUI (for proof-of-concept)
●
Mouse:
– Processed by rendering agent
– Position to be interpreted within graphical context
– Semantic user interaction delegated to AUI engine
Kris Van Hees & Jan Engelen 9/16

10. PUIR: Input processing (cont...)
●
Braille keyboard:
– Coded textual input translated into text
– Text processed as if entered on regular keyboard
– Limited cursor addressable positional input
– Positional input interpreted in context
– Semantic user interaction delegated to AUI engine
Kris Van Hees & Jan Engelen 10/16

11. Widgets
●
Two entities per widget:
– Abstract widget:
●
Operates entirely within the AUI engine
●
Implements user interaction semantics
●
Encapsulates data
– Widget presentation:
●
Usually one per rendering agent
●
Implements rendering in output modality
Kris Van Hees & Jan Engelen 11/16

12. Widgets (cont...)
●
Design:
– Define UI in abstract UI description
– Typically by means of design tool
– Rendering specific information can be included
Kris Van Hees & Jan Engelen 12/16

13. Widgets (cont...)
●
Runtime construction:
– Instantiation of UI based on AUI description
– For every widget:
●
Find built-in abstract widget class
●
If none, search in alternate locations
●
If none, UI cannot be instantiated
Kris Van Hees & Jan Engelen 13/16

14. Widgets (cont...)
●
Rendering:
– One-to-one correspondence between abstract widget
class and presentation class
– For every widget:
●
Find built-in widget presentation class
●
If none, search in alternate locations
●
If none and cosmetic widget, use core alternate
(more on this 2 slides from now...)
●
If none, use default place-holder widget
Kris Van Hees & Jan Engelen 14/16

15. Widgets (cont...)
●
Rendering (cont...):
– Rendering agent knows what widgets are supported
– Rendering agent can alert user appropriately
– Rendering agent can query information from AUI
about the unsupported widget
– Accidental user interaction is guarded against
Kris Van Hees & Jan Engelen 15/16

16. Cosmetic widgets
●
Widgets with a very specific presentation
●
Typically derivative of an existing core widget
●
User interaction semantics remain the same
●
AUI description specifies equivalence between cosmetic
widget and core widget
●
Core widget can act as default presentation when no
custom presentation class is found
Kris Van Hees & Jan Engelen 16/16

17. Cosmetic widgets (cont...)
●
Left-to-right:
– Four regular buttons
– VU-meter style widget: equivalent with numeric label
– Four knobs: equivalent with discrete slider
– Four indicator lights: equivalent with label
– Four regular buttons
Kris Van Hees & Jan Engelen 17/16

18. Conclusions
●
Parallel User Interface Rendering
– Powerful technique for Design-for-All
– Coherence between concurrent equivalent renderings
– Generic: not just for non-visual renderings
– Robust handling of custom widgets
– Support for custom widgets independent from
applications and specific uses
– Can be used in automated application testing as well
Kris Van Hees & Jan Engelen 18/16

19. Future...
●
Completion of proof-of-concept
●
More testing
●
Public release
●
Default presentation of custom widgets merely based on
abstract widget information
●
Remote operation of applications
At all stages, feedback from real users is be crucial!
Kris Van Hees & Jan Engelen 19/16

20. Kris Van Hees & Jan Engelen
Katholieke Universiteit Leuven
Department of Electrical Engineering
ESAT – SCD – DocArch
Kasteelpark Arenberg 10
B-3001 Heverlee
Belgium
kris@alchar.org, jan@docarch.be
Kris Van Hees & Jan Engelen 20/16