Satin, a toolkit for sketch-based applications at UIST 2000
•Télécharger en tant que PPT, PDF•
0 j'aime•168 vues
Some research on a toolkit I developed to facilitate the construction of sketch-based interaction. Presented at UIST 2000. Software support for making effective pen-based applications is currently rudimentary. To facilitate the creation of such applications, we have developed SATIN, a Java-based toolkit designed to support the creation of applications that leverage the informal nature of pens. This support includes a scenegraph for manipulating and rendering objects; support for zooming and rotating objects, switching between multiple views of an object, integration of pen input with interpreters, libraries for manipulating ink strokes, widgets optimized for pens, and compatibility with Java’s Swing toolkit. SATIN includes a generalized architecture for handling pen input, consisting of recognizers, interpreters, and multi-interpreters. In this paper, we describe the functionality and architecture of SATIN, using two applications built with SATIN as examples.
Recommandé
Recommandé
Contenu connexe
En vedette
En vedette (20)
Similaire à Satin, a toolkit for sketch-based applications at UIST 2000
Similaire à Satin, a toolkit for sketch-based applications at UIST 2000 (20)
Dernier
Dernier (20)
Satin, a toolkit for sketch-based applications at UIST 2000
- 1. Nov 06 2000 1 Jason Hong and James Landay University of California Berkeley Group for User Interface Research
- 2. Nov 06 2000 2 Motivation • Sketching is useful in many settings
- 3. Nov 06 2000 3 • Creative / Design • SILK • Music Notepad • Knight • Elec. Cocktail Napkin • Teddy • PatchWork Survey of Informal Ink Apps • Capture • ZenPad • NotePals • Dynomite • FiloChat • Others • Pegasus • XLibris • PerSketch • Kramer's Patches • Whiteboard • Tivoli • Flatland
- 4. Nov 06 2000 4 Example - Elec. Cocktail Napkin Gross and Do (UIST96)
- 5. Nov 06 2000 5 Example - Flatland Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)
- 6. Nov 06 2000 6 Common Features • Pen input as ink or gesture • Stroke is pen input from dragging • Ink is a stroke that stays on-screen • Gesture is a stroke that activates a command • Recognize ink as objects • Transformation / clean-up of ink • Immediate / deferred processing of ink erase me
- 7. Nov 06 2000 7 Common Features • Contain objects in addition to ink • Selecting and moving objects • Grouping of objects • Layering of objects
- 8. Nov 06 2000 8 Related Work • Commercial Software Support – Newton OS – PenPoint – Microsoft Windows for Pen Computing – Windows CE – Palm OS • Problems – Form-based and handwriting interfaces – Not easily extensible for building informal ink apps
- 9. Nov 06 2000 9 Related Work • Research Software Support – ArtKit (Henry et. al. 1990) – Garnet Pen Int. (Landay et. al. 1993) – Patches (Kramer, 1994) – OOPS (Mankoff et. al. 2000) • Problems – Need more reusable libraries for handling and processing strokes – Need more extensibility so lots of kinds of informal ink apps can be built
- 10. Nov 06 2000 10 Problems • Software infrastructure support for informal ink apps rudimentary • Pen-based interaction techniques difficult to implement • Similar to GUI development in 1980s – Lots of applications + interaction techniques – Need cohesive frameworks + toolkits
- 11. Nov 06 2000 11 Goals of SATIN • Toolkit support for informal ink apps – Provide reusable mechanisms for handling + processing strokes – Separate mechanism from policy – Be extensible for new kinds of apps • Reusable widgets for pens • Distribute this toolkit for use
- 12. Nov 06 2000 12 Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
- 13. Nov 06 2000 13 DENIM
- 14. Nov 06 2000 14 SketchySPICE
- 15. Nov 06 2000 15 SketchySPICE
- 16. Nov 06 2000 16 SketchySPICE
- 17. Nov 06 2000 17 Some Metrics SATIN DENIM Sketchy SPICE #source files 180 76 7 #methods 2192 642 63 #classes 220 131 32 #lines of code 20000 9000 1000 time 1 person 18 months 3 people 3 months 1 person 3 days
- 18. Nov 06 2000 18 Overview Motivation Survey of Informal Ink Applications Two Applications Built with SATIN SATIN Architecture Overview Conclusions
- 19. Nov 06 2000 19 Architecture Overview Java Core Classes Java Virtual Machine Java2D SATIN Swing Application SATIN relies on Java2D and Swing Applications use SATIN, can use Java2D and Swing
- 20. Nov 06 2000 20 Architecture Overview SATIN can be broken into twelve concepts Recognizers Interpreters Scenegraph Stroke Libraries Widgets Views Rendering Transitions Clipboard Notification Command Events
- 21. Nov 06 2000 21 Recognizers • Problem - many recognition techniques • Recognizers let new recognition technologies be plugged into SATIN • Given a stroke, return n-best list • No actions taken • SATIN comes with Rubine's Recognizer (implemented by Chris Long) Recognizer 0.86 0.65 0.63 Cut Copy Undo
- 22. Nov 06 2000 22 Interpreters • Problem – many ways of handling strokes • Interpreters let new ways of handling strokes be plugged into SATIN • e.g. straighten a line • e.g. issue a command Ink Interpreter Gesture Interpreter erase me
- 23. Nov 06 2000 23 Interpreters and Recognizers • Decoupled recognition (Recognizers) from actions (Interpreters) • Interpreters can use recognizers • Both are modules that can be plugged into apps built with SATIN Interpreter Recognizer Cut 0.86 0.65 0.63 Cut Copy Undo
- 24. Nov 06 2000 24 Multi-Interpreters • Problem – Hard to combine and extend existing interpreters • Multi-Interpreters are a collection of interpreters + dispatch policy – e.g. dispatch to chain of interpreters – e.g. disable some interpreters on context Default Multi Interpreter Intrp A Intrp Z… Semantic Zoom Multi Interpreter Intrp B Intrp A Intrp Z…Intrp B
- 25. Nov 06 2000 25 Phrase (Patch) Scenegraph Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke … Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke
- 26. Nov 06 2000 26 Stroke Dispatching • Needed an approach that enabled: – Individual objects handle strokes on own – Priority to gestures over ink • Default policy is top-down dispatch 1. Process stroke with its gesture interpreter 2. Re-dispatch stroke to one of its children 3. Process stroke with its ink interpreter 4. Handle stroke in object itself
- 27. Nov 06 2000 27 Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke
- 28. Nov 06 2000 28 Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke 1. Process stroke with gesture interpreter Sheet Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp …
- 29. Nov 06 2000 29 Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke
- 30. Nov 06 2000 30 Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Sheet Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp … 1. Process stroke with gesture interpreter
- 31. Nov 06 2000 31 Stroke Dispatching 2. Re-dispatch stroke to children Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)
- 32. Nov 06 2000 32 Stroke Dispatching 1. Process stroke with gesture interpreter Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)
- 33. Nov 06 2000 33 Stroke Dispatching 2. Re-dispatch stroke to children Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)
- 34. Nov 06 2000 34 Stroke Dispatching 3. Process stroke with ink interpreter Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch) Phrase Interpreter
- 35. Nov 06 2000 35 Stroke Dispatching 4. Handle stroke in object Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke Denim Sketch (Patch)
- 36. Nov 06 2000 36 Stroke Dispatching Phrase (Patch) Sheet Denim Label (Patch) Denim Sketch (Patch) Timed Stroke
- 37. Nov 06 2000 37 Example - Cut Shallow
- 38. Nov 06 2000 38 Example - Cut Deep
- 39. Nov 06 2000 39 Reusable Stroke Libraries Straighten Merge
- 40. Nov 06 2000 40 Reusable Stroke Libraries Split Simplify
- 41. Nov 06 2000 41 Widgets • Pie Menus • Pen PLAF • Swing "Pluggable Look And Feel" • Larger and clickable sliders • Single-click file opener
- 42. Nov 06 2000 42 Conclusions • SATIN - Java toolkit for informal ink apps – Extensible thru Recognizers, Interpreters, and Multi-Interpreters – Separates mechanism from policy for strokes – Offers reusable libraries for manipulating ink • Reusable widgets for pens
- 43. Nov 06 2000 43 Download SATIN http://guir.berkeley.edu/projects/satin http://guir.berkeley.edu Download SATIN at
- 44. Nov 06 2000 44 Extra Slides
- 45. Nov 06 2000 45 DENIM
- 46. Nov 06 2000 46 DENIM
- 47. Nov 06 2000 47 Survey of Informal Ink Apps • Elec. Cocktail Napkin • Flatland • Pegasus • FiloChat • Zenpad • PerSketch • Teddy • PatchWork • XLibris • NotePals • Dynomite • SILK • Tivoli • Kramer's Patches • Music Notepad • Knight
- 48. Nov 06 2000 48 Architecture Overview • Recognizers • Interpreters • Scenegraph • Stroke Libraries • Widgets • Views • Rendering • Transitions • Clipboard • Notifications • Commands • Events SATIN can be broken into twelve concepts
Notes de l'éditeur
- Themes Mechanism vs policy Motivation / Solution Introduce self A toolkit for creating informal ink-based applications Work done with James Landay at UC Berkeley Group for User Interface Research
- Motivation Sketching is a natural activity in many settings Two scanned-in sketches of a web site drawn on paper Informal nature of pens makes it easy to quickly draw rough pictures and diagrams w/o worrying about precision More and more computer-based apps are coming out that make use of sketching We call these "Informal Ink applications"
- Related Work We did a survey of informal ink applications As you can see, there has been a large amount of work in sketching interfaces in the research community These applications can be divided into creative / design, whiteboard, capture, and others Too many to go over each of these applications in detail Describe two of these to give you a flavor of what informal ink interfaces are like, Electronic Cocktail Napkin and Flatland
- Electronic Cocktail Freehand drawing tool for conceptual design Also does recognition of strokes drawn on screen (house, tree)
- Flatland Electronic whiteboard application Main workspace subdivided into segments, which are regions of space with interpretations Calculator interpreter on the left recognizes some ink as numbers and lets you do operations such as adding numbers together
- Some commercial software support out there for pens
- Problems Today Software support for creating informal ink applications is rudimentary In fact, most of the informal ink applications we surveyed were built from scratch Pen-based techniques - Gestures, Pie Menus This is one of the reasons that these techniques are not widespread In a situation similar to the 1980s with graphical user interfaces Many applications and many techniques exist, need to start creating cohesive frameworks and toolkits to support informal ink applications Need support for building the insides of applications!
- Build a toolkit that makes it easier to build the insides of an application The part where the interaction takes place (B. Myers) Toolkit is built in Java and targeted at desktop powered computers, currently not PDAs
- Glue Describe the features in SATIN first by showing some applications built using SATIN Then describe the architecture of SATIN by using some of these apps as examples
- Describe DENIM - Sketch-based web site design tool focused on the early stages of design Focus on features of SATIN not DENIM Center is Sheet - A Java Swing component that you can just draw in In the Sheet are three different kinds of objects: labels, ink, and arrows Labels (ex. "Home" and "Sonoma") Ink are the words inside of labels Arrows connect labels Left is a zoom slider - Clicking on a location or dragging it will change the zoom level SATIN does animated transitions for you - given a zoom factor you want to get to, it automatically calculates and animates the intermediate steps Take another look at the Sheet, b/c the next slide we'll zoom in a little
- Next application - SketchySPICE Focus on features of SATIN not SketchySPICE SPICE is a circuit CAD tool developed at UC Berkeley SketchySPICE has the added value of sounding like one of the Spice Girls SketchySPICE is a simple sketch-based circuit diagramming tool, lets you draw out AND and OR gates and clean them up
- Example of two-strokes for recognition (but this was built into the application, not the toolkit) Feedback of recognition
- Changing objects from sketchy to formal and back
- DENIM, first cut implemented by 3 people in 3 months SketchySPICE implemented in 3 days One reason these two applications were easier to build b/c SATIN provided reusable mechanisms for handling and processing ink
- SATIN is built using JDK 1.3, marketing name Java 2 version 1.3 Parts in gray are part of standard Java Runtime that everyone would have SATIN makes heavy use of the Java2D, a library for doing fairly advanced 2D graphics SATIN also uses a little bit of the Swing GUI toolkit, which we will see later on We try to abstract out interactions with Java2D, but advanced users can still use those libraries if desired To build an informal ink application, you would use SATIN and the Java Core Classes
- SATIN can be broken into twelve concepts Some of these concepts were added because they were in conventional toolkits Clipboard Command objects for undo and redo For purposes of time will only talk about the more novel concepts
- What we did in SATIN was to provide the interface for recognizers This lets you be able to plug-in other recognizers Currently SATIN ships with Rubine's Recognizer as implemented by Chris Long Don't have to use Recognizers though Decoupled Recognizers and Interpreters
- Interpreters are attached to Graphical Objects Currently SATIN only handles single strokes, doesn't handle multiple strokes at once or progressive strokes In SATIN, we call these Ink Interpreters In SATIN, we call these Gesture Interpreter Use Recognizers
- Interpreters can use recognizers but not required to do so
- Multi-interpreters are special cases of interpreters With Multi-Interpreters, you can have more than one interpreter attached to a Graphical Object
- All the parts in the DENIM section are either primitives provided by SATIN Or derived from SATIN primitives We will return to this scenegraph again when we talk about dispatching of strokes
- How is this different from Java dispatching? If step 1 fails, then go on to step 2, and so on… Take through an example with DENIM Why did we do this?
- Pan to the right gesture Gestures in DENIM are differentiated through the right pen button (Right as in left and right) This is set in DENIM, not in SATIN
- Pan to the right gesture Gestures in DENIM are differentiated through the right pen button (Right as in left and right) This is set in DENIM, not in SATIN
- Pan to the right gesture
- Pan to the right gesture
- Gets re-dispatched to the child in the scenegraph that completely contains the stroke Currently implemented, there is no tolerance for errors, ie stroke is dispatched to child that completely contains the stroke Will be fixed
- Gets re-dispatched to the child in the scenegraph that completely contains the stroke Currently implemented, there is no tolerance for errors, ie stroke is dispatched to child that completely contains the stroke Will be fixed
- Gets re-dispatched to the child in the scenegraph that completely contains the stroke Currently implemented, there is no tolerance for errors, ie stroke is dispatched to child that completely contains the stroke Will be fixed
- Gets re-dispatched to the child in the scenegraph that completely contains the stroke Currently implemented, there is no tolerance for errors, ie stroke is dispatched to child that completely contains the stroke Will be fixed
- Gets re-dispatched to the child in the scenegraph that completely contains the stroke Currently implemented, there is no tolerance for errors, ie stroke is dispatched to child that completely contains the stroke Will be fixed
- No gesture interpreter Can't re-dispatch from the sketch Sent to phrase interpreter, which tries to group together things that look like letters Since this doesn't look like writing, it fails
- So that's how stroke dispatching works. Something we learned while building DENIM with SATIN is deep / shallow Deep vs Shallow based on zoom level When zoomed out, do gestures shallowly When zoomed in, do gestures deeply Example
- There is support in SATIN for doing deep and shallow gestures now
- Libraries for doing simple manipulations on strokes These are mechanisms not policy Simplified merge, merge things near endpoints
- Again, these are library calls, mechanism not policy
- These are distributed as part of SATIN, but can also be used independently of SATIN In other words, you can use the pie menu or the pluggable look and feel without having to use SATIN
- Used by a research group at PARC, one researcher at a university in Canada, and by our research group About 60 downloads since our release August 11 Used in DENIM, with about 1500 downloads since May You too can download SATIN
- Semantic Zooming - Originally just showed labels, now we're showing entire panels (label + sketch) Panels also contain ink, and any strokes drawn inside the bounds of a page belong to that page Pie Menu - implemented as a Java Swing component Pie menu actually independent of SATIN, can be used in any application using Java Swing
- Zoomed in one more time to page level Highlighted is a phrase, important for arrows (would look strange if links were only letters instead of words)
- Related Work We did a survey of informal ink applications As you can see, there has been a large amount of work in sketching interfaces in the research community Too many to go over each of these applications in detail Describe two of these to give you a flavor of what informal ink interfaces are like, Electronic Cocktail Napkin and Flatland
- SATIN can be broken into twelve concepts Some of these concepts were added because they were in conventional toolkits Clipboard Command objects for undo and redo For purposes of time will only talk about the more novel concepts