1. A Grey-Box Approach for
Automated
GUI-Model Generation of Mobile
Applications
Wei Yang Mukul R. Prasad Tao Xie
Department of Computer Science, Software Systems Innovation group Department of Computer Science,
North Carolina State University, Fujitsu Labs. Of America North Carolina State University,
Raleigh, North Carolina Sunnyvale, CA Raleigh, North Carolina
3. Growing demand for high quality apps
Solution?
Automated Testing Tools!
3
4. GUI model
• GUI model is essential for developing
automated testing tools
– What does it represent?
– Why is such a model useful/important for
testing?
4
5. Sample App: Simple TippyTipper
Used to calculate tip amount for a meal
MENU MENU
5
6. GUI Model for Simple TippyTipper
Calculate
1 2
1 2 4
Menu
Menu
3 About 4
Settings
5a 5b
a1 10
11
a2 a2
01
a1
5c 5d 00
3 3
5
a1: Toggle exclude tax rate option.
a2: Toggle round up option.
6
7. Applications of GUI Model
• Classical Model-based testing applications
– Automatic (system-level) test-case generation
– Use as a driver for symbolic execution (SE) based testing
• More recent “crawl-and-compare” applications
– Cross-platform compatibility testing of an application
– Regression testing across different versions of the application
7
8. Related Work
• GUITAR: GUI Ripping for Windows, Java-based GUI
(desktop) apps, Atif et al.
A simplified event-flow model of simple tippytipper. Our GUI model of simple tippytipper.
Calculate
Click 3
Click 1 1 2
Click 2 Menu Menu
Click 5 Click 4 3 About 4
Settings
Click 6 5a 5b
Click 7 a1
11 10
Click 8 a2 a2
Click 9
5c 01
a1
5d 00
Click Click 0
Click
CLEAR DEL a1: Toggle exclude tax rate option.
a2: Toggle round up option.
Click
Calculate
8
9. Related Work(Cont.)
• Concolic Testing (SE) of Android Applications, Saswat et al.
• Random Testing of Android Applications, Cuixiong et al.
• Crawljax: Crawling of AJAX Web Applications, Ali et al.
• AndroidRipper: Stress Testing of Android
Applications, Domenico et al
9
11. ORBIT GUI Crawler: Overview
ORBIT GUI Model
Action
Android Action inference Mapping Dynamic Crawler
AUT
(source code)
11
12. Proposed GUI Model
• Visual Observable State
– Inspired by the UI-design principles espoused by
the Android team.
– Composition of the state
• Model
– A finite-state machine over visual observable
states with the user actions constituting the
transitions between these states
12
13. GUI Model for Simple TippyTipper
Calculate
1 2
1 2 4
Menu
Menu
3 About 4
Settings
5a 5b
a1 10
11
a2 a2
01
a1
5c 5d 00
3 3
5
a1: Toggle exclude tax rate option.
a2: Toggle round up option.
13
14. Illustration: State
These two states differ in
the state of this widget
Two different states
of Screen # 5b & 5d
5b 5d
14
15. Action Inference
• Approach:
– Locate all instances of event handler definitions in the code (sinks)
– Associate each with the corresponding widget instantiations (sources)
TippyTipper.java
View btn_delete = findViewById(R.id.btn_delete);
R.Id.java
…
Btn_delete.setOnClickListener(new onClickListener() {
public void onClick(View v) {
removeBillAmount();
FlurryAgent.onEvent(“Delete Button”);
}
});
…
Btn_delete.setOnClickListener(new onLongClickListener() {
public void onLongClick(View v) {
Inference: Widget btn_delete with Id = 0x7f0000a
clearBillAmount(); supports actions click and longClick
return true;
}
});
15
17. Dynamic Crawling: Illustration
Activity Setting Activity TippyTipper
Activity Total
Toggle
BACK
Toggle
Calculate
Toggle Toggle
BACK MENU
BACK MENU
Toggle Toggle
Toggle
SETTINGS
Toggle
17
19. Research question
• RQ1: Is the proposed GUI crawling algorithm more
efficient than a standard depth-first state-traversal
algorithm?
• RQ2: Are the widget and screen actions inferred by
static analysis effective in enhancing the behavior
covered by the generated model?
• RQ3: Can our tool generate a higher-quality model,
more efficiently, compared to other state-of-the-art
techniques?
19
20. Subjects used in the evaluation
Subject #LOC #A Category Purpose
TippyTipper 2238 5 Tool Dining tip calculator
OpenManager 1595 6 Business File manager for Android
Notepad 322 3 Productivity Note creation and management
TomDroid 3711 3 Business Online note-reading
Aarddict 4518 4 Books Aard Dictionary for Android
HelloAUT 234 1 Entertainment Color shapes
ContactManager 497 2 Productivity Contacts manager
ToDoManager 323 2 Productivity Create and manage task lists
• Our evaluation is based on subjects drawn from existing related tools
• We avoid bias by including all subjects used to evaluate Android
GUITAR and GUI Ripper in previous work.
20
23. Comparison with other tools
Subjects Monkey Android Android GUI ORBIT
GUITAR Ripper
Time Cov. Time Cov. Time Cov. Time Cov.(%)
(sec) (%) (sec) (%) (sec) (%) (sec)
TippyTipper 83 41 322 47 - - 198 78
OpenManager 90 29 - - - - 480 63
Notepad2 127 60 - - - - 102 82
Notepad0 122 59 - - - - 80 78
TomDroid 69 46 - - 529 40 340 70
AardDict 124 51 - - 694 27 173 65
HelloAUT 98 71 117 51 - - 156 86
ContactManager 90 53 247 61 - - 125 91
ToDoManager 115 71 194 71 - - 178 75
• Our crawler is 32%-75% faster while constructing a 5%-140% more
complete model than Android GUITAR and Android GUI Ripper.
23
24. Conclusions & Future Work
• Crawling algorithm
– Incorporate the context of event call to prune the order of the
sequence.
• GUI model
– Explore the possibility to generate a complete specification or test
oracle by using our driver and partial specification.
• Selection of subjects
– Inter-application communication of Android application.
• Manual effort
24
25. Thanks!
*This project was supported in part by NSF grants CCF-0845272, CCF-0915400, CNF-0958235,
CNS-1160603, and an NSA Science of Security Lablet Grant.
Notes de l'éditeur
Developers are required to develop high quality apps in order to be competitiveMobile apps are usually developed in relatively small-scale projects, which may not be able to support extensive and expensive manual testing
Classical Model-based testing applicationsAutomatic (system-level) test-case generationEach path in the model is a potential test-caseUse as a driver for symbolic execution (SE) based testingThe model provides the event sequences while explores the space of data-valuesMore recent “crawl-and-compare” applicationsCross-platform compatibility testing of an applicationDifferent mobile platforms or versions, desktop vs. mobileRegression testing across different versions of the applicationOther potential applications (future research)To complement more rigorous (computationally expensive) approaches such as SE for testingProvided model extraction can be done cheaply (efficiently)
GUITAR: GUI Ripping for Windows, Java-based GUI (desktop) apps.AtifMemon et al. U. MarylandCompletely event and component based modelNot suitable for capturing stateful behavior of Android apps.Completely black-box & dynamicCannot reverse engineer complex events other than simple tapsCrawljax: Crawling of AJAX Web ApplicationsAli Mesbah et al., T U Delft, Univ. of British ColumbiaFlat, state-based modelDoes not represent the structure of Android application: Very useful in testing & debuggingNOTE:In web-apps such information is not easily available/representableCompletely black-box and dynamic: Similar limitation as GUITARConcolic Testing (SE) of Android ApplicationsSaswatAnand et al., Georgia Tech., Stanford Univ.Uses Symb. Exec. to reverse engineer GUI action sequencesVery (compute) intensive and limited by length of sequencePath explosion since each sequence represented separately: No model !
GUI Model: What should it represent?All/Many use-cases of the application available to the user through the GUIScreen viewable by the userAction (on widgets) sequences exercisable by the user (through the GUI)Why is such a model useful/important (for Android applications) ?Mobile applications strongly GUI-driven/GUI-centricNot designed to be exercised without the GUITypically GUI encodes all permissible use-cases of the applicationLooking at the GUI alone is mostly sufficient to extract these use-casesMostly not necessary to look at/analyze application implementationJust executing it (through GUI) sufficient !
Visual Observable StateInspired by the UI-design principles espoused by the Android team. The Android User Experience Team suggests that developers should “make places in the app look distinct” to give users confidence that they know their way around the app.Composition of the stateThe hierarchy tree of the UI screen, as well as a vector of attribute values of each of the executable components.The structure of a GUI screen in Android is represented by a tree of different GUI components, called a hierarchy treeExecutable components are those GUI components support user actions (which are detected by our static analysis)ModelA finite-state machine over visual observable states with the user actions constituting the transitions between these states
A state corresponds to a specific set of widgets (i.e. a screen) of a specific component, with each widget in a specific statei.e., a state is identified by the states of its constituent widgetsThe state of a widget is denoted by the value of its attributes such as: enabled (true/false), visible(true/false), checked(true/false)
We make the observation that in the Android framework a user action is defined by either registering an appropriate event listener for it or, by inheriting the event-handling method of an Android-framework component. We term the former as registered action and the latter inherited action. We use the registered action as a demonstration here.For both these categories, identifying an action involves three basic steps: identify the place where an action is instantiated or registered; locate the component on which the action would be fired; extract an identifier of the component that the crawler can later use to recognize the corresponding object and fire the action.
Action DetectorWALAstatic-analysis frameworkintent-passing logicCreated based on our knowledge of the Android SDK.Builds a mapping of intent-sending methods and intent filters by analyzing the app’s source code and manifest file.Connects the sub call graphs into a partial connected call graph.Dynamic CrawlerDynamically-created GUI componentsTypically appear in Android containers like ListView, as a list of dynamically-created child components.System-generated GUI componentsTypically have system-defined IDs and predefined actions.
Crawling algorithmThe randomness in the choice of the next action to exploreSide effects of executionSelection of subjectsWe try to avoid bias by including all subjects used to evaluate Android GUITAR and GUI Ripper in previous workManual effortThe only manual work in our approach is to manually select attributes of executable components to compose the visual observable states for the GUI. This effort is a one-time effort for a mobile platform.