![Page 1: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/1.jpg)
StoryDroid: Automated Generation of Storyboard for Android Apps
Sen Chen, Lingling Fan,
Chunyang Chen, Ting Su, Wenhe Li,
Yang Liu, Lihua Xu
ICSE 2019Montréal, QC, Canada
![Page 2: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/2.jpg)
2
Over 3.8 million Android apps are available at Google Play Store.One single app category usually contains many similar Android apps.
Mobile apps are continuously increasing
One Single App Category
![Page 3: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/3.jpg)
3
Searching apps from Google Play StoreSocial Media Apps
Mobile apps are facing fierce competitions.
![Page 4: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/4.jpg)
4
Before developing a new mobile app
Development team usually endeavors painstaking efforts to review many existing apps (i.e., competitive analysis) with similar purposes.
![Page 5: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/5.jpg)
5
Similar purposes of competitive analysis
helps understand the competitors’ strengths and weaknesses
helps developers gain more insights on the actual implementation
reduces market risks before development
![Page 6: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/6.jpg)
6
Reviewing apps for competitive analysis
Eve: Product Manager
Bob: AppDeveloper
Functionalities
Login Main
Alice: UX/UIDesigner
UI design and layout code
Logic code
Login Layout Code
Login Activity Code
![Page 7: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/7.jpg)
However…
7
How to improve the reviewing efficiency?
Manual exploration of hundreds of existing similar apps by different roles is ineffective.
✔ It is a time-consuming task.
✔ it is difficult to explore all the functionalities.
![Page 8: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/8.jpg)
Storyboard
8
Storyboard of movies Storyboard of apps
![Page 9: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/9.jpg)
9
Storyboard of Android Apps
*ATG: Activity Transition Graph
![Page 10: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/10.jpg)
Challenges
10
✔ ATGs* are usually incomplete due to the limitation of current static analysis tools (e.g., IC3 [1])
✔ A pure static approach may miss parts of UIs that are dynamically rendered.
✔ The obfuscated activity names lack the semantics of the corresponding functionalities.
[1] Octeau et al., “Composite constant propagation: Application to Android intercomponent communication analysis,” in ICSE 2015.
![Page 11: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/11.jpg)
Our Solution - StoryDroid
11
1. ATG Extraction2. UI Page Rendering3. Semantic Name Inferring
![Page 12: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/12.jpg)
StoryDroid – Activity Transition Extraction
12
StoryDroid Enhances the ATG extraction ability of IC3, especially for fragments and inner classes.
StoryDroid leverages control- and data-flow analysis to obtain relatively complete ATG.
TransitionExtraction
ATG
![Page 13: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/13.jpg)
Activity Transition Extraction
13
TransitionExtraction
ATG
Main -> Reg Main -> PrefEditor -> AdvancedPrefEditor SearchPanel -> PartList
![Page 14: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/14.jpg)
StoryDroid – UI Page Rendering
14
StoryDroid translates dynamic and hybrid layouts to static layout (if needed) to render UI pages that users interact with.
Layout Type Checking
Layout Type Translating
UI Page
Rendering
![Page 15: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/15.jpg)
UI Page Rendering
15
Burger
Test
Layout Type Translation
Adapter data rendering
Layout Type Checking
Layout Type Translating
UI Page
Rendering
ListViewLayouts
![Page 16: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/16.jpg)
StoryDroid – Semantic Name Inferring
16
StoryDroid infers the semantic name for the obfuscated activity names by layout comparison.
Semantic Name Inferring
ATG
b
a
Main
LoginActivity
RegActivity
![Page 17: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/17.jpg)
17
Semantic Name Inferring
ATG
StoryDroid – Semantic Name Inferring
Layout Tree DB:4,426 F-Droid apps13,792 layout trees
Layout Tree:
Top 10 candidates
![Page 18: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/18.jpg)
18
Automated Generation of Storyboard
![Page 19: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/19.jpg)
Effectiveness Evaluation - Research Questions
● RQ1: Can StoryDroid extract a more complete ATG for an app, and achieve better activity coverage than the dynamic testing tool (i.e., Stoat)?
● RQ2: Can StoryDroid render UI pages with high similarity compared with the real screenshots?
● RQ3: Can StoryDroid infer semantic names for obfuscated activities?
19
![Page 20: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/20.jpg)
Effectiveness Evaluation - RQ1
20
Can StoryDroid extract a more complete ATG for an app, and achieve better activity coverage than the dynamic testing tool (i.e., Stoat)?
![Page 21: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/21.jpg)
Effectiveness Evaluation - RQ1
21
- Self-developed 10 apps- Capability of handling fragments and inner classes
![Page 22: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/22.jpg)
Effectiveness Evaluation - RQ1
22
StoryDroid outperforms IC3 on ATG extraction and covers 2 times more activities than Stoat with less time.
- 100 apps (i.e., 50 F-Droid and 50 Google Play)- Transitions pairs and activity coverage
87% 74%
![Page 23: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/23.jpg)
Effectiveness Evaluation - RQ2
23
Can StoryDroid render UI pages with high similarity compared with the real screenshots?
![Page 24: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/24.jpg)
Effectiveness Evaluation - RQ2
24
StoryDroid can render UI pages with high similarity (84%) to the real ones in our experiments.
- Similarity compared with the real screenshots- Metrics: MSE (Mean Squared Error) & MAE (Mean Absolute Error)
![Page 25: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/25.jpg)
Effectiveness Evaluation - RQ3
25
Can StoryDroid infer semantic names for obfuscated activities?
![Page 26: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/26.jpg)
Effectiveness Evaluation - RQ3
26
StoryDroid can infer semantic names with high accuracy for obfuscated activity names.
- 92 out of 100 activity names
![Page 27: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/27.jpg)
Usefulness Evaluation
27
Whether StoryDroid can help explore and understand the functionalities of apps effectively?
![Page 28: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/28.jpg)
Usefulness Evaluation – User Study
28
1. 4 apps with different number of activities (12-15 activities) from 2 categories (i.e., finance and tool), each category contains 2 apps
2. 8 participants including post-docs, Ph.D., and masters
3. For each category, each participant explored one app with StoryDroid, and the other without StoryDroid.
Compared with manual exploration, StoryDroid achieves 2 times more activity coverage with less time cost to help understand the app functionalities.
![Page 29: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/29.jpg)
Future applications based on StoryDroid
29
✔ Recommendation of UI design and code
✔ Guiding regression testing of apps
✔ Extracting commonalities across apps
![Page 30: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/30.jpg)
30
Summary
![Page 31: Automated Generation of Storyboard for Android Apps · Manual exploration of hundreds of existing similar apps by different roles is ineffective. It is a time-consuming task. it is](https://reader036.vdocument.in/reader036/viewer/2022071018/5fd2537fe729bb6a0641c8e5/html5/thumbnails/31.jpg)
31
Sen Chen
Nanyang Technological University
Contact: [email protected]
Homepage: https://sen-chen.github.io/
Thanks and Questions?
NTU, Singapore
Cyber Security Lab