Appaction: Automatic GUI Interaction for Mobile Apps via Holistic Widget Perception
In industrial practice, GUI (Graphic User Interface) testing of mobile apps still inevitably relies on huge manual efforts. The major efforts are those on understanding the GUIs, so that testing scripts can be written accordingly. Quality assurance could therefore be very labor-intensive, especially for modern commercial mobile apps, where one may include tremendous, diverse, and complex GUIs, e.g., those for placing orders of different commercial items. To reduce such human efforts, we propose Appaction, a learning-based automatic GUI interaction approach we developed for Meituan, one of the largest E-commerce providers with over 600 million users. Appaction can automatically analyze the target GUI and understand what each input of the GUI is about, so that corresponding valid inputs can be entered accordingly. To this end, Appaction adopts a multi-modal model to learn from human experiences in perceiving a GUI. This allows it to infer corresponding valid input events that can properly interact with the GUI. In this way, the target app can be effectively exercised. We present our experiences in Meituan on applying Appaction to popular commercial apps. We demonstrate the effectiveness of Appaction in GUI analysis, and it can perform correct interactions for numerous form pages.
Wed 6 DecDisplayed time zone: Pacific Time (US & Canada) change
11:00 - 12:30 | Testing IIIIndustry Papers / Demonstrations / Research Papers at Golden Gate C1 Chair(s): Tianyi Zhang Purdue University | ||
11:00 15mTalk | [Remote] Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support Research Papers Yibiao Yang State Key Laboratory for Novel Software Technology, Nanjing University, Maolin Sun Nanjing University, Yang Wang National Key Laboratory for Novel Software Technology, Nanjing University, Qingyang Li National Key Laboratory for Novel Software Technology, Nanjing University, Ming Wen Huazhong University of Science and Technology, Yuming Zhou Nanjing University Pre-print Media Attached | ||
11:15 7mTalk | [Remote] Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned Industry Papers Hassan Sartaj Simula Research Laboratory, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Tao Yue Beihang University, Kjetil Moberg Norwegian Health Authority DOI Pre-print Media Attached | ||
11:23 7mTalk | Helion: Enabling Natural Testing of Smart Homes Demonstrations Prianka Mandal William & Mary, Sunil Manandhar IBM T.J. Watson Research Center, Kaushal Kafle College of William & Mary, Kevin Moran University of Central Florida, Denys Poshyvanyk William & Mary, Adwait Nadkarni William & Mary Media Attached | ||
11:30 15mTalk | NeuRI: Diversifying DNN Generation via Inductive Rule Inference Research Papers Jiawei Liu University of Illinois at Urbana-Champaign, Jinjun Peng Columbia University, Yuyao Wang Nanjing University, Lingming Zhang University of Illinois at Urbana-Champaign Pre-print Media Attached | ||
11:45 15mTalk | Appaction: Automatic GUI Interaction for Mobile Apps via Holistic Widget Perception Industry Papers Yongxiang Hu Fudan University, China, Jiazhen Gu Fudan University, China, Shuqing Hu Fudan University, Yu Zhang Meituan, Wenjie Tian Meituan, Shiyu Guo Meituan, Chaoyi Chen Meituan, Yangfan Zhou Fudan University DOI Media Attached | ||
12:00 15mTalk | MuRS: Mutant Ranking and Suppression using Identifier Templates Industry Papers Zimin Chen KTH Royal Institute of Technology, Malgorzata Salawa Google, Manushree Vijayvergiya Google, Goran Petrović Google Inc, Marko Ivanković Google; Universität Passau, René Just University of Washington DOI Media Attached | ||
12:15 15mTalk | Outage-Watch: Early Prediction of Outages using Extreme Event Regularizer Research Papers Shubham Agarwal Adobe Research, Sarthak Chakraborty Adobe Research, Shaddy Garg Adobe Research, Sumit Bisht Amazon, Chahat Jain Traceable.ai, Ashritha Gonuguntla Cisco, Shiv Saini Adobe Research Media Attached |