Automata-based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android
Benchmarking software testing tools against known bugs is a classic approach to evaluating the tools’ bug finding abilities. However, this approach is difficult to give some clues on the tool-missed bugs to aid diagnosing the testing tools. As a result, heavy and ad hoc manual analysis is needed. In this work, in the setting of GUI testing for Android apps, we introduce an \emph{automata-based trace analysis} approach to tackling the key challenge of manual analysis, i.e., how to analyze the lengthy event traces generated by a testing tool against a missed bug to find the clues. Our \emph{key} idea is that, we model a bug in the form of a finite automaton which captures its bug-triggering traces; and match the event traces generated by the testing tool (which misses this bug) against this automaton to obtain the clues. Specifically, the clues are presented in the form of three designated automata-based coverage values. We apply our approach to enhance Themis, a representative benchmark suite for Android, to aid diagnosing GUI testing tools. Our extensive evaluation on nine state-of-the-art GUI testing tools and the involvement with several tool developers shows that our approach is \emph{feasible} and \emph{useful}. Our approach enables Themis+ (the enhanced benchmark suite) to provide the clues on the tool-missed bugs, and \emph{all} the Themis+’s clues are identical or useful, compared to the manual analysis results of tool developers. Moreover, the clues have helped find several tool weaknesses, which were unknown or unclear before. Based on the clues, two actively-developing industrial testing tools in our study have quickly made several optimizations and demonstrated their improved bug finding abilities. \emph{All} the tool developers give positive feedback on the usefulness and usability of Themis+’s clues.
Tue 5 DecDisplayed time zone: Pacific Time (US & Canada) change
16:00 - 18:00 | Fault Diagnosis and Root Cause Analysis IResearch Papers / Journal First / Industry Papers at Golden Gate C3 Chair(s): Akond Rahman Auburn University | ||
16:00 15mTalk | [Remote] Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-Modal Observability Data Research Papers Guangba Yu Sun Yat-Sen University, Pengfei Chen Sun Yat-Sen University, Yufeng Li Sun Yat-sen University, Hongyang Chen School of Computer Science and Engineering, Sun Yat-sen University, Xiaoyun Li Sun Yat-sen University, Zibin Zheng Sun Yat-sen University Pre-print | ||
16:15 15mFull-paper | [Remote] DiagConfig: Configuration Diagnosis of Performance Violations in Configurable Software Systems Research Papers Zhiming Chen Sun Yat-sen University, Pengfei Chen Sun Yat-Sen University, Guangba Yu Sun Yat-Sen University, Zilong He Sun Yat-Sen University, Genting Mai Sun Yat-sen University, Peipei Wang ByteDance Infrastructure System Lab Pre-print Media Attached | ||
16:30 15mTalk | [Remote] Pre-training Code Representation with Semantic Flow Graph for Effective Bug Localization Research Papers Media Attached | ||
16:45 15mTalk | [Remote] A Practical Human Labeling Method for Online Just-in-Time Software Defect Prediction Research Papers Liyan Song Southern University of Science and Technology, China, Leandro Minku University of Birmingham, Cong Teng Southern University of Science and Technology, Xin Yao Southern University of Science and Technology Pre-print Media Attached | ||
17:00 15mTalk | Trace Diagnostics for Signal-Based Temporal Properties Journal First Chaima Boufaied University of Ottawa, Claudio Menghi University of Bergamo; McMaster University, Domenico Bianculli University of Luxembourg, Lionel Briand University of Ottawa, Canada / University of Luxembourg, Luxembourg Media Attached | ||
17:15 15mTalk | TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice Systems Industry Papers Ruomeng Ding Microsoft, Chaoyun Zhang Microsoft, Lu Wang Microsoft Research, Yong Xu Microsoft Research, Minghua Ma Microsoft Research, Xiaomin Wu Microsoft, Meng Zhang , Qingjun Chen Microsoft 365, Xin Gao Microsoft 365, Xuedong Gao Microsoft 365, Hao Fan , Saravan Rajmohan Microsoft 365, Qingwei Lin Microsoft, Dongmei Zhang Microsoft Research DOI Media Attached | ||
17:30 15mTalk | Triggering Modes in Spectrum-Based Multi-location Fault Localization Industry Papers DOI Media Attached | ||
17:45 15mTalk | Automata-based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android Research Papers Enze Ma East China Normal University, Shan Huang East China Normal University, weigang he East China Normal University, Ting Su East China Normal University, Jue Wang Nanjing University, Huiyu Liu East China Normal University, Geguang Pu East China Normal University, Zhendong Su ETH Zurich Media Attached |