Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants
Mutation testing has been demonstrated to be one of the most powerful fault-revealing tools in the tester’s tool kit. Much previous work implicitly assumed it to be sufficient to re-compute mutant suites per release. Sadly, this makes mutation results inconsistent; mutant scores from each release cannot be directly compared, making it harder to measure test improvement. Furthermore, regular code change means that a mutant suite’s relevance will naturally degrade over time. We measure this degradation in relevance for 143,500 mutants in 4 non-trivial systems finding that, on overage, 52% degrade. We introduce a mutant brittleness measure and use it to audit software systems and their mutation suites. We also demonstrate how consistent-by-construction long-standing mutant suites can be identified with a 10x improvement in mutant relevance over an arbitrary test suite. Our results indicate that the research community should avoid the re-computation of mutant suites and focus, instead, on long-standing mutants, thereby improving the consistency and relevance of mutation testing.
Tue 5 DecDisplayed time zone: Pacific Time (US & Canada) change
11:00 - 12:30 | Testing IIdeas, Visions and Reflections / Research Papers / Journal First / Industry Papers at Golden Gate C1 Chair(s): Marcelo d'Amorim North Carolina State University | ||
11:00 15mTalk | [Remote] CAmpactor: A Novel and Effective Local Search Algorithm for Optimizing Pairwise Covering Arrays Research Papers Qiyuan Zhao Beihang University, Chuan Luo Beihang University, Shaowei Cai Institute of Software, Chinese Academy of Sciences, Wei Wu L3S Research Center, Leibniz University Hannover, Germany, Jinkun Lin Seed Math Technology Limited, Hongyu Zhang Chongqing University, Chunming Hu Beihang University DOI Pre-print Media Attached | ||
11:15 15mTalk | Accelerating Continuous Integration with Parallel Batch Testing Research Papers Emad Fallahzadeh Concordia University, Amir Hossein Bavand Concordia University, Peter Rigby Concordia University; Meta Pre-print Media Attached | ||
11:30 15mTalk | Keeping Mutation Test Suites Consistent and Relevant with Long-Standing Mutants Ideas, Visions and Reflections Milos Ojdanic University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Mark Harman Meta Platforms Inc. and UCL Media Attached | ||
11:45 15mTalk | DistXplore: Distribution-guided Testing for Evaluating and Enhancing Deep Learning Systems Research Papers Longtian Wang Xi'an Jiaotong University, Xiaofei Xie Singapore Management University, Xiaoning Du Monash University, Australia, Meng Tian Singapore Management University, Qing Guo IHPC and CFAR at A*STAR, Singapore, Yang Zheng TTE Lab, Huawei, Chao Shen Xi’an Jiaotong University Media Attached | ||
12:00 15mTalk | Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing Journal First Swaroopa Dola University of Virginia, Matthew B Dwyer University of Virginia, Mary Lou Soffa University of Virginia Media Attached | ||
12:15 15mTalk | A Unified Framework for Mini-game Testing: Experience on WeChat Industry Papers Chaozheng Wang The Chinese University of Hong Kong, Haochuan Lu Tencent, Cuiyun Gao The Chinese University of Hong Kong, Li Zongjie Hong Kong University of Science and Technology, Ting Xiong Tencent Inc., Yuetang Deng Tencent Inc. DOI Media Attached |