DeepDebugger: An Interactive Time-Travelling Debugging Approach for Deep Classifiers
A deep classifier is usually trained to (i) learn the numeric representation vector of samples and (ii) classify sample representations with learned classification boundaries. Time-travelling visualization, as an explainable AI technique, is designed to transform the model training dynamics into an animation of canvas with colorful dots and territories. Despite that the training dynamics of the high-level concepts such as sample representations and classification boundaries are now observable, the model developers can still be overwhelmed by tens of thousands of moving dots in hundreds of training epochs (i.e., frames in the animation), which makes them miss important training events such as abnormal movement dynamics (i.e., learning behavior) of certain samples.
In this work, we make the first attempt to develop the model time-travelling visualizers to the model time-travelling debuggers, for its practical use in model debugging tasks. Specifically, given an animation of model training dynamics of sample representation and classification landscape, we propose DeepDebugger solution to recommend the samples of user interest in a human-in-the-loop manner. On one hand, DeepDebugger monitors the training dynamics of samples and recommends suspicious samples based on the abnormality of their training dynamics and model prediction. On the other hand, our recommendation is interactive and fault-resilient for the model developers to explore the training process. By learning users’ feedback, DeepDebugger refines its recommendation to fit their intention. Our extensive experiments on applying DeepDebugger on the known time-travelling visualizers show that DeepDebugger can (1) detect the majority of the abnormal movement of the training samples on canvas; (2) significantly boost the recommendation performance of samples of interest (5-10X more accurate than the baselines) with the runtime overhead of 0.015s per feedback; (3) be resilient under the 3%, 5%, 10% mistaken user feedback. Our user study, consisting of 16 participants on two model debugging tasks, shows that the interactive recommendation of DeepDebugger can help the participants accomplish the debugging tasks by saving 18.1% completion time or boosting the performance by 20.3%.
Wed 6 DecDisplayed time zone: Pacific Time (US & Canada) change
16:00 - 18:00 | Fault Diagnosis and Root Cause Analysis IIIndustry Papers / Research Papers at Golden Gate A Chair(s): Yun Lin Shanghai Jiao Tong University | ||
16:00 15mTalk | DeepDebugger: An Interactive Time-Travelling Debugging Approach for Deep Classifiers Research Papers Xianglin Yang Shanghai Jiao Tong University; National University of Singapore, Yun Lin Shanghai Jiao Tong University, Yifan Zhang National University of Singapore, Linpeng Huang Shanghai Jiao Tong University, Jin Song Dong National University of Singapore, Hong Mei Peking University Media Attached | ||
16:15 15mTalk | AG3: Automated Game GUI Text Glitch Detection Based on Computer Vision Industry Papers Xiaoyun Liang ByteDance, Jiayi Qi ByteDance, Yongqiang Gao ByteDance, Chao Peng ByteDance, China, Ping Yang Bytedance Network Technology DOI Media Attached | ||
16:30 15mTalk | TransMap: Pinpointing Mistakes in Neural Code Translation Research Papers Bo Wang National University of Singapore, Ruishi Li National University of Singapore, Mingkai Li National University of Singapore, Prateek Saxena National University of Singapore Media Attached | ||
16:45 15mTalk | Dynamic Prediction of Delays in Software Projects Using Delay Patterns and Bayesian Modeling Research Papers Elvan Kula Delft University of Technology, Eric Greuter ING, Arie van Deursen Delft University of Technology, Georgios Gousios Endor Labs & Delft University of Technology Pre-print Media Attached | ||
17:00 15mTalk | Commit-level, Neural Vulnerability Detection and Assessment Research Papers Yi Li New Jersey Institute of Technology, Aashish Yadavally The University of Texas at Dallas, Jiaxing Zhang New Jersey Institute of Technology, Shaohua Wang Central University of Finance and Economics , Tien N. Nguyen University of Texas at Dallas Media Attached | ||
17:15 15mTalk | [Remote] Mining Resource-Operation Knowledge to Support Resource Leak Detection Research Papers Chong Wang Nanyang Technological University, Yiling Lou Fudan University, Xin Peng Fudan University, Jianan Liu Fudan University, Baihan Zou Fudan University Media Attached | ||
17:30 15mTalk | [Remote] Detection Is Better Than Cure: A Cloud Incidents Perspective Industry Papers Vaibhav Ganatra Microsoft, Anjaly Parayil Microsoft, Supriyo Ghosh Microsoft, Yu Kang Microsoft Research, Minghua Ma Microsoft Research, Chetan Bansal Microsoft Research, Suman Nath Microsoft Research, Jonathan Mace Microsoft DOI Media Attached | ||
17:45 7mTalk | [Remote] Diffusion-Based Time Series Data Imputation for Cloud Failure Prediction at Microsoft 365 Industry Papers Fangkai Yang Microsoft Research, Wenjie Yin KTH Royal Institute of Technology, Lu Wang Microsoft Research, Tianci Li Microsoft, Pu Zhao Microsoft Research, Bo Liu Beijing Institute of Technology, Paul Wang Microsoft 365, Bo Qiao Microsoft Research, Yudong Liu Microsoft Research, Mårten Björkman KTH Royal Institute of Technology, Saravan Rajmohan Microsoft 365, Qingwei Lin Microsoft, Dongmei Zhang Microsoft Research DOI Media Attached | ||