Assisting Static Analysis with Large Language Models: A ChatGPT Experiment
Recent advances of Large Language Models (LLMs), e.g., ChatGPT, exhibited strong capabilities of comprehending and responding to questions across a variety of domains. Surprisingly, ChatGPT even possesses a strong understanding of program code. In this paper, we investigate where and how LLMs can assist static analysis by asking appropriate questions. In particular, we target a specific bug-finding tool, which produces many false positives from the static analysis. Interestingly, in our evaluation, we find that these false positives can be effectively pruned by asking carefully constructed questions about function-level behaviors or function summaries. Specifically, with a pilot study of 20 false positives, we can successfully prune 8 out of 20 based on GPT-3.5, whereas GPT-4 had a near-perfect result of 16 out of 20, where the four failed ones are not currently considered/supported by our questions, e.g., involving concurrency. Additionally, it also identified one false negative case (a missed bug). We find LLMs a promising tool that can enable a more effective and efficient program analysis.
Thu 7 DecDisplayed time zone: Pacific Time (US & Canada) change
11:00 - 12:30 | Machine Learning IVResearch Papers / Ideas, Visions and Reflections / Industry Papers at Golden Gate C2 Chair(s): Diptikalyan Saha IBM Research India | ||
11:00 15mTalk | Dynamic Data Fault Localization for Deep Neural Networks Research Papers Yining Yin Nanjing University, China, Yang Feng Nanjing University, Shihao Weng Nanjing University, Zixi Liu Nanjing University, Yuan Yao Nanjing University, Yichi Zhang Nanjing University, Zhihong Zhao , Zhenyu Chen Nanjing University Media Attached | ||
11:15 15mTalk | Assisting Static Analysis with Large Language Models: A ChatGPT Experiment Ideas, Visions and Reflections Haonan Li University of California at Riverside, USA, Yu Hao University of California at Riverside, USA, Yizhuo Zhai University of California at Riverside, USA, Zhiyun Qian University of California at Riverside, USA Media Attached | ||
11:30 15mTalk | Understanding the Bug Characteristics and Fix Strategies of Federated Learning Systems Research Papers Xiaohu Du Huazhong University of Science and Technology, Xiao CHEN Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Jialun Cao Hong Kong University of Science and Technology, Ming Wen Huazhong University of Science and Technology, Shing-Chi Cheung Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hai Jin Huazhong University of Science and Technology Media Attached | ||
11:45 15mTalk | EvoCLINICAL: Evolving Cyber-Cyber Digital Twin with Active Transfer Learning for Automated Cancer Registry System Industry Papers Chengjie Lu Simula Research Laboratory; University of Oslo, Xu Qinghua Simula Research Laboratory; University of Oslo, Tao Yue Beihang University, Shaukat Ali Simula Research Laboratory and Oslo Metropolitan University, Thomas Schwitalla Cancer Registry of Norway, Jan F. Nygård Cancer Registry of Norway DOI Media Attached | ||
12:00 15mTalk | Learning Program Semantics for Vulnerability Detection via Vulnerability-specific Inter-procedural Slicing Research Papers bozhi wu Singapore Management University, Shangqing Liu Nanyang Technological University, Yang Xiao Institute of Information Engineering at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Zhiming Li Nanyang Technological University, Singapore, Jun Sun Singapore Management University, Shang-Wei Lin Nanyang Technological University Media Attached | ||
12:15 15mTalk | [Remote] DeepRover: A Query-efficient Blackbox Attack for Deep Neural Networks Research Papers Fuyuan Zhang Kyushu University, Xinwen Hu Hunan Normal University, Lei Ma The University of Tokyo / University of Alberta, Jianjun Zhao Kyushu University Media Attached | ||