‹Programming› 2024
Mon 11 - Fri 15 March 2024 Lund, Sweden
Tue 12 Mar 2024 13:15 - 13:45 at M:Teknodromen - Presentations and Panel

Feedback during programming is desirable, but its usefulness depends on immediacy and relevance to the task. Unit and regression testing are practices to ensure programmers can obtain feedback on their changes; however, running a large test suite is rarely fast, and only a few results are relevant.

Identifying which tests in a test suite are most relevant to a change helps detect defects earlier during programming and selecting tests that serve as examples to help programmers understand particular code.

In this work, we describe an approach to evaluate how well large language models (LLMs) and embedding models can judge the relevance of a test to a change. We construct a dataset by applying faulty variations of real-world code changes and measuring whether the model could nominate the failing tests beforehand.

We found that, while embedding models perform best on such a task, even simple information retrieval models are surprisingly competitive. In contrast, pre-trained LLMs are of limited use as they focus on confounding aspects like coding styles.

We argue that the high computational cost of AI models is not always justified, and tool developers should also consider non-AI models for code-related retrieval and recommendation tasks. Lastly, we generalize from unit tests to live examples and outline how our approach can benefit live programming environments.

Tue 12 Mar

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

13:15 - 15:00
Presentations and PanelProgramming with AI at M:Teknodromen
Faster Feedback with AI? -- A Test Prioritization Study
Programming with AI
Toni Mattis University of Potsdam; Hasso Plattner Institute, Lukas Böhme Hasso Plattner Institute, University of Potsdam, Potsdam, Germany, Eva Krebs Hasso Plattner Institute (HPI), University of Potsdam, Germany, Martin C. Rinard Massachusetts Institute of Technology, Robert Hirschfeld University of Potsdam; Hasso Plattner Institute
Extrapolating a programmer career - from Vim to LLM and beyond
Programming with AI
Industry Panel
Programming with AI
Markus Borg CodeScene, Gustaf Lundh Axis Communications, Mikael Lindberg Saab Kockums