‹Programming› 2024
Mon 11 - Fri 15 March 2024 Lund, Sweden
Thu 14 Mar 2024 13:30 - 14:00 at M:Teknodromen - Research Papers 5 Chair(s): Shigeru Chiba


Students often misunderstand programming problem descriptions. This can lead them to solve the wrong problem, which creates frustration, harms learning, and hurts their grades. Over the past few years, researchers have found that students can be made to better understand the problem by writing \emph{examples} before they start programming. These examples are checked against correct and wrong implementations provided by course staff, just as in mutation testing. This results in better student understanding of the problem as well as better test suites to accompany the program, both of which are desirable educational outcomes.


Producing these mutant implementations requires care. If there are too many, or they are too obscure, students will end up spending a lot of time on an unproductive task and also become frustrated. Instead, we want a small number mutants that each correspond to \emph{common problem misconceptions}. This paper presents a workflow with partial automation to produce mutants of this form which, notably, are \emph{not} those produced by mutation-testing tools.


We comb through student tests that \emph{fail} a correct implementation. The student misconceptions are embedded in these failures. We then use methods to \emph{semantically} cluster these failures. These clusters are then translated into ``conceptual mutants''. These can then be run against student data to determine whether we they are better than prior methods. Some of these processes also enjoy automation.

### Knowledge We find that student misconceptions illustrated by failing tests can be operationalized by the above process. The resulting mutants do much better at identifying student misconceptions.


Our findings are grounded in the quantitative evaluation of our clustering techniques and generated conceptual mutants. The former compares generated clusters to ground truth using standard cluster-correspondence measures, while the latter examines how conceptual mutants perform against student data.


Our work contributes a workflow, with some automation, to reduce the cost and increase the effectiveness of generating conceptually interesting mutants. Such mutants can both improve learning outcomes and reduce student frustration, leading to better educational outcomes. In the process, we also identify a variation of mutation testing not commonly discussed in the software literature.

Thu 14 Mar

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

13:30 - 14:30
Research Papers 5Research Papers at M:Teknodromen
Chair(s): Shigeru Chiba The University of Tokyo
Conceptual Mutation Testing for Student Programming MisconceptionsVol. 8
Research Papers
Siddhartha Prasad Brown University, Ben Greenman Brown University, USA, Tim Nelson Brown University, Shriram Krishnamurthi Brown University
Link to publication DOI
Arrays in Practice, An Empirical Study of Array Access Patterns on the JVMVol. 8
Research Papers
Beatrice Åkerblom Stockholm University, Elias Castegren Uppsala University
Link to publication DOI