Context

Students often misunderstand programming problem descriptions. This can lead them to solve the wrong problem, which creates frustration, harms learning, and hurts their grades. Over the past few years, researchers have found that students can be made to better understand the problem by writing \emph{examples} before they start programming. These examples are checked against correct and wrong implementations provided by course staff, just as in mutation testing. This results in better student understanding of the problem as well as better test suites to accompany the program, both of which are desirable educational outcomes.

Inquiry

Producing these mutant implementations requires care. If there are too many, or they are too obscure, students will end up spending a lot of time on an unproductive task and also become frustrated. Instead, we want a small number mutants that each correspond to \emph{common problem misconceptions}. This paper presents a workflow with partial automation to produce mutants of this form which, notably, are \emph{not} those produced by mutation-testing tools.

Approach

We comb through student tests that \emph{fail} a correct implementation. The student misconceptions are embedded in these failures. We then use methods to \emph{semantically} cluster these failures. These clusters are then translated into ``conceptual mutants''. These can then be run against student data to determine whether we they are better than prior methods. Some of these processes also enjoy automation.

### Knowledge We find that student misconceptions illustrated by failing tests can be operationalized by the above process. The resulting mutants do much better at identifying student misconceptions.

Grounding

Our findings are grounded in the quantitative evaluation of our clustering techniques and generated conceptual mutants. The former compares generated clusters to ground truth using standard cluster-correspondence measures, while the latter examines how conceptual mutants perform against student data.

Importance

Our work contributes a workflow, with some automation, to reduce the cost and increase the effectiveness of generating conceptually interesting mutants. Such mutants can both improve learning outcomes and reduce student frustration, leading to better educational outcomes. In the process, we also identify a variation of mutation testing not commonly discussed in the software literature.