MATmute was used to conduct a series of experiments that explore
Mutation Sensitivity Testing. For more information see the thesis
available here. A
condensed version of the same results can be found in the November/December
2009 issue of Computing in
Science and Engineering.
MATmute was demoed at CASCON
2008.
You can download a PDF of the (informative) poster here.
What
is
a code mutator and how can it help me?
A code mutator is a program which changes (i.e., mutates) source
code in a well-defined manner. When MATmute is asked to mutate an
.m file, it reads and parses parts of the target file and then
generates new .m
files which each contain one code mutation. These .m files are
known as mutants.
These mutants can be very useful for at least two reasons:
- If a piece of software is being well tested then coding mistakes
should be caught during the testing process. Therefore, that same
testing procedure should be able to detect (i.e., "kill") any mutants
which contain mistakes. However, if the testing procedure is
unable to detect the difference between the original programs and the
mutants then it illustrates a weakness in the testing procedure.
(In rare cases, a mutant may be functionally equivalent to the original
program, and, in that case, there is no flaw in the mutant for the
tests to detect. However, functionally equivalent mutants are
rare.)
- It is likely that every non-trivial sample of code has some
mistakes. Ideally one would prefer to eliminate all of these
mistakes, but because of the sheer amount of effort required to produce
fully correct code that approach is not usually feasible.
Mutation testing allows the code developer to perform a sensitivity
analysis on the code to quantify how certain classes of coding mistakes
effect the accuracy of their program.
Why is MATmute targeted at
computational scientists?
Code mutators have existed since the 1970s, but it has largely
remain a research tool used by software engineers who are developing
and evaluating testing methodologies for "traditional" software.
However, some researchers feel that mutation testing could be put to
valuable use by computational scientists who are in desperate need of
techniques to help them assess the quality of their code.
How does MATmute work?
MATmute consists of two main components. A set of Python
modules which are used to generate the mutant .m files, and some
.m files which are used by MATLAB in order to run the generated mutants.
When the
matmute command is
called in MATLAB it calls the Python modules with the name of the .m
file that is to be mutated. The mutants are then generated and
saved before control is passed back to the MATLAB function. This
function then uses the given tests to execute the
mutants. The ouputs from the tests run on the mutants are
compared with the outputs from the tests run on the original code to
measure the relative error that is introduced by the mutation.
The person who is testing the
software may then decide to alter or
augment their testing strategy by examining these results.
A similar procedure could be carried out by having a outside person
(someone who is not the tester) introduce mistakes in the code to see
if the tester is able to detect them. However, MATmute automates
this process is a way which is
reproducible
-- a feature that is particularly important in the context of
computational
science.
Project hosted by