Maven – PITest mutation testing example
In this article, we will show you how to use a Maven PIT mutation testing plugin to generate a mutation test coverage report for a Java project.
Tested with
- Maven 3.5.3
- JUnit 5.3.1
- PITest 1.4.3
Line coverage tools like JaCoCo is just telling whether the code is tested or covered, while this PITest mutation coverage tries to tell the effectiveness of the test.
1. Quick – Maven PITest Plugin
1.1 To enable PIT mutation testing, put this pitest-maven
in pom.xml
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.3</version>
<executions>
<execution>
<id>pit-report</id>
<!-- optional, this example attached the goal into mvn test phase -->
<phase>test</phase>
<goals>
<goal>mutationCoverage</goal>
</goals>
</execution>
</executions>
<!-- https://github.com/hcoles/pitest/issues/284 -->
<!-- Need this to support JUnit 5 -->
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>0.8</version>
</dependency>
</dependencies>
<configuration>
<targetClasses>
<param>com.mkyong.examples.*</param>
</targetClasses>
<targetTests>
<param>com.mkyong.examples.*</param>
</targetTests>
</configuration>
</plugin>
1.2 Run the PITest manually.
$ mvn clean org.pitest:pitest-maven:mutationCoverage
1.3 Above pom.xml
file attached the ‘mutationCoverage’ goal to Maven test phase. Now, when we run Maven test, it will trigger the PITest test automatically.
$ mvn clean test
1.4 Report will be generated at target/pit-reports/YYYYMMDDHHMI/*
2. What is Mutation Testing
2.1 The mutation testing is used to measure the effectiveness of the test.
The mutation testing is going to use mutators (switching math operators, change the return type, remove call and etc) to mutate / change the code into different mutations (create new code based on mutators), and check if the unit test will fail for the new mutations (mutation is killed).
The effectiveness of the tests can be a measure of how many mutations are killed.
2.2 For example, this code :
public boolean isPositive(int number) {
boolean result = false;
if (number >= 0) {
result = true;
}
return result;
}
By default, PITest will use different mutators to transform the above code into different mutations (new code) :
#1 Mutation – Changed conditional boundary (mutator)
public boolean isPositive(int number) {
boolean result = false;
// mutator - changed conditional boundary
if (number > 0) {
result = true;
}
return result;
}
#2 Mutation – Negated conditional (mutator)
public boolean isPositive(int number) {
boolean result = false;
// mutator - negated conditional
if (false) {
result = true;
}
return result;
}
#3 Mutation – Replaced return of integer sized value with (x == 0 ? 1 : 0) (mutator)
public boolean isPositive(int number) {
boolean result = false;
if (number > 0) {
result = true;
}
// mutator - (x == 0 ? 1 : 0)
return !result;
}
2.3 A Good unit test should fail (kill) all the mutations #1,#2,#3.
@Test
public void testPositive() {
CalculatorService obj = new CalculatorService();
assertEquals(true, obj.isPositive(10));
}
The above unit test will kill the mutation #2 and #3 (unit test is failed), but the mutation #1 is survived (unit test is passed).
2.4 Review the mutation #1 again. To fail (kill) this test (mutation), we should test the conditional boundary, the number zero.
public boolean isPositive(int number) {
boolean result = false;
// mutator - changed conditional boundary
if (number > 0) {
result = true;
}
return result;
}
Improving the unit test by testing the number zero.
@Test
public void testPositive() {
CalculatorService obj = new CalculatorService();
assertEquals(true, obj.isPositive(10));
//kill mutation #1
assertEquals(true, obj.isPositive(0));
}
Done, 100% mutation coverage now.
3. Maven + PITest Example
Another full Maven + PITest example, just for self reference.
3.1 A full pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mkyong.examples</groupId>
<artifactId>maven-mutation-testing</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<properties>
<!-- https://maven.apache.org/general.html#encoding-warning -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<junit.version>5.3.1</junit.version>
<pitest.version>1.4.3</pitest.version>
</properties>
<dependencies>
<!-- junit 5, unit test -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>maven-mutation-testing</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M1</version>
</plugin>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>${pitest.version}</version>
<executions>
<execution>
<id>pit-report</id>
<phase>test</phase>
<goals>
<goal>mutationCoverage</goal>
</goals>
</execution>
</executions>
<!-- https://github.com/hcoles/pitest/issues/284 -->
<!-- Need this to support JUnit 5 -->
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>0.8</version>
</dependency>
</dependencies>
<configuration>
<targetClasses>
<param>com.mkyong.examples.*</param>
</targetClasses>
<targetTests>
<param>com.mkyong.examples.*</param>
</targetTests>
</configuration>
</plugin>
</plugins>
</build>
</project>
3.2 Source Code
package com.mkyong.examples;
public class StockService {
private int qtyOnHand;
public StockService(int qtyOnHand) {
this.qtyOnHand = qtyOnHand;
}
private void isValidQty(int qty) {
if (qty < 0) {
throw new IllegalArgumentException("Quality should be positive!");
}
}
public int add(int qty) {
isValidQty(qty);
qtyOnHand = qtyOnHand + qty;
return qtyOnHand;
}
public int deduct(int qty) {
isValidQty(qty);
int newQty = qtyOnHand - qty;
if (newQty < 0) {
throw new IllegalArgumentException("Out of Stock!");
} else {
qtyOnHand = newQty;
}
return qtyOnHand;
}
}
3.3 Below unit test will kill all the mutations that are generated by PItest.
package com.mkyong.examples;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class TestStockService {
@DisplayName("Test deduct stock")
@Test
public void testDeduct() {
StockService obj = new StockService(100);
assertEquals(90, obj.deduct(10));
assertEquals(0, obj.deduct(90));
assertEquals(0, obj.deduct(0));
Assertions.assertThrows(IllegalArgumentException.class, () -> {
obj.deduct(-1);
});
Assertions.assertThrows(IllegalArgumentException.class, () -> {
obj.deduct(100);
});
}
@DisplayName("Test add stock")
@Test
public void testAdd() {
StockService obj = new StockService(100);
assertEquals(110, obj.add(10));
assertEquals(110, obj.add(0));
Assertions.assertThrows(IllegalArgumentException.class, () -> {
obj.add(-1);
});
}
}
3.4 Run it.
$ mvn clean org.pitest:pitest-maven:mutationCoverage
#or
$ mvn clean test
3.5 Review the report at target\pit-reports\${YYYYMMDDHHMI}\index.html
4. FAQs
4.1 Study the PITest Mutators, so that we understand how the mutation is generated.
4.2 This mutation testing is a time consuming task, always declares the classes that are needed for the mutation test.
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>${pitest.version}</version>
<configuration>
<targetClasses>
<param>com.mkyong.examples.*Calculator*</param>
<param>com.mkyong.examples.*Stock*</param>
</targetClasses>
<targetTests>
<param>com.mkyong.examples.*</param>
</targetTests>
</configuration>
</plugin>
Download Source Code
$ cd maven-mutation-testing
$ mvn clean org.pitest:pitest-maven:mutationCoverage
#or
$ mvn clean test
# view report at target/pit-reports/YYYYMMDDHHMI/index.html
“mvn clean org.pitest:pitest-maven:mutationCoverage”
causes an error:
“[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.3:mutationCoverage (default-cli) on project maven-mutation-testing: Execution default-cli of goal org.pitest:pitest-maven:1.4.3:mutationCoverage failed: No mutations found. This probably means there is an issue with either the supplied classpath or filters.”
it seems that the “target” folder should not be deleted
fix:
“mvn clean package org.pitest:pitest-maven:mutationCoverage”
[INFO] Mutating from C:\Example\maven-examples\maven-mutation-testing\target\classes
11:25:44 AM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
11:25:45 AM PIT >> INFO : MINION : Error: Could not find or load main class org.pitest.coverage.execute.CoverageMinion
Caused by: java.lang.ClassNotFoundException: org.pitest.coverage.execute.CoverageMinion
Will PITest create the mutant code on its own from the actual source code or the tester needs to create mutants explicitly?
yes.. based on mutators configured