Main Tutorials

Maven – PITest mutation testing example

In this article, we will show you how to use a Maven PIT mutation testing plugin to generate a mutation test coverage report for a Java project.

Tested with

  1. Maven 3.5.3
  2. JUnit 5.3.1
  3. PITest 1.4.3
Note
Line coverage tools like JaCoCo is just telling whether the code is tested or covered, while this PITest mutation coverage tries to tell the effectiveness of the test.

1. Quick – Maven PITest Plugin

1.1 To enable PIT mutation testing, put this pitest-maven in pom.xml

pom.xml

	<plugin>
		<groupId>org.pitest</groupId>
		<artifactId>pitest-maven</artifactId>
		<version>1.4.3</version>

		<executions>
			<execution>
				<id>pit-report</id>
				<!-- optional, this example attached the goal into mvn test phase -->
				<phase>test</phase>
				<goals>
					<goal>mutationCoverage</goal>
				</goals>
			</execution>
		</executions>

		<!-- https://github.com/hcoles/pitest/issues/284 -->
		<!-- Need this to support JUnit 5 -->
		<dependencies>
			<dependency>
				<groupId>org.pitest</groupId>
				<artifactId>pitest-junit5-plugin</artifactId>
				<version>0.8</version>
			</dependency>
		</dependencies>
		<configuration>
			<targetClasses>
				<param>com.mkyong.examples.*</param>
			</targetClasses>
			<targetTests>
				<param>com.mkyong.examples.*</param>
			</targetTests>
		</configuration>
	</plugin>

1.2 Run the PITest manually.


$ mvn clean org.pitest:pitest-maven:mutationCoverage

1.3 Above pom.xml file attached the ‘mutationCoverage’ goal to Maven test phase. Now, when we run Maven test, it will trigger the PITest test automatically.


$ mvn clean test

1.4 Report will be generated at target/pit-reports/YYYYMMDDHHMI/*

2. What is Mutation Testing

2.1 The mutation testing is used to measure the effectiveness of the test.

The mutation testing is going to use mutators (switching math operators, change the return type, remove call and etc) to mutate / change the code into different mutations (create new code based on mutators), and check if the unit test will fail for the new mutations (mutation is killed).

The effectiveness of the tests can be a measure of how many mutations are killed.

2.2 For example, this code :


	public boolean isPositive(int number) {

        boolean result = false;
        if (number >= 0) {
            result = true;
        }
        return result;
        
    }

By default, PITest will use different mutators to transform the above code into different mutations (new code) :

#1 Mutation – Changed conditional boundary (mutator)


	public boolean isPositive(int number) {

        boolean result = false;
		// mutator - changed conditional boundary
        if (number > 0) {
            result = true;
        }
        return result;
        
    }

#2 Mutation – Negated conditional (mutator)


	public boolean isPositive(int number) {

        boolean result = false;
		// mutator - negated conditional
        if (false) {
            result = true;
        }
        return result;
        
    }

#3 Mutation – Replaced return of integer sized value with (x == 0 ? 1 : 0) (mutator)


	public boolean isPositive(int number) {

        boolean result = false;
        if (number > 0) {
            result = true;
        }
		
		// mutator - (x == 0 ? 1 : 0)
        return !result;
        
    }

2.3 A Good unit test should fail (kill) all the mutations #1,#2,#3.


	@Test
    public void testPositive() {

        CalculatorService obj = new CalculatorService();
        assertEquals(true, obj.isPositive(10));

    }

The above unit test will kill the mutation #2 and #3 (unit test is failed), but the mutation #1 is survived (unit test is passed).

2.4 Review the mutation #1 again. To fail (kill) this test (mutation), we should test the conditional boundary, the number zero.


	public boolean isPositive(int number) {

        boolean result = false;
		// mutator - changed conditional boundary
        if (number > 0) {
            result = true;
        }
        return result;
        
    }

Improving the unit test by testing the number zero.


	@Test
    public void testPositive() {

        CalculatorService obj = new CalculatorService();
        assertEquals(true, obj.isPositive(10));
		//kill mutation #1
        assertEquals(true, obj.isPositive(0));

    }

Done, 100% mutation coverage now.

3. Maven + PITest Example

Another full Maven + PITest example, just for self reference.

3.1 A full pom.xml

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
		 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mkyong.examples</groupId>
    <artifactId>maven-mutation-testing</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <!-- https://maven.apache.org/general.html#encoding-warning -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <junit.version>5.3.1</junit.version>
        <pitest.version>1.4.3</pitest.version>
    </properties>

    <dependencies>

        <!-- junit 5, unit test -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

    </dependencies>
    <build>
        <finalName>maven-mutation-testing</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M1</version>
            </plugin>

            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>${pitest.version}</version>

                <executions>
                    <execution>
                        <id>pit-report</id>
                        <phase>test</phase>
                        <goals>
                            <goal>mutationCoverage</goal>
                        </goals>
                    </execution>
                </executions>

                <!-- https://github.com/hcoles/pitest/issues/284 -->
                <!-- Need this to support JUnit 5 -->
                <dependencies>
                    <dependency>
                        <groupId>org.pitest</groupId>
                        <artifactId>pitest-junit5-plugin</artifactId>
                        <version>0.8</version>
                    </dependency>
                </dependencies>
                <configuration>
                    <targetClasses>
                        <param>com.mkyong.examples.*</param>
                    </targetClasses>
                    <targetTests>
                        <param>com.mkyong.examples.*</param>
                    </targetTests>
                </configuration>
            </plugin>

        </plugins>
    </build>

</project>

3.2 Source Code

StockService.java

package com.mkyong.examples;

public class StockService {

    private int qtyOnHand;

    public StockService(int qtyOnHand) {
        this.qtyOnHand = qtyOnHand;
    }

    private void isValidQty(int qty) {
        if (qty < 0) {
            throw new IllegalArgumentException("Quality should be positive!");
        }
    }

    public int add(int qty) {

        isValidQty(qty);
        qtyOnHand = qtyOnHand + qty;
        return qtyOnHand;

    }

    public int deduct(int qty) {

        isValidQty(qty);

        int newQty = qtyOnHand - qty;
        if (newQty < 0) {
            throw new IllegalArgumentException("Out of Stock!");
        } else {
            qtyOnHand = newQty;
        }
        return qtyOnHand;

    }

}

3.3 Below unit test will kill all the mutations that are generated by PItest.

TestStockService.java

package com.mkyong.examples;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;

public class TestStockService {

    @DisplayName("Test deduct stock")
    @Test
    public void testDeduct() {
        StockService obj = new StockService(100);
        assertEquals(90, obj.deduct(10));
        assertEquals(0, obj.deduct(90));
        assertEquals(0, obj.deduct(0));

        Assertions.assertThrows(IllegalArgumentException.class, () -> {
            obj.deduct(-1);
        });

        Assertions.assertThrows(IllegalArgumentException.class, () -> {
            obj.deduct(100);
        });

    }

    @DisplayName("Test add stock")
    @Test
    public void testAdd() {
        StockService obj = new StockService(100);
        assertEquals(110, obj.add(10));
        assertEquals(110, obj.add(0));

        Assertions.assertThrows(IllegalArgumentException.class, () -> {
            obj.add(-1);
        });
    }


}

3.4 Run it.


$ mvn clean org.pitest:pitest-maven:mutationCoverage

#or 

$ mvn clean test

3.5 Review the report at target\pit-reports\${YYYYMMDDHHMI}\index.html

4. FAQs

4.1 Study the PITest Mutators, so that we understand how the mutation is generated.

4.2 This mutation testing is a time consuming task, always declares the classes that are needed for the mutation test.

pom.xml

	<plugin>
		<groupId>org.pitest</groupId>
		<artifactId>pitest-maven</artifactId>
		<version>${pitest.version}</version>

		<configuration>
			<targetClasses>
				<param>com.mkyong.examples.*Calculator*</param>
				<param>com.mkyong.examples.*Stock*</param>
			</targetClasses>
			<targetTests>
				<param>com.mkyong.examples.*</param>
			</targetTests>
		</configuration>
	</plugin>

Download Source Code

$ git clone https://github.com/mkyong/maven-examples.git
$ cd maven-mutation-testing

$ mvn clean org.pitest:pitest-maven:mutationCoverage
#or
$ mvn clean test

# view report at target/pit-reports/YYYYMMDDHHMI/index.html

References

  1. PITest mutation testing
  2. Wikipedia – Mutation testing
  3. Maven – JaCoCo code coverage example

About Author

author image
Founder of Mkyong.com, love Java and open source stuff. Follow him on Twitter. If you like my tutorials, consider make a donation to these charities.

Comments

Subscribe
Notify of
4 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
JohnKowalsky
2 years ago

“mvn clean org.pitest:pitest-maven:mutationCoverage”

causes an error:
“[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.3:mutationCoverage (default-cli) on project maven-mutation-testing: Execution default-cli of goal org.pitest:pitest-maven:1.4.3:mutationCoverage failed: No mutations found. This probably means there is an issue with either the supplied classpath or filters.”

it seems that the “target” folder should not be deleted

fix:
“mvn clean package org.pitest:pitest-maven:mutationCoverage”

Sandeep
3 years ago

[INFO] Mutating from C:\Example\maven-examples\maven-mutation-testing\target\classes
11:25:44 AM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
11:25:45 AM PIT >> INFO : MINION : Error: Could not find or load main class org.pitest.coverage.execute.CoverageMinion
Caused by: java.lang.ClassNotFoundException: org.pitest.coverage.execute.CoverageMinion

amreeta
4 years ago

Will PITest create the mutant code on its own from the actual source code or the tester needs to create mutants explicitly?

Mohsin Iqbal
4 years ago
Reply to  amreeta

yes.. based on mutators configured