Recently there was an interesting and unusual event in Warsaw for all enthusiasts of testing – “6 tastes of testing – flashtalks“. Instead of one long presentation common for WJUG meetings 6 guys gave 6 flashtalks (~15 minutes long presentations) about different aspects of testing.

The audience could listen about:

  • Fest Assertions – a set of assertions with fluent API for TestNG and JUnit by Michał Lewandowski,
  • JUnitPrams – a better way to write parameterized tests with JUnit by Paweł Lipiński,
  • Spock – a setting new standards testing framework for Java and Groovy code by Jakub Nabrdalik,
  • Geb – a Groovy way for acceptance testing with Web Driver and Page Object Pattern by Tomasz Kalkosiński,
  • asynchronous calls testing with Awaitility by Marcin Zajączkowski (a.k.a. me),
  • the idea how to use IoC and Guice in tests by Paweł Cesar Sanjuan Szklarz.

On my own speach I performed an experiment with a presentation format. Instead of a classical Impress/Power Point slides I used Vym – View Your Mind – an interesting tool for mind mapping with a build in slide editor. It allowed me to “animate” my mind map and talk about every its point (node). By the way I reported a dozen of feature requests for Vym, so there is a chance to make a slide editor even better on the next presentation :) . The downside of using such tool is a limited way how it could be exported to some neutral format. The picture below contains the whole map, but without code snippets.

Sleepless asynchronous calls testing with Awaitility

The people I asked were pleased with the presentations. They were cross-sectional and touched many different aspects briefly, but even though I knew covered topics before I noted down a few useful tips. The venue was tightly filled (about 100 people). In addition to interesting topics there were an ability to eat pizza and win an SSD disk. The event was completed with a networking session in the near restaurant. For those who cannot attend there is a video available (in Polish) on the WJUG’s You Tube channel.

Last Saturday together with 8 other mentors we were showing various Git-related technics on Git kata event for over 80 people.

Git kata was a free git workshop conducted in a kata form. Paraphrasing Wikipedia “A Git kata is an exercise in using Git which helps an user hone their skills through practice and repetition”. During sessions a mentor was showing selected Git aspects in practice providing listeners detailed comments on each performed step. The attenders could follow master’s steps using their own laptops.

There were various Git technics covered including:
- undoing changes (reset, revert, reflog),
- useful aliases, configuration tricks, Git internals and git prompt with fish shell,
- collaboration with other using public services (like GitHub, Bitbucket, GitLab, Gitorious) or patches via email/USB,
- submodules, filter branch and rerere.

In the past I was leading different programming katas, but I have never head about the idea to use it with Git. It sounded very interesting when I was asked to join a mentors team on Git kata. I hope I helped some people better understand internals of Git commands and in addition having two free slots I got to know (among other things) about rerere which can reduce number of manual merges (have you even heard about it before?). “branch.autosetuprebase always” flag which set a rebase as a default strategy on pull for a newly created branches or a “help.autocorrect 1” flag which automatically applies “did you mean” suggestions in case of typo. The event was very successful and I wonder if not to extend my training portfolio by a Git course.

Recently I wanted to configure an ability to run both TestNG and JUnit tests in one Maven module (project). At the end I managed how to do it clean and short, but before that I have found a few different solutions on the web (top 5 in Google) which part of them didn’t work and the rest applied to the earlier versions of Surefire plugin and was overly complicated (e.g. two separate executions). Therefore I decided to write this short post to show how it could be done in Surefire 2.13 – the newest version available in March 2013.

Mixing those tests frameworks in one module can be done just by adding both JUnit and TestNG as a plugin dependencies (not as project dependencies):

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>${surefire.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-testng</artifactId>
            <version>${surefire.version}</version>
        </dependency>
    </dependencies>
</plugin>

As the result both test types are executed.

[INFO] --- maven-surefire-plugin:2.13:test (default-test) @ junit-testng-poc ---
[INFO] Surefire report directory: /tmp/junit-testng-poc/target/surefire-reports
[INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running info.solidsoft.rnd.junit.testng.SampleJUnitTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.253 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0


-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: TestNG652Configurator
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.869 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Why to use TestNG with JUnit together? TestNG has a few features which are unavailable or less flexible in JUnit (just to mention a few: dependencies between tests and groups of tests (irreplaceable for integration tests with long startup), parametrized tests, concurrent execution or per suite/group/class init/shutdown operations). Therefore it is tempting to migrate existing tests from JUnit to TestNG. Having large code base it could be not so easy to migrate all of them at once and presented configuration allows to write the new tests in TestNG and rewrite the old tests when appropriate.

The whole working example can be found in my GitHub repository.

Btw, it is worth to mention that thanks to the fact TestNG generates reports also in the JUnit’s XML format all the tools compatible with JUnit (Jenkins, Sonar, …) will merge test results from JUnit and TestNG and display all of them properly.

Btw2, the same configuration works also with Failsafe plugin.

Btw3, thanks to the fact Spock Framework is under the hood a runner for JUnit presented trick can be used also to mix it with TestNG. This requires some additional work to integrate also Groovy and I plan to write about it in one of my future posts.

Mutation testing can efficiently detect places in code which are insufficiently covered by tests. The price we have to pay for it is time – number of mutations has to be tested with a set of unit tests. This time is much longer than calculating a “normal” code coverage. The newest PIT 0.29 provides a long awaiting feature – incremental analysis. When enabled PIT will store results from the previous runs on disk and track changes in the code and tests to avoid rerunning analyses which result should stay the same.

To start using incremental analysis it is necessary set historyInputLocation and historyOutputLocation configuration properties. For example in Pitest Maven Plugin it could be:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>0.29</version>
    <configuration>
        <threads>4</threads>
        <historyInputLocation>target/pitHistory.txt</historyInputLocation>
        <historyOutputLocation>target/pitHistory.txt</historyOutputLocation>
    </configuration>
</plugin>

It is worth to notice that for a basic usage (i.e. run analysis from time to time) input and output history locations will point to the same file. Therefor in Gradle plugin for PIT 0.29.0 there was added an additional parameter enableDefaultIncrementalAnalysis which when enabled automatically set historyInputLocation and historyOutputLocation to build/pitHistory.txt simplifying a configuration.

buildscript {
    (...)
    dependencies {
        classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:0.29.0'
        classpath 'org.pitest:pitest:0.29'
    }
}

apply plugin: 'pitest'

pitest {
    targetClasses = ['our.base.package.*']
    threads = 4
    enableDefaultIncrementalAnalysis = true
}

There is a number of optimizations already implemented in PIT. The author warns of potential errors which can be introduced in that way into the analysis, but although being currently an experimental feature it can dramatically reduce calculation time especially when used on very large codebases.

Update 2013-04-13: Added missing line which applies plugin to a project. Thanks to Bruno de Carvalho for report the issue.

Looking for a way to use mutation testing and PIT with your Gradle-based project? Your search is over. Recently released gradle-pitest-plugin makes it possible in a very comfortable way.

In short the idea with mutation testing is to modify the production code (introduce mutations) which should change its behavior (produce different results) and cause unit tests to fail. The lack of the failure may indicate that given part of the production code was not covered good enough by the tests. To read more about mutation testing take a look on my previous post or PIT webpage directly.

To start using PIT add following configuration to a build.gradle file in your project:

buildscript {
    repositories {
        mavenLocal()
        mavenCentral()
        //Needed to use a plugin JAR uploaded to GitHub (not available in a Maven repository)
        add(new org.apache.ivy.plugins.resolver.URLResolver()) {
            name = 'GitHub'
            addArtifactPattern 'http://cloud.github.com/downloads/szpak/[module]/[module]-[revision].[ext]'
        }
    }
    dependencies {
        classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:0.28.0'
        classpath 'org.pitest:pitest:0.28'
    }
}

This will add required dependencies to a build script together with a proper repositories configuration.

The second thing is to configure plugin itself.

pitest {
    targetClasses = ['our.base.package.*']
    threads = 4
}

The only required parameter is “targetClasses” – a package containing a code which should be mutated (usually the base package for your project), but in case your tests are written in a thread safe manner I encourage your to give a “threads” parameter a try. It can decrease a time required for mutation analysis dramatically. gradle-pitest-plugin supports all reasonable parameters available in PIT.

Having everything configured running mutation testing is as easy as:

gradle pitest

After a while you should see PIT summary similar to:

================================================================================
- Statistics
================================================================================
Generated 59 mutations Killed 52 (88%)
Ran 161 tests (2.73 tests per mutation)
================================================================================

Detailed reports with information about survived mutations and the corresponding parts of code is written to build/reports/pitest/ directory relative to your project root.

Sample PIT report

Sample PIT report

Btw, there is a version 0.29 of PIT just around the corner which provides such interesting features as incremental analysis (i.e. run only mutation tests for code that have been changed). I plan to write a post about it when released, so stay tuned.

Disclaimer. I am the author of gradle-pitest-plugin.

Mockito team provides two jars: mockito-core.jar and mockito-all.jar. Which is better to choose for Maven or Gradle-based project? Both of them can be downloaded manually from a project webpage and also are available in Maven Central Repository. In many Maven projects I have found that mockito-all.jar was used which isn’t the best option and I will shortly explain why.

mockito-all.jar besides Mockito itself contains also (as of 1.9.5) two dependencies: Hamcrest and Objenesis (let’s omit repackaged ASM and CGLIB for a moment). The reason was to have everything what is needed inside an one JAR to just put it on a classpath. It can look strange, but please remember than Mockito development started in times when pure Ant (without dependency management) was the most popular build system for Java projects and the all external JARs required by a project (i.e. our project’s dependencies and their dependencies) had to be downloaded manually and specified in a build script.

On the other hand mockito-core.jar is just Mockito classes (also with repackaged ASM and CGLIB). When using it with Maven or Gradle required dependencies (Hamcrest and Objenesis) are managed by those tools (downloaded automatically and put on a test classpath). It allows to override used versions (for example if our projects uses never, but backward compatible version), but what is more important those dependencies are not hidden inside mockito-all.jar what allows to detected possible version incompatibility with dependency analyze tools. This is much better solution when dependency managed tool is used in a project.

I asked to forget about ASM and CGLIB for a while. It is time to explain why. As classes from those projects are repackaged inside a Mockito JAR (i.e. put into different packages than the original ones) and they are treated by a class loader just as different classes. This means that from a dependency management point of view they are just another Mockito internal classes. The Mockito developers explain it as a way to avoid some not fixed problems with CGLIB and a class loader met in the early development phase.

This post is the third part of the series Beyond the Mockito refcard extending recently released my Mockito reference card.

Unit tests are very handy. They run fast and it is possible to execute hundreds or even thousands of them very often during development (especially useful when using TDD). Nevertheless from time to time we want to test the correctness of a part of an IoC container configuration and check how those components works together when managed by an IoC context. In this tutorial I will show how to do it in a convenient way.

The easiest solution could be to set up the whole “production” IoC context. Unfortunately depending on the size of a project it can take some time (long minutes) and in addition that configuration is likely to connect to external resources like a database or a message queue which makes restriction where those tests will be able to run. What can we do to fix it?

Very often to test interactions between selected components it is not needed to make the full featured integration tests. We can use two useful techniques together: limiting context scope (?) and mocking. The idea is to set up only needed components. This can be achieved by separate a minimal Spring configuration file using in tests (possible reusing some of existing production files). However very often there is a problem with direct dependencies which behaviors (?) are not needed in our tests, but are required to set up our beans. Doesn’t it look familiar to the situation meet also in unit tests? Mocks (and Mockito) to the rescue.

Our mocks have to be put into Spring context to be able to inject into other beans created by Spring. While it can be done using pure Spring mechanisms, there is a library which makes it much easier – Springockito written by Jakub Janczak.

Note. Pure unit tests are generally much better choice. Tests using IoC context should be only used when we really need to test behaviors within a context.

To start working with Springockito it is required to add its JAR to the project dependencies. Using Maven it could be:

<dependency>
    <groupId>org.kubek2k</groupId>
    <artifactId>springockito</artifactId>
    <version>1.0.4</version>
    <scope>test</scope>
</dependency>

With Gradle:

testCompile "org.kubek2k:springockito:1.0.4"

(change 1.0.4 to the latest released version of Springockito)

Note. The following examples are simplified to does not introduce unneeded distractions. They do not have a context configuration which is worth to test and it would be easier to use pure unit tests (without a Spring context).

Let’s take following configuration where plantWaterer object requires waterSource and waterScheduler beans:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

    <bean id="plantWaterer" class="info.solidsoft.refcard.mockito.PlantWaterer">
        <constructor-arg name="waterSource" ref="waterSource"/>
        <constructor-arg name="waterScheduler" ref="waterScheduler"/>
    </bean>

    <bean id="waterSource" class="info.solidsoft.refcard.mockito.RiverWaterSource">
        <constructor-arg ref="smallRiver"/>
    </bean>

    <bean id="waterScheduler" class="info.solidsoft.refcard.mockito.WaterScheduler"/>

</beans>

RiverWaterSource relies on an external resource which could be problematic during tests. To override it following configuration (with the additional namespace mockito) could be used.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:mockito="http://www.mockito.org/spring/mockito"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.mockito.org/spring/mockito https://bitbucket.org/kubek2k/springockito/raw/tip/springockito/src/main/resources/spring/mockito.xsd">

    <mockito:mock id="waterSource" class="info.solidsoft.refcard.mockito.WaterSource"/>

</beans>

Make a note that there is even no try to create the original bean. This is very useful in a situation where for example DAO would like to make a connection to a database during start up (which could take some time and is very fragile).

With Springockito it is also very easy to make a spy of existing Spring bean and inject it into another bean instead of the original bean for example to verify made interactions.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:mockito="http://www.mockito.org/spring/mockito"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.mockito.org/spring/mockito https://bitbucket.org/kubek2k/springockito/raw/tip/springockito/src/main/resources/spring/mockito.xsd">

    <mockito:spy beanName="waterSource"/>

</beans>

Mocks and spies created with a help of Springockito are just like “normal” mocks/spies which can be stubbed and verified. However there are two important differences. The instance of a mock/spy is created by Spring when a context is set up (in opposite to manual call Mockito.mock/spy() or using @Mock/@Spy annotation), so the easiest way to obtain it is to make an injection into our tests class (by name or by type).

@ContextConfiguration({<<configuration-files>>})
public class MockInjectionSpringockitoTest extends AbstractTestNGSpringContextTests {

    @Autowired
    private PlantWaterer plantWaterer;

    @Autowired	//it is safe - Springockito hid the original WaterSource bean
    private WaterSource waterSourceSpy;

    @Test
    public void shouldSpyInjectedObject() {
        //given - mocks/spies are already created, can be stubbed here

        //when
        plantWaterer.waterPlants();

        //then
        verify(waterSourceSpy).startWaterFlow();
    }
}

The seconds important difference is also related to the fact that a mock/spy is created with a context and the same instance is hold until the context shutdown. It can span across many tests or even test classes. It is important to properly reset/initialize mocks/spies manually before every test to avoid an impact of the operations done in previously executed tests. The issue has been already reported and hopefully will be resolved in the further Springockito versions.

Btw, in case of the following error:

org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace [http://www.mockito.org/spring/mockito]
Offending resource: class path resource [yourSpringConfig.xml]
	at org.springframework.beans.factory.parsing.FailFastProblemReporter.error(FailFastProblemReporter.java:68)
	at org.springframework.beans.factory.parsing.ReaderContext.error(ReaderContext.java:85)
(...)

make sure the Springockito JAR has been successfully added as a project dependency.

In case you prefer annotations over XML Springockito has also something for you. This is a version of Springockito which allow to do similar things, but directly in your test code.

To use it in a project it is required to add an springockito-annatations.jar to your project dependencies.

With Maven:

<dependency>
    <groupId>org.kubek2k</groupId>
    <artifactId>springockito-annotations</artifactId>
    <version>1.0.2</version>
</dependency>

With Gradle:

testCompile "org.kubek2k:springockito-annotations:1.0.2"

(change 1.0.2 to the latest released version of springockito-annotation – the numeration is not synchronized with springockito – it is an independent jar (as of 1.0.2))

@ContextConfiguration(loader = SpringockitoContextLoader.class, locations = {<<the-base-configuration-file>>})
public class MockInjectionSpringockitoTest extends AbstractTestNGSpringContextTests {

    @Autowired
    private PlantWaterer plantWaterer;

    @WrapWithSpy
    @Autowired	//to inject a spy into this field to make a verification
    private WaterSource waterSource; //spy

    @ReplaceWithMock //just replace the original bean with a mock, will be null
                     //it won't be stubbed - no need to inject
    private WateringScheduler wateringScheduler; //mock

    @Test
    public void shouldSpyInjectedObject() {
        //given - mocks/spies are already created, can be stubbed here

        //when
        plantWaterer.waterPlants();

        //then
        verify(waterSource).startWaterFlow();
    }
}

The trick here is to use a custom Spring context loader which checks (by name) if there is any bean which should be replaced with a mock or wrapped with a spy.

The code above causes waterSource bean defined in a configuration file to be wrapped with a spy and wateringScheduler bean to be replaced with a mock (the original wateringScheduler bean will not be created at all). They are both injected into plantWaterer. @ReplaceWithMock and @WrapWithSpy annotations are used only by Springockito to detect which beans should be replaced/wrapped. To get an instance of a mock/spy (to stub it or verify behavior) it is needed to use also an appropriate annotation (@Autowired, @Resource or @Inject) – see waterSource in the previous listing.

There is a limitation which force to use the same field name as a bean which should be replaced/wrapped. It should be fixed in the future.

This post is the second part of the series Beyond the Mockito refcard extending recently released my Mockito reference card.

Some time ago I’ve got an idea to make a short document briefly presenting the most important Mockito features with the real life examples. It was targeted at people wanting to quickly learn more about Mockito framework and those who use Mockito from time to time and need a small example to recall a syntax of a given construction. The result of my work was recently released by DZone as the Mockito refcard. Six – ready to print – pages filled in with condensed knowledge, a lot of examples and practical advises.

Why? This is my another action to promote practices helping to write high quality code guided by tests. Besides it always takes me a while to introduce a mocking concept and Mockito itself to a new junior developer and it is quite repetitive job. Having a Mockito refcard I can give it to that person and later talk what he/she was learned and practice it during the pair programming sessions. Pure long-term time savings :) .

As a supplement I also started a series of posts about Mockito, extending a refcard with the more advanced topics which did not fit into the base version.

I would like to take this opportunity to thank Paweł Lipiński who did a complete language review of a draft version and Brice Dutheil, Gordon Dickens and Tomek Kaczanowski for exchanging ideas how to make my refcard better. Krzysiek Jelski, Kamil Szymański, Marceli Kramarczuk and Kamil Żbikowski also contributed by reading a draft version and sending suggestions.

Writing the Mockito reference card I had an opportunity to take a closer look at the less popular, but very useful features of Mockito. Some of them were too advance or too rare in use to be described in a refcard which should be kept short. One of those things is SmartNull. Currently non void methods return “safe empty value” appropriate for known types (e.g.: 0, false, empty collection) or null in other cases. SmartNull to can be returned instead of pure null to receive a much more descriptive error message on NPE.

Instead of just a line when NullPointerException occurred:

java.lang.NullPointerException
    at PlantWaterer.generateNPE(PlantWaterer.java:24)
    at DefaultValuesTest.shouldReturnNicerErrorMessageOnNPE(DefaultValuesTest.java:64)

we have also got descriptive information what method was not stubbed:

org.mockito.exceptions.verification.SmartNullPointerException:
You have a NullPointerException here:
−> at PlantWaterer.generateNPE(PlantWaterer.java: 24)
because this method call was ∗not∗ stubbed correctly:
−> at PlantWaterer.generateNPE(PlantWaterer.java: 24)
wateringScheduler.returnNull();

    at PlantWaterer.generateNPE(PlantWaterer.java: 24)
    at DefaultValuesTest.shouldReturnNicerErrorMessageOnNPE(DefaultValuesTest.java:64)

A particular mock can be instructed to return SmartNull instead of the null value:

PlantWaterer plantWatererMock =
        mock(PlantWaterer.class, Mockito.RETURNS_SMART_NULLS);

or

@Mock(answer = Answers.RETURNS_SMART_NULLS)
private PlantWaterer plantWatererMock;

SmartNull will be probably the default bahavior in Mockito 2.0, but for the backward compatibility in 1.9.x it is necessary to tell every mock explicitly to use it. The need to write another piece of boilerplate code causes that almost nobody uses SmartNull despite it is a very useful feature. And there the second almost unknown element of Mockito enters the game – global configuration. Generally there is no need to configure Mockito. It just works. But for some rare cases the framework’s authors left a gate which allows to override the default configuration of a few core behaviors, including the default answer policy for unstubbed methods.

To make it work it is necessary to create org.mockito.configuration.MockitoConfiguration class (necessarily in that package) which implements IMockitoConfiguration interface. Usually it is comfortable to extend DefaultMockitoConfiguration class and only override desired behavior(s).

package org.mockito.configuration;

import org.mockito.internal.stubbing.defaultanswers.ReturnsSmartNulls;
import org.mockito.stubbing.Answer;

public class MockitoConfiguration extends DefaultMockitoConfiguration {

    public Answer<Object> getDefaultAnswer() {
        return new ReturnsSmartNulls();
    }
}

After that preparation we should get SmartNullPointerException with the verbose output instead of the pure NullPointerException for every mock in our module.

@Test(expectedExceptions = SmartNullPointerException.class)
public void shouldReturnNicerErrorMessageOnNPE() {
    //given
    //Mockito.RETURNS_SMART_NULLS not needed anymore
    WateringScheduler wateringSchedulerMock = mock(WateringScheduler.class);
    WaterSource waterSourceMock = mock(WaterSource.class);
    PlantWaterer plantWatererMock =
            new PlantWaterer(waterSourceMock, wateringSchedulerMock);

    //when
    plantWatererMock.generateNPE();

    //then
    //SmartNullPointerException exception expected
}

This post is the first part of the series Beyond the Mockito refcard.

Mutation testing is a technique which allows to discover which parts of our code are not covered by tests. It is similar to a code coverage, but mutation testing is not limited to the fact that a given line was executed during tests. The idea is to modify the production code (introduce mutations) which should change its behavior (produce different results) and cause unit tests to fail. The lack of the failure may indicate that given part was not covered good enough by the tests. The idea of mutation testing is quite old, but it is rather unpopular. Despite the fact I am rather experienced in testing I found it just recently reviewing a beta version of the new book about testing.

PIT is “a fast bytecode based mutation testing system for Java that makes it possible to test the effectiveness of your unit tests”. It is a very young project, but very promising. It offers a set of mutation operators which among others modify conditional statements, mathematical operations, return values and methods calls.

Starting with the recently released version 0.25 PIT (experimentally) supports TestNG based tests (in addition to JUnit based). To use it from Maven it is required to add pitest-maven plugin to pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>0.25</version>
<!--
    <configuration>
        <inScopeClasses>
            <param>info.solidsoft.blog.pitest.*</param>
        </inScopeClasses>
        <targetClasses>
            <param>info.solidsoft.blog.pitest.*</param>
        </targetClasses>
    </configuration>
-->
</plugin>

In many cases it would be enough. inScopeClasses (mutable classes and tests for running) and targetClasses (only candidates for mutation) by default use project groupId and usually can be omitted. There are several options that can be configured in the plugin configuration. “mvn org.pitest:pitest-maven:mutationCoverage” performs modified tests and generates mutation report which by default is saved in target/pit-reports/yyMMddHHmm directory.

A sample report (click to enlarge) for a specified class shows both line coverage and mutation coverage. Despite 100% line coverage (lines with light green background), PIT found that a test data set does not cover properly boundary conditions.

Sample PIT report

Sample PIT report