Posts Tagged ‘testing’

Learn how leverage Spock 1.2 to slice a Spring context of a legacy application writing integration tests.

Have you ever wanted, having some legacy application which you were starting to work on, to write some tests to get know what is going on and possibly be notified about regressions? That feeling when you want to instantiate a single class and it fails with NullPointerException. 6 replaced (with difficulty) dependencies later there are still some errors from the classes that you haven’t heard about before. Sounds familiar?

There are different techniques to deal with hidden dependencies. There is the whole dedicated book about that (and probably a few other that I haven’t read). Occasionally, it may be feasible to start with the integration tests and run through some process. It may be even more “entertaining” to see what exotic components are required to just setup the context, even if they are completely not needed in our case. Thank you (too wide and carelessly used) @ComponentScan :).

Injecting stubs/mocks inside the test context is a way to go as an emergency assistance (see the last paragraph, there are better, yet harder approaches). It can be achieved “manually” with an extra bean definition with the @Primary annotation (usually a reason to think twice before doing that) for every dependency at which level we want to make a cut of (or for every unneeded bean which is instantiated by the way). @MockBean placed on a field in a test is more handy, but still, it is needed to define a field in our tests and put the annotation on it (5? 10? 15 beans?). Spock 1.2 introduces somehow less know feature which may be useful here – @StubBeans.

Mockied dependencies Spring context

It can be used to simply provide a list of classes which (possible) instances should be replaced with stubs in the Spring test context. Of course before the real objects are being instantiated (to prevent for example NPE in a constructor). Thanks to that up to several lines of stubbing/mock injections:

@RunWith(SpringRunner.class) //Spring Boot + Mockito
@SpringBootTest //possibly some Spring configuration with @ComponentScan is imported in this legacy application
public class BasicPathReportGeneratorInLegacyApplicationITTest { //usual approach

    @MockBean
    private KafkaClient kafkaClientMock;

    @MockBean
    private FancySelfieEnhancer fancySelfieEnhancerMock;

    @MockBean
    private FastTwitterSubscriber fastTwitterSubscriberMock;

    @MockBean
    private WaterCoolerWaterLevelAterter waterCoolerWaterLevelAterterMock;

    @MockBean
    private NsaSilentNotifier nsaSilentNotifierMock;

    //a few more - remember, this is legacy application, genuine since 1999 ;)
    //...

    @Autowired
    private ReportGenerator reportGenerator;

    @Test
    public void shouldGenerateEmptyReportForEmptyInputData() {
        ...
    }
}

can be replaced with just one (long) line:

@SpringBootTest //possibly some Spring configuration with @ComponentScan is imported in this legacy application
@StubBeans([KafkaClient, FancySelfieEnhancer, FastTwitterSubscriber, WaterCoolerWaterLevelAterter, NsaSilentNotifier/(, ... */])
  //all classes of real beans which should be replaced with stubs
class BasicPathReportGeneratorInLegacyApplicationITSpec extends Specification {

    @Autowired
    private ReportGenerator reportGenerator

    def "should generate empty report for empty input data"() {
        ....
    }
}

(tested with Spock 1.2-RC2)

It’s worth to mention that @StubBeans is intended just to provide placeholders. In a situation it is required to provide stubbing and/or an invocation verification @SpringBean or @SpringSpy (also introduced in Spock 1.2) are better. I wrote more about it in my previous blog post.

There is one important aspect to emphasize. @StubBeans are handy to be used in a situation when we have some “legacy” project and want to start writing integration regression tests quickly to see the results. However, as a colleague of mine Darek Kaczyński brightly summarized, blindly replacing beans which “explode” in tests is just “sweeping problems under carpet”. After the initial phase, when we are starting to understand what is going on, it is a good moment to rethink the way the context – both in production and in tests – is created. The already mentioned too wide @ComponentScan is very often the root of all evil. An ability to setup a partial context and put it together (if needed) is a good place to start. Using @Profile or conditional beans are the very powerful mechanisms in tests (and not only there). @TestConfiguration and proper bean selection to improve context caching are something worth to keep in your mind. However, I started this article to present the new mechanism in Spock which might be useful in some cases and I want to keep it short. There could be an another, more generic blog post just about managing the Spring context in the integration tests. I have to seriously thing about it :).

Advertisements

Discover how to automatically inject Spock’s mocks and spies into the Spring context using Spock 1.2.

Coffee beans with 3 different beans

Stubs/mocks/spies in Spock (and their life cycle) have been always tightly coupled with the Spock Specification class. It was only possible to create them in a test class. Therefore, using shared, predefined mocks (in both unit and integration tests) was problematic.

The situation was slightly improved in Spock 1.1, but only with the brand new Spock 1.2 (1.2-RC1 as a time of writing) using the Spock mocking subsystem in Spring-based integration tests is as easy as using @SpringMock for Mockito mocks in Spring Boot. Let’s check it up.

Btw, to be more cutting edge in addition to Spock 1.2-RC1, I will be using Spring Boot 2.1.0.M2, Spring 5.1.0.RC2 and Groovy 2.5.2 (but everything should work with the stable versions of Spring (Boot) and Groovy 2.4).

One more thing. For the sake of simplicity, in this article, I will be using a term ‘mock’ to refer also stubs and spies. They differs in behavior, however, in a scope of injecting it into the Spring context in the Spock tests it usually doesn’t matter.

Spock 1.1 – manual way

Thanks to the work of Leonard Brünings, mocks in Spock were decoupled from the Specification class. It was finally possible to create them outside and to attach it later on into a running test. It was the cornerstone of using Spock mocks in the Spring (or any other) context.

In this sample code we have the ShipDatabase class which uses OwnShipIndex and EnemyShipIndex (of course injected by a constructor :) ) to return aggregated information about all known ships matched by name.

//@ContextConfiguration just for simplification, @(Test)Configuration is usually more convenient for Spring Boot tests
//Real beans can exist in the context or not
@ContextConfiguration(classes = [ShipDatabase, TestConfig/*, OwnShipIndex, EnemyShipIndex*/])
class ShipDatabase11ITSpec extends Specification {

    private static final String ENTERPRISE_D = "USS Enterprise (NCC-1701-D)"
    private static final String BORTAS_ENTERA = "IKS Bortas Entera"

    @Autowired
    private OwnShipIndex ownShipIndexMock

    @Autowired
    private EnemyShipIndex enemyShipIndexMock

    @Autowired
    private ShipDatabase shipDatabase

    def "should find ship in both indexes"() {
        given:
            ownShipIndexMock.findByName("Enter") >> [ENTERPRISE_D]
            enemyShipIndexMock.findByName("Enter") >> [BORTAS_ENTERA]
        when:
            List<String> foundShips = shipDatabase.findByName("Enter")
        then:
            foundShips == [ENTERPRISE_D, BORTAS_ENTERA]
    }

    static class TestConfig {
        private DetachedMockFactory detachedMockFactory = new DetachedMockFactory()

        @Bean
        @Primary    //if needed, beware of consequences
        OwnShipIndex ownShipIndexStub() {
            return detachedMockFactory.Stub(OwnShipIndex)
        }

        @Bean
        @Primary    //if needed, beware of consequences
        EnemyShipIndex enemyShipIndexStub() {
            return detachedMockFactory.Stub(EnemyShipIndex)
        }
    }
}

The mocks are created in a separate class (outside the Specification) and therefore DetachedMockFactory has to be used (or alternatively SpockMockFactoryBean). Those mocks have to be attached (and detached) to the test instance (the Specification instance), but it is automatically handled by the spock-spring module (as of 1.1). For generic mocks created externally also MockUtil.attachMock() and mockUtil.detachMock() would need to be used to make it work.

As a result it was possible to create and use mocks in the Spring context, but it was not very convenient and it was not commonly used.

Spock 1.2 – first class support

Spring Boot 1.4 brought the new quality to integration testing with (Mockito’s) mocks. It leveraged the idea, originally presented in Springockito back in 2012 (when the Spring configuration was mostly written in XML :) ) to automatically inject mocks (or spies) into the Spring (Boot) context. The Spring Boot team extended the idea and thanks to having it as the internally supported feature it (usually) works reliably just by adding an annotation or two in your test.

Similar annotation-based mechanism is built-in in Spock 1.2.

//@ContextConfiguration just for simplification, @(Test)Configuration is usually more convenient for Spring Boot tests
//Real beans can exist in the context or not
@ContextConfiguration(classes = [ShipDatabase/*, OwnShipIndex, EnemyShipIndex*/])
class ShipDatabaseITSpec extends Specification {

    private static final String ENTERPRISE_D = "USS Enterprise (NCC-1701-D)"
    private static final String BORTAS_ENTERA = "IKS Bortas Entera"

    @SpringBean
    private OwnShipIndex ownShipIndexMock = Stub()  //could be Mock() if needed

    @SpringBean
    private EnemyShipIndex enemyShipIndexMock = Stub()

    @Autowired
    private ShipDatabase shipDatabase

    def "should find ship in both indexes"() {
        given:
            ownShipIndexMock.findByName("Enter") >> [ENTERPRISE_D]
            enemyShipIndexMock.findByName("Enter") >> [BORTAS_ENTERA]
        when:
            List<String> foundShips = shipDatabase.findByName("Enter")
        then:
            foundShips == [ENTERPRISE_D, BORTAS_ENTERA]
    }
}

There is not much to be added. @SpringBean instructs Spock to inject a mock into a Spring context. Similarly, @SpringSpy wraps the real bean with a spy. In a case of @SpringBean it is required to initialize a field to let Spock know if we plan to use a stub or a mock.

In addition, there is also a more general annotation @StubBeans to replace all defined beans with stubs. However, I plan to cover it separately in an another blog post.

Limitations

For those of you who look forward to rewrite all Mockito’s mocks to Spock’s mocks in your Spock tests right after the lecture of this article there is a word of warning. Spock’s mocks – due to their nature and relation to Specification – have some limitations. The implementation under the hood creates a proxy which is injected into the Spring context which (potentially) replaces real beans (stubs/mocks) or wraps them (spies). That proxy is shared between all the tests in the particular test (specification) class. In fact, it also can span across other tests with the same bean/mock declarations in the situation Spring is able to cache the context (similar situation to Mockito’s mocks or Spring integration tests in general).

However, what is really important, a proxy is attached to a tests right before its execution and is detached right after it. Therefore, in fact, every test has it’s own mock instance (it cannot be applied to @Shared fields) and it is problematic for instance to group interactions from different tests and verify them together (which usually is quite sensible, but might lead to some duplication). Nevertheless, with using a setup block (or in-line stubbing) it is possible to share stubbing and interaction expectancy.

Summary

Spock 1.2 finally brings hassle-free Spock’s stubs/mocks/spies support for using them in the Spring context which is comparable with the one provided in Spring Boot for Mockito. It is just enough to add the spock-spring module to the project test dependencies. Despite some limitations, it is one point less for mixing native Spock’s mocking subsystem with external mocking frameworks (such as Mockito) in your Spock (integration) tests. And what is nice, it should work also in plain Spring Framework tests (not only Spring Boot tests). The same feature has been implemented for Guice (but I haven’t tested it).

Furthermore, Spock 1.2 brings also some other changes including better support for Java 9+ and it is worth to give it a try in your test suite (and of course report any potentially spotted regression bugs :) ).

One more good news. In addition to the Leonard’s work who made Spock 1.2 possible and a legion of bug reporters and PR contributors, since recently, there are also some other committers who are working on making Spock even better. Some of them you may know from some other popular FOSS projects. What is more, Spock 1.2 is (preliminary) planned to be the last version based on JUnit 4 and the next stable Spock version could be 2.0, leveraging JUnit 5 and (among others) its native ability to run tests in parallel.

The examples were written using Spock 1.2-RC1. It will be updated to 1.2-final once released. The source code is available from GitHub.

Btw, have you wonder if it is still worth using Spock in the time of JUnit 5? I try to help answer that question in my presentation which will be possible to see at JDD 2018, this October in Kraków, Poland. See you there.

JDD 2018 logo with date

The lead photo based on the Couleur‘s work published in Pixabay, CC0 1.0

Starting with the version 2.17.0 Mockito provides the official (built-in) support for managing a mocking life cycle if JUnit 5 is used.

mockito-junit5-logo

Getting started

To take advantage of the integration, the Mockito’s mockito-junit-jupiter dependency is required to be added next to the JUnit 5’s junit-platform-engine one (see below for details).

After that, the new Mockito extension MockitoExtension for JUnit 5 has to be enabled. And that’s enough. All the Mockito annotation should automatically start to work.

import org.junit.jupiter.api.Test;  //do not confuse with 'org.junit.Test'!
//other imports
import org.mockito.junit.jupiter.MockitoExtension;

@ExtendWith(MockitoExtension.class)
class SpaceShipJUnit5Test {

    @InjectMocks
    private SpaceShip spaceShip;

    @Mock
    private TacticalStation tacticalStation;

    @Mock
    private OperationsStation operationsStation;

    @Test
    void shouldInjectMocks() {
        assertThat(spaceShip).isNotNull();
        assertThat(tacticalStation).isNotNull();
        assertThat(operationsStation).isNotNull();
        assertThat(spaceShip.getTacticalStation()).isSameAs(tacticalStation);
        assertThat(spaceShip.getOperationsStation()).isSameAs(operationsStation);
    }
}

It’s nice that both a test class and test methods don’t need to be public anymore.

Please note. Having also JUnit 4 on a classpath (e.g. via junit-vintage-engine) for the “legacy” part of tests it is important to do not confuse org.junit.jupiter.api.Test with the old one org.junit.Test. It will not work.

Stubbing and verification

If for some reasons you are not a fan of AssertJ (although, I encourage you to at least give it a try) JUnit 5 provides a native assertion assertThrows (which is very similar to assertThatThrownBy() from AssertJ). It provides a meaningful error message in a case of an assertion failure.

@Test
void shouldMockSomething() {
    //given
    willThrow(SelfCheckException.class).given(tacticalStation).doSelfCheck();   //void method "given..will" not "when..then" cannot be used
    //when
    Executable e = () -> spaceShip.doSelfCheck();
    //then
    assertThrows(SelfCheckException.class, e);
}

I were not myself if I would not mention here that leveraging support for default methods in interfaces available in AssertJ and mockito-java8 a lot of static imports can be made redundant.

@ExtendWith(MockitoExtension.class)
class SpaceShipJUnit5Test implements WithAssertions, WithBDDMockito {
    ...
}

Tweaking default behavior

It also worth to point our that using the JUnit 5 extension Mockito by default works in the “strict mode”. It means that – for example – unneeded stubbing will fail the test. While very often it is a code smell, there are some cases where that test construction is desired. To change the default behavior an @MockitoSettings annotation can be used.

@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.WARN)
class SpaceShipJUnitAdvTest implements WithAssertions, WithBDDMockito {
    ....
}

Dependencies

As I already mentioned, to start using it is required to add a Mockito’s mockito-junit-jupiter dependency next to a JUnit 5’s junit-platform-engine one. In a Gradle build it could look like:

dependencies {
    testCompile 'org.junit.vintage:junit-platform-engine:5.1.0'
    testCompile 'org.mockito:mockito-junit-jupiter:2.17.2'  //mockito-core is implicitly added

    testCompile 'org.junit.vintage:junit-vintage-engine:5.1.0'  //for JUnit 4.12 test execution, if needed
    testCompile 'org.assertj:assertj-core:3.9.1'    //if you like it (you should ;) )
}

Please note. Due to a bug with injecting mocks via constructor into final fields that I have found writing this blog post, it is recommended to use at least version 2.17.2 instead of 2.17.0. That “development” version is not available in the Maven Central and the extra Bintray repository has to be added.

repositories {
    mavenCentral()
    maven { url "https://dl.bintray.com/mockito/maven" }    //for development versions of Mockito
}

In addition, it would be a waste to do not use a brand new native support for JUnit 5 test execution in Gradle 4.6+.

test {
    useJUnitPlatform()
}

IntelliJ IDEA has provided JUnit support since 2016.2 (JUnit 5 Milestone 2 at that time). Eclipse Oxygen also seems to add support for JUnit 5 recently.

mockito-junit5-idea-results

Summary

It is really nice to have a native support for JUnit 5 in Mockito. Not getting ahead there is ongoing work to make it even better.
The feature has been implemented by Christian Schwarz and polished by Tim van der Lippe with great assist from a few other people.

The source code is available from GitHub.

Btw, are you wondering how JUnit 5 compares with Spock? I will be talking about that at GeeCON 2018.

Geecon logo

Get know how fast (or slow) mutation testing in fact is (based on tangible results from the real FOSS projects) and how can it be optimized (with real implementation in PIT, a tool for Java projects).

One of the questions I’ve got after my first presentation about mutation testing with PIT at DevConf.cz in Brno years ago was “how long would it take in my project?”. I wasn’t able to answer precisely. The execution time depends on multiple factors. Size of the project, number of tests, kind of tests (unit/integration), just to mention a few. There are also some optimization techniques which can speed up the analysis. To give you a roughly idea about that, I collected the results from a mutation testing session in two popular FOSS projects performed with PIT.

I assume that you already know something about mutation testing. If not, please read that article before going further.

The fast and the furious 1910

Warm-up

As my guinea pigs I chose two popular open source projects – Assertj – which provides “fluent assertions for Java” and Joda-Time providing “a quality replacement for the Java date and time classes” which was a must for Java projects before Java 8 where similar solution became available out-of-box.

AssertJ can be named a medium size project according to FOSS standards. It has ~85K lines of production code and 150K lines of tests (as measured by Idea statistics plugin – including empty lines and comments). The code compilation takes ~2 seconds. Test execution ~12 seconds (almost 10_000 (mostly) unit tests). Joda-Time is somehow smaller with ~70K LOC in production and ~72K LOC in tests. Test execution takes ~3 seconds (~4200 tests).

I chose AssertJ and Joda-Time as they have a very solid test harness which consists (mostly) of unit tests – the best possible kind of tests for most of the cases. While integration tests can also be used with mutation testing (with some limitations and side effects) unit tests are the perfect candidate.

Brute force

Based on the further PIT analysis I estimated a number of possible to generate mutants at 8000 for AssertJ and 10000 for Joda-time. Of course it depends on the number of mutations available in your tool belt, however, to compare a brute force method with PIT the same number of mutants is accurate.

You may ask why there is greater number of mutants generated for Joda-time which has smaller codebase. It would be a fair question. PIT (or any other mutation testing tool) generates mutants in the lines where something can be “broken”. As Joda-time contains more “real logic” inside it was possible to generate more mutants than in AssertJ code where is a lot of delegation. In the other words, it’s usually possible to generate more mutants generated in a line with an if statement than in a line which just calls some void method in an another class. Therefore, number of mutations does not depend only on the number of lines.

Let’s get back to our calculations. AssertJ, brute force algorithm. For every mutation code has to be compiled – 8000 * 2 seconds and all tests have to be executed 8000 * 12 seconds. It gives 112000 seconds in total. It is ~1867 minutes, over 31 hours! Even for 5 times smaller Joda-time it is still: 10000 * 1 seconds + 10000 * 3 = 40000 seconds -> 667 minutes -> over 11 hours!

Wow, over 11 and 31 hours. For not so large projects. It is one of the reasons why mutation testing – first proposed by Richard Lipton in 1971 (over 40 years ago!) – had been an academic technique only. Let’s take a look how it could be optimized to make it conform to the reality in enterprise projects.

Optimizations

Test selection

Running all tests for every mutation it greatly ineffective. One of the naive approaches is an usage of a name convention – tests usually are named SomeProductionClass*Test. However, it tends to underestimate test suite effectiveness as no all tests are written in that way (especially integration/acceptance ones) and there could be also some typos. Another idea is a static call analysis. It’s more reliable, however, it completely skips reflection calls and in addition it can have a problem with polymorphism.

PIT in turns, leverages some other techniques. First of all, tests are executed in a “normal” mode and a standard code coverage metric is gathered. Thanks to that procedure, mutants with no “standard” coverage can be skipped. There is no chance to have them killed – no test even executes that given line. That technique can be especially beneficial if PIT is used in a project having small number of automated tests (low standard code coverage). What is more, PIT knows exactly which tests execute the particular line. Therefore, no test will ever be run against a mutant that it will not execute. This is essential to limit the number of tests executed per mutant. As a bonus, PIT (based on the first execution) is aware of tests execution time. Tests covering a given mutant can be reordered to have fast ones executed first. If a mutant has been killed, the subsequent (slower) tests can be skipped.

Those optimizations (among other technical solutions – such as creating mutations at the bytecode level instead of the source code one – which is must faster, albeit harder in implementation and also has some limitations) contributed to much faster mutation testing analysis in comparison to theoretical brute force mode.

Let’s compare the theoretical brute force analysis time with the one achieved with vanilla PIT configuration.

Execution time in minutes Brute force PIT (1 thread)
AssertJ 1866.67 14.15
Joda-time 666.67 11.65

Brute force mutation analysis vs PIT

Looking at the chart please notice that Y-axis has a logarithmic scale (!). It’s ~14 minutes for AssertJ (as opposed to ~1867 minutes – over 31 hours) and ~11 minutes for Joda-time (compared to ~667 minutes – over 11 hours). It’s ~130 and ~60 times faster respectively. Looks good, but it can be done even better.

Parallel execution

One of the consequences of breaking the Moore’s law years ago was an increment of number of cores in modern CPUs. Nowadays, 4 or 8 virtual cores is a standard of a developer workstation. Servers can have much more of them. PIT advertised as “a state of the art in mutation testing” in addition to aforementioned optimizations supports parallel execution. Let’s take a look how much it can be gained in our tests case.

Test environment

All tests were performed using a dedicated m4.4xlarge AWS instance (32 virtual cores – 2x Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz with 8 physical cores each – and 64GB RAM). Most of the executions were repeated to verify the achieved result, however, the methodology was far from one required in scientific researches (which was definitely not an aim of my work).

Raw results

AssertJ
Number of threads 1 2 4 6 8 10 12 16 20 24 32
Execution time in minutes 14.15 7.48 4.27 3.27 2.92 2.77 2.85 2.88 3.02 3.12 3.85
Number of timeout errors 17 17 17 17 17 17 17 18 31 31 260

AssertJ - PIT analysis time

Joda-time
Number of threads 1 2 4 8 12 16 20 24 32
Execution time in minutes 11.65 6.27 3.83 2.62 2.35 2.42 2.47 2.55 2.60
Number of timeout errors 49 49 50 49 49 49 49 48 51

Joda-time - PIT analysis time

Commented results

The aforementioned results clearly shows that having quite powerful machine it was possible to the reduce mutation testing analysis execution time over 5 times (in comparison to sequential analysis) thanks to doing multiple things at the same time. In the test subjects the saturation point was placed around 10-12 threads (out of 32 virtual cores). At that state the machine seemed to be (almost) fully loaded (see the screenshot below) by most of the time (excluding the initial and the last stage of the analysis). After certain point the number of timeout errors was increased significantly (in a case of AssertJ) which could confirming that the machine was overloaded. Unfortunately, I don’t have a good explanation why the system was saturated much below 32 virtual cores (16 physical cores) as PIT should not use more threads as defined and there was no extra parallelism in the tests itself. As reasonably suggested by Henry Coles, the author of PIT, next time I will attach a profiler to the analysis to try to find potential places to speed the things up even further.

CPU utilisation - PIT, 12 threads

CPU utilisation – PIT, 12 threads

Further optimizations

Having mutation testing performed regularly in large codebase, the amount of changed code is usually meaningless. To not to have to re-execute mutation on all the code, PIT incorporates the incremental mode. When enabled, PIT determines which mutant could have got a chance to get killed by the recent changes in the production code and tests. As a result, the analysis scope can be limited to those classes. This approach is perfect for running a local analysis by a developer on his/her workstation or in a change/pull request to determine the “quality” of the recently introduced changes. For large codebase savings on time execution can be tremendously. It is only worth to remember that the heuristic selecting which part of the analysis should be executed has some limitation and from time to time it is good to run the full analysis anyway.

Summary

With a few smart optimizations (here implemented in PIT) it is possible to reduce (a theoretical) brute force mutation testing execution time by even ~130 times (from 31 hours to less than 15 minutes in a case of AssertJ). Having a hi-end quad core laptop it is possible to cut another chunk thanks to parallel execution. However, only the power of very strong server machine (here a server with 32 virtual cores which is not uncommon among CI server executors used in large projects) allows to unleash the power of PIT to speed up the mutation testing analysis. The full analysis time was possible to reduce by 3 orders of magnitude (!) (from 31 hours to less than 3 minutes in a case of AssertJ). It, together with incremental analysis, can make mutation testing feasible* to use on a daily basis in the real life, large enterprise projects (especially with unit tests posed the majority of the tests used in a project – but it is a topic for an another blog post :) ).

Btw, what is your experience in using mutation testing in non-academic projects? Leave your comment below.

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock/JUnit/Mockito/AssertJ/PIT quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Mutation testing allows to check the quality (effectiveness) of automatic tests. PIT (aka pitest) is a leading mutation testing tool for Java environment. In my last blog post about PIT in January 2013 I have covered version 0.29. Since then the PIT development team has been busy and the 4 releases introduced various new features (besides fixed bugs). In this post I will cover the most important (in my opinion) changes in the project (up to recently released version 0.33).

PIT mutation testing tool logo

New features

– preliminary support for Java 8 bytecode – PIT can be used with code which contains Java 8 syntax and constructions (including lamdas)
– internal refactoring resulted in much faster “standard” line coverage calculation
– support for parametrized JUnit tests written with Spock (in Groovy) and JUnitParams
– ability to define a coverage threshold (both line and mutation) below which the build will fail
– ability to use PIT with Robolectric
– new Remove Conditionals Mutator (a conditional statement will always be true – not enabled by default as of 0.33)
– new Remove Increments Mutator (an increment operation will be removed – not enabled by default as of 0.33)
– ability to choose JVM to be used for mutation testing
– ability to run PIT only for locally changed files for Maven build with configured SCM plugin
– demanding users can define their own strategies for: test selection, output format and test prioritization – PIT provides extension points which allow to write custom implementations
– partial support for JUnit categories
– support for mutating static initializers in TestNG

In the meantime there were also releases of plugins/tools based on PIT. My plugin for Gradle was enhanced with the dynamic task dependencies resolution (just “gradle pitest” takes care about all the requisites in the Gradle build lifecycle) and support for the additional main and test source sets. Plugin for Eclipse has got (inter alia) a new mutation view and an ability to run PIT against all the projects in a workspace.

Not only releases

Besides new releases PIT has got brand new Bootstrap based webpage, the logo (see above) and the source code was migrated from Mercurial on Google Code to GitHub. The nice thing is that the move resulted in a few contributions withing the first weeks.

Henry Coles the author of PIT also started the new commercial project FaultSeed – “better mutation testing tools for the JVM” which will be based on PIT and has a goal to be 50% faster than PIT and support also Groovy and Scala. Very promising.

PIT (and mutation testing in general) becomes more and more popular and recently there were given a number of talks about it (including my talk at Developer Conference 2014slides). The number of questions on the project’s mailing list also significantly increased. And you, have you tried PIT in your project yet?

DevConf.cz 2014 logo