Get know how fast (or slow) mutation testing in fact is (based on tangible results from the real FOSS projects) and how can it be optimized (with real implementation in PIT, a tool for Java projects).

One of the questions I’ve got after my first presentation about mutation testing with PIT at DevConf.cz in Brno years ago was “how long would it take in my project?”. I wasn’t able to answer precisely. The execution time depends on multiple factors. Size of the project, number of tests, kind of tests (unit/integration), just to mention a few. There are also some optimization techniques which can speed up the analysis. To give you a roughly idea about that, I collected the results from a mutation testing session in two popular FOSS projects performed with PIT.

I assume that you already know something about mutation testing. If not, please read that article before going further.

The fast and the furious 1910

Warm-up

As my guinea pigs I chose two popular open source projects – Assertj – which provides “fluent assertions for Java” and Joda-Time providing “a quality replacement for the Java date and time classes” which was a must for Java projects before Java 8 where similar solution became available out-of-box.

AssertJ can be named a medium size project according to FOSS standards. It has ~85K lines of production code and 150K lines of tests (as measured by Idea statistics plugin – including empty lines and comments). The code compilation takes ~2 seconds. Test execution ~12 seconds (almost 10_000 (mostly) unit tests). Joda-Time is somehow smaller with ~70K LOC in production and ~72K LOC in tests. Test execution takes ~3 seconds (~4200 tests).

I chose AssertJ and Joda-Time as they have a very solid test harness which consists (mostly) of unit tests – the best possible kind of tests for most of the cases. While integration tests can also be used with mutation testing (with some limitations and side effects) unit tests are the perfect candidate.

Brute force

Based on the further PIT analysis I estimated a number of possible to generate mutants at 8000 for AssertJ and 10000 for Joda-time. Of course it depends on the number of mutations available in your tool belt, however, to compare a brute force method with PIT the same number of mutants is accurate.

You may ask why there is greater number of mutants generated for Joda-time which has smaller codebase. It would be a fair question. PIT (or any other mutation testing tool) generates mutants in the lines where something can be “broken”. As Joda-time contains more “real logic” inside it was possible to generate more mutants than in AssertJ code where is a lot of delegation. In the other words, it’s usually possible to generate more mutants generated in a line with an if statement than in a line which just calls some void method in an another class. Therefore, number of mutations does not depend only on the number of lines.

Let’s get back to our calculations. AssertJ, brute force algorithm. For every mutation code has to be compiled – 8000 * 2 seconds and all tests have to be executed 8000 * 12 seconds. It gives 112000 seconds in total. It is ~1867 minutes, over 31 hours! Even for 5 times smaller Joda-time it is still: 10000 * 1 seconds + 10000 * 3 = 40000 seconds -> 667 minutes -> over 11 hours!

Wow, over 11 and 31 hours. For not so large projects. It is one of the reasons why mutation testing – first proposed by Richard Lipton in 1971 (over 40 years ago!) – had been an academic technique only. Let’s take a look how it could be optimized to make it conform to the reality in enterprise projects.

Optimizations

Test selection

Running all tests for every mutation it greatly ineffective. One of the naive approaches is an usage of a name convention – tests usually are named SomeProductionClass*Test. However, it tends to underestimate test suite effectiveness as no all tests are written in that way (especially integration/acceptance ones) and there could be also some typos. Another idea is a static call analysis. It’s more reliable, however, it completely skips reflection calls and in addition it can have a problem with polymorphism.

PIT in turns, leverages some other techniques. First of all, tests are executed in a “normal” mode and a standard code coverage metric is gathered. Thanks to that procedure, mutants with no “standard” coverage can be skipped. There is no chance to have them killed – no test even executes that given line. That technique can be especially beneficial if PIT is used in a project having small number of automated tests (low standard code coverage). What is more, PIT knows exactly which tests execute the particular line. Therefore, no test will ever be run against a mutant that it will not execute. This is essential to limit the number of tests executed per mutant. As a bonus, PIT (based on the first execution) is aware of tests execution time. Tests covering a given mutant can be reordered to have fast ones executed first. If a mutant has been killed, the subsequent (slower) tests can be skipped.

Those optimizations (among other technical solutions – such as creating mutations at the bytecode level instead of the source code one – which is must faster, albeit harder in implementation and also has some limitations) contributed to much faster mutation testing analysis in comparison to theoretical brute force mode.

Let’s compare the theoretical brute force analysis time with the one achieved with vanilla PIT configuration.

Execution time in minutes Brute force PIT (1 thread)
AssertJ 1866.67 14.15
Joda-time 666.67 11.65

Brute force mutation analysis vs PIT

Looking at the chart please notice that Y-axis has a logarithmic scale (!). It’s ~14 minutes for AssertJ (as opposed to ~1867 minutes – over 31 hours) and ~11 minutes for Joda-time (compared to ~667 minutes – over 11 hours). It’s ~130 and ~60 times faster respectively. Looks good, but it can be done even better.

Parallel execution

One of the consequences of breaking the Moore’s law years ago was an increment of number of cores in modern CPUs. Nowadays, 4 or 8 virtual cores is a standard of a developer workstation. Servers can have much more of them. PIT advertised as “a state of the art in mutation testing” in addition to aforementioned optimizations supports parallel execution. Let’s take a look how much it can be gained in our tests case.

Test environment

All tests were performed using a dedicated m4.4xlarge AWS instance (32 virtual cores – 2x Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz with 8 physical cores each – and 64GB RAM). Most of the executions were repeated to verify the achieved result, however, the methodology was far from one required in scientific researches (which was definitely not an aim of my work).

Raw results

AssertJ
Number of threads 1 2 4 6 8 10 12 16 20 24 32
Execution time in minutes 14.15 7.48 4.27 3.27 2.92 2.77 2.85 2.88 3.02 3.12 3.85
Number of timeout errors 17 17 17 17 17 17 17 18 31 31 260

AssertJ - PIT analysis time

Joda-time
Number of threads 1 2 4 8 12 16 20 24 32
Execution time in minutes 11.65 6.27 3.83 2.62 2.35 2.42 2.47 2.55 2.60
Number of timeout errors 49 49 50 49 49 49 49 48 51

Joda-time - PIT analysis time

Commented results

The aforementioned results clearly shows that having quite powerful machine it was possible to the reduce mutation testing analysis execution time over 5 times (in comparison to sequential analysis) thanks to doing multiple things at the same time. In the test subjects the saturation point was placed around 10-12 threads (out of 32 virtual cores). At that state the machine seemed to be (almost) fully loaded (see the screenshot below) by most of the time (excluding the initial and the last stage of the analysis). After certain point the number of timeout errors was increased significantly (in a case of AssertJ) which could confirming that the machine was overloaded. Unfortunately, I don’t have a good explanation why the system was saturated much below 32 virtual cores (16 physical cores) as PIT should not use more threads as defined and there was no extra parallelism in the tests itself. As reasonably suggested by Henry Coles, the author of PIT, next time I will attach a profiler to the analysis to try to find potential places to speed the things up even further.

CPU utilisation - PIT, 12 threads

CPU utilisation – PIT, 12 threads

Further optimizations

Having mutation testing performed regularly in large codebase, the amount of changed code is usually meaningless. To not to have to re-execute mutation on all the code, PIT incorporates the incremental mode. When enabled, PIT determines which mutant could have got a chance to get killed by the recent changes in the production code and tests. As a result, the analysis scope can be limited to those classes. This approach is perfect for running a local analysis by a developer on his/her workstation or in a change/pull request to determine the “quality” of the recently introduced changes. For large codebase savings on time execution can be tremendously. It is only worth to remember that the heuristic selecting which part of the analysis should be executed has some limitation and from time to time it is good to run the full analysis anyway.

Summary

With a few smart optimizations (here implemented in PIT) it is possible to reduce (a theoretical) brute force mutation testing execution time by even ~130 times (from 31 hours to less than 15 minutes in a case of AssertJ). Having a hi-end quad core laptop it is possible to cut another chunk thanks to parallel execution. However, only the power of very strong server machine (here a server with 32 virtual cores which is not uncommon among CI server executors used in large projects) allows to unleash the power of PIT to speed up the mutation testing analysis. The full analysis time was possible to reduce by 3 orders of magnitude (!) (from 31 hours to less than 3 minutes in a case of AssertJ). It, together with incremental analysis, can make mutation testing feasible* to use on a daily basis in the real life, large enterprise projects (especially with unit tests posed the majority of the tests used in a project – but it is a topic for an another blog post :) ).

Btw, what is your experience in using mutation testing in non-academic projects? Leave your comment below.

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock/JUnit/Mockito/AssertJ/PIT quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Advertisements

Tune up your JUnit test class template for Idea with the BDD-like syntax, Java 8 and the Mockito-AssertJ duo.

Topics covered in this article may seem trivial. However, from my trainer experience I know that (unfortunately) it is not a common practice. Therefore, I decided to write this short blog post to propagate them and to be able to refer to it in the future.

given-when-then-template

My favorite testing framework for Java (and Groovy) is Spock. However, its mocks are not suitable for some purpose and I still use Mockito in various places. In addition, I still conduct a lot of my testing training in a JUnit/Mockito/AssertJ variant for teams which already have a test suite in that stack and would like to improve their skills without changing the known technology. Therefore, as an interlude, this blog post about testing in the pure Java style and propose how to tune up your JUnit testing framework assuming that you are already using Mockito and AssertJ (you should give them a try in the other case).

This blog post consists of tree parts. Firstly, I propose a BDD-style section-based test structure to keep your test more consist and more readable. Next, I explain how simplify – using the AssertJ and Mockito – constructions with Java 8. Last, but not least, I show how to configure it in IntelliJ IDEA as a default JUnit test (class) template (which isn’t as trivial as it should).

Part 1. BDD-style sections

Well written unit tests should meet several requirements (but it is a topic for a separate post). One of the useful practices is a clear separation into 3 code blocks with precisely defined responsibility. You can read more on that topic in my previous blog post.

As a repetition just the core rules presented in a short form:

  • given – an object under test initialization + stubs/mocks creation, stubbing and injection
  • when – an operation to test in a given test
  • then – received result assertion + mocks verification (if needed)
@Test
public void shouldXXX() {
  //given
  ...
  //when
  ...
  //then
  ...
}

That separation helps to keep tests short and focused on just one responsibility to test (in the end it’s just an unit test).

In Spock those sections are mandatory (*) – without them a test will not even compile. In JUnit there are just comments. However, having them in place encourage people to use them instead of having one big block of mess inside (especially useful for newbies in a testing area).

Btw, the mentioned given-when-then convention is based on (is a subset of) a much wider Behavior-Driven Development concept. You may encounter a similar division on 3 code blocks named arrange-act-assert which in general is an equivalent.

Part 2. Java 8 for AssertJ and Mockito

One of the features of Java 8 is an ability to put default methods in an interface. That can be used to simplify of calling static methods which is prevalent in the testing frameworks such as AssertJ and Mockito. The idea is simple. A test class willing to use a given framework can implement a dedicated interface to “see” those methods as its own methods on code completion in an IDE (instead of static methods from external class which require giving a class name before or a static import). Under the hood those default methods just delegate execution to static methods. You can read more about it in my other blog post.

AssertJ natively supports those construction starting with version 3.0.0. Mockito 1.10 and 2.x are Java 6 compatible and therefore it is required to use a 3rd-party project – mockito-java8 (which should be integrated into Mockito 3 – once available).

To benefit from easier method completion in Idea it is enough to implement two interfaces:

import info.solidsoft.mockito.java8.api.WithBDDMockito;
import org.assertj.core.api.WithAssertions;

class SampleTest implements WithAssertions, WithBDDMockito {

}

Part 3. Default template in Idea

I’m a big enthusiast of omnipresent automation. Wouldn’t it be good to have both given-when-then sections and extra interfaces automatically in place in your test classes? Let’s eliminate those boring things from our life.

Test method

Changing a JUnit test method is easy. One of the possible ways is “CTRL-SHIFT-A -> File Template -> Code” and a modification of JUnit4 Test Method to:

@org.junit.Test
public void should${NAME}() {
  //given
  ${BODY}
  //when
  //then
}

To add a new test in an existing test class just press ALT-INSERT and select (or type) JUnit4 Test Method.

Test class

With the whole test class the situation is a little bit more complicated. Idea provides a way to edit existing templates, however, it is used only if a test is generated with CTRL-SHIFT-T from a production class. It’s not very handy with TDD where a test should be created first. It would be good to have a new position “New JUnit test class” next to “Java class” displayed if ALT-INSERT is pressed being in a package view in a test context. Unfortunately, to do that a new plugin would need to be written (a sample implementation for Spock). As a workaround we can define a regular file template which (as a limitation) will be accessible everywhere (e.g. even in a resource directory).

Do “CTRL-SHIFT-A -> File Template -> Files”, press INSERT, name template “JUnit with AssertJ and Mockito Test”, set extension to “java” and paste the following template:

package ${PACKAGE_NAME};

import info.solidsoft.mockito.java8.api.WithBDDMockito;
import org.assertj.core.api.WithAssertions;

#parse("File Header.java") 
public class ${NAME} implements WithAssertions, WithBDDMockito {

}

Showcase

We are already set. Let’s check how it can look in practice (click to enlarge the animation).

idea-test-templates-in-action

Summary

I hope I convinced you to tune your test template to improve readability of your tests and to safe several keystrokes per test. In that case, please spend 4 minutes right now to configure it in your Idea. Depending on a number of tests written it may start to pay off sooner than you expect :).

Btw, at the beginning of October I will be giving a presentation about new features in Mockito 2 at JDD in Kraków.

JDD logo

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock/JUnit/Mockito/AssertJ quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Create better defined, shorter and maintainable unit tests with given-when-then sections.

Recently, I’ve been writing rather about more advanced concepts related to automatic testing (mostly related to Spock). However, conducting my testing training I clearly see that very often knowledge of particular tools is not the main problem. Even with Spock it is possible to write bloated and hard-to-maintain test, breaking (or not being aware of) good practices related to writing unit tests. Therefore, I decided to write about more fundamental things to promote them and by the way have a ready to use material to reference when coaching less experienced colleagues.

Introduction

Well written unit tests should meet several requirements and it is a topic for the whole series. In this blog post I would like to present a quite mature concept of dividing an unit test on 3 separate blocks with a strictly defined function (which in turn is a subset of Behavior-driven Development).

given-when-then to the victory

Unit tests are usually focused on testing some specific behavior of a given unit (usually one given class). As opposed to acceptance tests performed via UI, it is cheap (fast) to setup a class to test (a class under test) from scratch in an every test with stubs/mocks as its collaborators. Therefore, performance should not be a problem.

Sample test

To demonstrate the rules I will use a small example. ShipDictionary is a class providing an ability to search space ships based on particular criteria (by a part of a name, a production year, etc.). That dictionary is powered (energized) by different indexes of ships (ships in service, withdrawn from service, in production, etc.). In that one particular test it is tested an ability to search ship by a part of its name.

private static final String ENTERPRISE_D = "USS Enterprise (NCC-1701-D)";

@Test
public void shouldFindOwnShipByName() {
    //given
    ShipDatabase shipDatabase = new ShipDatabase(ownShipIndex, enemyShipIndex);
    given(ownShipIndex.findByName("Enterprise")).willReturn(singletonList(ENTERPRISE_D));
    //when
    List foundShips = shipDatabase.findByName("Enterprise");
    //then
    assertThat(foundShips).contains(ENTERPRISE_D);
}

given-when-then

The good habit which exists in both Test-driven and Behavior-driven Development methodologies is ‘a priori’ knowledge what will be tested (asserted) in a particular test case. It could be done in a more formal way (e.g. scenarios written in Cucumber/Gherkin for acceptance tests) or in a free form (e.g. ad hoc noted points or just an idea of what should be tested next). With that knowledge it should be quite easy to determine three crucial things (being a separated sections) of which the whole test will consist.

given – preparation

In the first section – called given – of a unit test it is required to create a real object instance on which the tested operation will be performed. In focused unit tests there is only one class in which the logic to be tested is placed. In addition, other objects required to perform a test (named collaborators) should be initialized as stubs/mocks and properly stubbed (if needed). All collaborators have to be also injected into the object under test which usually is combined with that object creation (as a constructor injection should be a preferred technique of dependency injection).

//given
ShipDatabase shipDatabase = new ShipDatabase(ownShipIndex, enemyShipIndex);
given(ownShipIndex.findByName("Enterprise")).willReturn(singletonList(ENTERPRISE_D));

when – execution

In the when section an operation to be tested is performed. In our case it is a search request followed by result memorization in a variable for further assertion.

//when
List foundShips = shipDatabase.findByName("Enterprise");

In most cases it is good to have just one operation in that section. More elements may suggest an attempt to test more than one operation which (possibly) could be divided into more tests.

then – assertion

The responsibility of the final section – then – is mostly an assertion of the previously received result. It should be equal to the expected value.

//then
assertThat(foundShips).contains(ENTERPRISE_D);

In addition, it may be necessary to perform a verification of method executions on declared mocks. It should not be a common practice as an assertion on received value in most cases is enough to confirm that code being tested works as expected (according to set boundaries). Nevertheless, especially with testing void methods it is required to verify that a particular method was executed with anticipated arguments.

AAA aka 3A – an alternative syntax

As I’ve already mentioned, BDD is a much wider concept which is especially handy for writing functional/acceptance tests with requirements defined upfront, (often) in a non technical form. An alternative test division syntax (with very similar meaning for the sections) is arrange-act-assert often abbreviated to AAA or 3A. If you don’t use BDD at all and three A letters are easier to remember for you than GWT, it’s perfectly fine to use it to create the same high quality unit tests.

Tuning & optimization

The process of matching used tools and methodologies to ongoing process of skill acquisition (aka the Dreyfus model) has been nicely described in the book Pragmatic Thinking and Learning: Refactor Your Wetware. Of course, in many cases it may be handy to use a simplified variant of a test with a given section moved to a setup/init/before section or initialized inline. The same may apply to when and then sections which could be merged together (into an expect section, especially in parameterized tests). Having some experience and fluency in writing unit tests it is perfectly valid to use shorthand and optimizations (especially testing some non-trivial cases). As long as the whole team understand the convention and is able to remember about basic assumptions regarding writing good unit tests.

Summary

Based on my experience in software development and as a trainer I clearly see that dividing (unit) tests into sections makes them shorter and more readable, especially having less experienced people in the team. It’s simpler to fill 3 sections with concisely defined responsibility than figure out and write everything in the tests at once. In closing, particularly for people reading only the first and the last sections of the article, here are condensed rules to follow:

  • given – an object under test initialization + stubs/mocks creation, stubbing and injection
  • when – an operation to test in a given test
  • then – received result assertion + mocks verification (if needed)

P.S. It is good to have a test template set in your IDE to safe a number of keystrokes required to write every test.
P.S.S. I you found this article useful you can let me know to motivate me to write more about unit test basics in the future.

Picture credits: Tomas Sobek, Openclipart, https://openclipart.org/detail/242959/old-scroll

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock/JUnit/Mockito/AssertJ quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Stubbing methods returning java.util.Optional with Spock is more tricky that you would probably expect. Get know how to do it efficiently.

by Infrogmation, Wikimedia Commons, public domain

Introduction

One of the nice features of the mocking framework in Spock is an ability to return sensible default values for unstubbed method calls made on stubs. Empty list for a method returning List, 0 for long, etc. Very handy if you don’t care about returned value in a particular test, but for example would like to prevent NullPointerException later in the flow. Unfortunately Spock 1.0 and 1.1-rc-2 (still compatible with Java 6) is completely not aware of types added in Java 8 (such as Optional or CompletableFutures). You may say “no problem” null is acceptable in many cases, but with Optional the situation is even worse.

Issue

Imaging the following code – method returning Optional and a try to use it in a test:

interface Repository<T> {
    Optional<T> getMaybeById(long id)
}

@Ignore("Broken")
def "should not fail on unstubbed call with Optional return type"() {
    given:
        Dao<Order> dao = Stub()
    when:
        dao.getMaybeById(5)
    then:
        noExceptionThrown()
}

You may think – null will be returned on the getMaybeById() call, but it’s not.

Expected no exception to be thrown, but got 'org.spockframework.mock.CannotCreateMockException'

    at spock.lang.Specification.noExceptionThrown(Specification.java:119)
    at info.solidsoft.blog.spock10.other.CustomDefaultResponseSpec.should not fail on unstubbed call with Optional return type(CustomDefaultResponseSpec.groovy:19)
Caused by: org.spockframework.mock.CannotCreateMockException: Cannot create mock for class java.util.Optional because Java mocks cannot mock final classes. If the code under test is written in Groovy, use a Groovy mock.
    at org.spockframework.mock.runtime.JavaMockFactory.createInternal(JavaMockFactory.java:49)
    at org.spockframework.mock.runtime.JavaMockFactory.create(JavaMockFactory.java:40)
(...)

The test fails at runtime as Spock is not able to stub java.util.Optional which is a final class:

CannotCreateMockException: Cannot create mock for class java.util.Optional
    because Java mocks cannot mock final classes.

What we can do?

Two workarounds

The EmptyOrDummyResponse factory class (which tries to be smart) is used by default for stubs when an ustubbed method is being called. However, it can be changed on demand during a stub creation:

def "should not fail on unstubbed call with Optional return type - workaround 1"() {
    given:
        Dao<Order> dao = Stub([defaultResponse: ZeroOrNullResponse.INSTANCE])
    when:
        dao.getMaybeById(5)
    then:
        noExceptionThrown()
}

This test will pass (getMaybeById() just returned null), but there is an easier way to achieve the same result.

Spock uses EmptyOrDummyResponse only for stubs (created with a Stub() method). For mocks (created with a Mock() method) the ZeroOrNullResponse factory is used (which makes sense as mocks should focus on interaction verification not just stubbing). Thanks to that a smart logic trying to return sensible default value is disabled in much simpler way:

def "should not fail on unstubbed call with Optional return type - workaround 2"() {
    given:
        Dao<Order> dao = Mock()
    when:
        dao.getMaybeById(5)
    then:
        noExceptionThrown()
}

However, this workaround is far from being perfect. Firstly, your colleagues may be surprised why a mock is created while only stubbing is performed (by the way, both stubbing and verifying interaction on the same mock is tricky itself in Spock, but this is a topic for an another blog post). Secondly, wouldn’t it be nice to have an empty optional (instead of null) returned by default?

Solution

In addition to an aforementioned way to use predefined factories for default return types Spock provides an ability to write a custom one. Let’s create EmptyOrDummyResponse-life factory which is aware of Java 8 types. In fact, the implementation is very straightforward:

class Java8EmptyOrDummyResponse implements IDefaultResponse {

    public static final Java8EmptyOrDummyResponse INSTANCE = new Java8EmptyOrDummyResponse()

    private Java8EmptyOrDummyResponse() {}

    @Override
    public Object respond(IMockInvocation invocation) {
        if (invocation.getMethod().getReturnType() == Optional) {
            return Optional.empty()
        }
        //possibly CompletableFutures.completedFuture(), dates and maybe others

        return EmptyOrDummyResponse.INSTANCE.respond(invocation)
    }
}

Our class implements an IDefaultResponse interface with one respond() method. Inside, we can apply custom logic for Optional, CompletableFutures and maybe other Java 8 specific types. As a fallback (for “standard” types) we switch to the original EmptyOrDummyResponse. This code works as expected:

@SuppressWarnings("GroovyPointlessBoolean")
def "should return empty Optional for unstubbed calls"() {
    given:
        Dao<Order> dao = Stub([defaultResponse: Java8EmptyOrDummy.INSTANCE])
    when:
        Optional<Order> result = dao.getMaybeById(5)
    then:
        result?.isPresent() == false    //NOT the same as !result?.isPresent()
}

Please pay attention to consider Groovy truth implementation while making assertions with Optional. !result?.isPresent() would be fulfilled also for null returned from a method.

However, maybe it would be good to simplify a Java 8 aware stub creation a little bit? To do that an extra method can be created:

private <T> T Stub8(Class<T> clazz) {
    return Stub([defaultResponse: Java8EmptyOrDummy.INSTANCE], clazz)
}

@SuppressWarnings("GroovyPointlessBoolean")
def "should return empty Optional for unstubbed calls with Stub8"() {
    given:
        Dao<Order> dao = Stub8(Dao)
    when:
        Optional<Order> result = dao.getMaybeById(5)
    then:
        result?.isPresent() == false    //NOT the same as !result?.isPresent()
}

Unfortunately in in that case an enhanced more compact stub creation syntax available in Spock 1.1 cannot be used with our Stub8() method. All because Spock will not be able to determine it’s type looking on he left side on assignment. In the end, however, it is much shorter than setting defaultResponse in an every stub creation.

Please note that due to Spock limitations that method cannot be put in a trait (or a separate class) and has to be defined in the current test or a custom base (super) class for all the tests (extending itself spock.lang.Specification), e.g.:

abstract class Java8AwareSpecification extends Specification {
    protected <T> T Stub8(Class<T> clazz) { ... }
}

class MyFancyTest extends Java8AwareSpecification { ... }

Summary

Thanks to exploring some Spock internals related to a stub and mock creation it was possible to enhance default strategy for smart responses for unstubbed calls to nicely support Java 8 features. This is just one of the topics I covered in my advanced “Interesting nooks and crannies of Spock you (may) have never seen before” presentation gave at Gr8Conf Europe 2016. You may want to see it :-).

Btw, the good news is that upcoming Spock 1.1(-rc-3) will contain native support for returning sensible default values for unstubbed Optional method calls.

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Learn how to visualize complex input parameters in parameterized tests in the way improving the readability of the test report.

Trimmed Hedge

By Tomwsulcer – CC0, Wikipedia

Introduction

Parameterized tests can simplify the way how the same functionality can be verified with different input parameters. Spock with its where block, data tables and data pipes makes it very easy to use in a very readable way. The input parameters are nicely presented in a test execution report (in IDE, Jenkins or generated HTML). They can be formatted in a desired order and completed with a custom constant message. It usually works flawlessly for simple objects (such as numbers, booleans, enums and strings). However, in the situation where complex objects are used (e.g. bigger value objects or custom classes) the whole output can be hijacked by just one very verbose parameter:

Very long full toString

or meaningless default toString() implementation: foo.bar.Unknown@78ac1102:

Meaningless Default toString in tests

Our sample code base

To present different available approaches I will use a very simplified version of an account & invoice related domain implemented with DDD in one of the projects I worked in.

The main class here is Invoice which represents an invoice :). The object is immutable (here with Groovy AST transformation, but it could be also achieved with Project Lombok or manually) which means that methods modifying a state return a new version of this class.

@Immutable
class Invoice {

    enum Status {
        ISSUED, PAID, OVERDUE, CANCELLED
    }

    Status status
    BigDecimal issuedAmount
    BigDecimal remainingAmount
    LocalDate issueDate
    //some other fields

    //different production methods

    //production toString() with all useful business fields
}

There is also Account class with an amountToPay() method returning amount to pay based in open invoices.

Naive approach (strongly not recommended)

As the first idea one could be tempted to modify toString() method implementation to, for example, display only 2 of the 10 fields in a class. However, it is a bad idea to change production toString() just for the better output in tests. What is more, other tests or error reporting in a production system can prefer to display more information. Luckily in Spock we have two nice techniques to cope with it.

Technique 1 – an extra formatting method

Test/specification names in Spock can be enhanced with #parameterName (not with $ character used internally by Groovy which is not allowed in a method name) placed in a test name or in an @Unroll annotation. In addition there is an ability to use object property value or call a parameterless method.

class AmountToPayInvoiceAccountSpec extends Specification {

    def "paid and cancelled invoices (#invoice.formatForTest()) should be ignored in current amount to pay"() {
        given:
            Account account = AccountTestFixture.createForInvoice(invoice)
        expect:
            account.amountToPay() == 0.0
        where:
            invoice << [paidInvoice, cancelledInvoice]
    }

    (...)
}

//required modifications in production code
class Invoice {

   (...)

   String formatForTest() {
       return "$issuedAmount: $status"
   }

Test specific production method - result

It’s nice to get just two fields from an object, however, in many cases we don’t want to add an artificial formatting method to production code just to be used in tests.

A tip. Don’t forget to enable unrolling of paremterized tests to instruct Spock to create a separate (sub)test for every input parameters set. It can be done manually by placing @Unroll annotation on an every parameterized method or at the class level. Alternatively the spock-global-unroll extension can be used to turn it on automatically in the whole project.

Technique 2 – an extra input parameter

Luckily, as an alternative it is possible to define an another artificial input parameter directly in a test. It looks like a ordinarily variable, but has access to a set of input parameters (for a given iteration) and can operate on they. That extra parameter is treated by Spock equally to others (however usually there is no need to reference to it in a test code – beside a test name).

class AmountToPayInvoiceAccountSpec extends Specification {

    @Shared
    private Invoice first = createOpenForAmount(200)
    @Shared
    private Invoice second = createOpenForAmount(300)

    def "current amount to pay (#expectedToPayAmount) should ignore paid and cancelled invoices (#invoicesDesc)"() {
        given:
            Account account = AccountTestFixture.createForInvoices(invoices)
        expect:
            account.amountToPay() == expectedToPayAmount
        where:
            invoices                   || expectedToPayAmount
            [paid(first), second]      || 300.0
            [first, cancelled(second)] || 200.0
            [first, second]            || 500.0

            invoicesDesc = createInvoicesDesc(invoices)
    }

    (...)
}

Implementation note. Methods createOpenForAmount() as well as paid() and cancelled() are implemented in a test specific InvoiceTestFixture class.

The result looks very nicely:
spock-formatting-input-parameters-test-specified-result

Just from the report it is pretty visible that there is a (regression) issue with handling CANELLED invoices. The assertion error is also helpful:

spock-formatting-input-parameters-test-specified-error-message

It’s worth to notice in this place that this technique can be also mixed with data pipes (in addition to data tables):

    where:
        invoice << [paidInvoice, cancelledInvoice]
        invoiceDesc = createInvoiceDesc(invoice)

A tip. Pay attention that in opposite to regular parameters in Spock the artificial one is created with an = operator not with <<.

Summary

The aforementioned techniques can be used to improve the readability of your test execution report. It’s useful during the development, but what is even more important it becomes indispensable if Spock is used for Behavior Driven Development and reports are read by, so called, Business People (i.e. need to be worded in the specific way).

[OT] The reason to bring up this topic is a fact that recently two colleagues of mine were struggling with that issue in their tests. Unfortunately they overlooked that slide in my advanced Spock presentation at Gr8Conf EU ;). Blessing in disguise, I was in the office to support them immediately. Nevertheless, not so long ago I’ve seen a presentation by Scott Hanselman about productivity. I liked the idea that every good question is worth to be answered on a blog. Replying privately (especially via email) usually can help only just one person. Writing a blog post and sending that person a link in addition can help other people struggling with the same issue.

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock quickly and efficiently? I conduct a condensed (unit) testing training which you may find useful.

Get know how to create mocks and spies in even more compact way with Spock 1.1

Introduction

Spock heavily leverages operator overloading (and Groovy magic) to provide very compact and highly readable specifications (tests) which wouldn’t be able to achieve in Java. This is very clearly visible among others in the whole mocking framework. However, preparing my Spock training I found a place for further improvements in a way how mocks and spies are created.

Shorter and shorter pencils

The Groovy way

Probably the most common way to create mocks (and spies) among devoted Groovy & Grails users is:

def dao = Mock(Dao)

The type inference in IDE works fine (there is type aware code completion). Nonetheless, this syntax is usually less readable for Java newcomers (people using Spock to tests production code written in Java) and in general for people preferring strong typing (including me).

The Java way

The same mock creation in the Java way would look as above:

Dao dao = Mock(Dao)

The first impression about this code snipped is – very verbose. Well, it is a Java way – why should we expect anything more ;).

The shorter Java way

As I already mentioned Spock leverages Groovy magic and the following construction works perfectly fine:

Dao dao = Mock()

Under the hood Spock uses a type used in the left side of an assignment to determine a type for which a mock should be created. Nominally everything is ok. Unfortunately there is one awkward limitation:

spock-1-0-mock-warning

IDE complains about unsafe type assignment and without getting deeper into the logic used in Spock it is justified. Luckily the situation is not hopeless.

The shorter Java way – Spock 1.1

Preparing practical exercises for my Spock training some time ago gave me an excuse to get into the details of implementation and after a few minutes I was able to improve the code to make it work cleanly in IDE (after a few years of living with that limitation!).

Dao dao = Mock()

spock-1-1-mock-no-warning2

No warning in IDE anymore.

Summary

Multiple times in my career I experienced a well known truth that preparing a presentation is very educational also for the presenter. In a case of a new 3-day long training it is even more noticeable – attendees have much more time to ask you uncomfortable question :). Not for the first time my preparations resulted in a new feature or an enhancement in some popular libraries or frameworks.

The last code snippet requires Spock in version 1.1 (which as a time of writing is available as the release candidate 3 – 1.1-rc-3 to not trigger a warning in IDE. There is a lot of new features in Spock 1.1 – why wouldn’t you give it a try? :)

Picture credits: GDJ from openclipart.org

Get know how to enable named method parameters support in a Gradle project

Introduction

Java 8 has introduced (among others) an ability to get a method parameter name at runtime. For backward compatibility (mostly with existing bytecode manipulation tools) it is required to enable it explicitly. The operation is as simple as an addition of a -parameters flag to a javac call in hello world tutorials. However, it turns out to be more enigmatic to configure in a Gradle project (especially for Gradle newcomers).

PensiveDuke

Gradle

To enable support for named method arguments it is required to set it for every java compilation task in a project. It can be easily attained with:

tasks.withType(JavaCompile) {
    options.compilerArgs << '-parameters'
}

For multi-project build the construction has to be applied on all the subprojects, e.g.:

subprojects {
    (...)
    tasks.withType(JavaCompile) {
        options.compilerArgs << '-parameters'
    }
}

Rationale

For me as a Gradle veteran and Gradle plugins author construction withType and passing different compilation or runtime JVM options is a bread and butter. However, I needed to explain it more than once to less Groovy experienced workmates, so for further reference (aka “Have you read my blog? ;-) ) I have written it down. As a justification for them I have to agree that as a time of writing this blog post the top Google results point to Gradle forum threads containing also “not so good” advises. Hopefully my article will be positioned higher :-).

Tested with Gradle 2.14 and OpenJDK 1.8.0_92.

Image credits: https://duke.kenai.com/

The simply way how buildscript dependencies (e.g. plugins) can be displayed and analyzed in Gradle

Introduction

This is the third part of my Gradle tricks mini-series related to visualization and analyze of dependencies. In the first post I presented a way how dependencies for all subprojects in multi-project build can be display. In the second I showed a technique of useful in tracking down not expected transitive dependencies in the project. This time less often used things, yet crucial in specific cases – buildscript dependencies.

Dependencies

Real use case

Buildscript dependencies contains plugins used in our project and their dependencies. It would seem nothing interesting unless you are a Gradle plugin developer, but it is not completely true. Once, as a consultant, I was investigating issue with NoSuchMethodException in a large project with custom build framework built on top of Gradle. The problem occurred only when one innocent, very popular open source plugin had been adding to the project. The same plugin worked fine in many other project in that company. In the end I was able to figure out that one of the dependencies used in buildSrc custom scripts overriding the same dependencies in older version from the plugin. As a result plugin failed at runtime with mentioned NoSuchMethodException. To achieve that I had to use my custom script as buildscript/classpath dependencies are completely ignored when ./gradlew dependencies or ./gradlew dependencyInsight is used.

Solution

The idea to write this post arose in at the beginning of 2015. I wanted to present my small Gradle task that using some internal Gradle mechanisms retrieves buildscript dependencies in display them to a console. The post was postponed and almost a year later I was positively surprised reading release notes for Gradle 2.10. The new buildEnvironment task was added.

$ ./gradlew buildEnvironment
:buildEnvironment

------------------------------------------------------------
Root project
------------------------------------------------------------

classpath
+--- com.bmuschko:gradle-nexus-plugin:2.3
\--- io.codearte.gradle.nexus:gradle-nexus-staging-plugin:0.5.3
     \--- org.codehaus.groovy.modules.http-builder:http-builder:0.7.1
          +--- org.apache.httpcomponents:httpclient:4.2.1
          |    +--- org.apache.httpcomponents:httpcore:4.2.1
          |    +--- commons-logging:commons-logging:1.1.1
          |    \--- commons-codec:commons-codec:1.6
          +--- net.sf.json-lib:json-lib:2.3
          |    +--- commons-beanutils:commons-beanutils:1.8.0
          |    |    \--- commons-logging:commons-logging:1.1.1
          |    +--- commons-collections:commons-collections:3.2.1
          |    +--- commons-lang:commons-lang:2.4
          |    +--- commons-logging:commons-logging:1.1.1
          |    \--- net.sf.ezmorph:ezmorph:1.0.6
          |         \--- commons-lang:commons-lang:2.3 -> 2.4
          +--- net.sourceforge.nekohtml:nekohtml:1.9.16
          \--- xml-resolver:xml-resolver:1.2

(*) - dependencies omitted (listed previously)

BUILD SUCCESSFUL

Total time: 1.38 secs

Two plugins and a pack of transitive dependencies to gradle-nexus-staging-plugin thanks to http-builder (maybe it would be good to replace it with Jodd?).

Summary

It is worth to be able to distinguish standard projects dependencies and buildscript dependencies. The new buildEnvironment task helps to deal with the latter. This in turn becomes essential when strange runtime errors start to show up.

Tested with Gradle 2.10.

Picture credits: Zeroturnaround.

Would it be useful to unroll all parameterized Spock tests automatically?

I’ve been always frustrated with the need to add @Unroll annotation to every parameterized test/feature (or at least at the class/specification level) to make unrolling works in Spock. It was even worse to deal with the code with already missing @Unroll annotations and cryptic test results. For backward compatibility unrolling will rather not be enabled by default in the foreseeable future, but luckily there is a quick solution.

Unroll

Photo: Christopher Michel, CC BY 2.0

Unroll for all and for free

To enable global unrolling it is only required to add spock-global-unroll.jar to your classpath:

testCompile 'info.solidsoft.spock:spock-global-unroll:0.5.0'

To make it easier to use spock-global-unroll with different Spock versions (like 1.0-groovy-2.0 and 1.0-groovy-2.3) the plugin does not have the compile dependency on Spock and a proper spock-core jar has to be explicitly defined in a build configuration. E.g.:

testCompile 'info.solidsoft.spock:spock-global-unroll:0.5.0'
testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'

That’s all. spock-global-unroll is a global extension which is activated automatically by Spock. All parameterized Spock tests are unrolled without the need to use @Unroll annotation.

Disabling automatic unrolling for a class

Automatic unrolling can be disabled for a particular class by putting @DisableGlobalUnroll on it.

The nice thing is that the @Unroll annotations manually placed at the test (feature) level can be used to unroll particular tests anyway (even if automatic unrolling has been disabled for given class).

@DisableGlobalUnroll
class PeselValidatorSpec extends Specification {

    //one big test for multiple input parameters
    def "should not be unrolled for some reasons PESEL #number"() { ... }

    (...)
}

Overriding default test name

To override default test name expanding (with #placeHolders in a test name) @Unroll annotation with a custom text can be used on the top of feature method or at the specification level.

@DisableGlobalUnroll
class PeselValidatorSpec extends Specification {

    //one big test for multiple input parameters
    def "should not be unrolled for some reasons PESEL #number"() { ... }

    //unrolled anyway
    @Unroll("PESEL '#pesel' should be #description")
    def "should validate PESEL correctness"() { ... }

    (...)
}

Summary

Being able to implement automatic tests unrolling within 15 minutes I decided to share it with the community – maybe there are others who don’t like to write boilerplate code :). The code written to achieve it has just a few lines of production code (of course there are also 3 test classes to verify if the extension works as expected :) ). This shows the power of Spock extensibility.

The complete source code is available from GitHub: https://github.com/szpak/spock-global-unroll

Update 20160521. I added automatic migration scripts in the project README to make a migration easier.

Btw, if you would like to find out more about “Interesting nooks and crannies of Spock” I will be speaking about them in May and June at GeeCON 2016 in Kraków, Gr8Conf 2016 in Copenhagen and Devoxx Poland again in Kraków.

Geecon big paw logo
GR8 Conf 2016 Europe
Devoxx Poland 2016 Speaker Badge

Self promotion. Would you like to improve your and your team testing skills and knowledge of Spock quickly and efficiently? I conduct condensed (unit) testing training which you may find useful.

Have you ever experienced the “Could not find property X on plugin extension Y” error with a freshly cloned GitHub project you wanted to contribute to?

Missing username, password or token to a service you may have never heard of? It usually happens when you try to do anything (like just build a project) not only when a given plugin (like an online code coverage tool) is used. I didn’t like to have to modify my environment to just provide a small fix to another open source project. It was annoying me and I wanted to change it. Starting with Gradle 2.13 it became possible. However, let’s start with the reasons (if you are interested only in the solution please move forward to the last 2 paragraphs).

Gradle logo

Why do I get “Could not find property…”?

Most of Gradle plugins need to be configured. Some properties can be set directly in build.gradle, but some others (especially credentials) are better to keep locally in ~/.gradle/gradle.properties. As a result, a plugin configuration sections often look like this:

bintray {
    user = project.getProperty('bintrayUser')
    key = project.getProperty('bintrayKey')
    ...
}

or that:

bintray {
    user = getProperty('bintrayUser')
    key = getProperty('bintrayKey')
    ...
}

or even shorter:

bintray {
    user = bintrayUser
    key = bintrayKey
    ...
}

It works fine for a project developer having bintrayUser and bintrayKey defined in its local configuration, but for every person not uploading to Bintray on their daily basis it fails with:

* What went wrong:
A problem occurred evaluating root project 'another-nice-open-source-project'.
> Could not find property 'bintrayKey' on com.jfrog.bintray.gradle.BintrayExtension_Decorated@2ecc563.

The result is that project.getProperty(), not to mentioned explicit assignment, just throws exception when a particular property is not found. The bad is that the code is executed in the configuration phrase. For that reason the execution of every task, even not related to that particular plugin (like gw tasks or gw wrapper) fails miserably.

As a workaround a guard check has to be performed:

bintray {    //Gradle <2.13
    user = hasProperty('bintrayUser') ? getProperty('bintrayUser') : ''
    key = hasProperty('bintrayKey') ? getProperty('bintrayKey') : ''
    ...
}

It doesn’t look good very compact. As an another option a dummy placeholder could be kept in project configuration, but starting with Gradle 2.13 there is a better way to cope with that.

project.findProperty()

Gradle 2.13 is the first version with my contribution of the new method project.findProperty(). It behaves the same as getProperty(), but instead of throwing an exception the null value is returned. This simplifies the assignment greatly:

bintray {    //Gradle 2.13+
    user = findProperty('bintrayUser') ?: ''
    key = findProperty('bintrayKey') ?: ''
    ...
}

Some people could say that Optional could be better as a returned value, but this is an API and Gradle supports Java older than 8.

Summary

For me findProperty is a method I’ve been very often looking for in Gradle. I regret that it took me over the year to make this pull request. Gradle 2.13 has been just released and version upgrades across projects will be performed gradually. It can take some time, but every project migrating to 2.13 will be able to simplify its configuration making the “Could not find property X on plugin Y” error message a remembrance of the past (of course unless you really need to configure particular plugin to use it :) ).

Tested with Gradle 2.13.