Posts Tagged ‘maven’

How to communicate with Maven Central/Nexus without using the password kept locally unencrypted (especially with Gradle, but not limited to it).

Rationale

Unfortunately, Gradle (and many other build tools) does not provide any mechanism to locally keep passwords encrypted (or at least encoded). Without that even such a simple activity like showing your global Gradle configuration (~/.gradle/gradle.properties) to a colleague it uncomfortable, not to mention more serious risks associated with storing passwords on a disk in a plain-text form (see among others Sony Pictures Entertainment hack). It is Gradle, so with all Groovy magic under the hood it would be possible to implement an integration with a system keyring on Linux to fetch a password, but I’m not aware of any existing plugin/mechanism to do that and I would rather prefer not to write it.

Another issue is that nowadays, in the world of ubiquitous automation and cloud environments it is common to use API keys which allow to perform given operation(s). However, its lost doesn’t provide an attacker a possibility to hijack the account (e.g. token cannot be used neither to log into an administration panel nor to change of email or password which requires additional authentication).

It is very important if you need to keep valid credentials on a CI server to make automatic or even continuous releases. Thanks to my gradle-nexus-staging-plugin there is no need to do any manual steps in Nexus GUI to promote artifacts to Maven Central, so this was the next issue I wanted to deal with for my private and our FOSS projects in Codearte.

Nexus API key generation

Internet search for “maven central api key” wasn’t helpful, so I started digging into Nexus REST API documentation and I’ve found that in fact there is a (non widely known) way to generate and use an API key (aka an auth token).

0. Log into Nexus hosting Sonatype OSS Repository Hosting (or your own instance of Nexus).
1. Click on your login name in right-upper corner and choose “Profile”.
2. From the drop-down list with “Summary” text select “User Token”.
3. Click “Access User Token”.

Generating API key in Nexus

Generating API key in Nexus

5. Enter your password
6. Copy and paste your API username and API key (into your ~/.gradle/gradle.properties file or a CI server configuration).
7. Work as usual with a little safer way.

Summary

It is good that using API keys is possible to deploy artifacts to Maven Central/Nexus and it is very easy to set it up. Someone could argue that the permission policy is coarse-grained (nothing or all operations except password/email change), but in my opinion it seems to be enough for the artifact repository system class. In addition, such an approach should work also with Sbt, Ivy, Leiningen and everything else that tries to upload artifacts into Maven Central (including Maven itself by removing limitations of the master password encryption with settings-security.xml). Hopefully, that post will make it widely known.

Quick tutorial how to promote/release artifacts in a Gradle project to Maven Central, without clicking in the Nexus GUI with Gradle Nexus Staging Plugin.

Introduction

Maven Central (aka The Central Repository) is (probably) the world’s largest set of open source artifacts used by Java and JVM-based projects. It was founded by the creators of Apache Maven and it has been serving artifacts since 2002. Nowadays there are some alternatives (listed below), but for many users Maven Central is still the primary source of project dependencies (and sometimes the only one whitelisted in the corporations).

The Central Repository logo

Problem

To perform the release to The Central Repository, Maven users can use Nexus Staging Maven Plugin – free, but not fully open source plugin. But with Gradle it was required to login into Nexus GUI and manually invoke two actions (close repository and release/promote repository). Quite boring and in addition highly problematic with the Continuous Delivery approach. Luckily Nexus exposes REST API which with some work allows to do the same. Gradle Nexus Staging Plugin arose to do that job.

Quick start

Important. Please pay attention that the prerequisite is to have an active and configured account in Sonatype OSSRH (OSS Repository Hosting) as well as Gradle project configured to publish release artifacts to staging repository. If you don’t have it already please follow a separate section for Gradle in the official guide.

To setup automatic release/promotion in your project add gradle-nexus-staging-plugin to the buildscript dependencies in your build.gradle file for root project:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath "io.codearte.gradle.nexus:gradle-nexus-staging-plugin:0.5.1"
    }
}

Apply the plugin:

apply plugin: 'io.codearte.nexus-staging'

Configure it:

nexusStaging {
    packageGroup = "org.mycompany.myproject"
    stagingProfileId = "yourStagingProfileId" //when not defined will be got from server using "packageGroup"
}

After successful archives upload (with maven, maven-publish or nexus plugin) to Sonatype OSSRH call:

./gradlew closeRepository promoteRepository

to close staging repository and promote/release it and its artifacts. If a synchronization with Maven Central was enabled the artifacts should automatically appear into Maven Central within several minutes.

Details

The plugin provides two main task:

  • closeRepository – closes the open repository with uploaded artifacts. There should be just one open repository available in the staging profile (possible old/broken repositories can be dropped with Nexus GUI)
  • promoteRepository – promotes/releases closed repository (required to put artifacts to Maven Central)

And one additional:

  • getStagingProfile – gets and displays staging profile id for a given package group. This is a diagnostic task to get the value and put it into the configuration closure as stagingProfileId. To see the result it is required to call gradle with --info switch.

It has to be mentioned that calling Nexus REST API ends immediately, but the closing operation takes a moment, so to make it possible to call closeRepository promoteRepository together there is a built-in retry mechanism.

The plugin is “upload mechanism agnostic” and can be used together with maven, maven-plugin or nexus plugins.

For more details and configuration parameters see project webpage or the working example in the plugin’s own release configuration.

Alternatives to Maven Central?

There is much younger, but promising alternative – Bintray which also allows to serve artifacts. It is free for open source projects and I personally had used it for some other projects and even created an automatic release mechanism for Bintray, Travis and Gradle. It works ok, but to put artifacts also to Maven Central it is required to store a private key used for singing on their servers and in addition provide Nexus credentials. It increases the risk to have them stolen and in Codearte we prefer to use private Jenkins instance to perform the release directly to Maven Central.

Summary

With Gradle Nexus Staging Plugin the whole release process to Maven Central can be performed with Gradle from a command line and with some additional work completely automatic from a CI server. No more buttons to push in Nexus GUI. In addition to Sonatype OSSRH the plugin can be also used with private Nexus instances with enabled staging repositories.

Btw, there possibly are many things that could be enhancement in the plugin. If you need something or found a bug feel free to use issue tracker to report that.

Thanks to Kuba Kubryński for motivation and help with analyzing the not very well documented Nexus REST API.

Quick tutorial how to configure Spock 1.0 with Groovy 2.4 using Maven and Gradle.

Spock 1.0 has been finally released. About new features and enhancements I already wrote two blog posts. One of the recent changes was a separation on artifacts designed for specific Groovy versions: 2.0, 2.2, 2.3 and 2.4 to minimize a chance to come across a binary incompatibility in runtime (in the past there were only versions for Groovy 1.8 and 2.0+). That was done suddenly and based on the messages on the mailing list it confused some people. After being twice asked to help properly configure two projects I decided to write a short post presenting how to configure Spock 1.0 with Groovy 2.4 in Maven and Gradle. It is also a great place to compare how much work is required to do it in those two very popular build systems.

Maven

Maven does not natively support other JVM languages (like Groovy or Scala). To use it in the Maven project it is required to use a third party plugin. For Groovy the best option seems to be GMavenPlus (a rewrite of no longer maintained GMaven plugin). An alternative is a plugin which allows to use Groovy-Eclipse compiler with Maven, but it is not using official groovyc and in the past there were problems with being up-to-date with the new releases/features of Groovy.

Sample configuration of GMavenPlus plugin could look like:

<plugin>
    <groupId>org.codehaus.gmavenplus</groupId>
    <artifactId>gmavenplus-plugin</artifactId>
    <version>1.4</version>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
                <goal>testCompile</goal>
            </goals>
        </execution>
    </executions>
</plugin>

As we want to write tests in Spock which recommends to name files with Spec suffix (from specification) in addition it is required to tell Surefire to look for tests also in those files:

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire.version}</version>
    <configuration>
        <includes>
            <include>**/*Spec.java</include> <!-- Yes, .java extension -->
            <include>**/*Test.java</include> <!-- Just in case of having also "normal" JUnit tests -->
        </includes>
    </configuration>
</plugin>

Please notice that it is needed to include **/*Spec.java not **/*Spec.groovy to make it work.

Also dependencies have to be added:

    <dependencies>
        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy-all</artifactId>
            <version>2.4.1</version>
        </dependency>
        <dependency>
            <groupId>org.spockframework</groupId>
            <artifactId>spock-core</artifactId>
            <version>1.0-groovy-2.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

It is very important to take a proper version of Spock. For Groovy 2.4 version 1.0-groovy-2.4 is required. For Groovy 2.3 version 1.0-groovy-2.3. In case of mistake Spock protests with a clear error message:

Could not instantiate global transform class
org.spockframework.compiler.SpockTransform specified at
jar:file:/home/foo/.../spock-core-1.0-groovy-2.3.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation
because of exception
org.spockframework.util.IncompatibleGroovyVersionException:
The Spock compiler plugin cannot execute because Spock 1.0.0-groovy-2.3 is
not compatible with Groovy 2.4.0. For more information, see
http://versioninfo.spockframework.org

Together with other mandatory pom.xml elements the file size increased to over 50 lines of XML. Quite much just for Groovy and Spock. Let’s see how complicated it is in Gradle.

Gradle

Gradle has built-in support for Groovy and Scala. Without further ado Groovy plugin just has to be applied.

apply plugin: 'groovy'

Next the dependencies has to be added:

compile 'org.codehaus.groovy:groovy-all:2.4.1'
testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'

and the information where Gradle should look for them:

repositories {
    mavenCentral()
}

Together with defining package group and version it took 15 lines of code in Groovy-based DSL.

Btw, in case of Gradle it is also very important to match Spock and Groovy version, e.g. Groovy 2.4.1 and Spock 1.0-groovy-2.4.

Summary

Thanks to embedded support for Groovy and compact DSL Gradle is preferred solution to start playing with Spock (and Groovy in general). Nevertheless if you prefer Apache Maven with a help of GMavenPlus (and XML) it is also possible to build project tested with Spock.

The minimal working project with Spock 1.0 and Groovy 2.4 configured in Maven and Gradle can be cloned from my GitHub.

Graphical comparison of Spock and Groovy configuration in Maven and Gradle

Bonus: Graphical comparison of Spock and Groovy configuration in Maven and Gradle

Note. I haven’t been using Maven in my project for over 2 years (I prefer Gradle), so if there is a better/easier way to configure Groovy and Spock with Maven just let me know in the comments.

Note 2. The configuration examples assume that Groovy is used only for tests and the production code is written in Java. It is possible to mix Groovy and Java code together, but then the configuration is a little more complicated.

Note 3. If you are interested in get know useful tips and tricks about using Spock Framework to test your Java and Groovy code I will have a presentation about that at 4Developers conference, April 20th, 2015.

Update 20150310. Redesigned summary.

Leonard Nimoy 1931-2015

Recently I wanted to configure an ability to run both TestNG and JUnit tests in one Maven module (project). At the end I managed how to do it clean and short, but before that I have found a few different solutions on the web (top 5 in Google) which part of them didn’t work and the rest applied to the earlier versions of Surefire plugin and was overly complicated (e.g. two separate executions). Therefore I decided to write this short post to show how it could be done in Surefire 2.13 – the newest version available in March 2013.

Mixing those tests frameworks in one module can be done just by adding both JUnit and TestNG as a plugin dependencies (not as project dependencies):

<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>${surefire.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-testng</artifactId>
            <version>${surefire.version}</version>
        </dependency>
    </dependencies>
</plugin>

As the result both test types are executed.

[INFO] --- maven-surefire-plugin:2.13:test (default-test) @ junit-testng-poc ---
[INFO] Surefire report directory: /tmp/junit-testng-poc/target/surefire-reports
[INFO] Using configured provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running info.solidsoft.rnd.junit.testng.SampleJUnitTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.253 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0


-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: TestNG652Configurator
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.869 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Why to use TestNG with JUnit together? TestNG has a few features which are unavailable or less flexible in JUnit (just to mention a few: dependencies between tests and groups of tests (irreplaceable for integration tests with long startup), parametrized tests, concurrent execution or per suite/group/class init/shutdown operations). Therefore it is tempting to migrate existing tests from JUnit to TestNG. Having large code base it could be not so easy to migrate all of them at once and presented configuration allows to write the new tests in TestNG and rewrite the old tests when appropriate.

The whole working example can be found in my GitHub repository.

Btw, it is worth to mention that thanks to the fact TestNG generates reports also in the JUnit’s XML format all the tools compatible with JUnit (Jenkins, Sonar, …) will merge test results from JUnit and TestNG and display all of them properly.

Btw2, the same configuration works also with Failsafe plugin.

Btw3, thanks to the fact Spock Framework is under the hood a runner for JUnit presented trick can be used also to mix it with TestNG. This requires some additional work to integrate also Groovy and I plan to write about it in one of my future posts.

Mutation testing can efficiently detect places in code which are insufficiently covered by tests. The price we have to pay for it is time – number of mutations has to be tested with a set of unit tests. This time is much longer than calculating a “normal” code coverage. The newest PIT 0.29 provides a long awaiting feature – incremental analysis. When enabled PIT will store results from the previous runs on disk and track changes in the code and tests to avoid rerunning analyses which result should stay the same.

To start using incremental analysis it is necessary set historyInputLocation and historyOutputLocation configuration properties. For example in Pitest Maven Plugin it could be:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>0.29</version>
    <configuration>
        <threads>4</threads>
        <historyInputLocation>target/pitHistory.txt</historyInputLocation>
        <historyOutputLocation>target/pitHistory.txt</historyOutputLocation>
    </configuration>
</plugin>

It is worth to notice that for a basic usage (i.e. run analysis from time to time) input and output history locations will point to the same file. Therefor in Gradle plugin for PIT 0.29.0 there was added an additional parameter enableDefaultIncrementalAnalysis which when enabled automatically set historyInputLocation and historyOutputLocation to build/pitHistory.txt simplifying a configuration.

buildscript {
    (...)
    dependencies {
        classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:0.29.0'
        classpath 'org.pitest:pitest:0.29'
    }
}

apply plugin: 'pitest'

pitest {
    targetClasses = ['our.base.package.*']
    threads = 4
    enableDefaultIncrementalAnalysis = true
}

There is a number of optimizations already implemented in PIT. The author warns of potential errors which can be introduced in that way into the analysis, but although being currently an experimental feature it can dramatically reduce calculation time especially when used on very large codebases.

Update 2013-04-13: Added missing line which applies plugin to a project. Thanks to Bruno de Carvalho for report the issue.

Mutation testing is a technique which allows to discover which parts of our code are not covered by tests. It is similar to a code coverage, but mutation testing is not limited to the fact that a given line was executed during tests. The idea is to modify the production code (introduce mutations) which should change its behavior (produce different results) and cause unit tests to fail. The lack of the failure may indicate that given part was not covered good enough by the tests. The idea of mutation testing is quite old, but it is rather unpopular. Despite the fact I am rather experienced in testing I found it just recently reviewing a beta version of the new book about testing.

PIT is “a fast bytecode based mutation testing system for Java that makes it possible to test the effectiveness of your unit tests”. It is a very young project, but very promising. It offers a set of mutation operators which among others modify conditional statements, mathematical operations, return values and methods calls.

Starting with the recently released version 0.25 PIT (experimentally) supports TestNG based tests (in addition to JUnit based). To use it from Maven it is required to add pitest-maven plugin to pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>0.25</version>
<!--
    <configuration>
        <inScopeClasses>
            <param>info.solidsoft.blog.pitest.*</param>
        </inScopeClasses>
        <targetClasses>
            <param>info.solidsoft.blog.pitest.*</param>
        </targetClasses>
    </configuration>
-->
</plugin>

In many cases it would be enough. inScopeClasses (mutable classes and tests for running) and targetClasses (only candidates for mutation) by default use project groupId and usually can be omitted. There are several options that can be configured in the plugin configuration. “mvn org.pitest:pitest-maven:mutationCoverage” performs modified tests and generates mutation report which by default is saved in target/pit-reports/yyMMddHHmm directory.

A sample report (click to enlarge) for a specified class shows both line coverage and mutation coverage. Despite 100% line coverage (lines with light green background), PIT found that a test data set does not cover properly boundary conditions.

Sample PIT report

Sample PIT report

To be useful automatic tests should run very fast. Otherwise will not be run often during development or even will be ignored in the default configuration on the developer workstations. The simplest rule it to write only small unit tests which test given class having the neighborhood mocked. Nevertheless sometimes it is useful/required to test something in IoC container context (Spring, CDI, Guice) or using embedded database (H2, HyperSQL, Derby). Unfortunately even a few tests of that kind can increase significantly overall test execution time. I had that situation in one of my projects and to prevent using skipTests flag I developed a solution using groups from TestNG and Maven Surefire plugin.

Tests were divided into three groups:
  – very fast real unit tests (all by default) – should be run very often during development (from IDE or by mvn test, mvn package)
  – slower integration but self-sufficient tests (setting up Spring context and/or using embedded H2 database) – should be run at least before commit/push or while working on given part (from IDE or by mvn integration-test, mvn install)
  – real integration tests (requiring access to remote servers – e.g. testing web services or REST) – should be run daily by CI server or by developers working on the integration (mvn install, mvn integration-test with additional profile enabled)

To achieve that given tests (or test classes) have to be marked as “self-integration” or “integration” (at method or class level):

    @Test(groups = "self-integration")
    public void shouldInitializeChainedAppInfoProperly() {
@Test(groups = "integration")
public class FancyWebServiceIntegrationTest {

Maven Surefire plugin should be configured to exclude “self-integration” and “integration” test groups from default execution and add “self-integration” to “integration-test phase:

    <build>
        <plugins>
            (...)
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${ver.surefire-plugin}</version>
                <executions>
                    <execution>
                        <id>default-test</id> <!-- to override default configuration - in fact: unit tests -->
                        <configuration>
                            <excludedGroups>self-integration,integration</excludedGroups>
                        </configuration>
                    </execution>
                    <execution>
                        <id>self-integration</id>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <groups>self-integration</groups>
                            <reportsDirectory>target/self-integration-surefire-reports/</reportsDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

In addition (if needed) the separate separate profile with “integration” test group configured in “integration-test” phase could be created.

    <profiles>
        (...)
        <profile>
            <id>integration</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-surefire-plugin</artifactId>
                        <version>${ver.surefire-plugin}</version>
                        <executions>
                            <execution>
                                <id>integration</id>
                                <phase>integration-test</phase>
                                <goals>
                                    <goal>test</goal>
                                </goals>
                                <configuration>
                                    <groups>integration</groups>
                                    <reportsDirectory>target/integration-surefire-reports/</reportsDirectory>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>

Working example can be find in AppInfo‘s artificial branch (pom.xml and sample test class). It’s easy to adopt it to your needs.

All three test groups have separate reports format to not override each other. As the extension it could probably possible to merge they into one aggregated test report.

TestNG is a testing framework created as an annotation driven alternative for JUnit 3 in times when “extends TestCase” was an indispensable part of writing tests. Even now it provides some interesting features like data providers, parallel tests execution or test groups. In the situation our tests are not executed from IDE it’s often useful to take a look at a test result in a HTML report. The original TestNG report looks… raw. What is more it is not very intuitive and readable. There is an alternative – ReportNG. It provides a better looking and more lucid HTML test reports.

More information about ReportNG can be found at its webpage, but when I tried to use for my AppInfo library in Maven builds running from a CI server I had a problem to find any at a glance guide how to use it with Maven. Fortunately there are samples for Ant and Gradle, so I was able to figure it out, but I hope with this post everyone wanting to use ReportNG with Maven will be able to achieve it without any problem within a few minutes.

First, the additional dependency has to be added to pom.xml:

<dependencies>
    <dependency>
        <groupId>org.uncommons</groupId>
        <artifactId>reportng</artifactId>
        <version>1.1.2</version>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>org.testng</groupId>
                <artifactId>testng</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    (...)
</dependencies>

Usually in our project a newer TestNG version is used, so that ReportNG dependency should be excluded.

Next, Surefire plugin has to be configured:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.5</version>
            <configuration>
                <properties>
                    <property>
                        <name>usedefaultlisteners</name>
                        <value>false</value>
                    </property>
                    <property>
                        <name>listener</name>
                        <value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
                    </property>
                </properties>
                <workingDirectory>target/</workingDirectory>
            </configuration>
        </plugin>
        (...)
    </plugins>
</build>

ReportNG uses two reporters pluggable into TestNG. JUnitXMLReporter generates XML summarize of running tests. It’s used for tools (like CI server). HTMLReporter creates human readable HTML report. Default TestNG listeners should be disabled.

After a test run I added also a workingDirectory property which causes that velocity.log (file created by Velocity engine used internally by ReportNG) is placed in a target instead of main project directory (and therefore it is deleted by the “mvn clean” command).

One more thing. Unfortunately ReportNG jar isn’t available in Maven Central Repository, so it could be required to add java.net repository in your settings.xml.

<repositories>
    <repository>
        <id>java-net</id>
        <url>http://download.java.net/maven/2</url>
    </repository>
    (...)
</repositories>

That’s all. Now “mvn clean test” should generate a nice looking HTML report for lots of tests covering our project :).

Update 2012-08-23. This post was written with TestNG 5.x in mind. With TestNG 6.0+ you can meet a problem with “ClassNotFoundException: com.google.inject.Module” exception. In that case Guice dependency needs to be added. Thanks to Alexander Schikora for pointing it out.

<dependency>
    <groupId>com.google.inject</groupId>
    <artifactId>guice</artifactId>
    <version>3.0</version>
    <scope>test</scope>
</dependency>