Easier work with Git submodules with in-place push access – Git Tricks #2

Posted: 2019-09-03 in Tricks & Tips
Tags: ,

Get know how to solve issue with pushing to submodules directly from the main repo while keeping the project easily cloneable by external contributors.

Introduction

The Git submodules mechanism is pretty handy to keep the source code of lousily related dependent software together in one Git repository while leaving their development separate. It is something like symlinks in the Unix world, but with an ability to refer also to the previous version. It’s quite popular in the projects using source code integration (instead of shared libraries) or to speed up development by making related changes in multiple repositories easier. It is not the only possible solution – Gradle, for instance, provides a composite build mechanism. Monorepo is an another approach, but it has its own limitations and it very problematic to use in FOSS projects developed by different people/teams.

As usual, I encountered that situation in one of my projects. As a big enthusiast of automatic code testing and Continuous Delivery, some time ago I have been working on improving the reliability of my (automatically released) gradle-nexus-staging-plugin. After each commit (so also before the release) I wanted to have the end-to-end tests executed to verify that the plugin is able to pass a simple (but real) project through the release process to Maven Central (aka The Central Repository).

I could move that test project to the plugin repository, but – well – it’s a distinct project which could be also released separately or replaced with some other project. In addition, testing two variants of releasing it was handy to use keep it in two branches “mounted” twice in my root repository.

gradle-nexus-staging-plugin
├── src
│   ├── funcTest
│   │   ├── groovy
│   │   │   └── ...
│   │   └── resources
│   │       └── sampleProjects
│   │           ├── nexus-at-minimal (submodule - master branch)
│   │           │   └── ...
│   │           ├── nexus-at-minimal-publish (submodule - publish branch)
│   │           │   └── ...
│   │           └── ...
│   ├── ...

Problem (aka Challenge)

gradle-nexus-staging-plugin contains a submodule nexus-at-minimal from a separate repo. Working on new fancy feature in GNSP it is needed to tweak the acceptance testing project. To make the development smoother it’s very useful to be able to commit introduces changes in the dependent project directly from the working copy of the main project. It works out of the box. However, later on we should also directly push them back to a separate repo of that dependent project (by stepping into the submodule and calling git push or – all-in-once by – using the git push --recurse-submodules=on-deamand parameter when pushing from the main repository). And then – in the long term – we encounter some inconvenience.

Let’s start by defining a submodule path to connect via SSH:

[submodule "src/funcTest/resources/sampleProjects/nexus-at-minimal"]
        path = src/funcTest/resources/sampleProjects/nexus-at-minimal
        url = git@gitlab.com:nexus-at/nexus-at-minimal.git

In general it works fine and pushing is allowed. However, any non-developer trying to clone that repo (and initialize submodules) gets:

...
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

It’s pretty bad in the open source (FOSS) development where the projects are publicly available other people are encouraged to download (clone) it, build and contribute.

The obvious remedy to the previous error is switching to HTTPS:

[submodule "src/funcTest/resources/sampleProjects/nexus-at-minimal"]
        path = src/funcTest/resources/sampleProjects/nexus-at-minimal
        url = https://gitlab.com/nexus-at/nexus-at-minimal.git

However then, to allow developers to commit changes directly to a submodule it is required to go through a separate HTTPS authentication which in the most cases is completely not use in favor of SSH (and would need to be configured independently with an access token).

Transparent solution

To keep external contributors happy the developers could manually change the url in .gitmodules from HTTPS to SSH for every cloned project. However, it’s quite tedious. The better solution is an usage of pushInsteadOf. For the aforementioned example, the developers need only to add to the global ~/.gitconfig configuration file:

[url "git@gitlab.com:nexus-at/"]
    pushInsteadOf = https://gitlab.com/nexus-at/

It effectively overrides the push URL to use SSH instead of HTTPS for the whole group in GitLab/GitHub, covering also submodules (where HTTPS – external contributors friendly scheme – is left as a default).

Simple solution for the same server

As I use GitLab for my profession activity, at that time, I was evaluating its usage also for my FOSS project. Therefor, two different servers (providers) used for the main project and a submodule (here GitHub and GitLab). However, very often it is placed on the same server or even in the same group. In that case the configuration can be simplified even more with relative paths:

[submodule "src/funcTest/resources/sampleProjects/nexus-at-minimal"]
        path = src/funcTest/resources/sampleProjects/nexus-at-minimal
        url = ../nexus-at-minimal.git

The protocol used to clone the main repository (SSH or HTTPS) will be used also for a submodule.

Summary

URL rewriting and conditional configuration (covered in the first part) are just a subset of options available in Git to make the development more flexible and simple. Simple, assuming we already found the required features and learned how to use it ;-).

Update 20190405. Added simple solution with a relative path.

The lead photo based on the Mohamed Hassan‘s work published in Pixabay, Pixabay License.

Comments
  1. […] >> Easier work with Git submodules with in-place push access – Git Tricks #2 [solidsoft.com] […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s