Spring – PM AND CODE

Continuous Integration using Gitlab CI with Docker

07/09/201707/09/2017
by zeisi

Gitlab.com

Most people I know prefer Github or a private server for hosting their private git repositories.
Nothing wrong with that, everyone has their own workflow.
I’ve been using Gitlab for a few years now, privately hosted. A few months ago, my Gitlab server had some problems after upgrading to a newer version. I needed the repositories back online fast, so I switched to using gitlab.com – and never looked back. A free account on gitlab.com gives you most features you’ll ever want, like built-in CI, issue tracker or even cycle analytics.
This wont be another Github vs. Gitlab article, in this post I want to highlight one of the features I’ve come to enjoy the most, Gitlab CI.

What we’re gonna build

For most of my projects I use Maven for building the packages and even for building the docker images.
I was always to lazy to set up a continuous integration system for my private projects, mostly because I was the only one commiting to the repositories, so nothing to integrate. For a recent project I was working together with someone and wanted automate the Junit tests and builds to catch any discrepancies early.

For me, the most annoying part of setting up unit tests was, that if you wanted to run tests against the database, you’d also have to manage that stuff. Still, it’s well worth the effort to make sure stuff like migration scripts work. Sure, you can just use a H2 in-memory database, which is fine for most simple databases, but as soon as you want to use specific database types (like Postgres JSONB), you’re stuck.
One of the advantages of Gitlab CI is that you can use docker images to run your test. So instead of setting up a special database instance / schema for our unit tests, we can just pull in a Postgres Docker image, spin it up and run our tests against that one!

You’ll also have to choose a docker image compatible to your production environment, because Gitlab CI will run the tests in them.

Setup

So to test the features we’ll talk about here, go to gitlab.com, register a free user and push a repository with a maven enabled project.
Got to your repository in the Gitlab web UI and click on Pipeline. Gitlab will help you setup your config.

.gitlab-ci.yml

If you add a .gitlab-ci.yml (notice the preceding dot) file to the root directory of your repository, and configure your GitLab project to use a Runner, then each commit or push triggers your CI pipeline.
In GitLab CI, Runners run the code defined in .gitlab-ci.yml . They are isolated (virtual) machines that pick up jobs through the coordinator API of GitLab CI.

A Runner can be specific to a certain project or serve any project in GitLab CI. A Runner that serves all projects is called a shared Runner.

We’ll use a shared runner, running docking. You can see which runners are available for your project by going to Setting -> Pipelines.

You can define when you want to trigger the build, on stuff like push to a specific branch or when pushing a tag or just timed.

To configure the build pipelines in Gitlab CI, create a file .gitlab-ci.yml in the root of your repository.

The .gitlab-ci.yml file tells the GitLab runner what to do. By default it runs a pipeline with three stages: build, test, and deploy.
You don’t need to use all three stages; stages with no jobs are simply ignored.

You’ll also have to choose a docker image compatible to your production environment, because Gitlab CI will run the tests in them.

As I use Maven 3.5 with JDK8 I’ll use the tag maven:3.5.0-jdk-8-alpine , but have a look at Docker Hub to find one that’s right for you.
Next, I want to make use of the new parallel query features in Postgres 9.6, so I’ll run my test against the tag postgres:latest .

Example

Here’s an example .gitlab-ci.yml – it’s a modified version of the maven template available on gitlab.com:

image: maven:3.5.0-jdk-8-alpine

services:
- postgres:latest

variables:
# Postgres container config, this sets up our test instance with a fresh database and the defined user
POSTGRES_DB: testdatabasename
POSTGRES_USER: testuser
POSTGRES_PASSWORD: "testpassword"
# Maven config
# This will supress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd`are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"

# Cache downloaded dependencies and plugins between builds.
# To keep cache across branches add 'key: "$CI_JOB_REF_NAME"'
cache:
paths:
- .m2/repository

build:
script: mvn clean package -DargLine="-Dspring.profiles.active=ci"

image: maven:3.5.0-jdk-8-alpine

services:

- postgres:latest

variables:

# Postgres container config, this sets up our test instance with a fresh database and the defined user

POSTGRES_DB: testdatabasename

POSTGRES_USER: testuser

POSTGRES_PASSWORD: "testpassword"

# Maven config

# This will supress any download for dependencies and plugins or upload messages which would clutter the console log.

# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.

MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"

# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used

# when running from the command line.

# `installAtEnd` and `deployAtEnd`are only effective with recent version of the corresponding plugins.

MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"

# Cache downloaded dependencies and plugins between builds.

# To keep cache across branches add 'key: "$CI_JOB_REF_NAME"'

cache:

paths:

- .m2/repository

build:

script: mvn clean package -DargLine="-Dspring.profiles.active=ci"

Let’s go through the example line by line:

1. This tell Gitlab CI what docker image we want our script to run on
3. Here we define any docker images our tests depend on. In this case, the latest available Postgres image.
6. When receiving a job from GitLab CI, the Runner prepares the build environment. Variables allow you to configure certain aspects of this environment.
8-10 The database and credentials config for our Postgres docker image. Naturally, your tests need to be configured to use these credentials, for more see below. Docker will provide use with a hostname, it will just be “postgres”
14-18 Common maven options
22. Caching the dependencies will speed up the build times.
24. A job definition, the job names are arbitrary.

Put this .gitlab-ci.yml into your repository root, push it and in the Gitlab web UI, go to Pipelines.
As soon as Gitlab sees the config it will run the pipeline.
Gitlab CI runs a linter on your config, so if you have a typo in it you’ll notice it immediately.

To view the log of the running process click on “Running”

Bonus – configure your tests in a Spring project to use the CI database

I make use of Spring Profiles here, as you can see in the parameter -DargLine="-Dspring.profiles.active=ci above.
This tells Spring to use the profile CI. For more on Spring Profiles see my post here.
I have a file application-ci.properties under src/test/ressource :

spring.datasource.url = jdbc:postgresql://postgres/testdatabasename
spring.datasource.username = testuser
spring.datasource.password = testpassword
spring.datasource.driverClassName: org.postgresql.Driver

spring.datasource.url = jdbc:postgresql://postgres/testdatabasename

spring.datasource.username = testuser

spring.datasource.password = testpassword

spring.datasource.driverClassName: org.postgresql.Driver

Category: Spring