Docker Swarm and Multicast

In one of my projects I had to use Docker Swarm instead of Kubernetes, as the operations department dictated it (mostly because it looked like less work for them).
As the project uses Hazelcast IMDG for some caching and some locks, I needed to find a Hazelcast discovery strategy that fitted my environment.

Docker Swarms overlay network uses libnetwork, which to this date, does not support multicast message transport. See this Github ticket for details.

The recommended peer discovery strategy for Hazelcast is Multicast, but as this was not supported by the environment I had to find alternatives.
I’ve tested the following alternatives:

hazelcast-docker-swarm-discovery-spi

By using this strategy, Hazelcast asks the Docker API the container is running on for peers. This did not work, as the Docker API also reported containers that were just starting or about to stop. This caused the Hazelcast library to run into connection timeouts when the container were starting up and caused the Swarm Manager to stop a container, adding to the problem – now the Docker API could report two containers that are about to be stopped. In the end this caused boot-stop loops in my project and made a flexible and resilient system impossible.

hazelcast-zookeeper

This strategy registeres new peers in Zookeeper, making discovery easy. But again, if a container was stopped, the registered peers entry was not removed from the zookeeper registry, causing dead peers to hinder spinning up new Hazelcast cluster members.

Weave Net

To circumvent the above mentioned problems and enable multicast across all Docker Swarm nodes, the Weave Net docker plugin was installed on them.

Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery. As Weave encapsulates the traffic, multicast also works across data centers, even if the network in-between does not support multicast routing.

Im using the Docker Swarm plugin version here, not the standalone version, because the standalone version does not support swarm mode.

Installing the Weave Net Plugin

Firewall Rules

If you’re using iptables on the Docker Swarm nodes, add the following firewall rules to iptables:

Install docker plugin

Run the following on all Swarm nodes to install the docker plugin and configure it with the current setups variables.

Make sure to disable the plugin before setting the configuration parameters. See here for full documentation on the plugin.

Test the docker plugin

After installing the plugin, run the following to check if all Weash Net peers could be discovered correctly.
Running weave status weave status gives you something like this for the connection status to other peers:

Using the Weave Net Plugin

Even thought the official documentation states that the plugin is called “weaveworks/net-plugin:latest_release“, I had to prepend the /store (resulting in “store/weaveworks/net-plugin:latest_release“) in my setup for docker to recognise the plugin correctly.

You can create a network with the following statment:

Or use the following as network definition in the stack file and let the swarm manager create the network for you.