Multicast with Docker Swarm

Docker Swarm and Multicast
In one of my projects I had to use Docker Swarm instead of Kubernetes, as the operations department dictated it (mostly because it looked like less work for them).
As the project uses Hazelcast IMDG for some caching and some locks, I needed to find a Hazelcast discovery strategy that fitted my environment.
Docker Swarms overlay network uses libnetwork, which to this date, does not support multicast message transport. See this Github ticket for details.
The recommended peer discovery strategy for Hazelcast is Multicast, but as this was not supported by the environment I had to find alternatives.
I’ve tested the following alternatives:
hazelcast-docker-swarm-discovery-spi
By using this strategy, Hazelcast asks the Docker API the container is running on for peers. This did not work, as the Docker API also reported containers that were just starting or about to stop. This caused the Hazelcast library to run into connection timeouts when the container were starting up and caused the Swarm Manager to stop a container, adding to the problem – now the Docker API could report two containers that are about to be stopped. In the end this caused boot-stop loops in my project and made a flexible and resilient system impossible.
hazelcast-zookeeper
This strategy registeres new peers in Zookeeper, making discovery easy. But again, if a container was stopped, the registered peers entry was not removed from the zookeeper registry, causing dead peers to hinder spinning up new Hazelcast cluster members.
Weave Net
To circumvent the above mentioned problems and enable multicast across all Docker Swarm nodes, the Weave Net docker plugin was installed on them.
Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery. As Weave encapsulates the traffic, multicast also works across data centers, even if the network in-between does not support multicast routing.
Im using the Docker Swarm plugin version here, not the standalone version, because the standalone version does not support swarm mode.
Installing the Weave Net Plugin
Firewall Rules
If you’re using iptables on the Docker Swarm nodes, add the following firewall rules to iptables:
1 2 3 4 5 6 7 8 |
# weave communication ports iptables -A INPUT -p tcp --dport 6783 -j ACCEPT iptables -A INPUT -p udp --dport 6783 -j ACCEPT iptables -A INPUT -p udp --dport 6784 -j ACCEPT # weave net metrics ports iptables -A INPUT -p tcp --dport 6781 -j ACCEPT iptables -A INPUT -p tcp --dport 6782 -j ACCEPT |
Makes sure to persist these rules, like in
/etc/systemd/scripts/iptables
(depending on your OS)
Install docker plugin
Run the following on all Swarm nodes to install the docker plugin and configure it with the current setups variables.
1 2 3 4 5 |
docker plugin install store/weaveworks/net-plugin:latest_release docker plugin disable store/weaveworks/net-plugin:latest_release docker plugin set store/weaveworks/net-plugin:latest_release WEAVE_PASSWORD=SomePassword docker plugin set store/weaveworks/net-plugin:latest_release WEAVE_MULTICAST=1 docker plugin enable store/weaveworks/net-plugin:latest_release |
Make sure to disable the plugin before setting the configuration parameters. See here for full documentation on the plugin.
Test the docker plugin
After installing the plugin, run the following to check if all Weash Net peers could be discovered correctly.
Running weave status
weave status gives you something like this for the connection status to other peers:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
root@docker-swarm01 [ ~ ] weave status Version: 2.4 Service: router Protocol: weave 1..2 Name: aa:bb:cc:dd:ee:ff(docker-swarm01) Encryption: enabled PeerDiscovery: enabled Targets: 3 Connections: 3 (3 established) Peers: 4 (with 12 established connections) TrustedSubnets: none Service: ipam Status: idle Range: 10.22.0.0/12 DefaultSubnet: 10.22.0.0/12 |
Notice that under
Connections
it states 3, this should correspondent to the count of other docker cluster nodes. In Peers
it says 4, this should correspondent the the overall docker cluster member count.
Using the Weave Net Plugin
Even thought the official documentation states that the plugin is called “weaveworks/net-plugin:latest_release
“, I had to prepend the /store
(resulting in “store/weaveworks/net-plugin:latest_release
“) in my setup for docker to recognise the plugin correctly.
You can create a network with the following statment:
1 |
docker network create --driver=store/weaveworks/net-plugin:latest_release example_net |
Or use the following as network definition in the stack file and let the swarm manager create the network for you.
1 2 3 4 5 6 7 8 9 10 11 |
version: '3.4' services: some-service: image: some_image .... networks: - example_net ... networks: example_net: driver: store/weaveworks/net-plugin:latest_release |
Troubleshooting
Weave has a very nice troubleshooting page, which should help you solve any problems quickly.