Two years ago, on a Friday morning at work, I bought my first Raspberry Pi.
I didn't really knew what I was gonna use it for but I really loved the idea of a pocket computer so I got one and for the better part of the year that followed I used it as a media server attached to a TV. All this was awesome but I wanted it to do so much more.
About the same time, we started experimenting with Docker at work and as any good developer that has a linux server attached to his TV, I wondered can i run Docker on this?, the answer is yes and it's a lot easier than I thought, after a quick Google search I found the guys at hypriot who specialized in building Docker ready system images for ARM devices.
As the time went on and I had the chance to explore the Docker ecosystem, the release of Raspberry Pi 3 and of a reminiscing course on distributed systems made me decided to finally build a Raspberry Pi stack.
Building the hardware
Building the stack is not that hard, I could've wired everything on a table and that would've been everything but I strive to be tidy and organized so I opted to get a stackable case.
Parts:
- 3 x Raspberry Pi 3 (as I already had a Raspberry Pi 2)
- 1 x ethernet switch TP-LINK TL-SF1005D
- 1 x 5 port usb power hub
- 4 x micro usb cable
- 4 x Ethernet cable
- 4 x 32 gb microsd card
- 2 x stackable case
Two weeks later when all the parts arrived, I could finally start building so I unpacked everything, striped the protective foil from the layers on the case and planned the building process which took an hour or so. The only exciting part was when my switch would not fit beside the power hub in the bottom layer of the case so... I cut the switch out of the case.
I don't have any photos from the build process as I'm writing this half a year after the events described, but here's the final product.
Building the system images
The guys at hypriot have an awesome getting started guide, I'll present here what I did the Mac OS X as that's what i needed.
Steps to reproduce:
First things first, download the latest image from the download page.
Go to your downloads folder.
cd ~/Downloads
Extract the zip, this should have a new .img file.
unzip hypriot-rpi-???.img.zip
Insert the SD card in your computer and find the disk alocated to it by running:
diskutil list
For this exampe I'll presume your SD card is mounted at /dev/disk4.
Let's unmount the disk and prepare for flashing.
diskutil unmountdisk /dev/disk4
Now we are ready to flash the SD card. We are going to use the dd command for this.
Before you execute the command below, make sure to
- replace the parameter after
if=with the path to the downloaded image - replace the parameter after
of=with the identifier of your SD card - make sure you put a r in front of disk as you can see in the example
sudo dd if=hypriot-rpi-???.img of=/dev/rdisk4 bs=1m
Now we only have to do the same thing for the other 3.
I did a quick search on their repo and found flash, it's a tool they developed to make this whole process simpler and you can set the WiFi credentials during flash time which skips the whole monitor and keyboard part making the process more lean.
For the master node, after inserting the SD card in the Raspberry Pi, connect it to a monitor, attach a keyboard to it and wait for it to boot you can set the WiFi connection by updating your /boot/occidentalis.txt to something like:
# hostname for your Hypriot Raspberry Pi:
hostname=rpi-0
# basic wireless networking options:
wifi_ssid=SSID
wifi_password=12345
And with this done we can access the rpi-0 through ssh.
ssh [email protected]
Configuring the networking layer
Even though each Raspberry Pi can connect has wireless, I did not want to clutter my network so I opted for a network architecture described in the diagram below.
Having the general idea in mind represented by a master node that acts as ethernet access point for the other 3 nodes connected to the switch and knowing that from now on everything happens on the master node, let's connect to it by ssh [email protected] assuming this is the ip assigned to the node from the router.
Like any good linux setup we start with updating the system.
sudo apt-get update && sudo apt-get upgrade
###### Configuring the DHCP Server
Install the dhcp server.
sudo apt-get install isc-dhcp-server
In the configuration file /etc/dhcp/dhcpd.conf comment the next 2 lines
#option domain-name "example.org";
#option domain-name-servers ns1.example.org, ns2.example.org;
And uncomment the next one
authoritative;
After that, at the bottom of the file define the new subnet that will serve the other nodes.
subnet 192.168.2.0 netmask 255.255.255.0 {
range 192.168.2.100 192.168.2.200;
option broadcast-address 192.168.2.255;
option router 192.168.2.1;
max-lease-time 7200;
option domain-name "rpi";
option domain-name-server 8.8.8.8, 8.8.4.4;
}
In /etc/default/isc-dhcp-server set which interface should the DHCP server manage.
INTERFACES="eth0"
After that we need to set the ip for the master node, /etc/network/interfaces.d/eth0 should look something like this after.
autho eth0
iface eth0 inet static
address 192.168.2.1
netmask 255.255.255.0
Reboot the master node for the dhcp configuration to load and connect back to it.
NAT Configuration
Inside the /etc/sysctl.conf we need to set the ip forwarding to true, for this we need to uncomment the following line.
net.ipv4.ip_forward=1
We activate the ipv4.ip_forwarding by running:
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
Run the following commands to create the network translation between the wifi port wlan0 and the ethernet port eth0.
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED
sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT
Check if everything is set as expected, should look something like this.
sudo iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-P POSTROUTING -o wlan0 -j MASQUERADE
sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i wlan -o eth0 -m state --state RELATED
-A FORWARD -i eth0 -o wlan0 -j ACCEPT
To make this happen on reboot (so you don't have to type it every time) run
sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"
And now we just have to tell the ethernet interface to pick the settings at reboot, in /etc/network/interfaces.d/eth0 update the ethernet interdace to something like this.
iface eth0 inet manual
address 192.168.2.1
netmask 255.255.255.0
post-up iptables-restore < /etc/iptables.ipv4.nat
Connect the first node to the switch and check inside the /var/lib/dhcp/dhcpd.leases to see which ip was assigned to the new node. You may see multiple entries in this that look something like this.
lease 192.168.2.102 {
......
hardware ethernet XX:XX:XX:XX:XX:XX;
......
}
Get the mac address and assign it a static ip for easier reference by adding in the dhcp config file /etc/dhcp/dhcpd.conf at the end, an entry that would look something like this.
host rpi-1 {
hardware ethernet XX:XX:XX:XX:XX:XX;
fixed-address 192.168.2.101;
}
Reboot the node and you could access it from the master with ssh [email protected]
Set the hostname in /boot/occidentalis.txt
# hostname for your Hypriot Raspberry Pi:
hostname=rpi-1
Now we can add the rest of the nodes by repeating the same process.
- connect to switch
- read the mac address from the dhcpd.lease on master
- assign a static ip address
- reboot the node
- ssh into it from the master
- set the hostname
- reboot again
Setting up the Docker Swarm
On the master node let's initialize the swarm by running
docker swarm init --advertise-addr 192.168.2.1
#Swarm initialized: current node (e216jshn25ckzbvmwlnh5jr3g) is now a manager.
#To add a worker to this swarm, run the following command:
docker swarm join \
--token some-very-secret-code-generated-by-docker \
192.168.2.1:2377
#To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
Now that the swarm is created we need to add the nodes to it by sshing into each one and joining them to the swarm.
docker swarm join \
--token some-very-secret-code-generated-by-docker \
192.168.2.1:2377
Checking if everything went ok.
$> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
1bcef6utixb0l0ca7gxuivsj0 rpi-1 Ready Active
38ciaotwjuritcdtn9npbnkuz rpi-2 Ready Active
sfasdsgxzfsdfsafa6as76655 rpi-3 Ready Active
e216jshn25ckzbvmwlnh5jr3g * rpi-0 Ready Active Leader
Now we're going to start a proxy service, I'm gonna use traefik.
docker service create --name proxy \
--constraint=node.role==manager \
--publish 80:80 --publish 8080:8080 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--network proxy hypriot/rpi-traefik \
--docker --docker.swarmmode --docker.domain=192.168.0.100 \
--docker.watch --web --logLevel=DEBUG
And a simple httpd container for demo purposes.
docker service create --name httpd \
--label traefik.port="80" \
--label traefik.backend="httpd" \
--label traefik.frontend.rule="PathPrefixStrip:/httpd" \
--label traefik.docker.network="proxy" \
--network ingress --network proxy \
--replicas 1 hypriot/rpi-busybox-httpd
Scale it up with docker service scale httpd=3 and check if everything is ok.
$> docker service ps httpd
NAME IMAGE NODE DESIRED STATE CURRENT STATE
httpd.1 rpi-busybox-httpd rpi-2 Running Running 6 minutes ago
httpd.2 rpi-busybox-httpd rpi-3 Running Running 4 minutes ago
httpd.3 rpi-busybox-httpd rpi-1 Running Running 4 minutes ago
Going to http://192.168.0.100:8080 we should see the traefik dashboard and it should look something like this.
Going to http://192.168.99.100/httpd we should see:
And it is done. We have a raspberry pi stack running docker swarm, a proxy that routes and load balances requests to nodes in the cluster and a service that serves an html page.
Thank you!
Come check out our other fun projects on Facebook, Twitter and Instagram!
