Satisfying a quirk of mine.

A while back, in my second year of university, during a course on digital and analog circuits, I had my first encounter with microcontrollers, learning all the intricacies of the Atmega8 family awoke my passion for IoT and so I bought my first Arduino Uno and a kit of wires, breadboards and some sensors with which everything started.

Two years later, on a Friday morning at work, I bought my first Raspberry Pi, I didn't really knew what I was gonna use it for but I really wanted one and for the better part of the year that followed I used it as a media server attached to a TV. All this was awesome but I wanted it to do so much more.

About the same time we started experimenting with Docker at work and as any good developer that has a linux server attached to his TV, I wondered can i run Docker on this?, the answer is yes and it's a lot easier than I thought, after a quick Google search I found the guys at hypriot who specialized in building Docker ready system images for ARM devices.

As the time went on and I had the chance to explore the Docker ecosystem, the release of Raspberry Pi 3 and reminiscing a course on distributed systems concluded in November last year when I decided to build a Raspberry Pi stack thus satisfying an old quirk of mine.

Building the hardware

Building the stack is not that hard, I could've wired everything on a table and that would've been everything, but I like my toys, as I like my code, tidy and organized so I opted to get a stackable case, even so I was still missing some parts... shopping time.

The almighty list:

3 x Raspberry Pi 3 (as I already had a Raspberry Pi 2)
1 x ethernet switch TP-LINK TL-SF1005D
1 x 5 port usb power hub
4 x micro usb cable
4 x Ethernet cable
4 x 32 gb microsd card
2 x stackable case

After an excruciating week waiting for the parts to arrive, an issue with the courier who almost returned my cases back and a whole in my pocket, I could finally start building the infamous stack. I unpacked everything, striped the protective foil from the layers on the case (who needs that type of protection anyway) and planned the building process which took an hour or so. The only exciting part was when my switch would not fit beside the power hub in the bottom layer of the case so... I cut the switch out of the case.

I don't have any photos from the build process as I'm writing this half a year after the events described, but here's the final product.

Hope you'll find it in yourself to forgive me.

Building the system images

The guys at hypriot have an awesome getting started guide, I'll present here what I did the Mac OS X as that's what i needed.

Steps to reproduce:

First things first, download the latest image from the download page.

Go to your downloads folder.

cd ~/Downloads

Extract the zip, this should have a new .img file.

unzip hypriot-rpi-???.img.zip

Insert the SD card in your computer and find the disk alocated to it by running:

diskutil list

For this exampe I'll presume your SD card is mounted at /dev/disk4.

Let's unmount the disk and prepare for flashing.

diskutil unmountdisk /dev/disk4

Now we are ready to flash the SD card. We are going to use the dd command for this.
Before you execute the command below, make sure to

replace the parameter after if= with the path to the downloaded image
replace the parameter after of= with the identifier of your SD card
make sure you put a r in front of disk as you can see in the example

sudo dd if=hypriot-rpi-???.img of=/dev/rdisk4 bs=1m

Now we only have to do the same thing for the other 3.

I did a quick search on their repo and found flash, it's a tool they developed to make this whole process simpler and you can set the WiFi credentials during flash time which skips the whole monitor and keyboard part of my rambling above and get's you straight to ssh which is nice, didn't test it tho...

For the master node, after inserting the SD card in the Raspberry Pi, connect it to a monitor, attach a keyboard to it and wait for it to boot you can set the WiFi connection by updating your /boot/occidentalis.txt to something like:

# hostname for your Hypriot Raspberry Pi:
hostname=rpi-0

# basic wireless networking options:
wifi_ssid=SSID
wifi_password=12345

And with this done we can access the rpi-0 through ssh.

ssh pirate@rpi-0.local

Configuring the networking layer

Even though each Raspberry Pi can connect has wireless, I did not want to clutter my network so I opted for a network architecture beautifully described in my sketch down below.

Having the general idea in mind represented by a master node that acts as ethernet access point for the other 3 nodes connected to the switch and knowing that from now on everything happens on the master node, let's connect to it by ssh pirate@192.168.0.100 assuming this is the ip assigned to the node from the router.

Like any good linux configuration we start with updating the system.

sudo apt-get update && sudo apt-get upgrade

###### Configuring the DHCP Server

Install the dhcp server.

sudo apt-get install lsc-dhcp-server

In the configuration file /etc/dhcp/dhcpd.conf comment the next 2 lines

#option domain-name "example.org";
#option domain-name-servers ns1.example.org, ns2.example.org;

And uncomment the next one

authoritative;

After that, at the bottom of the file define the new subnet that will serve the other nodes.

subnet 192.168.2.0 netmask 255.255.255.0 {
    range 192.168.2.100 192.168.2.200;
    option broadcast-address 192.168.2.255;
    option router 192.168.2.1;
    max-lease-time 7200;
    option domain-name "local";
    option domain-name-server 8.8.8.8, 8.8.4.4;
}

In /etc/default/isc-dhcp-server set which interface should the DHCP server manage.

INTERFACES="eth0"

After that we need to set the ip for the master node, /etc/network/interfaces.d/eth0 should look something like this after.

auto lo
allow-hotplug eth0
auto eth0
iface eth0 inet static
    address 192.168.2.1
    netmask 255.255.255.0

Reboot the master node for the dhcp configuration to load and connect back to it.

NAT Configuration

Inside the /etc/sysctl.conf we need to set the ip forwarding to true, for this we need to uncomment the following line.

net.ipv4.ip_forward=1

We activate the ipv4.ip_forwarding by running:

sudo sh -c "echo 1 > /proc/sys/ney/ipv4/ip_forward"

Run the following commands to create the network translation between the wifi port wlan0 and the ethernet port eth0.

sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED

sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT

Check if everything is set as expected, should look something like this.

sudo iptables -t nat -S

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-P POSTROUTING -o wlan0 -j MASQUERADE

sudo iptables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i wlan -o eth0 -m state --state RELATED
-A FORWARD -i eth0 -o wlan0 -j ACCEPT

To make this happen on reboot (so you don't have to type it every time) run

sudo sh -c "iptables-save > /etc/iptables.ipv4.nat"

And now we just have to tell the ethernet interface to pick the settings at reboot, in /etc/network/interfaces.d/eth0 update the ethernet interdace to something like this.

auto lo
allow-hotplug eth0
auto eth0
iface eth0 inet static
	address 192.168.2.1
	netmask 255.255.255.0
	post-up iptables-restore < /etc/iptables.ipv4.nat

Connect the first node to the switch and check inside the /var/lib/dhcp/dhcpd.leases to see which ip was assigned to the new node. You may see multiple entries in this that look something like this.

lease 192.168.2.102 {
    ......
    hardware ethernet XX:XX:XX:XX:XX:XX;
    ......
}

Get the mac address and assign it a static ip for easier reference by adding in the dhcp config file /etc/dhcp/dhcpd.conf at the end, an entry that would look something like this.

host rpi-1 {
    hardware ethernet XX:XX:XX:XX:XX:XX;
    fixed-address 192.168.2.101;
}

Reboot the node and you could access it from the master with ssh pirate@192.168.2.101

Set the hostname in /boot/occidentalis.txt

# hostname for your Hypriot Raspberry Pi:
hostname=rpi-1

Now we can add the rest of the nodes by repeating the same process.

connect to switch
read the mac address from the dhcpd.lease on master
assign a static ip address
reboot the node
ssh into it from the master
set the hostname
reboot again

Setting up the Docker Swarm

On the master node let's initialize the swarm by running

docker swarm init --advertise-addr 192.168.2.1

#Swarm initialized: current node (e216jshn25ckzbvmwlnh5jr3g) is now a manager.

#To add a worker to this swarm, run the following command:

docker swarm join \
    --token some-very-secret-code-generated-by-docker \
    192.168.2.1:2377

#To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```bash
Now that the swarm is created we need to add the nodes to it by sshing into each one and joining them to the swarm.
```bash
docker swarm join \
    --token some-very-secret-code-generated-by-docker \
    192.168.2.1:2377

Checking if everything went ok.

docker node ls

ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
1bcef6utixb0l0ca7gxuivsj0    rpi-1           Ready   Active
38ciaotwjuritcdtn9npbnkuz    rpi-2           Ready   Active
sfasdsgxzfsdfsafa6as76655    rpi-3           Ready   Active
e216jshn25ckzbvmwlnh5jr3g *  rpi-0           Ready   Active        Leader

Now we're going to start a proxy service, I'm gonna use traefik.

docker service create --name proxy \
    --constraint=node.role==manager \
    --publish 80:80 --publish 8080:8080 \
    --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
    --network proxy hypriot/rpi-traefik \
    --docker --docker.swarmmode --docker.domain=192.168.0.100 \
    --docker.watch --web --logLevel=DEBUG

And a simple httpd container for demo purposes.

docker service create --name httpd \
    --label traefik.port="80" \
    --label traefik.backend="httpd" \
    --label traefik.frontend.rule="PathPrefixStrip:/httpd" \
    --label traefik.docker.network="proxy" \
    --network ingress --network proxy \
    --replicas 1 hypriot/rpi-busybox-httpd

Scale it up with docker service scale httpd=3 and check if everything is ok.

docker service ps httpd

NAME     IMAGE              NODE   DESIRED STATE  CURRENT STATE              
httpd.1  rpi-busybox-httpd  rpi-2  Running        Running 6 minutes ago
httpd.2  rpi-busybox-httpd  rpi-3  Running        Running 4 minutes ago
httpd.3  rpi-busybox-httpd  rpi-1  Running        Running 4 minutes ago

Going to http://192.168.0.100:8080 we should see the traefik dashboard and it should look something like this.

Going to http://192.168.99.100/httpd we should see:

And it is done. We have a raspberry pi stack running docker swarm, a proxy that routes and load balances requests to nodes in the cluster and a service that serves an html page.

Thank you!