Clustered PyROS

It seems that one Pi is not enough. Our rover is, now, equipped with Adafriut's 9-dof breakout board with gyroscope/accelerometer and compass modules, touch screen, 8 x VL53L1X sensors, 4 AS5600 magnetic sensors, nRF24L01 for talking to wheels and another piece of hardware that goes to SPI interface.

9-dof module can be run on i²c bus, but it might saturate it and we already have VL53L1X modules that have to be run on i²c bus. One Raspberry Pi would need to read 4 magnetic sensors to steer wheels, 8 distance sensors to determine surroundings of the rover, read compass, gyro and accelerometer to try to deduct our position (which, itself is quite CPU heavy), draw on screen and have spare CPU capacity for running PyROS and challenge code. Quite a lot and some of them (9 dof data) are time sensitive.

Here is breakdown based on which buses different devices use:

• 9-dof can use i²c or SPI (two SPI devices - one for gyro/accelerometer and one for compass)
• 4 x AS5600 i²c with multiplexer
• 8 x VL53L1X i²c with multiplexer - two devices on same channel
• nRF24L01 SPI
• touch screen SPI
• another device on SPI

RPi SPI

So, if we go with a 9-dof on SPI bus, then we would need 5 SPI devices attached to the same bus. Fortunately RPi allows more than default 2 devices being attached to the same bus.

Richard has cracked that as well. To have more than two devices on SPI we need device-tree-compiler:

sudo apt-get install device-tree-compiler

Next is to fetch four-chip-selects-overlay.dts file and execute

dtc -@ -I dts -O dtb -o four-chip-selects.dtbo four-chip-selects-overlay.dts
sudo cp four-chip-selects.dtbo /boot/overlays

That will add two more devices on GPIOs 7 and 8, aside of default on GPIOs 24 and 25. See the four-chip-selects-overlay.dts file.

To enable the overlay you need to edit your /boot/config.txt file to include

dtoverlay=four-chip-selects

This will assign pins GPIOs 24 and 25 to CS2 and CS3 respectively.

You can specify different pins if you want

dtoverlay=four-chip-selects,cs2_pin=24,cs3_pin=25

But, 9-dof data is sensitive and nRF24L01 is quite chatty and can jump-in in the worst possible time. So, it makes sense to offload talking to wheels to separate Raspberry Pi. If talking to wheels goes to PiZero then steering wheels can go there (4 x As5600 and 2 GPIOs per wheel - 8 in total!), too; actually motion control itself. But, since we need (can't avoid) i²c multiplexer for AS5600 sensors, then it makes sense to move VL53L1X to the same device, too.

On the main RPi we'll be left with touch screen (very low frequency, not time issues) and 3x SPI for positioning (9-dof and another device) - four it total.

PyROS

Now, since code is going to be split across two devices the question was how to mange it. On one device all is managed by PyROS:

Wouldn't it be nice if the same, central control can be applied to second Raspberry Pi? Since Raspberry Pies are to be networked (look below) then there's no reason for PiZero to use Raspberry Pi 3's (B+) MQTT broker, too!

Total changes to the PyROS were minimal: each process id (name of service/program/agent) now can be prefixed with Pi we want it to go to and only PyROS main process that identifies itself with it (let's call it clusterId - or id within cluster) would react on message. Only 'global' message both react is ps command, but that's fine as result is collected from MQTT particular topic within timeout and if both respond at the same time - two messages are going to be delivered to the client and both processed and displayed. Also, if process id is not prefixed then only 'master' Raspberry Pi will react - in the same way as now.

Another simple change was to provide device identifier through exporter shell variable (CLUSTER_ID) and propagate it to all sub-processes PyROS is maintaining.

Now, PiZero detailing with wheels is called (imaginatively, right?) 'wheels' and uploading 'wheels' service to it looks like this:

pyros rover6 upload -s -r wheels:wheels wheels_service.py

Pi Networking

As mentioned above PiZero is going to be networked with Raspberry Pi 3. The simplest way seemed to be using USB for both power and networking. Fortunately PiZeros are well know that can provide 'ethernet gadget' (or 'ethernet USB device') and that is done by adding:

dtoverlay=dwc2

to /boot/config.txt and adding

modules-load=dwc2,g_ether

after rootwait to /boot/cmdline.txt file. After attaching PiZero to Raspberry Pi 3 new network interface called usb0 appeared and automatically local-link address was assigned to both Raspberry Pi 3 and PiZero devices. But, in our case we would really like to have static IP for Raspberry Pi 3 so its MQTT broker can be easily reached. Beside that, we would really like Raspberry Pi 3 to act as gateway and NAT our access to the rest of the world. That's slightly more involved:

First we need to allow IP Forwarding by uncommenting line net.ipv4.ip_forward=1 in /etc/sysctl.conf

Next is to setup IP tables to route packets. Manually it can be done by:

sudo iptables -A FORWARD -i usb0 -o wlan0 -j ACCEPT
sudo iptables -A FORWARD -i wlan0 -o usb0 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

But it is not permanent solution. The same setup can be 'dumped' to a file and that file would look like this:

# Generated by iptables-save v1.4.21 on Mon Jan 21 12:58:21 2019
*nat
:PREROUTING ACCEPT [10:739]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [5:572]
:POSTROUTING ACCEPT [2:344]
-A POSTROUTING -o wlan0 -j MASQUERADE
-A POSTROUTING -o wlan0 -j MASQUERADE
COMMIT
# Completed on Mon Jan 21 12:58:21 2019
# Generated by iptables-save v1.4.21 on Mon Jan 21 12:58:21 2019
*filter
:INPUT ACCEPT [8271:443643]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8108:437571]
-A FORWARD -i usb0 -o wlan0 -j ACCEPT
-A FORWARD -i wlan0 -o usb0 -j ACCEPT
-A FORWARD -i usb0 -o wlan0 -j ACCEPT
-A FORWARD -i wlan0 -o usb0 -j ACCEPT
COMMIT
# Completed on Mon Jan 21 12:58:21 2019

Save such file somewhere in /etc (for instance /etc/usb0_rules). Next is to ensure those rules are added at the time usb0 is attached. The New file /lib/dhcpcd/dhcpcd-hooks/80-usb0:

# usb0 up after assign ip address
if [ "$interface" = "usb0" ] && [ "$reason" = "STATIC" ]; then
        iptables-restore < /etc/iproute2/usb0_rules
fi

ensures that rules are run after usb0 interface is added. But we need to tell dhcpcd that we would like static IPs on usb0 interface. It can be done by adding following to /etc/dhcpcd.conf:

interface usb0
static ip_address=169.254.1.1/16

Now, we'll apply the same process for usb1 and usb2 and allow two more PiZero devices to be added, too. Why? Well, it is secret for now ;)

To control our rover we have our standard controllers. Last year we used a modified ps2 controller with a PiZeroW inside. This enabled us to remotely send packets through the WiFi. This was a relatively easy way of remotely driving the rover. However, it had some flaws. Firstly, because it used the WiFi hotspot that we had to carry around, it meant that the TCP packets would have to first be sent to the hotspot over WiFi, then to the rover over WiFi, and data was sent back through the same path. With lots of other 2.4GHz traffic on the spot at times we had latency of up to second or two!

So, this year we are planning to cut the corner, and connect a controller directly to the rover.

The way we will do this is with a (knockoff) PS3 controller, connected via Bluetooth to the rover. This is way better because there would be far less points in the packets route, and because its more direct, it should have a shorter travel time, meaning less delay. Also, there is no three way acknowledgement TCP robustness relies on. YAY!

So we set to work on coding this. First we had to connect controller to (the pair to) the Raspberry Pi. Then we, needed to make some code to actually utilise the controller connected in /dev/input/js0; the place that any Bluetooth/USB/Wired controller would connect to. Because on the modified PS2 controller with a PiZeroW we connected the controller inputs on the RPi in a similar way, to still appear as a controller on /dev/input/js0, we could easily just transfer the code. All that was needed was to knock off a few lines to disable the s1306 screen, because they were not needed, and just patch up the code to work with this controller.

All was working, but we noticed a problem. All the inputs were really delayed, with the delay increasing by the minute. Something wasn't right.

Silly Problem

Then we realised the problem. Because we still used PyROS to send packets internally in the rover (which was instantaneous) we needed to loop the controller service's thread, to process keys, buttons and sticks. This meant that we were sending packets at around 50 times a second, so we thought. However there was a little problem in PyROS's code. PyROS would wait a set amount of time before executing the next step, and it would achieve keeping this timing with a loop, waiting for the right time to strike. In the code below you can see our mistake. While PyROS was waiting for the next time to run the code's processes (to read the inputs) it would constantly read them anyway. This meant that we were sending packets at around 5000 times a second.

def loop(deltaTime, inner=None):
    for it in range(0, int(deltaTime / 0.002)):
        time.sleep(0.0015)
        if client is None:
            time.sleep(0.0005)
        else:
            client.loop(0.0005)

    if inner is not None:
        inner()

Because of it 'inner' code was executed with total delay of 2ms instead of originally expected 20ms and thus spamming drive with messages (or just stop). The drive service managed to process 10 times the volume of messages, but was giving four to eight times that much messages for the wheels service (one or two message per wheel - one for position and one for speed) which at peaks produced 400 extra messages per second. This meant that the wheels service would clog up with messages to execute, and not be able to execute them in the queue in time, until it has process the other hundreds of useless messages. A bit of a silly mistake there!