Content

1Abstract 1

2Operating System 3

2.1Download Operating System 3

2.2SD Card - Burn Image 4

3Configure The OS 5

3.1Edit CONFIG.TXT 5

3.2First Boot and OS Configurations 6

3.2.1Log in 6

3.2.2Configure OS 7

3.2.3Setup Ethernet 8

3.2.4Update the OS 8

3.3SSH - Putty 9

3.4SFTP - FILEZILLA 10

4Overall Architecture 11

5NODE.JS 12

5.1NPM 13

5.2SOCKET.IO 14

5.3WebSocket 15

5.4Serial Port 16

6FFMPEG 17

6.1Architecture 17

6.2USB Camera 18

6.3FFMPEG Transcoder and Streamer 19

6.4Streaming Example 20

7Custom Hat Firmware 21

8NODE Application 22

9Features 23

9.1Supported Features 23

9.2Limitations 23

9.3Features to be implemented 23

10Conclusions 24

1Abstract

This is a step by step guide on how to turn a virgin raspbian image into MazeRunner OS.

I need an operating system image for the Raspberry Pi 3 SBC with the following features:

Raspbian Stretch Lite - Official RPI OS without desktop environment
GPIO - Communicate with custom hat
Webserver - Serve html page locally
Video Stream - Real time low latency video from RPI to browser
Controls - Real time low latency bidirectional controls client<->server

2Operating System

2.1Download Operating System

FIrst step is to download an operating system and burn a SD card with the OS image.
Obvious choice is the Raspbian Operating System. Latest LTS release is the Raspbian Stretch that is a personalized version of the Debian Stretch that targets the Raspberry Pi 3. Previous version was the Raspbian Jessie.
https://www.raspberrypi.org/downloads/raspbian/
Originally I wanted an image with ROS, now I'm using NODE so the official OS image should do just as well. With ROS I run in trouble trying to install ROS. Problem is the armhf extension which is not available for all packages.
https://medium.com/@rosbots/ready-to-use-image-raspbian-stretch-ros-opencv-324d6f8dcd96
Unfortunately this version only runs ROS Kinetic Kame, not the latest ROS Melodic Morenia, alas this is the best I could get with my current efforts.
The current Maze Runner MVP Demo does not use ROS at all currently.

2.2SD Card - Burn Image

There are many tutorials to do this. Once it's done the SD card will have a FAT32 partition with the boot code and a linux partition with the operating system.
Instructions:

To ensure smooth operation delete existing partitions on the SD Card. Leave the volume with no partition nor file system.
Burn the image, usually in *.img format on the SD card. Do not open “computer” during this process because windows might mess up the writing process.

Once it's done you should see both volumes and be able to access to the boot partition.
Using Read it's possible to save the image. I did it as soon as I configured most stuffs, as I'll probably need to roll back to an earlier version. I also verified that cloning the card on a different Raspberry still works with no difference, other than the MAC address.

3Configure The OS

Tools needed to work with the OS without using dedicated keyboard and screen.

3.1Edit CONFIG.TXT

Before plugging in the card in the Raspberry Pi 3B+, several options need to be changed in the file BOOT/CONFIG.TXT in the FAT32 partition of the SD card. Edit can be done from windows.
Make sure the HDMI port is enabled. With my VGA->HDMI adapter the Raspberry Pi would output to video composite by default otherwise. You can disable it once you are done.

hdmi_force_hotplug=1

Configure the GPIO so that in a magical moment during boot process the GPIO pins are correctly configured. GPIO4 is the reset of my Raspberry Pi Shield. This has a nice side effect that the shield is reset if something goes wrong with the Raspberry Pi

gpio=4=op,dl

Configure the serial port not to work as serial console. It's needed by the custom hat.
*** Link to config.txt in github

3.2First Boot and OS Configurations

Put the SD card inside the RPI. Plug in Keyboard, Mouse and screen. If all goes well you should see the operating system console.

3.2.1Log in

For the raspbian with ROS image the credentials are:

Default username: “pi”
Default Password: “Rosbots!”

3.2.2Configure OS

Raspbian has a powerful wizard to help you with the configuration.

sudo raspi-config

Change User Password: use an easier password. I set “xxx”.
Network Options -> HostName: Set the name of the robot “Maze Runner”
Boot Options -> Console Autologin: Will boot logged to user “pi”
Localisation Option: Set the corret keyboard layout
Interfacing Options -> SSH: Enable SSH connections. From now on we can use PUTTY.
Advanced Options -> Expand File System: Make sure the image uses all the SD card. By default it uses only about 9GB and is 95% used.
Advanced Options -> Serial: Make sure the serial port is not used as serial terminal

3.2.3Setup Ethernet

I connect to the RPI using a wired LAN network. To configure network interfaces:

sudo nano /etc/dhcpd.conf

In the editor, uncomment static ip and gateway set ip and gateway. Otherwise you always have to discover IP before accessing with Putty or have to set a static rule on the router but it will go to waste each time you change Raspberry Pi since MAC address changes as well.

IP: 192.168.0.202
Gateway 192.168.0.1

3.2.4Update the OS

If all goes well, your RPI should now be interfaced with the internet, meaning you can update the operating system and the packages and get the latest version of everything. Which might break the following steps...

apt-get autoremove
apt-get upgrade
apt-get update

Reboot.

sudo reboot now

If all went well, now you can remove the screen keyboard and mouse.
Leave the LAN cable in.
Everything else will be done from a remote computer with SSH, leaving the Raspberry Pi headless as it will be in the robot.

3.3SSH - Putty

SSH allows to access command line of the raspberry remotely.
I use Putty with the following configurations:

IP: 192.168.0.202
User: “pi”
Pwd: “xxx”

I tested the same image on two cloned cards and two different raspberry pi, and it works headless after the initial configuration.
From here on you can disconnect screen keyboard and mouse, reboot and see if all still work.

3.4SFTP - FILEZILLA

FTP file transfer is going to be useful to easily move files from the RPI to the developer PC.
One option is to install and FTP client, but there is a better way: FTP over SSH the SFTP protocol.
This is the configuration page:

I setup two profiles.

pi, xxx as login information, which can't transfer most files
root, root in order to transfer all files

4Overall Architecture

This is the architecture of the Maze Runner Demo
NODE handles the web server and the web sockets
V4L2 and FFMPEG handle the transcoding
The custom hat powers the Raspberry, drive the motors and host an arduino shield slot
The client uses a web browser to control the robot

Architecture in deeper detail

5NODE.JS

The previous steps configured the Raspberry Pi to develop the application software.
NODE.JS combines http server and server side application, it's a modern way of building a webserver. A Raspbian Stretch package exists, install it.

sudo apt install -y nodejs

The webserver files can be safely hosted inside a subfolder inside the home folder.

/home/pi/my_node_webserver

Inside the home folder you put all the files needed by your web server, at minimum you have the webserver application itself and an html page

/home/pi/my_node_webserver/try.js
/home/pi/my_node_webserver/index.html

NODE can be executed by launching the command:

pi@MazeRunner:~ $ node try.js

NODE can also be launched in the background like any other application

pi@MazeRunner:~ $ node /home/pi/my_node_webserver/try.js &
[1] 2163
pi@MazeRunner:~ $

The call returns the process ID, which can be used to shut down NODE. Cutting power to the RaspberryPi while NODE is running runs the risk of breaking NODE, meaning it will hang with no feedback the next execution. To kill a running node:

pi@MazeRunner:~ $ sudo kill 2163

As long as NODE is running, your webserver will be running as well, and the client will be able to connect to it and receive the content of index.html

5.1NPM

NODE uses NPM as package manager.

It works like apt-get allowing packages to be added to node. Two packages are needed for the Maze Runner MVP demo:

socket.io: Provides low latency bidirectional controls server<->client
ws: Bare websocket used to stream the video from server to client
serialport: Use the UART interface in the GPIO connector

5.2SOCKET.IO

Maze Runner uses Socket.IO to provide bidirectional low latency communication from the web browser running on the client to the web server running on the Rasperry Pi.

To install the package:

pi@MazeRunner:~ $ sudo npm install socket.io

Socket.IO works by providing a javascript hosted by the package itself. You need to include the javascript inside the html page:

<script type="text/javascript" src="/socket.io/socket.io.js"></script>

Controls on the http page can be linked with websocket messages handled with the calls:

mysocket.on
mysocket.emit

The payload can be a json structure and allow for any number of unique messages to be handled by the stack. The calls works the same both on the server and the client.

To profile latency of the channel, a ping-pong can be implemented. Socket.IO achieve a 7mS round way trip latency, which is excellent.

5.3WebSocket

I'm already using websocket with Socket.IO, but Maze Runner uses the bare version as well. Since this is a MVP demo, the primary objective was to achieve a complete stack that achieve full functionality, even if such stack is inefficient or clunky.

Websocket are used to transport video streaming data from the server to the client.

Websockets can be installed with the NPM package manager with the following command:

pi@MazeRunner:~ $ sudo npm install ws

5.4Serial Port

The last component needed by the NODE demo is the serial port.
The custom hat is interfaced through the UART peripherals on the GPIO. In the CONFIG.TXT step, the serial port default use as serial monitor has been disabled, and can now be used to communicate with the custom hat.

First install a NODE package to interface with the GPIO UART:

pi@MazeRunner:~ $ sudo npm install serialport

Raspbian Stretch maps the UART GPIO as the following device:

/dev/ttyS0

That's it. Now NODE has all the required packages installed.

6FFMPEG

NODE takes care of serving html, css and js files to the client, but a streaming system is needed to send camera data to the client, and a player is needed to decode and show the data as low latency video inside the browser.

6.1Architecture

The MVP demo was meant to achieve a complete working system, not the optimal one. The video streaming stack is lacking achieving 300ms of latency, but is good enough to use. Great effort has gone into trying different system, and so far this is the best performing stack.

6.2USB Camera

The current stack is designed to work with a USB camera. The one used by the Maze Runner MVP demo is a logitech Webcam Pro 9000 that encodes a motion jpeg stream.
The V4L2 driver maps the camera as:

/dev/video0

Trust webcam (does not work):

USB\VID_093A&PID_2460&REV_0100

Any camera compatible with the V4L2 drivers embedded in Raspbian Stretch should work.

6.3FFMPEG Transcoder and Streamer

Server side, encoder and a video streaming server are required to send video streaming data to the client. The Maze Runner MVP demo uses FFMPEG for both.
To install FFMPEG execute the command:

sudo apt-get install ffmpeg

The difficult part is setting up the transcoding pipeline. The Maze runner MVP Demo transcode into a 640x480@20FPS stream using a mpeg1 codec with a TS encapsulation. The stream is sento to localhost.
The transcoder has to be launched by command line with this command:

pi@MazeRunner:~ $ ffmpeg -f v4l2 -framerate 20 -video_size 640x480 -i /dev/video0 -f mpegts -codec:v mpeg1video -s 640x480 -b:v 600k -bf 0 http://localhost:8080/mystream

NODE.JS takes care of listening to localhost for the video data and flipping them over a websocket to be sent to the client where a javascript player decodes them and show them inside the browser.
The javascript has to be hosted inside the webserver and served to the client when the main html page is loaded.

6.4Streaming Example

7Custom Hat Firmware

The custom hat interfaces and powers the Raspberry Pi 3B+ through the GPIO.

It can be programmed with the AVRisp MKII and the AVR Studio 7

The firmware can be found in this github repository. As with other parts of the Demo, the firmware only supports the bare minimum features.

8NODE Application

With the configuration of NODE.JS complete, the next step is install all files required which can be found on this github repository:

To launch the MazeRunner MVP Demo execute the commands:

pi@MazeRunner:~ $ node 2019-06-15-test-serial/demo.js &
[1] 2132
pi@MazeRunner:~ $ ffmpeg -f v4l2 -framerate 20 -video_size 640x480 -i /dev/video0 -f mpegts -codec:v mpeg1video -s 640x480 -b:v 600k -bf 0 http://localhost:8080/mystream

To stop the demo, press CTRL+C to stop the ffmpeg transcoder and kill the node application running in background using its pid number.

pi@MazeRunner:~ $ sudo kill 2132

To connect to the robot, you need to set insert the IP address in the address bar inside your browser. Doing so will display the interface.
The client sends the server key events.

'W' go forward

'S' go backward

'D' turn right

'A' turn left

'+' increase speed

'-' decrease speed

The server sends the client the local time.
The server sends the client a video stream.
This is the Maze Runner MVP Demo in action:

9Features

The Maze Runner MVP Demo was meant to demonstrate the minimum features required to achieve remote FPV control of a Raspberry Pi based robot through web browser.

9.1Supported Features

User can connect to the robot with a web browser over LAN
User can see a video stream from the robot. 640x480@20FPS@300ms
User can move the robot using wasd keys and +/- to set the speed with a latency of 7ms
User can see the robot local time
Controls up to four servos
Powered by a 3S 3Ah LIPO battery for about 6h run time

9.2Limitations

User needs to manually launch both the NODE web server and the ffmpeg stream connecting from SSH command line
User needs to manually shut down the NODE web server and the ffmpeg stream connecting from SSH command line to avoid SD card corruption
Serial communication from Custom Hat to Raspberry Pi does not work
Custom Hat needs a dedicated regulators for the servos
No wi-fi
Custom hat currently support only servo motion commands

9.3Features to be implemented

Move key processing inside the client. Do not send key events.
Implement wi-fi access point with static IP
Rework the video streaming stack. Get latency down and move processing inside the client.
Implement robot monitor features inside the web page (battery, position, ping, etc...)
Fix Custom Hat -> Raspberry Pi UART communication channel
Rework Custom hat firmware parser. It's barely holding together.

10Conclusions

The MazeRunner MVP Demo succeeds in its objective of demonstrating the bare minimum functionalities required to control a FPV robot from a browser.

Pages

HowTo - Maze Runner MVP Demo