1. Introduction
In the last few years, Artificial Intelligence (AI) has exponentially increased in everyday usage. However, each model typically requires a huge amount of resources, requiring sometimes an energy consumption of thousands of MWh for the training phase and of hundreds of MWh for the running phase [
1]. Although Artificial Neural Network (ANN) models are inspired by the brain, they are far from the operational efficiency of the human brain. In fact, the human brain consumes no more than 20 W of power [
2] to perform all kinds of complicated tasks compared to the classic AI systems that can require megawatts of energy. Similarly, robotic applications benefit greatly from the neuromorphic paradigm in terms of low-power and low-latency operations [
3]. This aspect has stimulated some university labs and companies to build so-called neuromorphic hardware, which is able to mimic biological behaviors in the operational principles of the hardware. Regarding bioinspired robotics, there are many different realizations related to the sensing, computing, and execution of mechanical actions, which are developed for solving tasks such as obstacle avoidance [
4,
5], object tracking [
4,
6], playing games such as rock–paper–scissors [
7], mimicking biological behavior [
8], and similar tasks. Regarding the computing platforms, both conventional and neuromorphic hardware have been implemented. For example, conventional platforms such as microcontrollers [
9,
10], FPGAs [
5], LEGO [
11], laptops [
12] and cloud computing have been reported. Regarding neuromorphic computing hardware [
13], devices such as SpiNNaker [
14], Loihi [
15], and TrueNorth [
16] are available, as well as custom-built platforms such as those that utilise memristive [
17] or spintronic devices [
18], but not all of them have been used in the context of robotic applications [
19]. Robotic platforms also require sensory inputs and actuators. Some platforms utilise conventional sensing devices such as ultrasound [
4], LED photodiodes, tactile [
20] sensors, etc. However, neuromorphic sensory devices have been developed as well, such as Dynamic Vision Sensors (DVSs or silicon retina), silicon cochlea, etc. [
21]. A DVS is a camera that produces signals based on light intensity changes and avoids unnecessary data generation [
22].
Models for processing information and decision making for neuromorphic robotics typically rely on the ANN paradigm, such as Spiking Neural Networks (SNNs) [
3,
23]. SNNs encode the information into asynchronously generated spikes, transmitted between the neurons of the network through synapses, replicating the natural behaviour of the brain. This aspect implicitly includes the time component (in the spike order), which is used for solving various dynamic tasks evolving in time, e.g., object tracking or obstacle avoidance.
Once the robot hardware and the neural network are set up, the difficult task of training the robot to learn the desired task or behaviour begins. Regarding learning algorithms for SNNs deployed on robotic platforms, several approaches have been proposed, such as Spike-Time-Dependent Plasticity (STDP), Reward- or Reinforcement-Based STDP (such as R-STDP [
24]), fast-weight STDP [
12], Conditioning [
11], and Dopamine-Modulated STDP [
4,
20]. STDP, in combination with lateral inhibition, is typically used for unsupervised learning [
25,
26]; however, for supervised learning, a reward signal is needed. Alternatively, for supervised learning, a well-known backpropagation algorithm could be used, but it is adapted for SNNs such as in SpikeProp [
27]. However, traditional learning algorithms are still a challenge when implemented for robotic applications.
The possibility of physically simulating the behaviour of biological systems leads to two main advantages: the implementation of brain-inspired low-power intelligent systems and the investigation of the brain’s operation via simulations. With these motivations, and with the awareness that dedicated neuromorphic hardware is not always readily available to those who wish to experiment with it—and even when available, the connectivity to external devices could be complicated—we investigated how to enable running real-time robotic applications based on SNNs using conventional low-power hardware.
For this purpose, we propose a neuromorphic robotic system that uses the Brian2 simulation platform, which runs on a low-power convectional CPU, specifically Raspberry Pi, in combination with a DVS camera, touch sensor, and a digital motor. The key novelty is to combine and implement two readily available and easy-to-use elements in a robotic setup: (i) Brian2 software for designing and running SNNs and (ii) a Raspberry Pi board for executing the SNN code and communication with external devices. Crucially, Brian2 has been proven to be able to cope with real-time input [
28].
There are Operating Systems (OSs) that are purpose-built for neuromorphic hardware (such as SpiNNaker, Loihi, and TrueNorth) and that could be optimised for the partitioning and mapping of SNNs [
29]. However, another approach (used in this work) is to rely on a conventional OS deployed on a microprocessor and simulation programming in common programming language (Python, C++, Java and so on) running on a PC.
As an experimental demonstrator, we implement a robot goalkeeper based on a DVS128 camera, working as an eye sensor, a Futaba S9257 actuator, working as an arm, and an infrared sensor, working as a touch sensor. All these devices are connected to a Raspberry Pi 5 board [
30], using USB and GPIO connections; see
Figure 1. On this low-power processor, we run an SNN model on a Virtual Machine, communicating with the sensors to predict the final goalkeeper position. The main purpose of this work is to demonstrate that we can use a conventional CPU in combination with a Virtual Machine for robotics applications as a replacement for a purpose-built neuromorphic hardware.
The paper is organised as follows. In
Section 2, we describe the methodology of our work, explaining how the sensors are connected to a Raspberry Pi 5 board, both physically and logically. In
Section 3, we report our results, measuring the accuracy in predicting the correct goalkeeper position, the power consumption of the whole system and the reaction time for real-time evaluation. Finally, in
Section 4, we compare our results with those of previously implemented systems addressing similar tasks.
3. Results
The proposed system, described above, has been designed to replicate the goalkeeper task on low-power SBC, allowing it to be powered by a small battery pack for mobility purposes. To evaluate the system performance’s different metrics, latency, accuracy, resources consumption and power consumption have been measured. In this section, we report the results of our tests for the real-time application of our system.
3.1. Latency
The prediction task latency in our system consists of the cumulative time required to receive the input signal, pre-process it, transfer it to the SNN, calculate the prediction of the goalkeeper position, communicate that information to the servo motor, and use the motor to move the arm to its final position. As mentioned in [
32], the DVS camera produces events each microsecond and needs an extra 1 μs to communicate it via the serial port. Since we run the model in time steps of 1 ms, we set the transmission rate from the DVS to 500 μs like in our previous realisations of the system [
6,
10], leaving enough time for the pre-processing step. As a consequence, we cannot consider latency time for receiving data from the DVS and elaborate it in the Control Unit for longer than 1 ms. The system then immediately forwards the input to the SNN model, which runs the simulations in 50 ms batches. At this point, the Network Operation object injects the received input at 1 ms time steps, with the response delay depending on the volume of incoming data. Additionally, while input is being injected, the output monitor continuously looks for output spikes, which are immediately forwarded. When the input is sufficient to trigger a spike in the output neurons, the minimum delay for the model answer is 1 ms for input and 1 ms for output. Considering negligible time for memory access and code execution, the computation latency time is about 3 ms. The remaining delay is attributed to the arm positioning (75 ms for 60° positioning + 1 ms for receiving instructions with the HTTP protocol), resulting in a maximum delay of 154 ms (optimised to 100 ms with position reset).
3.2. Resources Consumption
To measure the system performance, we ran the system on a 5 V battery pack (capacity 12,500 mAh) and executed an independent script to record the CPU power usage, consumption and temperature, memory consumption, and battery power consumption. The recording was conducted over 70 min, leaving the system idle for the first 5 min, followed by 1 h of model execution, and then 5 min of cooling down; see
Figure 5. The CPU drew a current of 4–7 A (average about 6 A) during the execution (CPU voltage of 1.2 V)—see panel (a). The CPU usage percentage immediately jumped to the maximum level and remained above 80% for the entire execution time—panel (b). The CPU temperature stabilised between 60 and 65 °C—panel (c). However, looking at the steps before and after the execution (idle and cooling down), the CPU ran at 2.2 A on average, meaning that the model consumes 3.8 A on average (i.e., 4.56 W at 1.2 V). For a complete system consumption, we measured the battery level change in percentage (panel (b)—red line). As we can see, the battery level linearly decreased to about 2/3 of its capacity after 1 h running, corresponding to an overall consumption rate of about 20 W (4 A at 5 V). The main memory (RAM) usage when all units were running ranged from 750 to 800 MB during the 1 h execution. However, during the idle phases (the first and last 5 min), the system memory usage was around 520 MB before the execution and 550 MB afterward. This indicates that the model initially consumed approximately 200 MB of memory, which increased by only 50 MB after 1 h of operation.
3.3. Accuracy
The accuracy of our system is the ability to predict the correct ball trajectory and stop the ball. In the online scenario, the DVS camera is placed 30 cm above the goal, with a 20 degree negative tilt, aimed at the space in front of the goal; see
Figure 6. The accuracy was measured counting the number of the blocked balls over 100 launches from 1 m distance with random speed and direction. To improve accuracy, when the decision is made, the system resets an inactivity timer that is used to replace the goalkeeper to the middle position. This simulates the goalkeeper behaviour of optimising the future intervention for intercepting the ball, reducing the arm positioning time to 75 ms. In case a new ball is coming before the timer expires, the predicted position is directly used to set the arm.
Apart from demonstrating the overall functionality of the system in real time, we also tested the system on a controlled simulated environment. In this (offline) scenario, the input consists of a ball image moving on a screen (instead of a real ball on the table) allowing us to accurately control the ball speed, directions, colour and the background. The rest of the system remains unchanged. The DVS camera is positioned in front of the screen and centered to the simulated Field of View (FOV) to capture the onscreen ball. For the reward signal (touch sensor), the Virtual MCU was replaced with an additional web service that communicates with the balls’ generator software, providing us with accurate information about the ball’s endpoint. This setup allows us to automatically calculate accuracy by validating predictions made before each ball reaches the end of the FOV with an additional maximum delay of 100 ms for arm positioning.
For this offline scenario, we tested the system under different background/target colors and two types of trajectories: (i) in lane, i.e., when the ball direction is precisely within the width of the goalkeeper’s pre-defined position (there are eight of theses positions in our experiment, each covers about 10°) and (ii) random straight trajectories. The results are shown in
Appendix D, and there is a summary in
Table 3. In the online scenario (i.e., with the real ball), we reached an overall accuracy of 80% of correctly predicted goalkeeper’s positions (SNN128)—a similar accuracy as with the ball moving on a screen from a random position. A confusion matrix result is shown in
Figure A5 of
Appendix D.
We note here that we have used a very simple SNN algorithm, since the aim of this initial work was to develop a proof-of-principle platform for our robotic system, and various optimisation and performance enhancement steps can be implemented later.
4. Discussion
In this work, we focused on the implementation of a neuromorphic robotic platform on a Single Board Computer running an SNN simulation. We configured a Raspberry Pi 5 board with 8 GB of RAM, running a Linux Ubuntu 23 OS. The main advantage of using the Raspberry Pi 5 SBC lies in the possibility of linking all the external components (DVS, servo and IR) to the provided USB and GPIO connection ports, resulting in a simple and compact system. This contrasts with previous projects like [
10], where multiple boards and devices were connected using a custom-built PCB. Furthermore, the integration of a USB-C port for powering the board allows the use of compact powerbanks running at the exact voltage without the necessity for level shifters and power regulator chips.
In
Table 4, we compare our system with several robotic platforms. First, we compare it with a neuromorphic robotic platform, which utilises a SpiNNaker board for running an SNN as well as a custom-built PCB with needed electronic components, which were designed for a similar task [
10] (named here spiNNaLink). Analysing the power consumption, our system consumes about 20 W in an unoptimised state. Unoptimised state means here with full installation of the system, WiFi on, no power safe mode, system services on. That is about three times more that the spiNNaLink system. The positioning time is practically the same as that of spiNNaLink and in line with the other projects. The achieved accuracy depends on the model running on the board, but it is comparable to previous results. For the SNN, we chose to implement a more selective model with a COBA equation to filter the incoming signal and to have a more accurate prediction. With a simple model that ignores capacitance, all the signals would be forwarded to the output, leaving the decision to a simple counting of the output neurons spikes.
4.1. Advantages
The proposed system is a combination of neuromorphic hardware, sensors, actuators and a Spiking Neural Network simulator, running on a low-power Single Board Computer (SBC). The choice of using an SBC to link all the external devices and to run the SNN model leads to several advantages. On the hardware side, the possibility of exploiting the native SBC connections increases the compatibility of external devices, avoiding the implementation of custom solutions and protocols. In fact, neuromorphic hardware like DVS cameras often rely on USB connections using their own driver, which are optimised to guarantee the maximum speed and robustness. In addition, the possibility to link sensors and actuators directly to the GPIO pins of the board avoids the implementation of custom interface boards, significantly reducing the transmission latency (no custom protocols are needed). Moreover, no level shifters or power regulators are necessary for the communications of external devices working with different voltages, resulting in a more compact and portable solution.
On the software side, our solution is based on the synchronisation of internal microservices that are driven by the predictive model in real time. Classical ANN and CNN models have been used before to solve similar, or more complicated, tasks utilising a more powerful and energy-demanding hardware like GPUs. In our system, we leverage the Brian2 framework for the simulation of an SNN model that is able to work in real time. Brian2 provides full control and customisation of the neural network model and avoids data loss due to dropping spikes, which are usually present in dedicated neuromorphic hardware. An SNN model, compared to CNN, is better adapted to cope with elaborate event-based inputs. It avoids loss of data that can be present in CNN models (due to a frame-based approach implicit in the model), also resulting in a slower prediction, which is otherwise crucial in real-time applications.
Our approach has a strong scalability. On the hardware side, it only depends on the number of available communication ports on the board. Diverse types of sensors can be added to the system, giving the opportunity to explore sensory fusion. On the software side, the scalability of the SNN, in terms of the number of neurons and synapses, is highly flexible because all simulations are conducted in software. In our case, there are no hardware constraints on the SNN (apart from the speed of execution) in contrast to neuromorphic computing, where the scalability of the network often depends on the scalability of the hardware as the network needs to be mapped directly to the available hardware. Finally, RaspiSNN works as a modular edge computer, allowing the development of a cluster of SBCs to improve the computational power and connectivity if needed.
4.2. Limitations
Despite the results described above, several limitations should be highlighted. Firstly, the real time performance of the model is affected by the SNN’s model complexity and structure. A smaller time-step increases the computation detail over time, affecting the delay inner simulations steps that can lead to overtime of the simulation against the real time. In this work, the minimum time-step that guarantees real-time synchronisation is ms. Additionally, the number of neurons and synapses, and their differential equation model, impacts the complexity and the computation time.
Second, the DVS input is susceptible to environmental conditions. Although the DVS is generally a robust vision system, factors such as artificial lighting, shadows, table background, and ball color can introduce noise, which may affect accuracy. One way to address this would be by integrating filters in the SNN model, to automatically adjust the neurons threshold based on noise, although this will influence the real-time execution.