Simultaneous Velocity and Texture Classification from a Neuromorphic Tactile Sensor Using Spiking Neural Networks

Abstract

The neuroTac, a neuromorphic visuo-tactile sensor that leverages the high temporal resolution of event-based cameras, is ideally suited to applications in robotic manipulators and prosthetic devices. In this paper, we pair the neuroTac with Spiking Neural Networks (SNNs) to achieve a movement-invariant neuromorphic tactile sensing method for robust texture classification. Alongside this, we demonstrate the ability of this approach to extract movement profiles from purely tactile data. Our systems achieve accuracies of 95% and 83% across their respective tasks (texture and movement classification). We then seek to reduce the size and spiking activity of our networks with the aim of deployment to edge neuromorphic hardware. This multi-objective optimisation investigation using Pareto frontiers highlights several design trade-offs, where high activity and large network sizes can both be reduced by up to 68% and 94% at the cost of slight decreases in accuracy (8%).

1. Introduction

Due to their rapid evolution, the proficiency of robotic manipulators and prosthetics lags far behind human performance and reliability, even for simple tasks. Physical interactions rely on a plethora of heterogeneous sensory inputs, often utilising both visual and tactile cues to perform basic tasks such as grasping, pouring and palpating. Expanding the sensory capabilities of manipulators is, therefore, an important issue for research as we look toward more competent systems. With the introduction of additional sensing modalities comes the delicate matter of developing algorithms to integrate and utilise these tactile senses, such that the robot is able to shape its interactions accordingly. The ability to classify texture is one such component of tactile perception that affords the manipulator richer knowledge of their environment and knowledge about how to approach, continue or avoid different interactions. More complex and capable systems often come at the cost of increased processing power, latency and/or weight. The application of brain-inspired neuromorphic hardware and processing techniques is well placed to facilitate further advancement, avoiding these common drawbacks by providing the potential to reduce power consumption [1,2,3] and offer increased performance in terms of latency [4,5] due to relative data sparsity.

Several processing platforms are available for the deployment of neuromorphic systems, including but not limited to Intel’s Loihi 1 [6] and its successor board Loihi 2 [7], IBM TrueNorth [8], and SpiNNaker [9]. These systems leverage neuromorphic processing techniques to provide energy-efficient edge platforms for the practical implementation of SNNs. Like any edge computing platform, constraints exist that limit the number of neurons and weighted connections they are able to update in real-time. Seeking to appropriately utilise these energy and computational benefits requires engineers to give careful consideration to design choices that impact memory and power consumption.

As we push toward a more human-level tactile interaction, the fusion of tactile sensing and computer-vision-based cues in robotic manipulation systems has yielded positive results for various tasks [10,11]. Within the domain of active prosthesis, vision-based cues are less useful, with the user providing visual input and control over the device. These factors, coupled with the potential occlusion of vision in real-world environments, encourages a move towards purely tactile-based sensing solutions.

In order to properly utilise these technological advances, edge neuromorphic solutions must be devised, presenting unique challenges to development. Limitations, such as constraints on how the textures are explored and the encoding of non-spiking inputs, may restrict their ability to perform in real-world classification tasks, especially where environments are much less structured. Accordingly, a pragmatic trade-off exists when engineering these systems based on their intended applications. Towards establishing an understanding of these trade-offs, we present the following contributions:

• The development of an end-to-end tactile neuromorphic system capable of movement-invariant texture classification.
• The simultaneous classification of the movement profile of a tactile sensor across different surfaces.
• A multi-objective optimisation analysis of network size, activity and accuracy to fit edge platform constraints.

This paper is structured as follows. Section 2 showcases related works in the field of artificial texture classification. Section 3 contains the equipment and experimental setup used to collect the tactile dataset used within this work. Section 4 demonstrates our preprocessing, network design, optimisations and analysis techniques. Section 5 details the results of our data collection, network optimisations, and analysis for our networks. Section 6 discusses our findings, the potential limitations of the work and recommendations for further work. Finally, Section 7 briefly summarises the work and contributions presented.

2. Related Works

Tactile texture classification has been widely explored using a multitude of different sensing technologies. These include capacitive [12], triboelectric [13], multimodal sensors such as the biotac [14,15], and optical sensors. Open-source optical tactile sensors such as the TacTip [16] and Gelsight [17] provide large spatial dimensionality in comparison to sensors of the same size while demonstrating sub-millimeter accuracy across a range of tactile tasks. While other sensor solutions may require additional encoding steps to convert their data into a spiking format for use with neuromorphic hardware, optical sensors are able to exploit event-based vision technology. Event-based cameras are able to operate at frequencies of ≤10 khz [18], outperforming standard video cameras by over 2 orders of magnitude. Prior works by the authors demonstrate the feasibility of the neuroTac sensor [19], a neuromorphic optical tactile sensor, for texture classification, using both traditional machine learning algorithms [20] and Spiking Neural Networks [2]. These works, while presenting high classification performances of ≤94%, are trained and tested on texture samples explored using limited movement profiles, constrained to a constant velocity and contact force. Other State of the Art (SOTA) works encounter similar issues with constrained experimental conditions. Coa et al. [21] present a zero-shot approach to identify previously unseen textures based on perceptual dimensions learned from training with other textures. Using an optical tactile sensor, this work achieves an accuracy of 83% when classifying these unseen textures. However, the collection of data within this experiment relies on the controlled pressing of a tactile sensor (Gelsight) against the texture surface. This methodology creates detailed tactile images of the texture surface but does not examine the surface during movement or the application shear forces, an important effect during tactile interactions that we seek to investigate here. Yang et al. [22] highlights the importance of these shear forces when identifying texture using tactile sensors.

In real-world applications, texture exploration cannot be assumed to utilise a consistent movement, especially when a user controls the motion of the sensor across the surface, as is the case with a prosthetic device. It is therefore advisable for the classification system to be either generalised across a wide range of movements, textures, and contact forces or to be able to perceive and utilise the movement of the sensor across the surface to inform its decisions. Hollins et al. [23] note how tactile afferent responses are dependant on the scanning speed used to interact with a given texture, while cortical neurons are able to disentangle these signals to allow for our innate ability for movement-invariant texture discrimination. Similarly, Boundy-Singer et al. [24] imply that different mechanisms within the brain work to compensate for different movements in order to maintain this high level of perception.

A simple approach toward movement-invariant texture discrimination is to train a machine learning model across a dataset utilising various exploratory patterns, allowing the training step to decouple the complex relationships between sensor output and texture features. Taunyazoz et al. [25] present results from a texture classification experiment using an iCub robot for data collection. They note how the compliance in the limbs of the robot allows for a single Degree of Freedom (DOF) to control the movement of the robot’s sensorised hand across the textures, with said compliance providing different perturbations for the collection of each sample. Across this heterogeneous dataset, the system presented achieves a classification accuracy of 98%, showing generalisation across the varied exploratory movements. A similar approach using a passive touch approach, where the texture is moved under the sensor rather than the sensor itself moving, is used within [26]. Both of these works utilise standard machine learning approaches to achieve these classification results. Gupta et al. [27] report accuracies of up to 98% when using a Support Vector Machine (SVM) to classify texture data from a neuromorphic sensor. Their experiment adjusted the angle at which the sensor was positioned, providing some perceived variance to the movement as it related to the sensor array.

Whereas many works seek to achieve texture classification regardless of exploratory movement, the classification of specific movements is an area not often explored within artificial tactile sensing literature. Kinesthesis within robotic systems is often performed by processing the output of a purpose built sensor [28,29]. While perhaps analogous to classifying known movements, specific movements must be detected amid noisy signals in the field of tactile slip detection. Works such as those presented in [30,31,32] must be able to disentangle task-specific tactile signals from movements caused by object slippage. The ability of these slip detection algorithms to detect slippage when interacting with previously unseen objects is also a key factor in their applicability to real-world interactions. A similar objective is sought within our work: the classification of different velocity profiles from a tactile sensor with the goal of this classification being invariant to contact surface.

Neuromorphic systems offer the potential for decreased latency and power efficiency while still providing strong performance in classification tasks. To use SNNs in different applications, one must consider all of these aspects during design and optimisation. Many works exist that investigate the impacts of different neuromorphic hardware [33,34,35], encoding schemes [36] and neuron models [37] but all seek to either maximise task performance or minimise one or more of power consumption, latency or memory footprint [2,38,39]. Optimising across these metrics becomes even more important when working on edge applications such as prosthetic devices. These optimisations are discussed within [40], where an exploration into spiking activity versus classification accuracy is performed for a neuromorphic tactile texture task. In this work, we introduce more dimensions to this optimisation problem by including metrics for SNN size, with memory usage being a key constraint on edge platforms.

3. Experimental Setup

3.1. neuroTac Sensor

The tactile sensor used within this work is the neuroTac optical tactile sensor first presented in [19]. The sensor mimics human tactile perception by combining the biomimetic design of the open-source TacTip sensor [16] with an event-based camera. The sensor’s soft, domed surface is embedded with internal pin-like structures with markers to emulate the distributed discrete nature of mechanoreceptors in human skin. The deflection of these markers, as the result of deformation of the sensor’s skin, is tracked by the event-based camera to produce a rich, natively spiking output in response to contact. The version of the neuroTac used in this experiment utilises a DVXplorer (Inivation) event-camera, a higher resolution camera than that used within prior works with the neuroTac [19,20,41], and a tip with 331 pins laid out in two-dimensional concentric circles, stereographically mapped to the inside of the sensor skin (see Figure 1).

3.2. Dataset Collection

Within this work, we expand on the range of naturalistic textures explored in previous works by the authors towards a more comprehensive tactile dataset [2,20]. Based on prior studies, humans identify textures using distinct tactile dimensions [42]. These dimensions are roughness, hardness, temperature and stickiness. Our dataset aims to include textures that encompass the full range of each perceptual dimension. Figure 2 shows the textures that are explored within this work.

In order to collect the datasets used during this work, a Franka Research 3 robotic arm (RoboDK Software S.L, Barcelona, Spain) is used to move the neuroTac across the textures with a constrained downwards force within the bounds of $F=1.51\mathrm{N}\pm 0.5\mathrm{N}$, approximating forces recorded for human tactile texture exploration by Smith et al. [43]. The downward force applied by the tip of the sensor is verified by a ROBOTIQ FT 300-S Force Torque Sensor (ROBOTIQ Europe, Vaulx-en-Velin, France) at both the beginning and end of the sample. The setup for this movement is shown in Figure 1.

Within our experiment, the sensor is moved across the textures using different velocity profiles ($Vel\_Prof$): constant velocity, constant acceleration and constant deceleration. Each are explored at different starting velocities or rates of change, as shown in Figure 3. The velocities achieved by these movement profiles (≤60 mm/s) are consistent with human exploratory patterns for texture, as presented in [44].

Our collected dataset comprised of 100 iterations of each texture, at each $Vel\_Prof$, resulted in 14,400 samples (12 textures $\times 12$$Vel\_Prof$$\times 100$ iterations). A separated test set was initially split from the dataset at a ratio of 0.2. From the remaining 80% of the dataset, a train/validation split of 0.6/0.4 was applied. This resulted in a test split of 2880 samples, a training split of 6912, and a validation split of 4608. For the results in this work, the classification performance is given by the accuracy of the networks when classifying across the test set. Details of the collected and preprocessed datasets are shown in

Table 1. Features of raw and preprocessed datasets.

Dataset	Number of Samples	Number of Textures	Number of Movements	Spatial Resolution (Pixels)	Sample Length (ms)
Raw Data	14,400	12	12	640 × 480	1400–4800
Processed Data	-	-	-	78 × 78	1000

.

4. Proposed Method

4.1. Networks

Preprocessing

The large input space provided by the inivation DVXplorer event camera within the neuroTac (640 × 480) must be reduced in order to increase processing throughput and for eventual deployment to Loihi2. Loihi2 is a neuromorphic processing platform designed specifically for the deployment of SNNs [6]. This reduction is performed by a combination of cropping and pooling of the camera frame. Cropping is performed to remove pixels on the edges of the frame that do not have a view of the sensors internal pins. A pooling operation is performed to further reduce the input space. This operation is inspired by the neuromorphic pooling algorithm outlined in Rizzo et al. [45], as has been utilised in prior work [2]. A pooling kernel (of size $k_x\ast k_y$) is shifted a set distance ($stride$) over the 2D input space. At each timestep, if the kernel contains a number of spikes above a given threshold $p_{thresh}$, it outputs a spike to the corresponding pixel in the output mapping. By tuning the kernel size ($k_x$ and $k_y$), $stride$, and spiking threshold ($p_{thresh}$), we are able to reduce spatial dimensionality and perform basic noise filtering within one operation. The following parameters were used for the pooling of samples within this work: $k_x=4$, $k_y=4$, and $stride=4$, $p_{thresh}=1$. This pooling resulted in an output size of 78 × 78. We found that further pooling resulted in an unacceptably large degradation of performance.

Moving the sensor across the surface with different $Vel\_Prof$ could lead to samples with inconsistent temporal lengths, i.e., higher velocity movements, moved across textures of consistent sizes, will inherently result in shorter sample lengths. An option to combat the effects of this is to pad shorter samples to maintain a sample length more consistent with the slower movements. This, however, is undesirable as the network may learn to identify samples due to the absence of spikes at the end of certain samples rather than learning the features present within the spiking patterns. Therefore, samples collected from the sensor are cropped in the temporal domain to maintain an even sample length across all $Vel\_Prof$. A beneficial side effect of this temporal cropping is an overall reduction in dataset size, leading to faster training. For the training of our networks in this article, samples were cropped to 1000 ms.

Figure 4 demonstrates how the observed spiking patterns change for a given sample during each step of this preprocessing.

4.2. Training

Within this work, we train our SNNs using the SLAYER algorithm [46] employed by the Lava framework (https://github.com/lava-nc/lava-dl accessed on 7 January 2024). This algorithm allows for the training of SNNs in a process similar to gradient back propagation. Our networks all comprise densely connected layers of current-based (CUBA) Leaky-Integrate and Fire (LIF) neurons [47]. The neuron parameters are set within Lava at 1.25 voltage threshold, 0.03 voltage decay constant, 0.25 current decay constant, and 0.03 time constant ($\tau$). The parameters of these neurons are kept consistent for each model. These values were found through manual tuning with a random dropout applied to 10% of neurons in each layer of each network to reduce overfitting. Each network was trained for 100 epochs with an initial learning rate of 0.01 that decays by 33% every 20 epochs. Our networks all feature a neuron in the output layer for each of the potential output classifications, with classification determined by the output neuron with the highest number of spikes accrued within the sample time.

Data sparsity is a driving factor behind the power efficiency of neuromorphic hardware. While analogous to higher amounts of information being processed, high spiking activity within networks correlates to higher energy overheads, even when running on purpose built neuromorphic hardware [48,49]. Here, a potential trade-off exists between information processing and energy efficiency. Previously mentioned limitations present within edge hardware further constrains the size and complexity of the networks we seek to train and deploy. With the goals of deploying to the edge in the future, we investigate the effects of altering network size and spiking activity within this work.

Several training parameters can be altered in order to elicit different behaviours from the network. The Spike Rate loss function used by Lava SLAYER utilises a $truerate$ parameter ($0<truerate<1$) to control the network’s spiking activity. This parameter acts as a set point, encouraging the network to obtain a target spiking rate in the correctly classified neuron. For example, at $truerate=1.0$ the loss function will encourage the network to output a spike within the correct output neuron at every time step. The error is then calculated based on this set point and the network’s ability to create this level of spiking activity. Increasing this set point, therefore, increases the overall spiking activity of the network. An opposing $falserate$ parameter works conversely to reduce the spiking activity in the incorrect output neurons. This $falserate$ parameter remains set at $0.02$ for all networks within this work.

4.2.1. Texture Classifier

Our first method for movement-invariant texture classification is the training of a network using data from all $Vel\_Prof$, for all textures, without considering the difference in $Vel\_Prof$. This, ideally, results in a simple, effective and robust classification system without the complications of further features.

An input layer of size 6084, followed by a single hidden layer and a final output layer of size 12, provides one output neuron per texture. The size of the hidden layer was selected based on a grid search detailed in Figure 5 and

Table 2. Search Space for the performed training parameter grid search. The Peak Values shown here are the training parameters that produced the highest accuracy classifiers.

Parameter	Search Space	Classifier	Peak Value
Hidden Layer Size	$S=\{x\mid x=25n, n\in \mathbb{N}, x\le 500\}$	Texture	425
		$Vel\_Prof$	450
True Rate	$S=\{0.2+0.1n\mid n\in {\mathbb{N}}_0, 0\le n\le 7\}$	Texture	0.9
		$Vel\_Prof$	0.3

.

4.2.2. Velocity Profile Classifier

Here, we introduce a $Vel\_Prof$ classification network. The structure of this network is similar to that of the texture classifier and is trained using the same data as the texture classifier network. This network, however, is trained to classify the sample as one of the 12 movement profiles described in Section 3.2.

The same grid search used for the texture classifier was performed upon this network, again shown in Figure 5 and Table 2.

4.3. Metrics

As mentioned previously, when appraising our SNNs for their utility on edge platforms we must also consider the constraints of this application domain. In order to achieve this, we use the following metrics that account for performance, the management of resources, and power consumption.

4.3.1. Accuracy

An objective measure of classification performance calculated as a ratio of true-positive ($TP$) and true-negative ($TN$) classifications to the number of samples within the test set (N). This is shown below in Equation (1).

$$Accuracy=\frac{TP+TN}{N}$$

(1)

4.3.2. Total Number of Weights

The memory resource limitations of edge platforms often limit the size of the network we are able to construct. The total number of weighted connections ($W_t$) within our network is an easily comparable metric that we actively seek to minimise within this work. Equation (2) shows the calculation for this metric given that our network layers are densely connected.

$$W_t=\sum _{l=0}^{L-1}{i}_l\ast i_{l+1}$$

(2)

where $W_t$ is the total number of of weighted connections in the network, L is the number of layers in the network, and $i_l$ is the number of neurons in layer l.

4.3.3. Spiking Activity

As discussed above (Section 4.2), spiking activity is analogous to power consumption. With one of the advantages of neuromorphic hardware being its relatively low energy footprint, we also seek to minimise the spiking activity within our networks. Spiking activity (S) is defined in Equation (3).

$$S=\frac{{\sum }_{n=0}^N{\sum }_{i=0}^I{s}_i}{N}$$

(3)

where n is the current sample, N is the total number of samples in the test set, I is the total number of neurons in the network, and $s_i$ is the number of spikes that occur for neuron i within the current sample.

4.4. Optimisation Techniques

4.4.1. Grid Search

A grid search optimisation is performed on each of our networks in order to determine the highest performing networks in terms of accuracy across the search space. The full search space and optimisation results from the performed grid search are shown within Table 2. From the results of our grid search, different analysis techniques allow us to analyse potential trade-offs when choosing a network to deploy for a specific application.

4.4.2. Pareto Frontier Analysis

Pareto frontiers [50] present a series of equally efficient solutions along a multi-dimensional search space, providing insights into potential design trade-offs when optimising for specific metrics. We look to find Pareto front approximations from our grid search, enabling us to find optimal solutions that provide trade-offs between each of our three metrics: accuracy, the average spiking activity (S) of the network, and the total number of weights present in the network ($W_t$). We form Pareto frontiers for both our texture and $Vel\_Prof$ classifiers, seeking to find a series of equal solutions depending on the choice of metric. We do so using a skyline query [51].

5. Results

5.1. Data Inspection

Figure 6 shows several raster plots comparing the data output from the neuroTac sensor when moved across both different textures with the same $Vel\_Prof$ and the effects different $Vel\_Prof$ have on the output from the same texture.

Figure 6a,b showcases the variance in spiking activity produced at the same $Vel\_Prof$ depending on the method of exploration. Visual inspection implies a large loss of data due to temporal clipping; however, prior works have proven that initial movement of the neuroTac across a texture is sufficient to accurately classify [2,20].

Large differences in spiking patterns are also observed for the same texture across different movement profiles. Figure 6c,d showcases this with the higher speed achieved by $Vel\_Prof$ 3, evidenced by a shorter sample length when compared to profile 2. As previously mentioned, the temporal clipping in our preprocessing is implemented to avoid this imbalance.

5.2. Grid Search Results

In order to determine the optimal hidden layer size of our texture, $Vel\_Prof$, we perform a grid search using the parameters noted in Table 2. The results of this optimisation are shown in both

Table 3. Comparison of our networks analysed in this work. Bold values indicate optimal values for that metric across the performed tests.

Classifier	Method	Accuracy	S ($\times {10}^6$)	${\mathit{W}}_{\mathit{t}}$ ($\times {10}^6$)
Texture	Peak	0.95	9.07	2.59
	Pareto Accuracy	0.95	9.07	2.59
	Pareto Spiking Activity	0.68	0.13	0.15
	Pareto Network Size	0.87	0.75	0.15
$Vel\_Prof$	Peak	0.83	1.70	2.74
	Pareto Accuracy	0.83	1.70	2.74
	Pareto Spiking Activity	0.42	0.08	0.15
	Pareto Network Size	0.71	0.54	0.15

and Figure 5. These finalised values indicate that, generally, towards the upper limits of the search space is where the classifiers achieve optimal accuracy. Increasing network activity, via an increase in target $truerate$, is also shown to increase the accuracy of our texture and $Vel\_Prof$ classifiers.

Our texture classifier generally achieves high accuracies up to $0.95$ accuracy. As intuitively expected, classification performance tends to increase alongside an increase in network size and spiking activity, showing respective Spearman’s rank correlation coefficients with accuracy of $\rho =0.51$ and $\rho =0.89$, respectively. However, classification performance tends to plateau during the grid search, with classification performance improving $\approx 1\%$ as hidden layer size increases between 200 and 500 nodes. This may indicate that the weights present within smaller networks contain sufficient capacity to learn the correct spatio-temporal features from our data.

Our $Vel\_Prof$ classifier provides slightly lower accuracies of up to $0.82$. Spearman’s rank correlations within this grid search again yield strong correlations between both network size and accuracy ($\rho =0.71$), and spiking activity and accuracy ($\rho =0.81$). As with the texture classifier, this grid search also appears to converge, with smaller networks (hidden layer size of 125) reporting accuracies only $\approx 1\%$ lower than the large 500 hidden layer networks.

Figure 7 presents the confusion matrices for our highest accuracy texture and $Vel\_Prof$ classifiers. While the texture classification network performs well across all textures, its highest rates of error come from the MDF, Denim and Satin textures. The confusion shown between the two wooded textures of MDF and plywood is perhaps expected. Our profile classifier demonstrates its lowest classification performance among the linear acceleration profiles (4–7), with $77\%$ accuracy recorded for these profiles; a similar performance is seen for triangular acceleration (9–12), with an accuracy of $78\%$. Comparatively high performance from the linear $Vel\_Prof$ (0–3) gives respective accuracies of $95\%$.

5.3. Pareto Front Analysis

Figure 8 shows a 3D scatter plot of our grid search, with Pareto frontier tests indicated. These frontier tests are deemed as non-dominated solutions that provide equal trade-offs, for the three metrics we look to optimise. While many tests are shown to exist along these frontiers, we summarise the solutions within Table 3, presenting the points along the frontier where one may choose to fully minimise or maximise one of our metrics. For both our texture and the $Vel\_Prof$ classifiers, where there are several solutions along the frontier that minimise a given metric, we prioritise the solution that returns the highest accuracy.

5.4. Comparative Analysis

Table 3 encompasses the solutions found through our methodologies.

Pareto front analysis of our texture classifier grid search returns the same optimal solution when searching the frontier for maximum accuracy. When looking to minimise spiking activity, we see a decrease of $98.6\%$ in S, alongside a similar decrease in $W_t$ ($94\%$ decrease), when compared to the peak accuracy solution. These gains come at the cost of accuracy with a decrease in performance of $28\%$. Searching for minimum network size solutions along the frontier results in a classifier that is able to again reduce $W_t$ by $94\%$ and reduce S by $91\%$ (with a smaller loss of $8\%$ in accuracy).

Similar results are shown for our $Vel\_Prof$ classifier. The Pareto front includes the peak network parameters at the apex of accuracy maximisation. Minimising S along this front results in a respective $95\%$ and $94\%$ reduction in S and $W_t$, at the expense of a large $49\%$ reduction in accuracy. Finding the smallest network from this test results in the same $94\%$ reduction in $W_t$; however, the comparatively low $68\%$ reduction in S results in classification performance only $14\%$ below peak accuracy.

6. Discussion

This work has presented a series of SNNs able to classify a variety of naturalistic textures or movements based on input from a neuromorphic tactile sensor. We then provide an investigation into the effects of network size and spiking activity on the classification performance of our systems, with the goal of finding solutions that are able to deploy to constrained edge neuromorphic platforms.

From our grid search detailed in Section 5.2, we can see that the larger texture and $Vel\_Prof$ classifiers achieved a higher performance than smaller networks, regardless of spiking activity. This is expected as larger networks contain a higher capacity for learning. Interestingly, despite having a much larger input space afforded by the DVXplorer camera, the hidden layer size for both networks trended towards the network size observed in prior work [2].

As observed within Figure 5, MDF and Plywood provide the highest confusion during classification across all texture classifiers. We assume that both being similar wooden textures leads to this confusion and posit that that may also lead to classification confusion within human subjects. While this confusion is discussed here, these occurrences are minimal with all texture classifiers reporting ≥95% accuracy.

Relatively low classification performance was seen for our $Vel\_Prof$ classifiers when compared to our texture classifiers (peak 82%). From the confusion matrices of our peak performance network, our classifier seems to perform well at identifying different constant velocities (profiles 0–3). The main confusion occurs within profiles that require acceleration across the texture. The confusion seen within $Vel\_Prof$ sets (4–7, 8–11) indicates that the system responds well to identifying the movement of the sensor across the texture, only failing to correctly classify the precise acceleration/deceleration for the said profile. Interestingly, the confusion displayed by $Vel\_Prof$ 5 (37%) with profiles 6 and 7 further indicates that the network is getting confused within each movement profile shape. We hypothesise that this confusion is potentially due to the temporal cropping of our data to the first 1000 ms. Figure 3 showcases how, for our constant acceleration and triangular-shaped velocity profiles, similar trajectories are followed during this period of each sample, whereas constant velocity profiles appear to be more differentiable.

Our Pareto frontier analysis of our networks has presented a viable method for design engineers to tailor power, size and performance based on the requirements of different edge applications. For applications such as active prosthetics, where power consumption is often cited as a key design consideration [52,53], our analysis provides a solution that has an S 98.6% lower than a solution found by simply optimising a SNN for accuracy. Our Pareto solutions that provide the smallest, lowest spiking networks among our texture classifiers exceed the human baseline accuracy of ≈65% [2,20] despite their relatively poor performance when compared to the peak accuracy classifiers. This indicates that human-level texture classification is possible even on constrained edge environments.

Additional steps to further reduce $W_t$ such as pruning are not investigated during this work, with a focus primarily on parameters used to train our networks. These techniques could be applied to networks trained in order to further reduce network size, and in turn spiking activity, for even more highly constrained environments. These techniques have proven to be effective for reducing the size of large scale spiking networks [54].

The integration of our texture and $Vel\_Prof$ classifiers into a combined tactile system is a logical future step for this work. Prior studies have demonstrated how combining both proprioceptive and tactile senses using Neural Networks (NNs) generally leads to an increase in texture classification performance [22,28]. Alternative approaches using reservoir computing paradigms could also be explored due to their largely interconnected neuronal structure mimicking areas of the brain [55]. These architectures have also previously been explored on edge neuromorphic platforms, underpinning their utility [56].

Finally, we plan to deploy the networks developed within this work to Loihi2, a neuromorphic processor designed for Spiking Neural Network implementation [7]. This will enable embedded processing within robotic hands that require rapid and low power processing solutions.

7. Conclusions

In this work, we present an end-to-end system utilising a neuromorphic tactile sensor and a Spiking Neural Network to classify naturalistic textures in a movement-invariant manner. A parallel network utilising the same sensory input is also introduced to classify movements. Our classifiers achieve accuracies of $95\%$ and $82\%$ for the texture and velocity profiles ($Vel\_Prof$), respectively. After performing a grid search of training parameters controlling network size and spiking activity, we analyse the Pareto frontier to discuss potential design trade-offs for the deployment of these networks to edge hardware environments. In doing so, we are able to reduce the size of our texture classification network by up to $94\%$ and spiking activity by up to $68\%$ while still maintaining an accuracy above human baseline performance ($87\%$).