Spike-timing based coding in neuromimetic tactile system enables dynamic object classification
Data files
Mar 12, 2024 version files 768.55 KB
-
Fig2C.xlsx
-
Fig2D.xlsx
-
Fig2F.xlsx
-
Fig3A.xlsx
-
Fig3B.xlsx
-
Fig3C.xlsx
-
Fig3D.xlsx
-
Fig3F.xlsx
-
Fig3G.xlsx
-
Fig4A.xlsx
-
Fig4B.xlsx
-
Fig4C.xlsx
-
Fig4D.xlsx
-
Fig4E.xlsx
-
Fig4F.xlsx
-
README.md
Abstract
Coding dynamic tactile information in spike timing is essential to human haptic exploration and dexterous object manipulation. Conventional electronic skins generate frames of tactile signals upon interaction with objects and are unfortunately ill-suited for efficient coding of temporal information and rapid feature extraction. Here, we report a neuromorphic tactile system that uses spike timing, especially the first-spike timing, to code dynamic tactile information about touch and grasp. This strategy enables the system to seamlessly code highly dynamic information with millisecond temporal resolution on par with the biological nervous system, yielding dynamic extraction of tactile features. Upon interaction with objects, the system rapidly classifies them in the initial phase of touch and grasp, thus paving the way to fast tactile feedback desired for neuro-robotics and neuro-prosthetics.
README: Spike-Timing Based Coding in Neuromimetic Tactile System Enables Dynamic Object Classification
Explanation of Figures
- Fig 2C: Typical spike trains in the ensembles of afferents (with indices 29 through 32 and 36 through 39) on the fingertip in response to touching the hard hemisphere (left) and soft hemisphere (right). The spike trains have been digitalized. The center of spike trains is intentionally aligned to t=2s just for illustration.
- Fig 2D: Activity over population, which is the time-dependent spike rate by counting spikes in all afferents within a bin size of 0.01 s, in response to touching the hard hemisphere (left) and soft hemisphere (right).
- Fig 2F: 1st-spike patterns (radar maps) illustrating the 1st-spike sequences and latencies of the ensemble of afferents, in response to touching the hard hemisphere (left) and soft hemisphere (right). The distance from the pole defines the relative 1st-spike latency to t0, while the angle denotes the artificial afferent that fired the 1st spike. Here, all unrecruited afferents are set at t=10s just for illustration. The moment that the first activated afferent fires its first spike is intentionally set at t0=0s.
- Fig 3A: Visualization of the outputs of the trained network of spiking neurons for testing samples using t-SNE in response to touching four surfaces varied in curvature and stiffness. Each point represents the outputs of a testing sample. Symbol denotes the actual label of the sample. There are 10 testing samples in each surface category.
- Fig 3B: (Top) time-dependent classification accuracy of all testing samples that were classified according to curvature, stiffness, and both curvature and stiffness, respectively. (Bottom) time-dependent first-spike count and total spike count, both of which were accumulated over time. The spike count counts spikes from all afferents. t=0 denotes the moment that the first activated afferent fires the first spike.
- Fig 3C: Time-dependent classification accuracy of the testing samples in each surface category, that were classified according to curvature and stiffness, respectively.
- Fig 3D: Statistics of the time needed for half of the testing samples in each category being classified correctly as curvature or stiffness.
- Fig 3F: Stimulating-spike-train output frequencies for surfaces in each category with different stiffness.
- Fig 3G: Correlation between hindlimb twitching angle and surfaces’ stiffness, which is calculated based on the stimulation frequency used.
- Fig 4A: Typical time-dependent activity over population during grasping an apple by subject 1. t=0 denotes the moment that the first activated afferent fires its 1st spike.
- Fig 4B: Typical 1st-spike pattern of grasping the apple. The tactile afferents are re-sequenced to show the cooperativity between different parts of the hand during grasping. All unrecruited afferents stayed at t=0. The moment that the first activated afferent fires its first spike is intentionally set at t0=0.1s just for illustration. So, the timing of all spikes is increased by 0.1s accordingly.
- Fig 4C: Visualization of the outputs of the trained network of spiking neurons using t-SNE in response to the grasp of 22 objects. Each point represents the output of one grasp event, of which the object’s actual label is denoted by the corresponding symbol.
- Fig 4D: Time-dependent classification accuracy and 1st-spike count during grasping.
- Fig 4E: Maximum first-spike latency, averaged over testing samples in each object category, plotted against the classification time of a = 52%. Maximum first-spike latency is the latency between first and last appeared first spikes in all afferents. Linear fitting result and correlation analyzing results are also presented.
- Fig 4F: Normalized first-spike count (a = 52%), plotted against the corresponding classification time. Normalized first-spike count is the first-spike count at the moment normalized by the total recruited neuron number when the object is fully grasped. Linear fitting result and correlation analyzing results are also presented.
Description of the data and file structure
Data in this upload are organized based on the figure they appear in the publication: Spike-timing based coding in neuromimetic tactile system enables dynamic object classification
Sharing/Access information
Links to other publicly accessible locations of the data:
N/a
Other sources:
N/a
Methods
The method is provided in SM. In brief, the raw data of spike trains in voltage were generated and collected by using the e-skin that touched surfaces and grasped objects. The raw data was converted to digital spike trains with 1 as a spike and 0 as non-spike. The processed spike trains were analyzed to study the encoding performance of the system and were used to train and test the spiking neural networks for classification tasks. The tabular data provided are corresponding to the main text figures (i.e., Fig 2, Fig 3 and Fig 4).