Data from: Decoding intended speech with an intracortical brain-computer interface in a person with longstanding anarthria and locked-in syndrome
Abstract
Intracortical brain-computer interfaces (iBCIs) for decoding intended speech have provided individuals with ALS and severe dysarthria an intuitive method for high-throughput communication. These advances have been demonstrated in individuals who are still able to vocalize and move speech articulators. Here, we decoded intended speech from an individual with longstanding anarthria, locked-in syndrome, and ventilator dependence due to advanced symptoms of ALS. We found that phonemes, words, and higher-order language units could be decoded well above chance. While sentence decoding accuracy was below that of demonstrations in participants with dysarthria, we can attain an extensive characterization of the neural signals underlying speech in a person with locked-in syndrome and, through our results, identify several directions for future improvement. These include closed-loop speech imagery training and decoding linguistic (rather than phonemic) units from neural signals in the middle precentral gyrus. Overall, these results demonstrate that speech decoding from the motor cortex may be feasible in people with anarthria and ventilator dependence. For individuals with longstanding anarthria, a purely phoneme-based decoding approach may lack the accuracy necessary to support independent use as a primary means of communication; however, additional linguistic information embedded within neural signals may provide a route to augment the performance of speech decoders.
Dataset DOI: 10.5061/dryad.vq83bk481
Description of the data and file structure
Anarthric Speech iBCI Dataset
Overview
These data are reported in Jude et al. 2026 and consist of separate sessions (listed below in “Session Table”) across a BrainGate2 clinical trial participant with anarthria and locked-in syndrome.
All data files are .mat files that can be loaded with MATLAB or Python (using the scipy function scipy.io.loadmat). Each .mat file is a single session block.
Alongside each .mat file, a .png image shows normalized neural features (threshold crossings and spike power) from a single trial in each .mat file block.
Accompanying code is available at https://github.com/justin-jude/anarthria-speech-bci.
Trial-by-trial neural features and corresponding ground truth sentence pairs can be extracted from .mat files using the following function: https://github.com/justin-jude/anarthria-speech-bci/blob/main/utils/mat_to_tfrecord.py
Blocks run in each session are listed below per participant:
Participant T17 Isolated Orofacial Movements, Phonemes and individual Words
t17.2024.03.25 (Trial day 25)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 5 | Isolated Phonemes | 80 |
| 6 | Isolated Phonemes | 80 |
| 7 | Isolated Phonemes | 80 |
| 8 | Isolated Phonemes | 80 |
| 9 | Isolated Phonemes | 80 |
| 10 | Isolated Phonemes | 80 |
| 11 | Isolated Phonemes | 80 |
| 12 | Isolated Phonemes | 80 |
| 13 | Isolated Phonemes | 80 |
| 14 | Isolated Orofacial Movements | 34 |
| 15 | Isolated Orofacial Movements | 34 |
| 16 | Isolated Orofacial Movements | 34 |
| 17 | Isolated Orofacial Movements | 34 |
| 18 | Isolated Orofacial Movements | 34 |
t17.2024.03.26 (Trial day 26)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 10 | Individual Words | 50 |
| 14 | Individual Words | 50 |
| 15 | Individual Words | 50 |
| 16 | Individual Words | 50 |
| 17 | Individual Words | 50 |
| 19 | Individual Words | 50 |
| 20 | Individual Words | 50 |
| 21 | Individual Words | 50 |
| 22 | Individual Words | 50 |
Participant T17 Phased Task
t17.2024.05.23 (Trial day 84)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | Phonemes | 50 |
| 2 | Words | 50 |
| 3 | Short Sentences | 50 |
| 5 | Sentences | 50 |
Participant T17 Sentence Speech
t17.2024.03.26 (Trial day 26)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 23 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 24 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 25 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 26 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 27 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
t17.2024.04.02 (Trial day 33)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 4 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 9 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 10 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 11 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 12 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 13 | 125k-word Vocabulary Sentences (Closed Loop) | 10 |
t17.2024.04.15 (Trial day 46)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 5 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 9 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 10 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 11 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 12 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 13 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
t17.2024.04.16 (Trial day 47)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 9 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 10 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 11 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 12 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 13 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
t17.2024.04.17 (Trial day 48)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 4 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Closed Loop) | 14 |
| 8 | 125k-word Vocabulary Sentences (Closed Loop) | 11 |
| 10 | 125k-word Vocabulary Sentences (Closed Loop) | 5 |
t17.2024.05.06 (Trial day 67)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 125k-word Vocabulary Sentences (Open Loop) | 13 |
| 4 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 9 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 10 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
t17.2024.05.07 (Trial day 68)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 4 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 9 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 10 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 11 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
t17.2024.05.13 (Trial day 74)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 11 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 12 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
t17.2024.05.14 (Trial day 75)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Closed Loop) | 17 |
| 9 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
t17.2024.05.28 (Trial day 89)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 4 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 125k-word Vocabulary Sentences (Open Loop) | 40 |
| 8 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
| 9 | 125k-word Vocabulary Sentences (Closed Loop) | 40 |
t17.2024.08.12 (Trial day 164)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 4 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 7 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 8 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
| 9 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
| 10 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
t17.2024.08.13 (Trial day 165)
| Block Number | Block Description | Number of Trials |
|---|---|---|
| 1 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 2 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 3 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 5 | 50-word Vocabulary Sentences (Open Loop) | 40 |
| 6 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
| 7 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
| 8 | 50-word Vocabulary Sentences (Closed Loop) | 40 |
Dataset Description
Neural voltage time series signals were recorded and digitized (30 kHz, 16 bits per sample) using the NeuroPort system (Blackrock Neurotech) attached to three percutaneous connectors on the participant’s head, and transmitted via three mini-HMDI cables (one to each Neuroplex headstage), attached to two Gemini hubs (Blackrock Neurotech), prior to final processing via a Neural Signal Processor (NSP) (Blackrock Neurotech). Packets of neural data were streamed from the NSP to our processing pipeline. Signals were analog filtered (4th order Butterworth with corners at 250 Hz to 5 kHz) using the Scipy python library (scipy.signal.filtfilt).
Electrode-specific thresholds were calculated using filtered 30kHz data once these calculated references had been applied to identify spike events. Thresholds were set at -3.5 times the standard deviation of the voltage signal per electrode. The number of non-causal threshold crossings (ncTX) were computed by counting the number of times the filtered neural time series crossed these calculated thresholds (threshold crossing rates). Additionally, spike band power was computed by taking the sum of squared voltages observed during each 10ms time bin.
T17 had six 64-electrode microelectrode arrays (Blackrock Microsystems; 1.5 mm electrode length) placed in the left precentral gyrus; two arrays were placed in the dorsal precentral gyrus (area 6d), two arrays were placed in the ventral precentral gyrus (area 6v), and two arrays were placed in middle precentral gyrus (area 55b). T17 was diagnosed with ALS 3 years prior to the start of this study. T17 has tetraplegia, anarthria, and ventilator dependence; with only remaining volitional motor control being over their extraocular muscles.
Isolated Orofacial/Phoneme/Word Movement blocks pertain to individual trials orofacial movements, phonemes or individual words.
Phased Task
Participant T17 was asked to memorize a phoneme, word, short sentences or sentence in the first reading phase of a phased task. In the second phase, the participant was asked to internally speak (without vocalizing) the memorized language unit. In the last phase, the participant was asked to attempt to speak with articulation the memorized language unit.
Files and variables
Data.zip contains the folder t17 containing all listed session folders. Within each session folder (t17.2024.XX.XX), there is a brainToText folder, then a RedisMat folder in which the .mat files can be found for each session block.
Variables in .mat files
- participant: Session participant is T17.
- session_name: Date of session.
- session_description: Indicates a speech session.
- block_number: Block number corresponding to those listed above.
- block_description: Indicates a speech session.
- binned_neural_threshold_crossings: Number of threshold crossings observed in each 10ms bin.
- binned_neural_spike_band_power: Mean averaged sum of square voltages in each 10ms bin.
- binned_neural_nsp_timestamp: Timestamp in nanoseconds assigned to each 10ms bin by the Neural Signal Processor.
- binned_neural_redis_clock: Timestamp in milliseconds assigned to each 10ms bin by BRAND.
- cue : Ground Truth sentence in Sentence Speech trials.
- decoder_logit_output: In Sentence Speech trials, the RNN output probabilities for each phoneme. Predictions occur every 20 milliseconds (i.e., every other index).
- decoder_signal: Flag indicating start or end of decoding RNN decoding loop.
- decoder_output_redis_clock: Timestamp in milliseconds corresponding to each decoder logit output.
- ngram_decoder_partial_output: Real-time 5-gram language model output, updated continuously as sentence is spoken throughout a trial.
- ngram_decoder_final_output: Final decoded sentence from 5-gram language model for each trial.
- norm_channel_means: Mean values of 512 features (T17) used for decoding - used for online z-scoring to maintain stability.
- norm_channel_stds: Standard deviations of values of 512 features (T17) used for decoding - used for online z-scoring to maintain stability.
- norm_redis_times: Redis timestamps for z-scoring Redis stream.
- delay_duration_ms: Time in milliseconds when cued sentence is shown before speech begins.
- inter_trial_duration_ms: Time in milliseconds between trials.
- trial_start_redis_time: Timestamp in milliseconds signalling the start of a trial before a cue is shown in the delay period, assigned by BRAND.
- trial_start_nsp_neural_time: Timestamp in nanoseconds signalling the start of a trial before a cue is shown in the delay period, assigned by the Neural Signal Processor.
- trial_start_nsp_analog_time: Timestamp in nanoseconds signalling the start of a trial before a cue is shown in the delay period, assigned by the analog microphone component of the Neural Signal Processor.
- go_cue_redis_time: Timestamp in milliseconds signalling to the participant to start speaking after the initial delay period, assigned by BRAND.
- go_cue_nsp_neural_time: Timestamp in nanoseconds signalling to the participant to start speaking after the initial delay period, assigned by the Neural Signal Processor.
- go_cue_nsp_analog_time: Timestamp in nanoseconds signalling to the participant to start speaking after the initial delay period, assigned by the analog microphone component of the Neural Signal Processor.
- trial_end_redis_time: Timestamp in milliseconds indicating the end of a trial, assigned by BRAND. A trial is ended in Sentence Speech trials through a button press.
- trial_end_nsp_neural_time: Timestamp in nanoseconds indicating the end of a trial, assigned by the Neural Signal Processor.
- trial_end_nsp_analog_time: Timestamp in nanoseconds indicating the end of a trial, assigned by the analog component of the Neural Signal Processor.
- microphone_data: Microphone audio recorded during trials.
- microphone_nsp_time: Microphone audio timestamps in nanoseconds.
Language Model
The 5-gram phoneme-based language model used to produce sentences from RNN phoneme probabilities can be obtained here: https://datadryad.org/downloads/file_stream/2547359. This is used in the analysis scripts to obtain a final word error rate for decoding of sequences of phonemes.
Install the WeNet-based Kaldi language modelling software from here first (just the third step): https://github.com/fwillett/speechBCI/tree/main/LanguageModelDecoder
Extract Data.zip and languageModel_5gram.tar.gz to the same folder as the software repo (https://github.com/justin-jude/anarthria-speech-bci) as this is where the analysis scripts will search for both the data and the language model.
Code/software
https://github.com/justin-jude/anarthria-speech-bci
Access information
Other publicly accessible locations of the data:
- None
Data was derived from the following sources:
- Real-time recordings
Human subjects data
No personally identifiable information (PII) is contained within this data. Only neural features and spoken sentences are present. No sentences spoken reveal personal information. Informed consent for data sharing was obtained.
