Data and code from: Human learning of noninvasive brain-computer interfaces via manifold geometry
Data files
Apr 01, 2026 version files 38.82 GB
-
avatarRT_subject_data.zip
38.82 GB
-
README.md
21.50 KB
Abstract
Brain-computer interfaces (BCIs) promise to restore and enhance a wide range of human capabilities. However, a barrier to the adoption of BCIs is that learning to control them can be difficult and slow, with inconsistent results across users. We hypothesized that human BCI learning could be accelerated by leveraging the naturally occurring geometric structure of brain activity, or its intrinsic manifold, extracted using a data-diffusion process. We trained participants on a noninvasive BCI that allowed them to gain real-time control of an avatar in a virtual reality game by modulating functional magnetic resonance imaging (fMRI) activity in brain regions that support spatial navigation. We then perturbed the mapping between fMRI activity patterns and the movement of the avatar to test our manifold hypothesis. When the new mapping relied on directions of significant variance along the intrinsic manifold, participants succeeded in regaining control of the BCI by aligning their brain activity within the manifold. When the new mapping did not rely on the intrinsic manifold, participants could not learn to control the avatar again. These findings show that the manifold geometry of brain activity constrains human learning of a complex cognitive task in higher-order brain regions. Manifold geometry may be a critical ingredient for unlocking the potential of future human neurotechnologies.
Busch, E. L., Fincke, E. C., Lajoie, G., Krishnaswamy, S., & Turk-Browne, N. B. (2026). Human learning of noninvasive brain-computer interfaces via manifold geometry. Nature Neuroscience.
This directory contains raw and preprocessed neuroimaging and behavioral data used for the analyses in the manuscript. To analyze this data, please refer to https://github.com/ericabusch/avatarRT_analysis. The analysis notebook at that repository utilizes these files to recreate the figures reported in the paper. For this purpose, the contents of this directory are expected to be placed in a folder called data/avatarRT. The notebook can be adapted to explore other analyses, which in some cases will create new files.
All files with the suffix .nii.gz or .nii can be opened using FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki) or nilearn in python. Files with the suffix .npy can be loaded in Python using the NumPy library (numpy.load()). Model files with the suffix .pt can be loaded using PyTorch (torch.load()). Behavioral .csv and .tsv files can be read with any standard spreadsheet software or Python (e.g., pandas.read_csv()).
Scan sequences
Data were acquired on a 3T Siemens Prisma scanner with a 64-channel head coil at the Brain Imaging Center at Yale University.
- BOLD (gradient-echo EPI): TR = 2 s; TE = 30 ms; voxel size = 3 mm isotropic; FA = 8°; iPAT GRAPPA acceleration factor = 2; distance factor = 25 %. Approximately 300 volumes (~10 minutes) were collected per run.
- Spin-echo field maps (AP/PA): TR = 5 s; TE = 80 ms; otherwise matching the EPI sequence. Acquired in both anterior-to-posterior (AP) and posterior-to-anterior (PA) phase-encoding directions for susceptibility distortion correction.
- T1w (MPRAGE): TR = 2.5 s; TE = 2.9 ms; voxel size = 1 mm isotropic; FA = 8°; iPAT GRAPPA acceleration factor = 2; 176 sagittal slices. Collected during the joystick session only.
- T2w (fast spin echo): TR = 3.2 s; TE = 565 ms; voxel size = 1 mm isotropic; iPAT GRAPPA acceleration factor = 2; 176 sagittal slices; variable flip angle. Collected during the joystick session only.
Experimental design
Participants (N = 18 after exclusions) completed four or five fMRI sessions, each consisting of four functional runs (~10 minutes each), on separate days (mean intersession interval = 27.24 hours; maximum = 48 hours; all sessions completed within 7 days).
During all sessions, participants performed a custom video game task (Unity3d, Version 2019.4.12f1) in which a human avatar navigated a virtual outdoor arena toward a goal flag. The avatar's movement was controlled either by a joystick (Session 1) or by the participant's brain activity measured via real-time fMRI neurofeedback (Sessions 2–5). See demos at: https://github.com/ericabusch/avatarRT_task. Session types are defined as follows:
- Session 1 (ses_01) — Joystick (calibration): Participants steered the avatar using an MR-compatible joystick (Current Designs Tethyx). No neurofeedback was provided. Data from this session were used to fit the T-PHATE manifold and train the Manifold Regularized Autoencoder (MRAE) that formed the basis of the real-time BCI pipeline used in subsequent sessions.
- Session 2 (ses_02) — Intrinsic Manifold (IM) neurofeedback: The BCI mapped the participant's real-time fMRI activity onto the first component of their T-PHATE manifold (the direction of greatest variance). All participants received the IM session as the first neurofeedback (second overall) session.
- Sessions 3–4 (ses_03, ses_04) — Perturbation neurofeedback (counterbalanced): These sessions used either a Within-Manifold Perturbation (WMP) or an Outside-Manifold Perturbation (OMP) decoder, counterbalanced across participants. The WMP mapped brain activity onto the second principal component of the T-PHATE manifold (second greatest variance), whereas the OMP mapped onto the lowest-ranked component (least variance in the 20-dimensional space). Each perturbation session began with one run of IM re-calibration, followed by three runs of the perturbation condition.
- Session 5 (ses_05) — Additional session (subject-specific): One participant (avatarRT_su_06) did not reach BrainControl plateau during Session 2 and therefore repeated the IM session, completing five sessions total instead of four. For this participant, ses_03 corresponds to a second IM session, and ses_04–ses_05 correspond to WMP and OMP (counterbalanced as for all other participants).
Counterbalancing: The following participants received WMP in Session 3 and OMP in Session 4: [SUBJECT IDs — to be filled in]. All remaining participants received OMP in Session 3 and WMP in Session 4.
The navigation network used as the neurofeedback target ROI was defined from a Neurosynth meta-analysis of 77 studies using the search term "navigation" (FDR = 0.01; bottom 10 % of z-statistics zeroed out; minimum cluster size = 10 voxels), resulting in a mean of 1,354 voxels per participant (range = 1,058–1,599). The MNI-space navigation mask is publicly available at https://github.com/ericabusch/avatarRT_analysis. Each subject-specific mask file can be found within the avatarRT_sub_XX/reference directory.
Description of the data and file structure
Data are organized by subject. Each subject directory follows the naming convention avatarRT_sub_XX and contains the following top-level folders:
session_tracker.csv
Top-level file containing one row per subject with session assignment and metadata. Columns:
| Column | Description | Units/Scale |
|---|---|---|
subject_ID |
Subject directory name | string |
subject_number |
Numeric subject identifier | integer |
omp_session |
Session number in which the OMP condition was run | integer |
wmp_session |
Session number in which the WMP condition was run | integer |
im_session |
Session number in which the IM condition was run | integer |
simulated |
Whether this subject's data were simulated | binary (0/1) |
age |
subject age in months | integer |
omp_component |
T-PHATE component index used for the OMP decoder | integer |
wmp_component |
T-PHATE component index used for the WMP decoder | integer |
n_runs_per_session |
Number of functional runs per session | integer |
runs_to_exclude_xnat |
notes any discrepancies between XNAT data files and eventual analyses | string |
runs_to_exclude_rtc |
notes any discrepancies between rtcloud and eventual analyses | string |
notes |
Freeform notes, things to keep in mind after data collection | string |
model/
Contains the trained Manifold Regularizing Autoencoder (MRAE) for each subject, fit to the Session 1 joystick data. The MRAE was trained with two objectives: (1) to reconstruct input fMRI data in voxel space, and (2) to learn a latent representation that closely resembles the data's T-PHATE manifold geometry. This allows the frozen encoder to extend the T-PHATE manifold to new incoming real-time data points at each TR during neurofeedback sessions.
bottleneck.npy: NumPy array containing the 20-dimensional T-PHATE manifold coordinates (bottleneck activations of the MRAE encoder) for the first 80 % of samples [TRs] from the Session 1 joystick data. Shape: [TRs × 20].modelSpec.txt: Text file specifying the architecture and hyperparameters of the MRAE used for this subject.state_dict.pt: PyTorch state dictionary containing the trained weights of the MRAE model.
reference/
Contains reference images and transformation matrices used to align each session's functional data to a common space (Session 1 native functional space) during real-time preprocessing.
mask.nii.gz: Binary mask defining the participant-specific navigation ROI in Session 1 native functional space. Derived by inverting the MNI-to-native registration and applying it to the Neurosynth navigation z-statistic map, then thresholding and clustering as described above.navigation_native.nii.gz: The Neurosynth navigation z-statistic map registered to the participant's native functional space, prior to binarization.ses_01_brain_mask.nii.gz: Whole-brain binary mask derived from Session 1.ses_01_example_func.nii.gz: Mean functional reference image from Session 1 (V_ref), used as the alignment target for all subsequent sessions and runs.ses_XX_run_YY_reference_volume.nii: The first functional volume acquired in run YY of session XX, smoothed with a 5-mm FWHM Gaussian kernel and saved as that run's motion correction reference volume (V_1).ses_XX_run_YY_reference_volume_to_ses_01_space.mat: FSL-format 4×4 affine transformation matrix (T) aligning V_1 from run YY of session XX to V_ref (Session 1 space). Applied during real-time preprocessing so that all incoming BOLD volumes are projected into the coordinate space in which the navigation ROI mask was defined.
ses_01/ — Joystick (Calibration) Session
Contains anatomical scans, raw and preprocessed functional data, and field maps from the joystick session used to learn the participant's neural manifold.
ses_01/anat/
High-resolution anatomical images collected during Session 1 and skull-stripped using FSL's Brain Extraction Tool (BET).
avatarRT_sub_XX_T1w_brain.nii.gz: Skull-stripped T1-weighted (MPRAGE) anatomical image.avatarRT_sub_XX_T2w_brain.nii.gz: Skull-stripped T2-weighted anatomical image.
ses_01/data/
Empty for Session 1. No real-time BCI data stream was generated during the joystick session.
ses_01/fmap/
Spin-echo field map images used for EPI susceptibility distortion correction in Session 1, processed with FSL's topup.
ses_01/func/
Raw and preprocessed BOLD functional data from Session 1.
avatarRT_sub_XX_task-joystick_run-YY_bold.nii/.json: Raw BOLD data and BIDS-format JSON sidecar for each run of the joystick task.avatarRT_sub_XX_task-joystick_run-YY_bold_preproc_v2_native.nii.gz: Preprocessed BOLD data for each run, registered to Session 1 native functional space. Preprocessing was performed using FSL FEAT (version 6.00) and included: skull stripping (BET), susceptibility distortion correction (topup), high-pass temporal filtering (100 s cutoff), head motion correction (MCFLIRT), slice timing correction, spatial smoothing (5-mm FWHM Gaussian kernel), registration to the participant's T1w anatomical scan (boundary-based registration, BBR).
ses_02/ – ses_05/ — Neurofeedback Sessions
Sessions 2 through 5 share a common folder structure. Each session folder contains behavioral data, processed real-time neural data streams, field maps, and raw and preprocessed functional images from the neurofeedback BCI task.
ses_XX/behav/
Behavioral and regressor files for each run of the neurofeedback session.
-
events_master_revised_v2.csv: TR-by-TR behavioral timeseries for the full session. One row per TR. Columns:Column Description Units/Scale TriggerCounttriggers counted coming in from scanner integer (1 index) data_TRs_0idxTR index (zero-based) TR number (TriggerCount-1) trial_numberTrial number; -1 during inter-trial intervals integer (-1 = ITI) isTrialWhether this TR falls within a trial or ITI binary (0/1) x_coordAvatar x-position in the arena location in unity grid z_coordAvatar z-position in the arena location in unity grid dist2goalEuclidean distance from avatar to goal flag x_coord_normNormalized avatar x-position (x_coord - mean(x_coords_during_trial)) standardized units z_coord_normNormalized avatar z-position (z_coord - mean(z_coords_during_trial)) standardized units demeaned_dist2goalDistance to goal, mean-subtracted runRun number within session integer -
avatarRT_sub_XX_ses_YY_run_ZZ_timeseries_regressors.csv: TR-by-TR regressor timeseries for a single run. One row per TR. Columns:Column Description Units/Scale TRTR index (one-based) TR number TR_0idxTR index (zero-based) TR number trialTrial number; -1 during inter-trial intervals integer (-1 = ITI) z_coord_normNormalized avatar z-position (x_coord - mean(x_coords_during_trial)) x_coord_normNormalized avatar x-position (z_coord - mean(z_coords_during_trial)) z_coordAvatar z-position in the arena location in unity grid x_coordAvatar x-position in the arena location in unity grid isTrialWhether this TR falls within a trial binary (0/1) dist2goalEuclidean distance from avatar to goal flag float demeaned_dist2goalDistance to goal, mean-subtracted float -
avatarRT_sub_XX_ses_YY_run_ZZ_trial_regressors.csv: Trial-level summary for each trial in this run. One row per trial. Columns:Column Description Units/Scale trialTrial number (zero-based) integer startTRTR at which the trial began (one-based) TR number endTRTR at which the trial ended (one-based) TR number nTRsNumber of TRs in the trial integer brain_controlBrainControl score at the end of the trial successWhether the participant reached the goal flag binary (0/1) error_rateDistance over the shortest path possible that the avatar actually traveled float perturbationDecoder/condition used for this trial string (IM / WMP / OMP) -
avatarRT_sub_XX_ses_YY_run_ZZ_regressors.png: Visualization of the motion and task regressors for quality checking.
ses_XX/data/
Processed neural data streams recorded at each TR during real-time BCI operation. These files capture three successive stages of the real-time processing pipeline applied to each incoming whole-brain DICOM volume:
- The volume was smoothed (5-mm FWHM), motion-corrected to V_1 (MCFLIRT), aligned to V_ref via transformation matrix T, and masked with
mask.nii.gz. Voxel-wise activity was then z-scored using a running estimate of each voxel's mean and standard deviation across all prior TRs in the run. - The z-scored, masked data were passed through the frozen MRAE encoder to produce a 20-dimensional T-PHATE manifold embedding.
- The embedding was projected (dot product) onto the BCI component for that session (C_IM, C_WMP, or C_OMP) and linearly scaled to a movement angle.
avatarRT_sub_XX_run_ZZ_masked_data.npy: NumPy array of shape [TRs × voxels] containing the real-time fMRI data after ROI masking, spatial alignment to Session 1 space, and running z-score normalization (pipeline step 1).avatarRT_sub_XX_run_ZZ_projected_data.npy: NumPy array of shape [TRs × 20] containing the masked data projected onto the 20-dimensional T-PHATE manifold via the MRAE encoder (pipeline step 2).avatarRT_sub_XX_run_ZZ_decoded_angles.npy: NumPy array of shape [TRs] containing the BCI output value at each TR: the projection of the manifold-embedded data onto the session's BCI component, linearly scaled to the range [−1, 1] using the 1st and 99th percentile bounds estimated from the Session 1 held-out test set. Values of −1 and +1 correspond to avatar turning angles of −90° and +90°, respectively (pipeline step 3).
ses_XX/fmap/
Spin-echo field maps and FSL topup distortion correction outputs for this session.
Note: The fmap/ folder for Session 3 of some subjects contains additional intermediate files from an extended brain masking pipeline (prefixed fieldmap_rads_brain_tmp_premask_*). These are intermediate outputs from iterative masking steps and are not required for downstream analyses.
ses_XX/func/
Raw and preprocessed BOLD functional data from the neurofeedback session.
sub-avatarRT_sub_XX_ses_YY_task-avatarRT_run-ZZ_bold.nii/.json: Raw BOLD data and BIDS-format JSON sidecar for each neurofeedback run.sub-avatarRT_sub_XX_ses_YY_task-avatarRT_run-ZZ_events.tsv: BIDS-format events file for each run, with columns for onset, duration, and trial-level behavioral variables.avatarRT_sub_XX_task-RT_run-ZZ_bold_preproc_v2_native.nii.gz: Preprocessed BOLD data for each run in Session 1 native functional space, generated using the same FSL FEAT pipeline described for Session 1.
Sharing/Access information
The MNI-space navigation network ROI mask and task demo videos are available at https://github.com/ericabusch/avatarRT_task/demo and https://github.com/ericabusch/avatarRT_analysis.
The T-PHATE package used for neural manifold learning in this study is citable on Zenodo: https://doi.org/10.5281/zenodo.7637522 and can be installed using pip: https://pypi.org/project/TPHATE/. The MRAE package is available on Github: https://github.com/ericabusch/MRAE.
Code/Software
Analysis scripts for reproducing all figures and results from the manuscript are available at https://github.com/ericabusch/avatarRT_analysis. Key software dependencies include:
- FSL (version 6.0.5): https://fsl.fmrib.ox.ac.uk/fsl/fslwiki
- Python (≥ 3.8) with NumPy, PyTorch, and the T-PHATE/MRAE packages
- Unity3d (version 2019.4.12f1): https://unity3d.com
Questions about the data or analyses can be directed to Erica Busch (erica.busch@yale.edu; permanent email: ericalindseybusch@gmail.com) or Nicholas Turk-Browne (nicholas.turk-browne@yale.edu).
Human subjects data
All participants explicitly consented to publishing de-identified data in the public domain. To de-identify the fMRI data, raw DICOM files were converted to NIfTI format using dcm2niix, which scrubbed identifying header information during conversion. Facial features were removed from scans through skull stripping with FSL's BET (Brain Extraction Tool), eliminating the risk of face reconstruction from high-resolution anatomical images
