A visual cortical prosthesis(VCP) has long been proposed as astrategy for restoringuseful vision to the blind, under the assumptionthat visual percepts of small spots of lightproduced withelectrical stimulation of visual cortex (phosphenes) will combine into coherent percepts of visual forms, like pixels on a video screen.We tested analternative strategy in which shapes were traced on the surface of visual cortex by stimulating electrodes in dynamic sequence. In both sighted and blind participants, dynamic stimulation enabled accurate recognition of letter shapespredicted by the brain’s spatial map ofthe visual world.Forms could be presented and recognized rapidlyby blind participants, up to 86 forms per minute.These findings demonstrate that a brainprosthetic can produce coherent perceptsof visualforms.

We investigated the use of new electrical stimulation paradigms to produce form perception in sighted and blind subjects. Traditional approaches to electrical stimulation of visual cortex have been restricted both due to the small number of electrodes implanted, and the limited way in which groups of electrodes were activated simultaneously in an attempt to produce form vision. In an attempt to overcome these restrictions and produce more effective form vision, we used two new techniques: 1) dynamic stimulation, in which a series of electrodes defining a form are rapidly activated in sequence rather than simultaneously, and 2) current steering, in which two electrodes located nearby are activated simultaneously at various current ratios to produce ‘virtual electrodes’ in between the physical electrodes.

Sighted subjects were epilepsy patients undergoing invasive monitoring in the epilepsy monitoring unit (EMU). In those subjects, we recorded receptive fields for each electrode, and screened each electrode to see which electrodes produced phosphenes when electrical stimulation was delivered. For each electrode that produced a phosphene, we determined the approximate threshold current required to produce a phosphene, and current values slightly above threshold were used when investigating dynamic pattern stimulation. Once electrodes producing phosphenes and threshold currents were established we picked various sets of electrodes to stimulate in dynamic stimulation patterns in an attempt to convey particular forms to the subject. We had the subjects give a verbal report of what they saw and make drawings on a touchscreen placed in front of them. In some subjects, we additionally had them perform alternative forced choice discrimination tasks in which we presented different dynamic stimulation sequences, corresponding to different perceived characters or letters, in random order, and the subject reported which character they perceived on each trial.

The blind subjects used in this study were one subject (BAA) who was implanted with a Neuropace stimulation device with 8 electrodes over their occipital lobe as part of an early feasibility assessment prior to the Orion Visual Cortical Prosthesis Clinical Trial sponsored by Second Sight Medical Inc. The other subject (03281) was one of six subjects enrolled in the actual Orion clinical trial, and this subject had an array of 60 electrodes implanted over their occipital lobe. In blind subjects, we tested each electrode to see which produced phosphenes when stimulated in isolation and at what current magnitudes. We then selected dynamic stimulation sequences to produce the percept of lines in visual space or of particular forms or characters. As with sighted subjects, subjects verbally described what they saw and we recorded drawings they made on a touchscreen in response to the various stimulation sequences. In addition, we conducted forced choice tasks in which the subjects discriminated between forms. In subject BAA, we also conducted current steering experiments in which nearby electrodes on the cortex were stimulated simultaneously at various current ratios to create ‘virtual electrodes’ in between the two physical electrodes. Finally, in subject BAA we also examined how quickly we could deliver dynamic electrical stimulation sequences to the subject to allow rapid discrimination between various forms.

A more detailed description of all methods is included in a separate file titled ‘FullMethods.pdf’.

Notes for usage of datasets associated with “Dynamic Stimulation of Visual Cortex Produces Form Vision in Sighted and Blind Humans”

(This information is also is included as 'UsageNotes.pdf' in the main folder of our submission)

First: Download the zip archive ‘DynamicStimulationDataAndCode.zip’. Expand the file, and keep the directory structure that results without moving files to new locations. Then the below notes apply to the data and code found in each directory.

CurrentSteering03281 folder:

Current steering experiments presented in this report were conducted with subject 03-281, a blind subject enrolled in the Orion clinical trial at the Baylor College of Medicine test site. Two types of basic current steering experiments were performed. In the first experiment, we used current steering between two physical electrodes to generate three virtual electrodes in between the two physical electrodes. In the second type of experiment, we used what we refer to as continuously variable dynamic current steering (CVDCS) in which the ratio of currents used across two electrodes was varied in small steps approximately every 8ms which was the fastest allowed by the Orion system. There were five physical electrodes arranged in a row on the electrode array, and CVDCS was used first between the first pair of electrodes, then the second pair of electrodes, and so on, so as to rapidly move the locus of activation across the entire row of electrodes including many intervening points between each pair of physical electrodes.

In this repository we include:

‘CurrentSteeringInfo.xslx’ This file contains information about the dynamic electrical stimulation sequences used for testing current steering between two electrodes, including the current ratios for each condition, and the location of the resulting phosphenes.

The Matlab file ‘03281_CSPairData.mat’ contains the data for the experiment testing current steering between two electrodes. In this file, the ‘pair.trials’ data structure contains the electrical stimulation parameters, monitor settings, and drawing data for each trial. The x and y fields contain the original subject drawing, and xDeg and yDeg are the drawing data converted into degrees of visual space. The ‘segment’ field is used to divide the drawings into the discrete drawings in different locations on the screen made during single trials, since the subject was drawing more than one phosphene per trial on the touchscreen.

The Matlab file ‘03-281_CVDCSData.mat’ contains data from the experiment testing continuously variable current steering across a row of five electrodes. The ‘cvdcs.trials’ data structure contains the electrical stimulation parameters, monitor settings, and drawing data for each trial. The fields x and y contain the original drawings made by the subject and the xDeg and yDeg fields contain the drawings converted into units of degrees of visual space.

The Matlab file ‘03281_PlotCSPair.m’ is used to plot the results from the two-electrode current steering experiment. The code imports data from ‘03281_CSPairData.mat’ and generates a plot showing the average location for each of the five phosphenes the subject perceived in this experiment with error ellipses. Two of the phosphene locations correspond to the physical electrodes used, and three resulting from the current steering conditions. The units are azimuth and elevation in degrees of visual space.

Matlab files ‘errorEllipse.m’ and ‘getSegmentCenter’ are helper functions required for ‘03281_PlotCSPair’. They are used to align drawings made on separate trials, and to obtain error ellipse estimates for each point.

The file ‘03281_PlotCVDCS.m’ is used to plot results from the experiment with continuously variable current steering. This code imports data from ‘03-281_CVDCSData.mat’ and generates a plot of the drawing made by the subject on one trial of the experiment. In this experiment, the subject was asked to trace the overall line they perceived rather than plot the location of individual phosphenes, so there is just one line segment. The units are azimuth and elevation in degrees of visual space.

‘Figure2.pdf’ This is Figure 2 from our Cell manuscript. It shows the results from the two electrode current steering experiment, and the location of the electrodes on the subjects occipital lobe.

‘Figure3.pdf’ This is Figure 3 from our Cell manuscript. It shows the results from the continuously variable dynamic current steering experiment, and the location of the electrodes on the subjects occipital lobe.

ReceptiveFieldsYBN folder:

For most experiments shown in this report, individual phosphene locations were used to plan dynamic stimulation sequences. However, in sighted subject YBN, receptive field (RF) locations were used instead. We used receptive field mapping and analysis methods that we have used in before in other reports (Yoshor et al 2007; Bosking et al 2017). During RF mapping the subject performed a letter detection task at the central fixation point while small checkerboard stimuli were flashed briefly in various locations on the screen.

In this repository we include:

‘SuppFigure1.pdf’ This is supplementary figure 1 from our Cell manuscript. It shows sample receptive field data information for two of the electrodes on the electrode array used in subject YBN.

Data subfolder:

‘YBNDatafile051_ch105.mat’ … ‘YBNDatafile051_ch128.mat’  The data subfolder contains a series of files containing the raw voltage recording obtained during the RF mapping in subject YBN for each of the 24 mini-electrodes located over the medial wall of the occipital lobe. Channel 105-128 corresponds to mini-electrodes 1-24.

‘YBNDatafile051_timeStamp.mat’ This file containing the timestamps for the presentation of visual stimuli and other events during the experiment ‘YBNDatafile051_timeStamp.mat’.

Code subfolder:

Matlab analysis function “generateRF.m”. This function is called with one of the channels 105-128 specified as an input argument to obtain the RF for that channel. For example: ‘generateRF(105)’. The remaining files are helper functions necessary for generateRF to execute. The code generates two figures. The first shows the average voltage traces across trials for each of the positions in visual space tested. The second shows the result of fitting a two-dimensional gaussian to the average root mean square signal for each position in visual space. The remaining files in this folder are the helper functions required to run ‘generateRF.m’.

Dynamic electrical stimulation used to evoke letter percepts in sighted subjects

We used dynamic sequences of electrical stimulation of multiple electrodes to attempt to produce perception of specific characters or letters in sighted epilepsy monitoring unit (EMU) subjects.

In this repository we include:

CharacterDrawingYAY folder:

‘SubjectYAYcurrentSteeringExample.mp4’ This is a slightly longer version of Supplementary Movie 1 included with the Cell manuscript. The subject draws the pattern they perceived following a dynamic current steering stimulation sequence, and provides additional description of the percept.

‘SupplementaryMovie1.mp4’  This is a slightly edited version of the above movie, included with Cell manuscript.

‘Figure4.pdf’ This is Figure 4 from our Cell manuscript. Panels E-H shows the location of the electrodes on the brain in this subject, the location of the phosphenes when each electrode was stimulated in isolation, the stimulation sequence used, and the subject drawing.

CharacterDiscriminationYBN folder:

‘BehavioralDataYBN’ This Excel file gives information from a four alternative forced choice (4AFC) discrimination experiment in which the subject was asked to discriminate between four different character patterns (V,C,N,S) based on four different dynamic electrical stimulation patterns. Before this experiment was run, we had asked to the subject to draw the pattern perceived resulting from each of the four stimulation patterns (this is in Figure 5 in the Cell manuscript). Note that during the initial drawing of characters associated with each sequence the subject reported one of the sequences as a “U” but later reported this same dynamic stimulation sequence as a “V” during the 4AFC task. The spreadsheet shows the electrode sequence used on each trial, the target or expected character for that trial, the character the subject actually reported perceiving, and whether the trial was correct or wrong.

‘Figure4.pdf’ This is Figure 4 from our Cell manuscript. Panels A-D show the location of the electrodes on this subject, the receptive fields for each electrode on the array, the stimulation sequence, and the drawings made by the subject for four different dynamic sequences.

Dynamic electrical stimulation used to evoke letter percepts in blind subjects

We used dynamic electrical stimulation to convey letters in two blind subjects. Subject BAA had a Neuropace device implanted, with eight electrodes located over her occipital cortex, as part of an early feasibility assessment for the Orion clinical trial sponsored by Second Sight Medical Inc. As previously mentioned, Subject 03-281 is a participant in the Orion clinical trial enrolled at the Baylor College of Medicine test site.

In this repository we include:

CharacterDiscriminationBAA folder:

Subject BAA made drawings of seven different characters perceived as a result of seven different dynamic electrical stimulation sequences, and later performed a five alternative forced choice task using five of these patterns.

‘BAAindividualPhospheneDrawings.pdf’ This file shows the location of each of the phosphenes that the subject perceived when electrodes were stimulated one at a time. These locations were used to plan the dynamic sequences used to convey various characters.

‘BAAcharacterDrawingsOriginal.mat’ This file contains the drawings the subject made for each of the 7 different electrical stimulation sequences with four trials per sequence. The subject identified each sequence with a particular character (backwards G, N, backwards R, upside down U, upside down V, W, Z). There are 28 data structures corresponding to the data from each trial, for each character. For example, ‘xy_G_T1_Original’ contains the x and y coordinates from trial 1 for the electrical stimulation sequence that produced the percept of the backwards G character.

‘BAAcharacterDrawingsResampled.mat’ This file contains the same drawing data, but resampled to have the same number of x, y coordinates (100) for each drawing. This transformation is used to prepare the data for the correlation and MDS analysis.

‘CorrDataBAA.mat’ This file contains a matrix with the correlation between each possible pair of drawings. This is the input to the MDS analysis.

‘corr2_allPairsBAA.m’ This Matlab file is the code used to generate the correlation matrix from the resampled drawings.

‘corr2_allPairsOneRefBAA.m’ This Matlab file is a helper function required by ‘corr2_allPairsBAA.m’.

‘MDS_analysisBAA.m’ This Matlab file is used to perform the actual MDS analysis using the correlation matrix. It outputs a figure showing the separation of the different character groups in multi-dimensional space.

‘BehavioralDataBAA5AFC.xlsx’ This Excel file contains information from the 5AFC discrimination experiment in which the subject discriminated between five different characters corresponding to five different dynamic stimulation sequences. The characters used were backwards R, Z, upside down V, W, and upside down U. The spreadsheet shows the stimulation sequence, expected or target character, and actual reported character for each trial.

‘Figure5.pdf’ This is Figure 5 from our Cell manuscript. It shows the subject plotting on the touchscreen, location of the electrodes on her brain, location of individual phosphenes for each electrode stimulated in isolation, the electrical stimulation sequence timing, drawings made by the subject in response to seven different dynamic stimulation sequences, corresponding to seven different characters, and the results of MDS analysis on those drawings.

‘SupplementaryMovie2.mp4’ This is supplementary movie 2 included with our Cell manuscript. It shows the subject plotting the location of individual phosphenes, and then drawing the characters she perceived with certain dynamic stimulation sequences.

CharacterDiscrimination03281 folder:

Subject 03281 performed a 4AFC task in which he both drew the characters he perceived and gave a verbal report of what character he perceived on each trial. There were four different dynamic stimulation sequences which he perceived as the characters M, N, U, and W.

‘M1_resampled.mat’, ‘N1_resampled.mat’, ‘U1_resampled.mat’, ‘W1_resampled.mat’ etc.  We include the Matlab data files for each of ten trials for each of the four dynamic sequences corresponding to the four different charactes the subject perceived (M, N, U, W). Each file contains the original x and y coordinates for the drawing from one trial, and the resampled coordinates used as input for the MDS analysis.

‘CorrData03281.mat’ This file contains a matrix with the correlation between each possible pair of drawings. This is the input to the MDS analysis.

‘corr2_allPairs03281.m’ This Matlab file is the code used to generate the correlation matrix from the resampled drawings.

‘corr2_allPairsOneRef03281.m’ This Matlab file is a helper function required by ‘corr2_allPairs03281.m’.

‘MDS_analysis03281.m’ This Matlab file is used to perform the actual MDS analysis using the correlation matrix. It outputs a figure showing the separation of the different character groups in multi-dimensional space.

‘ElectrodeInfo4AFCsubject03281.xslx’ This Excel files gives the electrodes and currents used in the 4AFC character discrimination experiment.

‘CharacterDiscrimBehData03281.xlsx’ This Excel file gives the behavioral data including expected or target character, and actual verbal report from the subject for each trial of the 4AFC discrimination experiment.

‘03281_4AFC_AlignLetters.m’ This file contains the Matlab code used to align character drawings obtained on each trial.

‘removeCellOffset.m’ This is a helper function required by ‘03281_4AFC_AlignLetters’.

‘03281_4AFC_LetterDiscrimData.mat’ This Matlab file contains the data from each trial of the 4AFC discrimination experiment. The ‘data.trials’ structure contains the electrical stimulation parameters including the sequence of electrodes used, monitor settings, target character (letter and stimCharacterID), actual subject response (response), and drawing data (xDeg, yDeg). The stimCharacterID for each character used are M = 1, N =2, U = 3, W = 4.

‘Figure6.pdf’ This is Figure 6 from our Cell manuscript. It shows the subject drawing one of the characters perceived on the touchscreen, location of electrodes on his brain, location of the individual phosphenes, the timing of the stimulation sequences used, the subject drawings for each of the four dynamic sequences corresponding to four different characters, the results of the MDS analysis, and the percent correct for rapid direction discrimination (2AFC) and character discrimination (3AFC).

‘SupplementaryMovie3.mp4’ This is supplementary movie 3 from our Cell manuscript. It shows the subject performing the 4AFC character discrimination task.

RapidFormDiscrimination03281 folder:

In blind subject 03-281 we conducted two types of experiments to assess the ability of subjects to use dynamic sequences to rapidly discriminate between different forms. In one experiment, we used continuously variable dynamic current steering sequences to present a rapidly drawn line in visual space and varied whether the progression moved in the upward or downward direction. In the other experiment, dynamic sequences tracing four different characters were presented rapidly to the subject and the subject gave a verbal report indicating which character was perceived on each trial.

In this repository we include:

‘03281_2AFC_DirectionStreamData.mat’ This Matlab file contains the data from the rapid direction discrimination task. The ‘data.trials’ structure contains the electrical stimulation information, monitor settings, expected or target direction (up or down), and the actual verbal report from the subject. stimDirectionID 1 corresponds to the up direction, stimDirectionID 2 correspond to the down direction.

‘03281_3AFC_LetterStreamData.mat’ This Matlab file contains the data from the rapid letter discrimination task. The ‘data.trials’ structure contains the electrical stimulation information, monitor settings, expected or target character (U, C, or backwards C), and the actual verbal report from the subject. stimCharacterID 1 corresponds to the the letter C, stimCharacterID 2 correspond to the backwards C, and stimCharacterID 3 corresponds to the letter U.

‘RapidCharacterDiscrimBehData.xlsx’ This Excel file contains a summary of the behavioral data from the rapid character discrimination task including the target or expected character for each trial and the actual subject report.

‘RapidDirectionDiscrimBehData.xlsx’ This Excel file contains a summary of the behavioral data from the rapid direction discrimination task including the target or expected character for each trial and the actual subject report.

‘RapidCharacterDiscrim03281.mov’ This movie shows the subject performance during the rapid character discrimination task.

‘RapidCharacterDiscrim03281withOverlays.mp4’ This is the same movie but with overlays we added to show the target character based on the dynamic electrical stimulation sequence used on each trial.

‘RapidDirectionDiscrimBlock1.mov’ This movie shows the subject performance during the first block of ten trials of the rapid direction discrimination task. Additional movies are included for the second and third block of ten trials.

‘RapidDirectionDiscrimBlock1withOVerlays.mp4’ These movies are the same as above but with overlays we added to show the expected or target direction for each trial.

‘SupplementaryMovie4.mp4’ This is supplementary movie 4 from our Cell manuscript. It shows the subject performing one block of the rapid direction discrimination 2AFC task, and the entire rapid character discrimination 3AFC task.