Schizophrenia is a severe psychiatric disorder associated with a wide range of cognitive and neurophysiological dysfunctions and long-term social difficulties. Early detection is expected to reduce the burden of disease by initiating early treatment. In this paper, we test the hypothesis that the integration of multiple simultaneous acquisitions of neuroimaging, behavioral, and clinical information will be better for the prediction of early psychosis than unimodal recordings. We propose a novel framework to investigate the neural underpinnings of the early psychosis symptoms (that can develop into Schizophrenia with age) using multimodal acquisitions of neural and behavioral recordings including functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and facial features. Our data acquisition paradigm is based on live face-to-face interaction in order to study the neural correlates of social cognition in first-episode psychosis (FEP). We propose a novel deep representation learning framework, Neural-PRISM, for learning joint multimodal compressed representations combining neural as well as behavioral recordings. These learned representations are subsequently used to describe, classify, and predict the severity of early psychosis in patients, as measured by the Positive and Negative Syndrome Scale (PANSS) and Global Assessment of Functioning (GAF) scores to evaluate the impact of symptomatology. We found that incorporating joint multimodal representations from fNIRS and EEG along with behavioral recordings enhances classification between typical controls and FEP individuals (significant improvements between 10 − 20%). Additionally, our results suggest that geometric and topological features such as curvatures and path signatures of the embedded trajectories of brain activity enable the detection of discriminatory neural characteristics in early psychosis.

The proposed method employs dyads that include one individual who serves as the live expressive face stimulus and the other partner categorized as either typically developed (TD) or first episode psychosis (FEP) patient. Dyads faced each from across a table at a distance of approximately 140 cm and table-mounted eye-tracking systems were positioned to measure continuous eye movements of the subject. Functional NIRS and EEG data were also synchronized and continuously acquired hemodynamic and electrocortical responses of the subject during the experiment. The dyads were separated by a “smart glass” in the center of the table that controlled face gaze times (the glass was transparent during gaze periods) and “rest times” (the glass was opaque during rest) (Hirsch, X. Zhang, Noah, and Bhattacharya, 2023).

Paradigm

The dyads were seated 140 cm across a table from each other. A "Smart Glass" (glass that is capable of alternating its appearance between opaque and transparent upon application of an appropriate voltage) panel was positioned in the middle of the table 70 cm away from each participant. In both conditions of direct and diverted face gaze, the subject was instructed to gaze at the eyes of their partner who watches emotionally balanced movie clips followed by direct or diverted gaze towards the subject's face. In the direct face gaze condition, dyads had a direct face-to-face view of each other. On the other hand, in the diverted face gaze condition the stimulus looks at the subject's shoulder.

The actor watches a 4-second movie (joyful or sad) and then looks at the partner's (subject's) eyes or his shoulders (diverted face gaze) for 5 seconds. This sequence of tasks was repeated twice for each pair. Then there is a 12-second rest period when the smart glass is made opaque. The same process (30 seconds) is repeated three times for each condition. The subjects were instructed to watch the actor's (stimulus) face all the time. The actor was instructed to watch short movies followed by direct or diverted gaze towards the subject.

FNIRS, EEG, and Facial Action Units data collection

Data were collected via simultaneous fNIRS, EEG, and Facial Action Units while individuals viewed another human partner’s face.
fNIRS Data were collected via a multichannel continuous-wave system (LABNIRS, Shimadzu Corporation, Kyoto, Japan) consisting of forty emitter-detector optode pairs. During the task, optodes were connected to a cap placed on the participant’s head based on size to fit comfortably. For consistency in cortical coverage, the middle anterior optode was placed 1 cm above the nasion; the middle posterior optode was placed in line with the inion; and the CZ optode was aligned with the anatomical CZ. After cap placement, hair was cleared from optode holders using a lighted fiber-optic probe (Daiso, Hiroshima, Japan) prior to optode placement. Optodes were arranged in a matrix, contacting the scalp, enabling the acquisition of 128 channels. After optode placement and prior to beginning the experiment, the signal-to-noise ratio was assessed by measuring the attenuation of light for each channel, with adjustments made as needed(Noah et al., 2015; Tachibana et al., 2011).

FNIRS signal acquisition, optode localization, and signal processing were similar to methods described previously (Dravida et al., 2020; Hirsch et al., 2017, 2022; Kelley et al., 2021; Noah et al., 2020).

Eye movements were recorded using a desk-mounted Tobii Pro (Stockholm, Sweden) X3-120 eye-tracking system placed 70cm in front and slightly below the participant’s face. Eye behavior was recorded at 120 Hz. The Eye tracker was calibrated for each participant using a transparent plane with three dots placed around the face of the partner. Participants were instructed to look at each dot in turn, and each gaze angle was recorded. Calibration was confirmed by having participants look at each eye and the nose of the partner and confirming alignment. Synchronized scene video capturing the participant’s view of the partner was recorded at 30 Hz with a resolution of 1280x720 pixels using a Logitech c920 camera (Lausanne, Switzerland) positioned directly behind and above the participant's head. This enabled tagging of participant-looking behavior within a manually placed “face box.”
EEG data were acquired via a 256-Hz, 32-electrode dual-bio amplifer g.USB multi-amp system (g.tec Medical Engineering, Austria). The electrode layout was adapted from the 10-10 system to accommodate optode placement on the fNIRS cap. Saline conducting gel was manually placed for each electrode after optode placement to ensure scalp contact. Scalp contact was manually reviewed per electrode using a digital oscilloscope, and adjustments were made as needed.

Recording of optode locations

After completion of the tasks, the locations of optodes and electrodes were recorded for each participant using the Structure Sensor scanner (Boulder, CO, USA) which creates a 3D model (.obj file) of the participant’s head and cap. Locations of the standard anatomical landmarks nasion, inion, cz, t3, and t4 as well as optode locations were manually placed on the 3D model using either MATLAB. Electrode locations were then determined by calculating the midpoint between surrounding optodes. Locations were then corrected for cap drift using custom MATLAB scripts which rotated optode and electrode locations around the Montreal Neurological Institute (MNI) X-axis from the left ear towards the midline (Eggebrecht et al., 2012; Okamoto & Dan, 2005). This was done to bring the cz optode in line with the anatomical cz according to the original placement to account for the stereotyped tilting of the cap towards the left ear that could occur during optode removal.

Environmental/experimental conditions: During the experiment, the overhead lights of the room were extinguished. An experimenter was present out of the view of the participant during each run. Two directed lights were used to fully illuminate the partner’s face and eliminate shadows. Two diffuse bar lights were used to softly illuminate the opaque SmartGlass during no-face-viewing periods to suppress the participant’s reflection. Participants only reported being able to see the rough outline of their heads in the SmartGlass, with no internal features visible. These were on throughout run durations.

Data included in the .tar file are raw, unprocessed data.

Data is presented in a generic text file format, making them broadly accessible for analysis in a wide array of software. The software utilized by the lab is described here. fNIRS data were collected via a multichannel continuous-wave LABNIRS system, producing OMM files and converted to text files which can be analyzed using MathWorks MATLAB with the NIRS-SPM(Ye et al., 2009) package. EEG data were collected with a 256-Hz, 32-electrode dual-bio amplifier g.USB Amp system (G. Tec Medical Engineering, Austria) and are analyzable through MathWorks MATLAB with the EEGLAB extension(Swartz Center for Computational Neuroscience, California, USA)(Delorme & Makeig, 2004). Facial Action Unit data were obtained using OpenFace software (Baltrušaitis et. al).

During the experimental setup, optode holders were cleared of hair using a lighted fiberoptic wand. This ensured that scalp contact was made. After fNIRS optode placement and prior to beginning the experiment, the signal-to-noise ratio was assessed by measuring the attenuation of light for each channel, with manual adjustments made as needed. After optode placement, saline conducting gel was placed in EEG electrodes manually to ensure scalp contact. Scalp contact was reviewed using an oscilloscope, and adjustments to scalp connectivity were made as needed. Eye tracking calibration was completed using Tobii Pro Lab’s calibration protocol, with a vertical plane placed in alignment with the end of the partner's nose which placed three calibration points around the edge of the partner's face which participants were instructed to view in time.

Quality-assurance procedures performed on the data: Optode and electrode connectivity were reviewed and adjusted prior to starting the experiment and were further monitored over the course of the experiment so that connectivity issues could be addressed between runs if needed. After data collection, EEG data were manually reviewed by eye and bad channels were removed and replaced with an interpolation calculated from the remaining channels. Channel 32 was removed from all subjects due to stereotyped connectivity issues from the cap design.

People involved with sample collection, processing, analysis, and/or submission: Dr. J. Adam Noah was responsible for the maintenance of the EEG, fNIRS, and eye-tracking hardware and software. He also assisted in the design and implementation of the experiment, as well as data collection and analyses. Dr. Xian Zhang assisted in data analysis and software upkeep, as well as produced the code responsible for running the experiment and synchronizing the modalities. Dr. Rahul Singh participated in data collection, conducted analyses on the data, and produced this document. Dr. Joy Hirsch oversaw the project.