This readme file was generated on 2024-09-20 by Dr. Megan Kelley.

GENERAL INFORMATION

Title of Dataset:

Author Information
Name: Megan Kelley
ORCID: 0000-0001-8612-5215
Institution: Yale School of Medicine
Address: 300 George Street, Suite 902, New Haven, CT, 06511, USA
Email: megan.kelley@yale.edu

Principal Investigator Information
Name: Joy Hirsch
ORCID: 0000-0002-1418-6489
Institution: Yale School of Medicine
Address: 300 George Street, Suite 902, New Haven, CT, 06511, USA
Email:

Author/Alternate Contact Information
Name: J. Adam Noah
ORCID: 0000-0001-9773-2790
Institution: Yale School of Medicine
Address: 300 George Street, Suite 902, New Haven, CT, 06511, USA
Email: adam.noah@yale.edu

Date of data collection: Approximate collection dates are 2022-01-01 through 2023-02-01.

Geographic location of data collection: 300 George Str, New Haven, CT, United States.

Information about funding sources that supported the collection of the data:
Title: Basic neural processing mechanisms of live human face viewing
Source: NIH: 1F99NS129174-01A1
PI: Kelley, Megan

Title: Mechanisms of Interpersonal Social Communication: Dual-Brain fNIRS Investigation
Source:  NIMH - R01 MH107513
PI:  Joy Hirsch, Ph.D.

Title: Mechanisms of Dynamic Neural Coupling during Face-to-Face Expressions of Emotion
Source: NIMH - R01MH119430
PI: Joy Hirsch, Ph.D.

Title: Neural Mechanisms for Social Interactions and Eye Contact in ASD
Source: NIMH, 1R01MH111629-01
PI: Joy Hirsch, Ph.D.

SHARING/ACCESS INFORMATION

Licenses/restrictions placed on the data: None.

Links to publications that cite or use the data: None

Links to other publicly accessible locations of the data: None

Links/relationships to ancillary data sets: None

Was data derived from another source? No

Recommended citation for this dataset:
Kelley, Megan S.; Noah, J. Adam; Zhang, Xian; Hirsch, Joy (2024). Simultaneous fNIRS, EEG, and eye tracking data during real live human face viewing and robot face viewing, with different degrees of continuity. [Dataset]. Dryad.

DATA & FILE OVERVIEW
For this data set, we have included three files: 1: a data.tar file that includes all raw and exported data collected during the experiment; 2) this readme.md file; and 3) and text file, FileExistTable.csv, which is an overview of the dataset indicating which files are present or missing for each subject.
During data collection subjects completed two data recording visits on separate days. On one visit, the task was completed with a human partner and on the other visit, with a robot partner. Partner order was counterbalanced per subject.
Each visit consisted of two runs each of four conditions (described below). The order of conditions was counterbalanced across subjects, with each subject completing the same condition order during both visits.

File List:
The types of files included are briefly listed below. For full details on names and details specific to each file, see DATA-SPECIFIC INFORMATION section.

• FNIRS data files: csv files containing oxyhemoglobin, deoxyhemoglobin, and total concentration for each channel at each time point. Data were collected with a 6msec sample time per channel.
• FNIRS channel location files: csv file containing MNI coordinates for each fNIRS channel.
• EEG data files: csv files containing scalp voltage for each channel, at a sampling rate of 256hz.
• EEG channel location files: csv file containing the MNI coordinates for each EEG channel.
• Eye tracking data files: csv file containing the location, timestamp, and classification of eye behavior during the tasks. These data were collected at 120Hz.

File names follow a format indicating visit type, condition, and run. The format for each file name is included in DATA-SPECIFIC INFORMATION section.

Visit H: the task was completed with a human partner. Both fNIRS data and channel location files will start with H for human.
Visit R: the task was completed with a robot partner. Both fNIRS data and channel location file will start with R for robot.

For each visit there are four conditions consisting of 15 events. In all conditions, events consisted of 3200ms of face viewing time, subdivided differently depending on condition.
Condition 1 - NF: also called the “no flicker (NF)” condition, events consist of one stimulus epoch that is 3200 ms.
Condition 2 - LF: also called the “long epoch with flicker (LF)” condition, events consist of two stimulus epochs that are 1600 ms, with a 200 ms disruption between them.
Condition 3 - SF: also called the “short epoch with flicker (SF)” condition, events consist of four stimulus epochs that are 800ms, with a 200 ms disruption between each.
Condition 4 - XF: also called the “extra short epoch with flicker (XF)” condition, events consist of eight stimulus epochs that are 400ms, with a 200 ms disruption between each.
In all cases, the stimulus is a view of the partner’s face, enabled by the smart glass divider between the participant and the partner turning transparent.

Filenames contain a three letter phrase indicating condition and visit name. For example, NFR indicates the NF condition (condition 1) for the robot visit. XFH indicates the XF condition (condition 4) for the Human visit.

Each condition is conducted twice per partner. The filename will contain run1 or run2.

Additional related data collected that was not included in the current data package: None.

Are there multiple versions of the dataset? No
If yes, name of file(s) that was updated:
Why was the file updated?
When was the file updated?

METHODOLOGICAL INFORMATION
Description of methods used for collection/generation of data:

Paradigm
Participants completed two runs of a face viewing paradigm with two partners. The face viewing paradigm consisted of alternating periods of face viewing and no face viewing. Participants were seated 140 centimeters (cm) in front of and facing a partner with a “Smart Glass” divider (Hohofilm, Shanghai, City) at the midpoint between them. The divider can be made transparent or opaque under computer control, with a switching latency of ~10ms. Participants faced straight ahead, keeping a consistent head posture while the opacity of the smart glass was toggled between transparent and opaque, allowing or preventing them from seeing their partner. When they could see their partner, participants were instructed to look at the face, allowing their eyes to move as felt comfortable and natural, and when they could not see it, they were instructed to keep their eyes open and focused on a dot centered on the divider.
Runs consisted of five task blocks alternating with 15-s rest blocks. Task blocks were divided into three face-viewing events which alterna¬ted with 3-s no-face-viewing periods . Face viewing events for all conditions consisted of 3200ms of total face viewing time each; however, the continuity of view varied based on condition. During condition 1, face viewing events consisted of a single 3200ms epoch of continuous view of the partners face with no disruptions. During condition 2, the 3200ms of face viewing time was subdivided into two 1600ms face viewing epochs with a single 200ms disruption to face view. Condition 3 events consisted of four 800ms face viewing epochs with three 200ms disruptions to face view. Condition 4 events consisted of eight 400ms face viewing epochs with seven 200ms disruptions to face view. Disruptions occurred by “flickering” the smart glass from transparent to opaque for 200ms, and then quickly back to transparent.

The face viewing paradigm was a modification on similar methods described previously(Dravida et al., 2020; Hirsch et al., 2017, 2022; Kelley et al., 2021; Noah et al., 2020; Parker et al., 2023).

FNIRS, EEG, and Eye tracking data collection
Data were collected via simultaneous fNIRS, EEG, and eye tracking while individuals viewed either another real human partner’s face or a robot partners face.
fNIRS Data were collected via a multichannel continuous-wave system (LABNIRS, Shimadzu Corporation, Kyoto, Japan) consisting of forty emitter-detector optode pairs. During the task, optodes were connected to a cap placed on the participant’s head based on size to fit comfortably. For consistency in cortical coverage, the middle anterior optode was placed 1 cm above the nasion; the middle posterior optode was placed in line with the inion; and the CZ optode was aligned with the anatomical CZ. After cap placement, hair was cleared from optode holders using a lighted fiber-optic probe (Daiso, Hiroshima, Japan) prior to optode placement. Optodes were arranged in a matrix, contacting the scalp, enabling acquisition of 128 channels. After optode placement and prior to beginning the experiment, signal-to-noise ratio was assessed by measuring attenuation of light for each channel, with adjustments made as needed(Noah et al., 2015; Tachibana et al., 2011).\
FNIRS signal acquisition, optode localization, and signal processing were similar to methods described previously(Dravida et al., 2020; Hirsch et al., 2017, 2022; Kelley et al., 2021; Noah et al., 2020).\
Eye-movements were recorded using a desk-mounted Tobii Pro (Stockholm, Sweden) X3-120 eye-tracking system placed 70cm in front and slightly below the participant’s face. Eye behavior was recorded at 120 Hz. The Eye tracker was calibrated for each participant using a transparent plane with three dots placed around the face of the partner. Participants were instructed to look at each dot in turn, and each gaze angle was recorded. Calibration was confirmed by having participants look at each eye and the nose of the partner and confirming alignment. Synchronized scene video capturing the participant’s view of the partner was recorded at 30 Hz with a resolution of 1280x720 pixels using a Logitech c920 camera (Lausanne, Switzerland) positioned directly behind and above the participants’ head. This enabled tagging of participant looking behavior within a manually placed “face box.”
EEG data were acquired via a 256-Hz, 32-electrode dual-bioamplifer g.USB multi-amp system (g.tec Medical Engineering, Austria). The electrode layout was adapted from the 10-10 system to accommodate optode placement on the fNIRS cap. Saline conducting gel was manually placed for each electrode after optode placement to ensure scalp contact. Scalp contact was manually reviewed per electrode using a digital oscilloscope, and adjustments were made as needed.

Recording of optode locations
After completion of the tasks, locations of optodes and electrodes were recorded for each participant using the Structure Sensor scanner (Boulder, CO, USA) which creates a 3D model (.obj file) of the participant’s head and cap. Locations of the standard anatomical landmarks nasion, inion, cz, t3, and t4 as well as optode locations were manually placed on the 3D model using either MATLAB. Electrode locations were then determined by calculating the midpoint between surrounding optodes. Locations were then corrected for cap drift using custom MATLAB scripts which rotated optode and electrode locations around the Montreal Neurological Institute (MNI) X-axis from left ear towards the midline (Eggebrecht et al., 2012; Okamoto & Dan, 2005). This was done to bring the cz optode in line with the anatomical cz according to original placement to account for stereotyped tilting of the cap towards the left ear that could occur during optode removal.

Environmental/experimental conditions: During the experiment, the overhead lights of the room were extinguished. An experimenter was present out of the view of the participant during each run. Two directed lights were used to fully luminate the partner’s face and eliminate shadows. Two diffuse bar-lights were used to softly illuminate the opaque SmartGlass during no-face-viewing periods to suppress the participant’s reflection. Participants only reported being able to see the rough outline of their head in the SmartGlass, with no internal features visible. These were on throughout run durations. Partners had comparable luminance (Human partner: 42.4 lumens; Robot partner: 42.6 lumens), with slightly lower luminance than the opaque smart glass (44.0 lumens). Luminance was measured with a luxmeter positioned at a typical eye height, placed 70cm from and facing the smart glass divider and 140cm from the partner.

Methods for processing the data: Data included in the .tar file are raw, unprocessed data.

Instrument- or software-specific information needed to interpret the data:
Data is presented in a generic text file format, making them broadly accessibly for analysis in a wide array of software. The software utilized by the lab are described here. fNIRS data were collected via a multichannel continuous-wave LABNIRS system, producing OMM files and converted to text files which can be analyzed using MathWorks MATLAB with the NIRS-SPM(Ye et al., 2009) package. EEG data were collected with a 256-Hz, 32-electrode dual-bioamplifer g.USB Amp system (G. Tec Medical Engineering, Austria) and are analyzable through MathWorks MATLAB with the EEGLAB extension(Swartz Center for Computational Neuroscience, California, USA)(Delorme & Makeig, 2004). Eye-tracking data were collected Tobii Pro (Stockholm, Sweden) X3-120 eye-tracking system and are analyzable with Tobii Pro Labs.

Standards and calibration information, if appropriate:
During experimental set up, optode holders were cleared of hair using a lighted fiberoptic wand. This ensured that scalp contact was made. After fNIRS optode placement and prior to beginning the experiment, signal-to-noise ratio was assessed by measuring attenuation of light for each channel, with manual adjustments made as needed. After optode placement, saline conducting gel was placed in EEG electrodes manually to ensure scalp contact. Scalp contact was reviewed using an oscilloscope, and adjustments to scalp connectivity were made as needed. Eye tracking calibration was completed using Tobii Pro Lab’s calibration protocol, with a vertical plane placed in alignment with the end of the partners nose which placed three calibration points around the edge of the partners face which participants were instructed to view in time.

Describe any quality-assurance procedures performed on the data: Optode and electrode connectivity were reviewed and adjusted prior to starting the experiment and were further monitored over the course of the experiment so that connectivity issues could be addressed between runs if needed. Eye tracking calibration was assessed by confirming that recorded looking behavior corresponded with accurate location. After data collection, EEG data were manually reviewed by eye and bad channels were removed and replaced with an interpolation calculated from the remaining channels. Channel 32 was removed from all subjects do to stereotyped connectivity issues from the cap design. Eye tracking was calibrated at the start of each visit using a transparent plane with three dots placed in alignment with the tip of the nose of the partner. Participants were instructed to focus on each dot in turn in order to calibrate looking behavior, and then calibration was confirmed by ensuring that the eyetracker conformed with locations the participant was instructed to look at.

People involved with sample collection, processing, analysis, and/or submission: Dr. Megan Kelley designed the experiment and participated in data collection, conducted analyses on the data, and produced this document. Lab post-graduate associates including Sophie Gardephe and Julianna Brenner were primarily responsible for fNIRS, EEG, and eye tracking set up and data collection, with assistance from other lab members as needed. Dr. J. Adam Noah was responsible for maintenance of the EEG, fNIRS, and eye tracking hardware and software. He also assisted in the design and implementation of the experiment, as well as data collection and analyses. Dr. Xian Zhang assisted in data analysis and software upkeep, as well as produced the code responsible for running the experiment and synchronizing the modalities. Dr. Joy Hirsch oversaw the project.

DATA-SPECIFIC INFORMATION
FileExistTable.csv is an organizational table indicating which files are present or missing.

• Each column corresponds to a subject.
• Each row corresponds to a file.
• Values in the table are binary indicating the presence or absence of each file for each subject.

For all file names, the following are used as place holders:

• VISIT indicates the type of partner the task is with, robot or human. The value of visit can be either R or H.
• ID indicates subject number, ranging from 01 to 21.
• CONDITION can be 1, 2, 3, or 4, corresponding with NF, LF, SF, and XF conditions respectively.
• RUNID is either the first or second run, and can be either 1 or 2.

fNIRS data
The format of the name of fNIRS files:
VISIT_ID_cCONDITION_rRUNID_fnirs.csv

A description of the contents of fNIRS files:

• Each row is a sample.
• Column 1 is time in seconds.
• Column 2 is trigger. There are two types of triggers in the files. The main ones relevant to the data set are those with a trigger value >0 and < 3000, which indicates the onset of the stimulus.
• Column 3 is the oxyhemoglobin concentration of ch1.
• Column 4 is the de-oxyhemoglobin concentration of ch1.
• Column 5 is the total-oxyhemoglobin concentration of ch1.
• Column 6 is the oxyhemoglobin concentration of ch2.
• Column 7 is the de-oxyhemoglobin concentration of ch2.
• […]
• The final column is the total-oxyhemoglobin concentration of ch134.

Columns indicated by […] continue in the pattern of three columns per channel corresponding to oxyhemoglobin, de-oxyhemoglobin, and total-oxyhemoglobin concentration in that order. There are a total of 134 channels.

The format of the name of fNIRS channel location files:
VISIT_ID_xyz.csv

A description of the contents of fNIRS channel location files:

• Columns 1, 2, and 3 correspond to the MNI X – Y – Z coordinates, respectively.
• Rows correspond to 134 channels, 134 rows for 134 channels

EEG data
The format of the name of EEG data files:
VISIT_ID_EEGData.csv

A description of the contents of EEG data files:

• Columns correspond to EEG channels.
• Rows correspond to samples, at a sample rate of 256hz.

The format of the name of EEG channel location files:
VISIT_ID_EEGxyz.csv

A description of the contents of EEG channel location files:

• Column 1, 2, and 3 correspond with MNI X – Y – Z coordinates, respectively.
• Rows correspond with channels. Channel names are, in row order: fp1, fp2, af3, af4, f7, f3, fz, f4, f8, fc5, fc1, fc2, fc6, t7, c3, cz, c4, t8, cp5, cp1, cp2, cp6, p7, p3, pz, p4, p8, po3, po4, o1, oz, and o2.

The format of the name of EEG event file:
VISIT_ID_EEGevent.csv

A description of the contents of EEG event files:

• Rows correspond to each event.
• Column 1 is the sample number
• Column 2 is the condition, using the following legend.
  1 = NF : the duration of one stimulus is 3200 ms
  2 = LF : the duration of one stimulus is 1600 ms
  3 = SF : the duration of one stimulus is 0800 ms
  4 = XF : the duration of one stimulus is 0400 ms

Eye Tracking data
The format of the name of the eye tracking data file:
VISIT_ID_cCONDITION_rRUN_eyetracking.csv

A description of the contents of eye-tracking files:

• Rows correspond to samples
• Columns 1 and 2 are time information, with 1 being time since the recording was started and 2 being the timestamp on the computer
• Columns 3-22 are descriptive information about the recording, software, date, and calibration accuracy. They will be identical for all samples in a file.\
  o Column 3 contains technical information about the acquisition sensor
  o Columns 4-12 are self-explanatory
  o Columns 13 and 14 contain software details including the name of the filter used to classify fixation behavior and the software version number. The filter used was the default Tobii Pro filter.
  o Columns 15 and 16 contains the spatial resolution of the recording
  o Columns 17-22 contain average calibration and precision metrics.
   Accuracy is the average difference between the real stimuli position and the measured gaze position when looking at that stimuli position, given in mm. This is found in column 17 in millimeters and column 20 in degrees
   Precision is the ability of the eye tracker to reliably reproduce the same gaze point measurement when looking at the same stimuli position, measured in Root Mean Square (RMS), found in column 19 in mm and 22 in degrees. Precision SD is the spread of the measurements taken when looking at the same stimuli position. This is found in column 18 in mm and 21 in degrees.
• Column 23 is the timestamp on the eye tracker.
• Columns 24 and 25 contain event information, which will include triggers indicating when the smart glass was transparent and thus the face was visible and when the smart glass turned opaque, occluding the face. Column 24 contains the classification of the event, in terms of whether it was an event entered using a keyboard (e.g. a button press). Column 25 contains the actual value of the event. Events were marked using a custom python UDP broadcast tool that communicated directly between the computer running the paradigm and the computer running the Tobii Pro Lab Software.
• Columns 26-31 contain the XY coordinates of the gaze points in the plane of calibration. 26 and 27 contain coordinates of the average of the two eyes, 28 and 29 of the left eye alone, and 30 and 31 of the right eye alone.
• Columns 32-37 are the XYZ coordinates of the gaze vector between the left and right eye and the point in the environment that is in focus
• Columns 38 and 39 are the pupil diameter of the left and right eyes.
• Columns 40 and 41 are the validity of the left and right eyes. Valid is 1 and no eye present is a value of 0.
• Columns 42-47 are the XYZ coordinates of the left and right eyes in the Display Area Coordinate System (DACS). DACS is a 3D coordinate system with its origin in the top left corner of the stimuli area on the plane of calibration, with Y pointing downward (a coordinate that is lower on the stimuli area will have a higher Y-value), X pointing horizontally across the calibration plane, and Z pointing towards the participant (A gaze point closer to the participant than the calibration plane will have a positive value). They are given in millimeters. See https://connect.tobii.com/s/article/Display-Area-Coordinate-System-DACS?language=en_US for description.
• Columns 48-53 are the xyz coordinates of the gaze points in the media coordinate system (MCS). These columns should not be used for analysis as the definition of media was not standardized between participants.
• Column 54 is the classified eye behavior for each data point. Possible values include Saccade, Fixation, Microsaccade, Tremor, Drift, Smooth pursuit, Vergence, and Vestibular-ocular reflex. Descriptions of each can be found at the following link: https://www.tobii.com/resource-center/learn-articles/types-of-eye-movements
• Column 55 is the duration of the discrete behavioral event containing the data point. This value will be the same for all samples classified as belonging to the same event.
• Column 56 numeric indicator corresponding with the event type in column 55.
• Columns 57 and 58 are the XY coordinates of the fixated location on the calibration plane.
• Column 59 not used in this experiment
  Missing Data
  Eye-tracking calibration could not be obtained on all subjects due to wearing of glasses or other factors. In the case of non-calibration, data is marked as missing.
  Other missing data is due to equipment malfunction during experiment.

References
Delorme, A., & Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 9–21.
Dravida, S., Noah, J. A., Zhang, X., & Hirsch, J. (2020). Joint Attention During Live Person-to-Person Contact Activates rTPJ, Including a Sub-Component Associated With Spontaneous Eye-to-Eye Contact. Frontiers in Human Neuroscience, 14. https://doi.org/10.3389/fnhum.2020.00201
Eggebrecht, A. T., White, B. R., Ferradal, S. L., Chen, C., Zhan, Y., Snyder, A. Z., Dehghani, H., & Culver, J. P. (2012). A quantitative spatial comparison of high-density diffuse optical tomography and fmri cortical mapping. Neuroimage, 61(4), 1120–1128. https://doi.org/10.1016/j.neuroimage.2012.01.124
Hirsch, J., Zhang, X., Noah, J. A., Dravida, S., Naples, A., Tiede, M., Wolf, J. M., & McPartland, J. C. (2022). Neural correlates of eye contact and social function in autism spectrum disorder. PLOS ONE, 17(11), e0265798. https://doi.org/10.1371/journal.pone.0265798
Hirsch, J., Zhang, X., Noah, J. A., & Ono, Y. (2017). Frontal, temporal, and parietal systems synchronize within and across brains during live eye-to-eye contact. NeuroImage, 157, 314–330. https://doi.org/10.1016/j.neuroimage.2017.06.018
Kelley, M., Noah, J. A., Zhang, X., Scassellati, B., & Hirsch, J. (2021). Comparison of human social brain activity during eye-contact with another human and a humanoid robot. Frontiers in Robotics and AI.
Noah, J. A., Ono, Y., Nomoto, Y., Shimada, S., Tachibana, A., Zhang, X., Bronner, S., & Hirsch, J. (2015). fMRI Validation of fNIRS Measurements During a Naturalistic Task. Journal of Visualized Experiments : JoVE, 100. https://doi.org/10.3791/52116
Noah, J. A., Zhang, X., Dravida, S., Ono, Y., Naples, A., McPartland, J. C., & Hirsch, J. (2020). Real-time eye-to-eye contact is associated with cross-brain neural coupling in angular gyrus. Frontiers in Human Neuroscience, 14, 19. https://doi.org/10.3389/fnhum.2020.00019
Okamoto, M., & Dan, I. (2005). Automated cortical projection of head-surface locations for transcranial functional brain mapping. Neuroimage, 26(1), 18–28. https://doi.org/10.1016/j.neuroimage.2005.01.018
Parker, T. C., Zhang, X., Noah, J. A., Tiede, M., Scassellati, B., Kelley, M., McPartland, J., & Hirsch, J. (2023). Neural and visual processing of social gaze cueing in typical and ASD adults. medRxiv, 2023.01. 30.23284243.
Tachibana, A., Noah, J. A., Bronner, S., Ono, Y., & Onozuka, M. (2011). Parietal and temporal activity during a multimodal dance video game: An fNIRS study. Neuroscience Letters, 503(2), 125–130. https://doi.org/10.1016/j.neulet.2011.08.023
Ye, J. C., Tak, S., Jang, K. E., Jung, J., & Jang, J. (2009). NIRS-SPM: Statistical parametric mapping for near-infrared spectroscopy. NeuroImage, 44(2), 428–447. https://doi.org/10.1016/j.neuroimage.2008.08.036