Real-world human-robot interaction data with robotic pets in user homes in the United States and South Korea

Bennett, Casey1 ; Sabanovic, Selma2 ; Stanojevic, Cedomir3; Piatt, Jennifer2; Henkel, Zachary4; Baugus, Kenna4; Bethel, Cindy4 ; Kim, Seongcheol5 ; Lee, Jinjae5

Published Oct 17, 2023; Updated Jan 02, 2024 on Dryad. https://doi.org/10.5061/dryad.tb2rbp078

Data files

Oct 17, 2023 version files 89.67 MB

preprocessed_data(KR_New_collar)_Ver3.zip
50.21 MB
preprocessed_data(US_New_collar)_Ver3.zip
39.45 MB
README.md
5.66 KB

Jan 02, 2024 version files 89.67 MB

preprocessed_data(KR_New_collar)_Ver3.zip
50.21 MB
preprocessed_data(US_New_collar)_Ver3.zip
39.45 MB
README.md
9.44 KB

Abstract

Socially-assistive robots (SARs) hold significant potential to transform the management of chronic healthcare conditions (e.g. diabetes, Alzheimer’s, dementia) outside the clinic walls. However doing so entails embedding such autonomous robots into people’s daily lives and home living environments, which are deeply shaped by the cultural and geographic locations within which they are situated. That begs the question of whether we can design autonomous interactive behaviors between SARs and humans based on universal machine learning (ML) and deep learning (DL) models of robotic sensor data that would work across such diverse environments. To investigate this, we conducted a long-term user study with 26 participants across two diverse locations (the United States and South Korea) with SARs deployed in each user’s home for several weeks. We collected robotic sensor data every second of every day, combined with sophisticated ecological momentary assessment (EMA) sampling techniques, to generate a large-scale dataset of over 270 million data points representing 173 hours of randomly-sampled naturalistic interaction data between the human and robot pet. Interaction behaviors included activities like playing, petting, talking, cooking, etc.

README: Real-World Human-Robot Interaction Data with Robotic Pets in User Homes in the United States and South Korea

https://doi.org/10.5061/dryad.tb2rbp078

The study included 26 participants, 13 from South Korea and 13 from the United States. The participants were drawn from the general population aged 20-35 and living alone, approximately 70% of the sample was female. The robot included sensors that could detect light, sound, movement, indoor air quality, and other environmental health data in the vicinity of the robot (please refer to associated published papers for details). While sensor data was collected via the collars, self-reported interaction behavior modalities were collected simultaneously using the Expiwell EMA mobile app (https://www.expiwell.com/).

Description of the data and file structure

Sensor data from the robot ("feature" data) was collected roughly 9 times per second, every minute of every day, across the three- week deployment period. Meanwhile, the interaction modality data ("target" data) was collected via the EMA app randomly during each day about ongoing interactions during the preceding 15 minute window. The dataset thus includes the target interaction modality for prediction, as well as a series of columns of various sensor features. A detailed description of each individual column in the datasets is provided in the table below. Note that participants sometimes reported multiple modalities occurring during the same interaction period (on average roughly 2 modality types per interaction), which requires the data to be pre-processed appropriately. In our case, we chose to turn the dataset into a series of binary classification problems (e.g. petting vs not-petting), so that was a non-issue. Essentially this is time-series data, where all the rows from the 15 minute interaction can be used to predict the interaction modality of that interaction.

There are two separate files in the dataset, one for participants from South Korea and one for participants for the United States. The data was collected using the same protocol in both locations. However, due to some software changes in between the two phases, the sampling frequency for the sensor data may be slightly higher for the US data than Korean data. As such 5 seconds worth of data equates to roughly 80 rows (US) but roughly 50 rows (Korea). We addressed that by sub-sampling the US data during pre-processing in the Python code, which is mentioned in the "Code/Software" section below.

Column	Description
RowID	Unique ID for each row (XX_XX_XX), where the 1st XX indicates nationality, the 2nd XX indicates participant ID, and the 3rd XX indicates the interaction ID for each 15 minute interaction
Real_time	Timestamp indicating when sensors were read, down to the second level
accX	Motion amount from accelerometer in X (lateral) direction (meters/sec²)
accY	Motion amount from accelerometer in Y (lateral) direction (meters/sec²)
accZ	Motion amount from accelerometer in Z (up/down) direction (meters/sec²)
chord	Absolute amount of rotational motion since previous reading (calculated from X/Y/Z)
arc	Absolute amount of rotational motion since previous reading (calculated from X/Y/Z)
orientation	The orientation of the robot at a given time, using accelerometer manufacturer-specified thresholds. These numerical codes are equivalent to the orientation category labels below
orientation_cat	Specific orientation categories detected (labels), using accelerometer manufacturer-specified thresholds
ir	Infrared reading (ambient light value), scaled from completely dark (0) to blinding light (1024)
full	Alternate ambient light value (equivalent to ir), same scale as ir column
iaq	Indoor air quality reading (scaled), measured in ppm
staticIaq	Indoor air quality estimate (unscaled), scaled IAQ value is probably better, measured in ppm
pressure	Air pressure reading, measured in millibars
gasResistance	Electrical flow resistance in the presence of specific target gases. They are widely used in a variety of applications, such as environmental monitoring, industrial safety, and air quality control. Measured as pascal per m/s
temperature	Indoor air temperature (celsius)
humidity	Air humidity reading (relative percent)
co2Equivalent	Indoor CO2 levels (ppm)
breathVocEquivalent	Volatile organic compounds (ppm), such as those released by cooking, hairspray, etc.
audioLevel	Ambient sound levels (dBV, typically 0-100 scale in our device setup)
sound_cat	Discretized audio levels into categories (quiet, moderate, loud)
Modality_cat	Interaction behavior modality (numerical code), used as the Target variable prediction
Modality	Interaction behavior modality (label), same as Modality_cat but text label rather than numerical code

Sharing/Access information

If you make use of this data, please cite:

Bennett C.C., Sabanovic S., Stanojevic C., Henkel Z., Kim S., Lee J., Baugus K., Piatt J.A., Yu J., Oh j., Collins S., and Bethel C. (2023). Enabling robotic pets to autonomously adapt their own behaviors to enhance therapeutic effects: A data-driven approach. IEEE International Symposium on Robot and Human interactive Communication (RO-MAN), pp.1-8.

Code/Software

Machine Learning & Deep Learning was performed using custom code written in Python. That code also includes the necessary data pre-processing steps as well. The code is included with the datasets.