Real-world human-robot interaction data with robotic pets in user homes in the United States and South Korea
Data files
Oct 17, 2023 version files 89.67 MB
-
preprocessed_data(KR_New_collar)_Ver3.zip
50.21 MB
-
preprocessed_data(US_New_collar)_Ver3.zip
39.45 MB
-
README.md
5.66 KB
Jan 02, 2024 version files 89.67 MB
-
preprocessed_data(KR_New_collar)_Ver3.zip
50.21 MB
-
preprocessed_data(US_New_collar)_Ver3.zip
39.45 MB
-
README.md
9.44 KB
Abstract
Socially-assistive robots (SARs) hold significant potential to transform the management of chronic healthcare conditions (e.g. diabetes, Alzheimer’s, dementia) outside the clinic walls. However doing so entails embedding such autonomous robots into people’s daily lives and home living environments, which are deeply shaped by the cultural and geographic locations within which they are situated. That begs the question of whether we can design autonomous interactive behaviors between SARs and humans based on universal machine learning (ML) and deep learning (DL) models of robotic sensor data that would work across such diverse environments. To investigate this, we conducted a long-term user study with 26 participants across two diverse locations (the United States and South Korea) with SARs deployed in each user’s home for several weeks. We collected robotic sensor data every second of every day, combined with sophisticated ecological momentary assessment (EMA) sampling techniques, to generate a large-scale dataset of over 270 million data points representing 173 hours of randomly-sampled naturalistic interaction data between the human and robot pet. Interaction behaviors included activities like playing, petting, talking, cooking, etc.
README: Real-World Human-Robot Interaction Data with Robotic Pets in User Homes in the United States and South Korea
https://doi.org/10.5061/dryad.tb2rbp078
The study included 26 participants, 13 from South Korea and 13 from the United States. The participants were drawn from the general population aged 20-35 and living alone, approximately 70% of the sample was female. The robot included sensors that could detect light, sound, movement, indoor air quality, and other environmental health data in the vicinity of the robot (please refer to associated published papers for details). While sensor data was collected via the collars, self-reported interaction behavior modalities were collected simultaneously using the Expiwell EMA mobile app (https://www.expiwell.com/).
Description of the data and file structure
Sensor data from the robot ("feature" data) was collected roughly 9 times per second, every minute of every day, across the three- week deployment period. Meanwhile, the interaction modality data ("target" data) was collected via the EMA app randomly during each day about ongoing interactions during the preceding 15 minute window. The dataset thus includes the target interaction modality for prediction, as well as a series of columns of various sensor features. A detailed description of each individual column in the datasets is provided in the table below. Note that participants sometimes reported multiple modalities occurring during the same interaction period (on average roughly 2 modality types per interaction), which requires the data to be pre-processed appropriately. In our case, we chose to turn the dataset into a series of binary classification problems (e.g. petting vs not-petting), so that was a non-issue. Essentially this is time-series data, where all the rows from the 15 minute interaction can be used to predict the interaction modality of that interaction.
There are two separate files in the dataset, one for participants from South Korea and one for participants for the United States. The data was collected using the same protocol in both locations. However, due to some software changes in between the two phases, the sampling frequency for the sensor data may be slightly higher for the US data than Korean data. As such 5 seconds worth of data equates to roughly 80 rows (US) but roughly 50 rows (Korea). We addressed that by sub-sampling the US data during pre-processing in the Python code, which is mentioned in the "Code/Software" section below.
| Column | Description |
|---|---|
| RowID | Unique ID for each row (XX_XX_XX), where the 1st XX indicates nationality, the 2nd XX indicates participant ID, and the 3rd XX indicates the interaction ID for each 15 minute interaction |
| Real_time | Timestamp indicating when sensors were read, down to the second level |
| accX | Motion amount from accelerometer in X (lateral) direction (meters/sec2) |
| accY | Motion amount from accelerometer in Y (lateral) direction (meters/sec2) |
| accZ | Motion amount from accelerometer in Z (up/down) direction (meters/sec2) |
| chord | Absolute amount of rotational motion since previous reading (calculated from X/Y/Z) |
| arc | Absolute amount of rotational motion since previous reading (calculated from X/Y/Z) |
| orientation | The orientation of the robot at a given time, using accelerometer manufacturer-specified thresholds. These numerical codes are equivalent to the orientation category labels below |
| orientation_cat | Specific orientation categories detected (labels), using accelerometer manufacturer-specified thresholds |
| ir | Infrared reading (ambient light value), scaled from completely dark (0) to blinding light (1024) |
| full | Alternate ambient light value (equivalent to ir), same scale as ir column |
| iaq | Indoor air quality reading (scaled), measured in ppm |
| staticIaq | Indoor air quality estimate (unscaled), scaled IAQ value is probably better, measured in ppm |
| pressure | Air pressure reading, measured in millibars |
| gasResistance | Electrical flow resistance in the presence of specific target gases. They are widely used in a variety of applications, such as environmental monitoring, industrial safety, and air quality control. Measured as pascal per m/s |
| temperature | Indoor air temperature (celsius) |
| humidity | Air humidity reading (relative percent) |
| co2Equivalent | Indoor CO2 levels (ppm) |
| breathVocEquivalent | Volatile organic compounds (ppm), such as those released by cooking, hairspray, etc. |
| audioLevel | Ambient sound levels (dBV, typically 0-100 scale in our device setup) |
| sound_cat | Discretized audio levels into categories (quiet, moderate, loud) |
| Modality_cat | Interaction behavior modality (numerical code), used as the Target variable prediction |
| Modality | Interaction behavior modality (label), same as Modality_cat but text label rather than numerical code |
Sharing/Access information
If you make use of this data, please cite:
Bennett C.C., Sabanovic S., Stanojevic C., Henkel Z., Kim S., Lee J., Baugus K., Piatt J.A., Yu J., Oh j., Collins S., and Bethel C. (2023). Enabling robotic pets to autonomously adapt their own behaviors to enhance therapeutic effects: A data-driven approach. IEEE International Symposium on Robot and Human interactive Communication (RO-MAN), pp.1-8.
Code/Software
Machine Learning & Deep Learning was performed using custom code written in Python. That code also includes the necessary data pre-processing steps as well. The code is included with the datasets.
Methods
Data was collected from robot sensors during deployment of the robots in user homes over a period of 3 weeks, using a sampling technique called ecological momentary assessment (EMA) in order to generate realistic real-word interaction data.