Code for: Sleepless in America: A social sensing study of pandemic-era sleeplessness using nighttime social media data

Gong, Xi1 ; Liu, Lin 2 ; Lu, Yujian2; Zhang, Guiming1; Huang, Xiao3; Lin, Yan1

Published Jun 12, 2026; Updated Jul 21, 2026 on Dryad. https://doi.org/10.5061/dryad.wh70rxx41

Data files

Jun 12, 2026 version files 2.75 MB

paper_code.ipynb

2.75 MB
README.md

6.27 KB

Jul 21, 2026 version files 2.74 MB

paper_code.ipynb

2.73 MB
README.md

6.27 KB

Abstract

Sleeplessness is a widespread public health concern that was further intensified during the COVID-19 pandemic, yet large-scale, real-time monitoring of sleep disturbance remains limited. This study introduces a scalable social sensing framework that leverages temporally filtered nighttime social media activity (10:00 p.m. - 6:00 a.m. local time) to infer population-level patterns of sleep disturbance. To demonstrate the framework, we conducted a comprehensive analysis of geotagged tweets posted by users in the United States from March 1, 2020 to June 30, 2021, using pre-pandemic tweets as a baseline for comparison. Spatial, temporal, sentiment, and topical patterns were analyzed to characterize nighttime sleep disturbances at the U.S. state level. Three distinct temporal patterns of nighttime disturbance were identified during the pandemic, including single peak, multiple peaks, and smooth patterns, indicating heterogeneous impacts across states. Correlations between nighttime tweet ratios and daily new COVID-19 case counts varied across states but were similar among states with comparable demographic and economic characteristics. In addition, more negative emotions expressed in nighttime tweets were associated with increased sleep disturbance. Nighttime tweets were more likely to focus on local news and events. Greater topical diversity among the population was associated with reduced negative emotional impacts during the pandemic. Overall, the findings demonstrate that nighttime social media data provide an effective and scalable social sensing approach for assessing sleeplessness across large geographic regions and extended time periods. Beyond sleep-related applications, this framework offers a transferable method for monitoring other forms of nighttime human activity and broader social dynamics, including crime, disaster response, and social unrest.

Dataset DOI: 10.5061/dryad.wh70rxx41

Description of the data and file structure

This repository contains the analysis code used to reproduce the main data processing steps, statistical analyses, and figures for the manuscript. The code analyzes geotagged social media posts collected before and during the COVID-19 pandemic in the United States to examine nighttime tweeting activity as a proxy for sleep disturbance.

The uploaded file paper_code.ipynb is a Jupyter Notebook containing the analysis workflow.

Required software

The code was written in Python and run as a Jupyter Notebook.

Recommended environment:

Python 3.9 or later
Jupyter Notebook or JupyterLab

Required Python packages:

pandas
numpy
matplotlib
scipy
scikit-learn

These packages can be installed using:

pip install pandas numpy matplotlib scipy scikit-learn jupyter

Expected file structure

The notebook expects a folder named processedData in the same directory as the notebook.

Expected input files include:

processedData/
    us_prepandemic_with_localTime.csv
    us_pandemic_with_localTime.csv
    US_State_summary_covid19_confirmed.csv
    pandemic_day_with_emotions.csv
    pandemic_night_with_emotions.csv
    prePandemic_day_with_emotions.csv
    prePandemic_night_with_emotions.csv

The notebook also creates output folders during execution, including:

figures/
monthly_disturbance_night/

Input data description

The processed tweet-derived files used by the notebook contain variables needed for the analysis, including date, local time, state abbreviation, state name, language, user follower/friend count fields used for filtering, and sentiment scores where applicable.

The COVID-19 case file is expected to contain state-level cumulative confirmed COVID-19 case counts in wide format, with dates as columns.

Because the Dryad submission includes code only, these input files are not included in the repository. De-identified and aggregated derivative datasets may be requested from the corresponding author.

Main analysis steps

The notebook performs the following analyses:

Loads processed pre-pandemic and pandemic tweet-derived files.
Filters common individual users based on follower/friend ratio and follower count.
Splits tweets into daytime and nighttime groups using local time.
Computes daily state-level daytime and nighttime tweet counts.
Calculates nighttime tweet ratios.
Selects states with sufficient valid data coverage.
Processes state-level COVID-19 case data.
Generates figures showing COVID-19 case trends and daily tweet quantities.
Computes monthly disturbance-night measures using pre-pandemic monthly baselines.
Performs k-means clustering of state-level disturbance-night patterns.
Analyzes the relationship between nighttime tweet ratios and COVID-19 case trends.
Performs sentiment analysis using precomputed sentiment scores.
Generates additional figures for monthly nighttime tweet counts and hourly tweet distributions.

Key definitions used in the code

Daytime tweets are defined as tweets posted after 06:00 and before 22:00 local time.

Nighttime tweets are defined as tweets posted at or before 06:00, or at or after 22:00 local time.

A valid state-day is defined as a state-day with more than 30 geotagged tweets.

A valid state-month is defined as a state-month with more than 10 valid days.

Disturbance-night measures are calculated by comparing pandemic nighttime tweet ratios against pre-pandemic monthly baseline nighttime tweet ratios.

Files and variables

Important data availability note

The Dryad submission contains analysis code only. The raw tweet-level data are not publicly deposited because they may contain information related to individual social media users and could pose privacy or re-identification risks.

The notebook was originally run using processed tweet-derived data files and COVID-19 case data. De-identified and aggregated derivative datasets necessary to run the code may be made available from the corresponding author upon reasonable request, subject to applicable ethical, legal, and platform-policy restrictions.

No raw tweet text, usernames, user IDs, exact user-level coordinates, or other direct identifiers are included in this Dryad submission.

Code/software

How to run the code

Install the required Python packages.
Place the required input files in a folder named processedData.
Open paper_code.ipynb using Jupyter Notebook or JupyterLab.
Run the notebook cells sequentially from top to bottom.
Generated figures and intermediate files will be saved in the figures and monthly_disturbance_night folders.

Outputs

The notebook generates intermediate summary files and figure files used in the manuscript. Output files include state-level monthly disturbance-night files and figures related to COVID-19 trends, nighttime tweet activity, disturbance-night patterns, sentiment analysis, and hourly tweet distributions.

Access information

Reproducibility note

The analysis code is provided to document and reproduce the computational workflow. Full reproduction requires access to the processed de-identified and aggregated derivative datasets, which are not publicly deposited due to privacy and platform-policy considerations. These data may be requested from the corresponding author (xigong@psu.edu).

Human subjects data

Yes. The original study used publicly available social media posts, which may contain information related to individual human subjects. However, the Dryad submission contains analysis code only and does not include raw tweet-level data, tweet text, usernames, user IDs, exact coordinates, timestamps linked to individual users, or other potentially identifying information. De-identified and aggregated derivative datasets needed to run the code may be made available from the corresponding author upon reasonable request, subject to applicable ethical, legal, and platform-policy restrictions.