Data from: Large language model-based evaluation of the impact of gender in medical research
Data files
Jul 27, 2026 version files 111.26 MB
-
broad_subjects.json
2.56 KB
-
Medicine_Authorship.zip
89.67 MB
-
README.md
11.37 KB
-
supplementary-data-1.xlsx
21.48 MB
-
supplementary-data-2.xlsx
84.21 KB
-
supplementary-data-descriptions.pdf
22.30 KB
Abstract
Objective: Gender disparities in academic medicine have been previously reported, but prior analyses have relied on either manual labor or fixed databases of name-gender pairs that fail to generalize across different populations and cultures. The objective of this work is to evaluate the utility of large language models (LLMs) as a potential tool to facilitate systematic bibliometric analysis of academic research trends.
Materials and methods: We introduce an LLM-based pipeline that aggregates gender labels from multiple LLM instances to predict the genders of manuscript authors based on their first names.
Results: Our proposed method outperforms alternative algorithms relying on lookup from finite databases of name-gender pairs, while also offering the scalability to tens of millions of authors that is unfeasible with other manual, human-based methods alone.
Discussion and conclusion: Our results suggest that LLMs can be a powerful tool to scalably track gender-based trends in academic medical research.
Dataset DOI: 10.5061/dryad.bvq83bkq5
Description of the data and file structure
To construct our dataset, we first queried the PubMed Central (PMC) database for manuscripts published in an NLM-catalogued journal. For each manuscript, we used the Entrez programming utilities API to fetch its author list, year of publication, and DOI. We restricted our analysis to manuscripts published between January 2015 and September 2025.
To quantitatively measure a journal's academic impact, we used the SCImago Journal Rank (SJR) published by the SCImago Research Group, which measures a journal's citation impact over a 3-year period. For each manuscript published in a journal, we labelled a manuscript entry with the journal's SJR from the year immediately prior to the publication year of the manuscript. The SJR metric can therefore be thought of as a proxy for the impact of an academic journal. We also recorded the Open Access (OA) status of each journal, which was also publicly provided by the SCImago Research Group.
We then queried the OpenCitations API (v1.2.0) to obtain a manuscript's total number of citations; the number of "self-citations" (defined as a citation from a separate manuscript that shares at least one author, which is tracked in the OpenCitations database); and the total number of references.
Finally, we used the Llama-3.1 8B Model from Meta AI (meta-llama/Llama-3.1-8B-Instruct) to predict the gender of a manuscript author given their first name.
Files and variables
File: Medicine_Authorship.zip
Description: A ZIP file containing the following folders and files:
All Authors/: A folder of Parquet files that map the LLM-predicted gender of all authors of a manuscript to the associated manuscript metadata, such as manuscript year of publication, open access status, associated journal SJR score, number of citations, and number of references, for each medical specialty.First Author/: A folder of Parquet files that map the LLM-predicted gender of a manuscript's first author to the associated manuscript metadata, such as manuscript year of publication, open access status, associated journal SJR score, number of citations, and number of references, for each medical specialty.Last Author/: A folder of Parquet files that map the LLM-predicted gender of a manuscript's last author to the associated manuscript metadata, such as manuscript year of publication, open access status, associated journal SJR score, number of citations, and number of references, for each medical specialty.Fractional Gender Analysis/: A folder of Parquet files that map the fraction of male and female authors within each manuscript for each medical specialty.Titles/: A folder of CSV files that map the title of each manuscript in our dataset to the genders of the first and last author of the manuscript for each medical specialty.Gender Failure Mode Analysis.csv: A table of human-analyzed failure modes of LLM-based gender predictions. Each row is an observed failure, and columns B-C are the human-annotated failure mode categories and associated descriptions of the failure mode.
File: broad_subjects.json
Description: A json file that maps medical specialties to their corresponding NIH-defined Broad Subject(s) and associated metadata.
File: supplementary-data-descriptions.pdf
Description: A PDF file that describes the data in the files supplementary-data-1.xlsx and supplementary-data-2.xlsx.
File: supplementary-data-1.xlsx
Description: Source data for all figures associated with our manuscript. See below for sheet-level descriptions of the data file. Note that genders can be one of male, female, or unknown, and male and female percentages may not sum to 100% since unknown-gendered authors are counted in the deminator. The workbook contains 28 worksheets described below:
- Figure 1: A figure that summarizes the data curation process used in the associated manuscript. There is no source data associated with this figure.
- Figure 2: Compares the accuracy of each annotation method (rows) on the SSA, Pinyin, and Global name benchmarks. Column A is the gender annotation method. Columns B-D are the accuracy of each method computed as (# correct gender) / (# correct gender + # incorrect gender). Columns E-G are the accuracy of each gender, with the number of unknown genders also included in the denominator. Columns H-J are the percentages of names assigned to either a male or female label (i.e., not "unknown"). Columns L-T break down the percentages of correct, incorrect, and unknown genders by method for each benchmark. Columns U-W show the macro-average percentages across the aforementioned three benchmarks.
- Figure 3: Analyzes how the percentage of male and female authors changes by specialty and over time. Column A corresponds to the set of authors under consideration. Column B is the medical specialty. Column C is the total number of manuscript authors that are in each category. Columns D-K correspond to the percentages of each gender within each group and computed statistics based on how the percentages change over time.
- Supplementary Figure 1: A per-specialty Pareto chart, where each column is a medical specialty, and each number in the column corresponds to the number of articles published in a given journal in that specialty from January 2015 to September 2025. The non-blank entries per column correspond to the number of journals in that specialty.
- Supplementary Figure 2: For each gender prediction method, we report the probability of assigning the wrong label, conditioned on a name's true gender, on each benchmark. Each row is a prediction method, and each column is the probability of a wrong gender label conditioned on the true correct gender for the true gender and dataset pair in the column.
- Supplementary Figure 3: For each gender prediction method, we report the probability of assigning the unknown label, conditioned on a name's true gender, on each benchmark. Each row is a prediction method, and each column is the probability of a wrong gender label conditioned on the true correct gender for the true gender and dataset pair in the column.
- Supplementary Figure 4 - Female: Number of female authors of manuscripts published in a given year (column) and medical specialty (row).
- Supplementary Figure 4 - Male: Number of male authors of manuscripts published in a given year (column) and medical specialty (row).
- Supplementary Figure 5 -
Specialty: (13 sheets)
For each specialty (corresponding to each sheet), we report the proportion of a manuscript's author list (row) that is female (column A), male (column B), and the total number of manuscript authors (column C). Note that the row counts are equal to the total number of manuscripts published within each specialty in our bibliometric study. - Supplementary Figure 5 - p**-values**: the per-specialty p-values from a z-test for intra-group gender homophily (column B) for each medical specialty (column A). All values were extremely small, indicating significant same-gender clustering in every specialty.
- Supplementary Figure 6: Statistics (columns) for the total number of citations, self-citations, and total references for manuscripts, split by the gender of the first vs last author, with a two-sided unpaired t-test per metric. (A citation = a later paper citing this one; a self-citation = a citation from a paper sharing at least 1 author; a reference = a work cited within this manuscript.)
- Supplementary Figure 7: Column A corresponds to the set of authors under consideration. Column B is the medical specialty. Column C (resp., D) is the percentage of manuscripts published in open-access (OA) journals for female (resp., male) authors. Columns E-J report computed statistics regarding OA journal publication frequency by gender. Columns K-L (resp., M-N) report statistics on the mean SJR journal ranking of journals publishing manuscripts from female (resp., male) authors. Column O reports the t-test significance for the difference in SJR.
- Supplementary Figure 8A: Cross-cultural F1 performance by region. Mean F1 score of each gender-prediction method within each world region (from the Global benchmark, macro-averaged over male/female, then averaged within region weighted by each country's name count). Each row is a gender prediction method, and each column is a region of the world. The value of a cell is the mean F1 score for a given method in a given geographic region.
- Supplementary Figure 8B: Within-region difference in cross-cultural F1 performance by gender. Each row is a gender prediction method, and each column is a region of the world. The value of a cell is the difference in mean F1 between male and female names in a given geographic region.
- Supplementary Figure 8C: Comparing the performance of gender prediction methods on Western vs non-Western names. Each row is a header and each column either Western names or non-Western names. Each cell is the mean F1 score for that method (row) on that subset of names (column).
- Supplementary Figure 8 Countries to Regions: A mapping from country ISO 3166-1 alpha-2 country codes (column A) to the nominal regions (column B) used in Supplementary Figure 8.
File: supplementary-data-2.xlsx
Description: For each medical specialty, we report the journals categorized under the corresponding Broad Subject Term from the National Library of Medicine (NLM), along with the total number of manuscripts published by each journal between January 2015 and September 2025, inclusive. We then further stratify the total manuscript count by the LLM-predicted genders of the first and last authors: the column "[First]/[Last] Count" refers to the number of manuscripts published in the journal with a predicted first-author gender of [First] and last-author gender of [Last], where M corresponds to male and F to female. The column "Other Count" refers to the number of manuscripts published in the journal where the gender of either the first and/or last author could not be determined. We focus on the 3 journals with the greatest number of published manuscripts in our study period for each medical specialty in Supplementary Table 6, and on the 3 journals with the highest SCImago Journal Rank (SJR, a measure of journal impact) in Supplementary Table 7.
Each sheet in the data file corresponds to data for a given medical specialty (title of the sheet). Each row is a journal counted in that specialty, and each column is the number of manuscripts published in that journal from January 2015 to September 2025 FG/LG authors, where FG is the gender of the first author of a manuscript and LG is the gender of the last author of the manuscript. Note that within each row, columns C-G add to column B.
Code/software
Microsoft Excel and any standard Parquet data loader, such as pandas in Python, are required software to view our data.
