KeyBench: A resource for extracting Keywords from scientific papers
Data files
Jul 13, 2026 version files 91.27 MB
-
keywords7.zip
91.26 MB
-
README.md
3.90 KB
Abstract
This resource is intended to encourage the community to build tools to extract keywords. Many papers on ArXiv list keywords on the first page, typically near the abstract. The proposed resource scrapes these keywords, and uses them as the gold standard for a prediction task. An evaluation compares three baselines for predicting the gold standard: (1) VitaLity, (2) topics from Semantic Scholar and (3) a chatbot. We challenge the community to beat these baselines. To make it easier for the community to build such systems, the resource includes a number of fields for 117,877 papers from Semantic Scholar: (a) title, (b) abstract, (c) authors, (d) citing sentences, and (e) ids to make it easier to join with data in resources such as ArXiv, Semantic Scholar, ACL Anthology, PubMed, etc. We hope the community will show that citing sentences are useful because of the wisdom of the crowd (good keywords are likely to be used by many authors).
Availability
This data is posted in two places:
- https://doi.org/10.5061/dryad.bk3j9kdsn
- https://drive.google.com/file/d/17LEj3Ybwyt_oDq-vjcxmOo065lhC7KVo/view?usp=sharing
This dataset is associated with a paper that is under review (not published).
More Details
See GitHub
A draft paper is posted here
The JSON object in keywords7.zip is a python dict. The keys are corpusids (primary keys in Semantic Scholar). Since VitaLITy does not provide join keys into Semantic Scholar, those corpusids were inferred by an (imperfect) heuristic process based on titles.
Vitality provides many values in addition to keywords. Many of those other fields are in the json as well.
The paper mentions $G$, $V$ and $S$ labels:
| Symbol | Description | json field | Source of Data |
|---|---|---|---|
| $G$ | Gold labels | keywords_from_pdf |
Scrape PDFs from ArXiv |
| $V$ | VitaLITy labels | VitaLITy |
VitaLITy |
| $S$ | Semantic Scholar Topics | keywords_from_s2 |
Files on AWS (see below) |
| Citing sentences | citing_sentences |
Semantic Scholar Bulk Download | |
| IDs for joining with other resources | externalids |
Semantic Scholar Bulk Download | |
| URL to Semantic Scholar | url |
Semantic Scholar Bulk Download | |
| Title | title |
Semantic Scholar Bulk Download | |
| Abstract | abstract |
Semantic Scholar Bulk Download | |
| Authors | authors |
Semantic Scholar Bulk Download |
There are two types of labels under $S$: $S_m$ (mention) and $S_c$ (cited for). See discussion in paper
for more details about those labels. These labels are based on three publicly available files from AWS:
The task is to predict the $G$ labels from titles, abstracts and citing sentences in the json file.
Many values are missing (not in json), as discussed in the draft paper.
The github mentioned above provides:
- code for evaluation
- code for scraping pdfs
- code for using bots to extract keywords from pdfs