Durably reducing conspiracy beliefs through dialogues with AI
Data files
Aug 29, 2024 version files 71.40 MB
Abstract
Conspiracy theory beliefs are notoriously persistent. Influential theories propose they fulfill important psychological needs, thus resisting counterevidence. Yet previous failures in correcting conspiracy beliefs may be due to counterevidence being insufficiently compelling and tailored. To evaluate this possibility, we leverage developments in generative artificial intelligence and engaged 2,190 conspiracy believers in personalized evidence-based dialogues with GPT-4 Turbo. The intervention reduced conspiracy belief by ~20%. The effect remained 2 months later, generalized across a wide range of conspiracy theories, and occurred even among participants with deeply entrenched beliefs. Although the dialogues focused on a single conspiracy, they nonetheless diminished belief in unrelated conspiracies and shifted conspiracy-related behavioral intentions. These findings suggest that many conspiracy theory believers can revise their views if presented with sufficiently compelling evidence.
README: Durably reducing conspiracy beliefs through dialogues with AI
https://doi.org/10.5061/dryad.v6wwpzh4h
This repository contains data, code, and supplementary information.
Datasets
These data are required to run the RMarkdown file containing the analytic code associated with this manuscript. All data reported in the manuscript are included other than the raw conversational data, which potentially contains personally identifiable information and is therefore edited to remove identifiable features.
- AllDataForPublication.PPI.csv: Dataset containing all participant responses and results of NLP analyses (with personally identifying responses removed)
- AllDataFiltered.Cluster.PPI.csv: Dataset containing all participant responses and results of NLP analyses, only provided to exactly replicate cluster analysis in RMarkdown
- embeddings_allData.rds: Text embeddings of conspiracy theories (do not change, merely output of text-embeddings-large)
- *reduced_embeddings.rds: *Text embeddings of conspiracy theories, reduced with PCA to retain 85% of variance (note: this is necessary for the RMarkdown file to run, but is otherwise likely not of interest)
- AI_Strategy_Ratings_2024-04-30.csv: AI strategies used in each conversation. Labels should be self-explanatory.
Codebook
| Variable Name | Label/Description |
|---|---|
| X | Row number |
| userResponse_combined | Raw text responses to both questions soliciting conspiracy beliefs (concatenated) |
| conRestatement | GPT-4 summary of userResponse_combined |
| Pre_Belief_Specific | Pre-treatment belief strength in focal conspiracy theory (0-100). |
| Post_Belief_Specific | Post-treatment belief strength (0-100) |
| GPTResponse | Opening AI message of the conversation |
| userSecondResponse | Opening user message |
| GPTResponse2 | Second AI message |
| userThirdResponse | Second user message |
| GPTResponse3 | Third AI message |
| Q9 | Final user message |
| dbscan_cluster | Conspiracy type |
| DV_BeliefChange_Specific | Change score (pre-post) |
| FailedWritingScreener | Indicates if participants failed the initial writing screener. |
| FailedAttentionCheck | Indicates if participants failed the attention check question. |
| CompletedOnPhone | Indicates if the survey was entered on a phone. |
| DiscontinuedPriorToTreatment | Indicates if participants discontinued the survey before receiving any treatment. |
| ProvidedInvalidConspiracy | Indicates if the provided conspiracy theory was invalid according to GPT-4 |
| ProvidedValidConspiracy | Indicates if the provided conspiracy theory was valid according to GPT-4 |
| LowConfidence | Indicates if participants reported low confidence in their conspiracy theory. |
| AIProvidedInaccurateSummary | Indicates if participants indicated the AI-generated summary of the conspiracy theory was not accurate. |
| Duration (in seconds) | Length to completion |
| BCTI_pre_matrix_1 | Belief in Conspiracy Theories Inventory, Item 1, pre-treatment |
| BCTI_pre_matrix_2 | Belief in Conspiracy Theories Inventory, Item 2, pre-treatment |
| BCTI_pre_matrix_3 | Belief in Conspiracy Theories Inventory, Item 3, pre-treatment |
| BCTI_pre_matrix_4 | Belief in Conspiracy Theories Inventory, Item 4, pre-treatment |
| BCTI_pre_matrix_5 | Belief in Conspiracy Theories Inventory, Item 5, pre-treatment |
| BCTI_pre_matrix_6 | Belief in Conspiracy Theories Inventory, Item 6, pre-treatment |
| BCTI_pre_matrix_7 | Belief in Conspiracy Theories Inventory, Item 7, pre-treatment |
| BCTI_pre_matrix_8 | Belief in Conspiracy Theories Inventory, Item 8, pre-treatment |
| BCTI_pre_matrix_9 | Belief in Conspiracy Theories Inventory, Item 9, pre-treatment |
| BCTI_pre_matrix_10 | Belief in Conspiracy Theories Inventory, Item 10, pre-treatment |
| BCTI_pre_matrix_11 | Belief in Conspiracy Theories Inventory, Item 11, pre-treatment |
| BCTI_pre_matrix_12 | Belief in Conspiracy Theories Inventory, Item 12, pre-treatment |
| BCTI_pre_matrix_13 | Belief in Conspiracy Theories Inventory, Item 13, pre-treatment |
| BCTI_pre_matrix_14 | Belief in Conspiracy Theories Inventory, Item 14, pre-treatment |
| BCTI_pre_matrix_15 | Belief in Conspiracy Theories Inventory, Item 15, pre-treatment |
| WritingScreener | Participant’s answer to writing screener question |
| Age | Year of birth |
| nicks | Attention check question |
| DemRep_C | Democratic vs. Republican preference |
| PartyAffil | Political party affiliation |
| religion | Belief in god (0-10) |
| genai_fam_1 | Familiarity with generative AI |
| genai_use_1 | Past usage of generative AI |
| genai_trust | Trust in generative AI |
| Sureness_1 | Pre-treatment belief in focal conspiracy |
| AI_check | Did AI provide an accurate summary? |
| BCTI_matrix_post_1 | Belief in Conspiracy Theories Inventory, Item 1, post-treatment |
| BCTI_matrix_post_2 | Belief in Conspiracy Theories Inventory, Item 2, post-treatment |
| BCTI_matrix_post_3 | Belief in Conspiracy Theories Inventory, Item 3, post-treatment |
| BCTI_matrix_post_4 | Belief in Conspiracy Theories Inventory, Item 4, post-treatment |
| BCTI_matrix_post_5 | Belief in Conspiracy Theories Inventory, Item 5, post-treatment |
| BCTI_matrix_post_6 | Belief in Conspiracy Theories Inventory, Item 6, post-treatment |
| BCTI_matrix_post_7 | Belief in Conspiracy Theories Inventory, Item 7, post-treatment |
| BCTI_matrix_post_8 | Belief in Conspiracy Theories Inventory, Item 8, post-treatment |
| BCTI_matrix_post_9 | Belief in Conspiracy Theories Inventory, Item 9, post-treatment |
| BCTI_matrix_post_10 | Belief in Conspiracy Theories Inventory, Item 10, post-treatment |
| BCTI_matrix_post_11 | Belief in Conspiracy Theories Inventory, Item 11, post-treatment |
| BCTI_matrix_post_12 | Belief in Conspiracy Theories Inventory, Item 12, post-treatment |
| BCTI_matrix_post_13 | Belief in Conspiracy Theories Inventory, Item 13, post-treatment |
| BCTI_matrix_post_14 | Belief in Conspiracy Theories Inventory, Item 14, post-treatment |
| BCTI_matrix_post_15 | Belief in Conspiracy Theories Inventory, Item 15, post-treatment |
| participantId | Participant identifier |
| InitialWritingScreener | Raw text of writing screener responses |
| conspiracyTheory | GPT-summarized statement of conspiracy |
| userBeliefLevel | Pre-treatment belief in focal conspiracy |
| messageType | Experimental condition |
| controlType | Type of control |
| followUp_Duration (in seconds) | 10-day follow-up variables have the prefix "followUp_" and 2-month follow-up variables have the prefix "followUp2_", otherwise all naming conventions are maintained. |
| followUp_SpecificConspiracyDV_1 | |
| followUp_BCTI_matrix_1 | |
| followUp_BCTI_matrix_2 | |
| followUp_BCTI_matrix_3 | |
| followUp_BCTI_matrix_4 | |
| followUp_BCTI_matrix_5 | |
| followUp_BCTI_matrix_6 | |
| followUp_BCTI_matrix_7 | |
| followUp_BCTI_matrix_8 | |
| followUp_BCTI_matrix_9 | |
| followUp_BCTI_matrix_10 | |
| followUp_BCTI_matrix_11 | |
| followUp_BCTI_matrix_12 | |
| followUp_BCTI_matrix_13 | |
| followUp_BCTI_matrix_14 | |
| followUp_BCTI_matrix_15 | |
| followUp_conRestatement | |
| followUp2_Duration (in seconds) | |
| followUp2_SpecificConspiracyDV_1 | |
| followUp2_BCTI_matrix_1 | |
| followUp2_BCTI_matrix_2 | |
| followUp2_BCTI_matrix_3 | |
| followUp2_BCTI_matrix_4 | |
| followUp2_BCTI_matrix_5 | |
| followUp2_BCTI_matrix_6 | |
| followUp2_BCTI_matrix_7 | |
| followUp2_BCTI_matrix_8 | |
| followUp2_BCTI_matrix_9 | |
| followUp2_BCTI_matrix_10 | |
| followUp2_BCTI_matrix_11 | |
| followUp2_BCTI_matrix_12 | |
| followUp2_BCTI_matrix_13 | |
| followUp2_BCTI_matrix_14 | |
| followUp2_BCTI_matrix_15 | |
| followUp2_conRestatement | |
| device_type | |
| API_Conspiracy_Checker | Less stringent GPT classification of conspiracy (yes or no) |
| GPT_CoT_Conspiracy_Checker | More stringent GPT classification of conspiracy (yes or no) |
| GPT_CoT_PlausibilityRating | GPT classification of conspiracy’s plausibility |
| exc1 | Reason for exclusion in Study 1 Time 1 |
| arm | Experimental condition in Study 1 Time 1 |
| attrit1 | Attrited in Study 1 Time 1 |
| fow1 | Entered Study 1 Time 2 |
| exc2 | Reason for exclusion in Study 1 Time 2 |
| attrited | Attrited in Study 1 Time 2 |
| fow2 | Completed Study 1 Time 3 |
| exc3 | Reason for exclusion in Study 1 Time 3 |
| StudyNumber | Study number (1, 2, 3) |
| BCTI_pre | Belief in Conspiracy Theories Inventory, pre-treatment score for false conspiracies |
| BCTI_pre_true | Belief in Conspiracy Theories Inventory, pre-treatment score for true conspiracies |
| BCTI_post | Belief in Conspiracy Theories Inventory, post-treatment score for false conspiracies |
| BCTI_post_true | Belief in Conspiracy Theories Inventory, post-treatment score for true conspiracies |
| BCTI_Difference | Belief in Conspiracy Theories Inventory, difference score |
| Pre_Belief_General | Belief in Conspiracy Theories Inventory, pre-treatment score for false conspiracies |
| Post_Belief_General | Belief in Conspiracy Theories Inventory, post-treatment score for false conspiracies |
| BCTI_Discern_Pre | Belief in Conspiracy Theories Inventory, pre-treatment difference score (true vs. false) |
| BCTI_Discern_Post | Belief in Conspiracy Theories Inventory, post-treatment difference score (true vs. false) |
| BCTI_NWO_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_COVID_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_PearlHarbor_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_AIDS_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_MLK_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_Moon_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_Area51_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_911_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_JFK_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_Roswell_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_PrinDiana_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_OKC_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_CocaCola_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_EnergyTech_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_UKDrug_Match | Did participants focal conspiracy lexically match for BCTI item? |
| BCTI_pre_overlap | BCTI pre-treatment score removing items that overlapped with participants focal conspiracy |
| BCTI_post_overlap | BCTI post-treatment score removing items that overlapped with participants focal conspiracy |
| BCTI_pre_overlap_gt50 | BCTI pre-treatment score removing items that overlapped with participants focal conspiracy and removing items endorsed < 50/100 |
| BCTI_post_overlap_gt50 | BCTI post-treatment score removing items that overlapped with participants focal conspiracy and removing items endorsed < 50/100 |
| BCTI_fu | BCTI Study 1 Time 2 score |
| BCTI_10Day_overlap | BCTI Study 1 Time 2 score, removing items that overlapped with participants focal conspiracy |
| BCTI_10Day_overlap_gt50 | BCTI Study 1 Time 2 score, removing items that overlapped with participants focal conspiracy and removing items endorsed < 50/100 |
| BCTI_fu2mo | BCTI Study 1 Time 3 score |
| BCTI_2Mo_overlap | BCTI Study 1 Time 3 score, removing items that overlapped with participants focal conspiracy |
| BCTI_2Mo_overlap_gt50 | BCTI Study 1 Time 3 score, removing items that overlapped with participants focal conspiracy and removing items endorsed < 50/100 |
| IH | Intellectual Humility Scale |
| AOT | Actively Open-minded Thinking Scale |
| Importance | Perceived importance of the conspiracy theory to the participant (0-8) |
| Misinformation_Resis | Intentions to ignore or unfollow social media accounts espousing the focal conspiracy (1-3) |
| Social_Influence2 | Willingness to ignore or argue against people who believe the focal conspiracy (1-5) |
| Protests | Intentions to join pro-conspiracy protests (1-5) |
| GeneralTrust | Societal trust score |
| PersonalTrust | Personal trust score |
| DV_BeliefChange_General | Difference score for BCTI |
| Cat_50_Sureness | Focal conspiracy went below 50/100? |
| PercentChange | Percentage of focal conspiracy change, relative to initial level |
| DV_BeliefChange_Specific_Post2FU | Difference score for focal belief change, Study 1 Time 2 to Time 3 |
| droppedFollowUp | Returned for Study 1 Time 2? |
| ExperimentalCondition | Experimental condition |
| FourCategoryCondition | Experimental condition, including control type |
| Education_Cat | Educational attainment |
| PartyCat | Political party affiliation |
| AgeYears | Age |
| Race_Cat | Race |
| Extremism | Political extremism (1-3, folded over from political ideology score) |
| InstitutionalTrust | Institutional Trust Score |
| OpenendedResponseWordCount | Word count for conspiracy solicitation |
| ParticipantResponseWordCount_r1 | Participant’s response length in round 1 of conversation |
| ParticipantResponseWordCount_r2 | Participant’s response length in round 2 of conversation |
| ParticipantResponseWordCount_r3 | Participant’s response length in round 3 of conversation |
| GPTResponseWordCount_r1 | GPT response length in round 1 of conversation |
| GPTResponseWordCount_r2 | GPT response length in round 2 of conversation |
| GPTResponseWordCount_r3 | GPT response length in round 3 of conversation |
| GenderCat | Gender |
| StudyNumberChar | Study number (1 vs 2) |
Conversation and text variables are only provided for users who did not reveal personally identifying information and who were included in the DBSCAN analysis. Hence, NA values are present in the dataset that were not present in the original dataset. Moreover, given that many participants were screened out throughout the each experiment (and in our analytic process), NA values are present throughout the datasets.