Research Letter
April 24, 2024
Characteristics of X (Formerly Twitter) Community Notes Addressing COVID-19 Vaccine Misinformation
Matthew R. Allen, Nimit Desai, Aiden Namazi, et al
JAMA. 2024;331(19):1670-1672. doi:10.1001/jama.2024.4800
Social media can magnify health misinformation, especially about vaccination.1 Platform countermeasures have included censoring, shadowbanning (limiting distribution without disclosure), and adding warning labels to problematic content. Yet, evaluating these countermeasures is challenging due to restrictive public disclosures about their inner workings.2
In late 2022, X (formerly Twitter) introduced Community Notes, a crowdsourced misinformation countermeasure. Anonymous volunteer contributors independently identify posts containing misinformation and propose corrections called “notes.” Notes labeled as helpful by contributors who disagreed on past notes (to rely on a diversity of perspectives) are shown alongside the original post.3 Because Community Notes is open source, we were able to evaluate the topics, accuracy, and credibility of notes addressing COVID-19 vaccination.
Methods
Notes from the first year of Community Notes (December 12, 2022, to December 12, 2023) were obtained from X’s public data page. We filtered for notes that were visible on X that mentioned “vaccin*” and “covid*” or “coronavirus.”
A random sample of notes was double-annotated by M.R.A. and N.D. to determine topic, accuracy, and credibility. Open coding, which entails deriving labels from review of raw data, was used to determine the primary topic of each note. Axial coding was used to resolve open codes into 4 overarching topics (adverse events, conspiracy, vaccine recommendations, and vaccine effectiveness); all labeled notes were resolved to a primary subject label, or the 4 topics described all annotated notes. Notes were categorized as entirely (scientifically supported), partially (scientifically debated), or not (scientifically unsupported) accurate. Annotators were instructed to use their training, experience, and primary sources to evaluate accuracy. Since notes require citations, top-level domains in citations were rated as having high (primary sources, such as peer-reviewed journals or government websites), moderate (reputable secondary sources, such as major news outlets or fact checkers), or low (less reputable secondary sources, such as blogs or tabloids) credibility. When notes cited multiple sources, the highest credibility domain was used. Annotations were reviewed and disagreements adjudicated by a third clinician-author (D.M.S.).
Weekly rates of notes, the prevalence of note labels with bootstrapped 95% CIs, and total view counts for noted posts were computed with Python, version 3. The study, using public data (45 CFR §46), was exempted from ethical review.
Results
Of the 45 783 notes made visible on X, 657 mentioned COVID-19 vaccination. Monthly rates increased from 22 to 186 notes during the study (Figure). Of the 205 randomly sampled notes, there was strong agreement on note topics (90% agreement, Cohen κ = 0.83), source credibility (87% agreement, Cohen κ = 0.77), and accuracy (96% agreement, Cohen κ = 0.90) before resolving disagreements.
The predominant note topic was adverse events (51%; 95% CI, 44%-58%), followed by conspiracy theories (37%; 95% CI, 31%-44%), vaccine recommendations (7%; 95% CI, 4%-11%), and vaccine effectiveness (5%; 95% CI, 2%-8%). Ninety-seven percent (95% CI, 96%-99%) of notes were entirely accurate, 2% (95% CI, 0%-4%) partially accurate, and 0.5% (95% CI, 0%-1%) inaccurate. Forty-nine percent (95% CI, 42%-56%) of notes cited high, 44% (95% CI, 37%-51%) moderate, and 7% (95% CI, 4%-11%) low credibility sources.
Post view data were available for 189 of 205 posts, totaling 201 281 364 views (mean number of views, 1 064 981; 95% CI, 689 821-1 548 471). Example notes are provided in the Table.
Discussion
A sample of Community Notes added to posts on X containing COVID-19 vaccination misinformation primarily addressed adverse events and conspiracy theories, were accurate, cited moderate and high credibility sources, and were attached to posts viewed hundreds of millions of times.
The US Food and Drug Administration commissioner recently urged health professionals to redouble their vaccine education efforts.4 The small number of notes addressing posts with COVID-19 vaccine misinformation suggests opportunities for health professionals to contribute to this mission via participating in Community Notes.
The primary limitation of this study is that only note quality was studied, but these attributes are predictive of effectiveness (eg, higher credibility yields greater persuasiveness5). Additional limitations include a narrow focus on COVID-19 vaccination, a small sample, human judgments were used to assess accuracy, user engagement with notes was not studied, and effects on perceptions or behaviors were not studied.
Investigations of other health topics and note influence (including unintended effects6) are needed. More social media firms should open-source their misinformation countermeasures for evaluation by independent scientists to illuminate, foster public trust in, and scale the most effective strategies.
