#academicsky

Thanks @shugars.bsky.social !!!

Thank you for fighting for our public land @wyden.senate.gov !

our sources included Russian, English, and Ukrainian

and by you!! @jasong.bsky.social really enjoyed working with you on this paper

Our replication package is available at this link: doi.org/10.7910/DVN/...

This article will appear in print in the August SMR special issue on generative AI. Special thanks to Daniel Karell and @thomasdavidson.bsky.social for organizing this special issue! All the articles in the issue are fantastic, please read them all.

Our paper shows that large language models can be used for complex labeling tasks unattainable by previous measures. We furthermore find fine tuning to hold particular promise for these types of tasks. We also provide a framework for out-of-sample validation of our rare event estimand.

We benchmarked the performance of our approach against a range of existing measures of related estimands. Our measure outperformed these relevant alternatives. We show the performance of our large language model estimators (“SBERT-LLM”) versus considered alternatives in the table below.

We use this method in a case study of U.S. news website coverage of the war in Ukraine. We show that low quality U.S. news sites were more likely than mainstream U.S. news sites to have overlapping claims (narrative similarity) with Russian newspapers.

We leverage recent advances in NLP to measure whether two newspaper articles are making the same claims about the same underlying subjects. We use document embeddings to reduce the number of comparisons to a tractable number and large language models for pair annotation.

Researchers are often interested in tracking the flow of ideas and claims across texts. This is a very challenging target to estimate, however, as due to copyright and journalistic norms authors will often reuse information and ideas without using the same words, phrases or even language.

Thanks Erik!