How can one understand the spread of ideas across text data? This is a key measurement problem in sociological inquiry, from the study of how interest groups shape media discourse, to the spread of policy across institutions, to the diffusion of organizational structures and institution themselves. To study how ideas and narratives diffuse across text, we must first develop a method to identify whether texts share the same information and narratives, rather than the same broad themes or exact features. We propose a novel approach to measure this quantity of interest, which we call “narrative similarity,” by using large language models to distill texts to their core ideas and then compare the similarity of claims rather than of words, phrases, or sentences. The result is an estimand much closer to narrative similarity than what is possible with past relevant alternatives, including exact text reuse, which returns lexically similar documents; topic modeling, which returns topically similar documents; or an array of alternative approaches. We devise an approach to providing out-of-sample measures of performance (precision, recall, F1) and show that our approach outperforms relevant alternatives by a large margin. We apply our approach to an important case study: The spread of Russian claims about the development of a Ukrainian bioweapons program in U.S. mainstream and fringe news websites. While we focus on news in this application, our approach can be applied more broadly to the study of propaganda, misinformation, diffusion of policy and cultural objects, among other topics.

中文翻译：

---

量化不同语言的叙述相似性

如何理解思想在文本数据中的传播？这是社会学研究中的一个关键测量问题，从研究利益集团如何塑造媒体话语，到政策在机构中的传播，再到组织结构和机构本身的传播。要研究思想和叙述如何在文本中传播，我们必须首先开发一种方法来识别文本是否共享相同的信息和叙述，而不是相同的广泛主题或确切特征。我们提出了一种新的方法来衡量这种兴趣量，我们称之为“叙事相似性”，通过使用大型语言模型将文本提炼为其核心思想，然后比较声明的相似性，而不是单词、短语或句子的相似性。结果是 estimand 比过去的相关替代方案更接近叙事相似性，包括返回词汇相似文档的精确文本重用;主题建模，返回主题相似的文档;或一系列替代方法。我们设计了一种方法来提供性能的样本外测量（精度、召回率、F1），并表明我们的方法大大优于相关替代方案。我们将我们的方法应用于一个重要的案例研究：俄罗斯关于开发乌克兰生物武器计划的说法在美国主流和边缘新闻网站上传播。虽然我们在这个应用程序中专注于新闻，但我们的方法可以更广泛地应用于宣传、错误信息、政策传播和文化物品等主题的研究。