Generative artificial intelligence (AI) has the potential to revolutionize social science research. However, researchers face the difficult challenge of choosing a specific AI model, often without social science-specific guidance. To demonstrate the importance of this choice, we present an evaluation of the effect of alignment, or human-driven modification, on the ability of large language models (LLMs) to simulate the attitudes of human populations (sometimes called silicon sampling ). We benchmark aligned and unaligned versions of six open-source LLMs against each other and compare them to similar responses by humans. Our results suggest that model alignment impacts output in predictable ways, with implications for prompting, task completion, and the substantive content of LLM-based results. We conclude that researchers must be aware of the complex ways in which model training affects their research and carefully consider model choice for each project. We discuss future steps to improve how social scientists work with generative AI tools.

中文翻译：

---

在社会科学研究中平衡大型语言模型对齐和算法保真度

生成式人工智能 （AI） 有可能彻底改变社会科学研究。然而，研究人员面临着选择特定 AI 模型的艰巨挑战，而且通常没有社会科学特定的指导。为了证明这种选择的重要性，我们评估了对齐或人工驱动的修改对大型语言模型 （LLM） 模拟人类态度的能力（有时称为硅采样）的影响。我们将六个开源 LLM 的对齐和未对齐版本相互比较，并将它们与人类的类似响应进行比较。我们的结果表明，模型对齐以可预测的方式影响输出，对提示、任务完成和基于 LLM 的结果的实质性内容产生影响。我们得出的结论是，研究人员必须意识到模型训练影响其研究的复杂方式，并仔细考虑每个项目的模型选择。我们讨论了改进社会科学家使用生成式 AI 工具的方式的未来步骤。