NASP International and Interdisciplinary Seminars

Replacing experts with LLMs when analyzing political texts
Kenneth Benoit (Singapore Management University)

Chair
Fabio Franchino (University of Milan) 

11 April 2025, h. 14:30
Room B | Graduate School
Via Pace 10, Milan

Abstract
In this paper, co-authored by Scott De Marchi, Conor Laver, Michael Laver, and Jinshuai Ma, we explore the use of large language models (LLMs) as a scalable alternative to human experts in analysing political texts, such as party manifestos. Traditional methods, like expert surveys and crowdsourcing, are limited by cost and scalability. We propose a systematic method to deploy LLMs, such as GPT, Gemini, and Claude, to estimate party policy positions, replicating tasks typically performed by human experts. For content validation, compare LLM-generated estimates with costly expert survey data, across key dimensions like taxation, social policies, immigration, and the environment. Our results demonstrate that LLMs can reliably replicate expert judgments, significantly reducing the need for human intervention in large-scale political text analysis. This approach not only makes political text analysis more accessible but also enables researchers to analyse large datasets in multiple languages, overcoming many of the limitations of traditional expert-based methods.

This seminar is part of the NASP International and Interdisciplinary Seminars Series 2025.