Narrative License
When the language of a study becomes more certain, more general, or more dramatic than the evidence actually supports.
Understanding Research Narratives
Research is not only communicated through data tables and methods sections. It also lives in the stories scholars tell about what their results mean. Sometimes those stories stay close to the evidence. Sometimes they drift beyond it.
Narrative License is not always a product of fraud or deliberate deception. In fact, we believe malintent reflects a minority of occurrences. More often, it grows out of the human desire for a clear story and academic incentives that reward novelty and confidence over careful, hedged research.
In our lab, we address these problems across three fronts:
- Measuring the prevalence of Narrative License within the academic literature
- Assessing how LLMs may inadvertently exacerbate but also be used to mitigate its spread
- Developing solutions for more faithful science communication
CASE STUDY
Causal Language
One major form of Narrative License appears when researchers use causal language for studies with associational findings. In a large-scale analysis of 194,631 cross-sectional articles across the social sciences, we found that causal wording in titles and abstracts is both common and growing sharply over time.
LLMs
Part of the Problem,
Part of the Solution
Large language models are increasingly used to summarize and understand scientific research. But they do not just repeat findings; they often subtly reshape them. In our work, LLM summaries frequently made research sound more certain, removing hedging, strengthening causal claims, or using more confident language than the underlying studies.
This becomes even more concerning when sycophancy enters the picture. Models can shift their summaries depending on the user’s attitudes and beliefs, meaning the same paper may be described differently depending on who is asking.
Solutions & Ongoing Work
Building Better Guardrails
We are currently evaluating prompting strategies and creating simple tools that can improve science communication.
Measuring other forms of Narrative License at scale
Assessing associations with engagement metrics and citations
Tracking how Narrative License in papers spreads to popular media
Constructing a public-facing dashboard for easy detection
Interested in working at the CSSLab?
Click here regarding research assistant opportunities for Penn students