Metascience

 

 

As the behavioral and social sciences have expanded, so too have the challenges in how research is organized, evaluated, and communicated: Research is often siloed across disciplines, key concepts are defined inconsistently, and findings from different studies are not always combined effectively. In this landscape, collaborative approaches that aggregate evidence promise to improve communication with policymakers and the public. Another challenge is that the research community itself faces issues of diversity and representation. We use computational tools, including large language models and large-scale analyses of scientific publications, to take stock of these problems and develop solutions.

 

Mapping and Measuring Science

Psychology uses thousands of concepts and even more measures to capture them, but the relationships between them are often unclear. The same label can refer to different things, and different labels sometimes describe the same thing—this is known as the "jingle–jangle" problem. In our research, we use language models to place constructs and measures from the psychological literature into a shared space, where similar items sit close together. This space helps bring order to psychology’s conceptual landscape: It reveals clusters and key dimensions in the field, clarifies conceptual confusion by detecting when measures or labels overlap or diverge in meaning, and highlights gaps where new measures or constructs are needed.

Clarifying the mapping between constructs and measures is particularly important in personality psychology, and our work addresses this issue using methodologically distinct approaches. For example, a large-scale analysis of 48 impulsivity measures used patterns in people’s responses to identify a single, stable general factor that predicted real-world behavior better than any individual scale. Together, these complementary lines of work demonstrate that constructs like impulsivity that were long considered fragmented or unwieldy can be systematically mapped and integrated.
 
We also use large-scale analyses of scientific publications to map entire research areas, revealing how they are structured, where they overlap, and which questions remain underexplored. For instance, our interactive map of reinforcement learning research lets users explore clusters of work, see how strands connect, and discover areas that are densely researched versus those that are still emerging.

 

Large Language Models Transforming Science

Large language models are changing the way research is done, from literature review and hypothesis generation to data analysis. We study both the opportunities and the risks of this development, advocating for open-source models that allow researchers to understand and reproduce each step. Our work includes practical tutorials, new methods that harness large language models for scientific clarification, and broader reflection on what it means for AI to become a core part of research. Dirk Wulff discusses these themes in an interview on the MPIB website and a feature in Max Planck Research. 

We also use methods from behavioral science to understand the large language models themselves, using psychological tasks to assess whether and how they mirror human cognition. This kind of testing is critical for making the best use of these models and for anticipating their impact on society.

 

Diversity and Representation in Science

In many social science disciplines, women outnumber men among students and early-career researchers, yet remain underrepresented at senior levels. Using large-scale publication data to analyze the academic careers of psychology researchers, we have found persistent early-career losses among women that cannot be explained by differences in performance or institutional setting.

We also investigate subtler inequities—for example, how name changes following marriage can fragment a researcher’s publication record and reduce their perceived impact. By making these patterns visible through data, we aim to inform fairer evaluation practices.

 

Science for Policy and Communication

Research can only make a difference if it is aligned with societal needs and reaches the people who can use it. We develop tools and methods that bridge the gap between scientific evidence and real-world decision making.

One example is our open-source software text2sdg, which automatically detects references to the United Nations Sustainable Development Goals in scientific publications and other documents. Instead of relying on a single classification system, text2sdg combines multiple approaches in an “ensemble model,” providing more accurate and less biased labeling. This makes it easier to monitor how science contributes to global sustainability efforts and to identify where more work is needed. 

 

We also develop methods for harvesting distributed knowledge at scale. Using large language models to extract concepts from citizens’ open-ended answers to survey questions, we can surface policy perspectives that traditional polls might miss. And on the science communication side, we advocate for moving beyond the model of individual scientists broadcasting their own findings toward collaborative, technology-supported approaches that aggregate evidence and expert judgments more effectively—for example, resources like the COVID-19 Vaccine Communication Handbook, which brought together international experts to synthesize research into practical guidance for policymakers, practitioners, and the public.

 

References

• Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., Shiffrin, R. M., Gershman, S. J., Popov, V., Bender, E. M., Marelli, M., Botvinick, M. M., Akata, Z., & Schulz, E. (2025). (2025). How should the advancement of large language models affect the practice of science? PNAS, 122(5), e2401227121. https://doi.org/10.1073/pnas.2401227121
• Holford, D., Fasce, A., Tapper, K., Demko, M., Lewandowsky, S., Hahn, U., Abels, C. M., Al-Rawi, A., Alladin, S., Sonia Boender, T., Bruns, H., Fischer, H., Gilde, C., Hanel, P. H. P., Herzog, S. M., Kause, A., Lehmann, S., Nurse, M. S., Orr, C., … Wulf, M. (2023). Science communication as a collective intelligence endeavor: A manifesto and examples for implementation. Science Communication, 45(4), 539–554. https://doi.org/10.1177/10755470231162634
• Huang, Y., Luan, S., Wu, B., Li, Y., Wu, J., Chen, W., & Hertwig, R. (2024). Impulsivity is a stable, measurable, and predictive psychological trait. PNAS, 121(24), e2321758121. https://doi.org/10.1073/pnas.2321758121
• Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024). A tutorial on open-source large language models for behavioral science. Behavior Research Methods, 56(8), 8214–8237. https://doi.org/10.3758/s13428-024-02455-8
• Lopez-Lopez, E., & Herzog, S. M. (2025). Surfacing citizens’ policy perspectives at scale in the age of large language models. Behavioral Science & Policy, 11(1), 10–21. https://doi.org/10.1177/23794607251346
• Meier, D. S., Mata, R., & Wulff, D. U. (2025). text2sdg: An R package to monitor sustainable development goals from text. The R Journal, 16(1), 83–95. https://doi.org/10.32614/RJ-2024-005
• Thoma, A. I., Bolenz, F., Tiede, K., Yang, Y., Palminteri, S., Hertwig, R., & Wulff, D. U. (2025). Mapping the landscape of behavioral reinforcement learning research. PsyArXiv. https://doi.org/10.31234/osf.io/6c2va_v1
• Tikhomirova, T., & Wulff, D. U. (2026). Where meaning lives: Layer-wise accessibility of psycholinguistic features in encoder and decoder language models. https://arxiv.org/abs/2601.0379 
• Wulff, D. U., Hussain, Z., & Mata, R. (2024). The behavioral and social sciences need open LLMs. OSF. https://doi.org/10.31219/osf.io/ybvzs
• Wulff, D. U., & Mata, R. (2025). Advancing longstanding challenges in cognitive science with language models. arXiv. https://arxiv.org/abs/2511.00206
• Wulff, D. U., & Mata, R. (2025). Escaping the jingle-jangle jungle. Current Directions in Psychological Science, 35(2), 59–65. https://doi.org/10.1177/09637214251382083
• Wulff, D. U., & Mata, R. (2025). Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement. Nature Human Behaviour, 9, 944–954. https://doi.org/10.1038/s41562-024-02089-y
• Wulff, D. U., Meier, D. S., & Mata, R. (2024). Using novel data and ensemble models to improve automated labeling of Sustainable Development Goals. Sustainability Science, 19(5), 1773–1787. https://doi.org/10.1007/s11625-024-01516-3
• Zhao, X., Hertwig, R., & Wulff, D. U. (2025). Sustainable Development Goals in psychology. arXiv. https://arxiv.org/abs/2512.08628
• Zhao, X., Thoma, A. I., Hertwig, R., & Wulff, D. U. (2025). Mapping the gender attrition gap in academic psychology. arXiv. https://arxiv.org/abs/2510.13273