Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science

Linden, Audrey and Honekopp, Johannes (2021) Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science. Perspectives on Psychological Science, 16 (2). pp. 358-376. ISSN 1745-6916

[img]
Preview
Text (Final published version)
1745691620964193.pdf - Published Version
Available under License Creative Commons Attribution Non-commercial 4.0.

Download (370kB) | Preview
[img]
Preview
Text (Advance online version)
1745691620964193.pdf - Published Version
Available under License Creative Commons Attribution Non-commercial 4.0.

Download (369kB) | Preview
[img]
Preview
Text
Heterogeneity and Progress accepted.pdf - Accepted Version

Download (596kB) | Preview
Official URL: https://doi.org/10.1177/1745691620964193

Abstract

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

Item Type: Article
Uncontrolled Keywords: meta-analysis, heterogeneity, replication, statistical power, philosophy of science, psychological research
Subjects: C800 Psychology
Department: Faculties > Health and Life Sciences > Psychology
Depositing User: Elena Carlaw
Date Deposited: 19 May 2020 08:36
Last Modified: 31 Jul 2021 16:07
URI: http://nrl.northumbria.ac.uk/id/eprint/43180

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics