Heterogeneity of research results as a new perspective to assess and promote progress in psychological science

Linden, Audrey and Honekopp, Johannes (2020) Heterogeneity of research results as a new perspective to assess and promote progress in psychological science. Perspectives on Psychological Science. ISSN 1745-6916 (In Press)

[img]
Preview
Text
Heterogeneity and Progress accepted.pdf - Accepted Version

Download (596kB) | Preview

Abstract

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed, and therefore a lack of understanding of the subject matter. Consequently, typical levels of heterogeneity offer a useful but neglected perspective on levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large, for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. Based on insights from the history and philosophy of science, we argue that reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice towards this end.

Item Type: Article
Uncontrolled Keywords: meta-analysis, heterogeneity, replication, statistical power, philosophy of science, psychological research
Subjects: C800 Psychology
Department: Faculties > Health and Life Sciences > Psychology
Depositing User: Elena Carlaw
Date Deposited: 19 May 2020 08:36
Last Modified: 19 May 2020 08:45
URI: http://nrl.northumbria.ac.uk/id/eprint/43180

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics