Skip to main content
2023 ESA Annual Meeting Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back

Contributed Talk

Session: : Statistics

COS 314-1 - Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Thursday, August 10, 2023
3:30 PM – 3:45 PM PDT
Location: E145
Abstract: In ecology and evolutionary biology results often differ among studies, and these differences are typically far greater than expected from sampling error alone. Differences among studies may be due to ‘true biological heterogeneity’, which comes from working in different systems with meaningful biological differences. However, researchers also differ in their methods, including the many decisions they make regarding statistical analyses. Our study is the first empirical evaluation of the effects of analytical choice on variability in statistical effects in ecology and evolutionary biology. We obtained two unpublished datasets and we recruited 145 independent teams of scientists to conduct analyses of these datasets to answer pre-specified research questions. We found substantial heterogeneity in statistical effects among the analyses of one data set, and only modestly less heterogeneity among the analyses of the other data set. The causes of this heterogeneity remain largely unexplained. Surprisingly, the quality of analyses, as assessed by independent reviewers, appeared only weakly related to variability in results. Our work raises important meta-scientific questions about the factors that influence the analytical heterogeneity among results. Further, because our results suggest that even well-designed analyses may often provide substantially different results from each other, we hope our work leads researchers to a thoughtful debate about how we should conduct and interpret statistical analyses in ecology and evolutionary biology and other probabilistic disciplines.