Allegedly Does Not Replicate

Run the exact same Google Trends query tomorrow and you may get a different time series — because Google quietly redraws its sample every day. So what happens to published results built on that data?In this episode, Lester Lusher tells us about "When Samples Shape Significance," his new project with four of his PhD students. The team identified 73 papers in top general-interest economics journals that use Google Trends, took 24 with usable replication packages, and re-ran each paper's exact queries 40 times — same terms, same locations, same time windows, same code. If results were solid, t-statistics should bounce around the published value, shrinking about half the time. Instead they shrank 70% of the time, dropping 31% on average, with some papers losing statistical significance in nearly every redraw. The strongest predictors of fragility: researcher degrees of freedom — transforming the index, using only part of the downloaded series, running many queries — and noisy queries whose resamples barely correlate. No p-hacking required; just the quiet, ordinary decisions that survive into print. Lester even found one of his own papers in the sample, and tells us — honestly — how it fared.We also get into what this means for referees ("show me the same query, resampled"), whether pre-registration would fix it, why every robustness exercise in economics seems to land at 50–60%, and what this implies for census samples, unemployment statistics, and every other dataset that is a random draw.Lester Lusher is a professor of economics at the University of Pittsburgh, working in applied microeconomics, with work spanning education, labor, and the credibility of empirical research

What's the data-generating model? The theoretical problem of replication.

Before we argue about whether a finding replicates, we should ask a harder question: did we ever know what we were estimating? This session makes the case that many findings are "not even wrong" — and that in economics, the theory often gets written after the results. If we can't state the data-generating model, what exactly is a replication testing?

Leading the session: Nate Breznau is a researcher at the German Institute for Adult Education — Leibniz Centre for Lifelong Learning in Bonn. He works on metascience and open science, with a focus on transparency, reproducibility, and inclusivity in research — including the landmark crowdsourced studies showing how much results vary across analysts.

Recorded at the Institute for Replication's Barcelona Workshop, July 18, 2026.

Allegedly Doesn't Replicate is a podcast by the Institute for Replication, where we put research to the test — again.

In this episode, we talk with Adam Salisbury from GiveWell about what funders like GiveWell and Open Philanthropy are interested in, how they view replication, and their stance as funders. This is all in the context of a study I4R conducted for them. We reached out to many researchers funded by GiveWell to see if they had data and code available, and found that very few did. Join us to learn more!

Olle Folke and Joop Adema join us to explain how a single mis-specified regression turned a modest seasonal blip into a dramatic claim about Sweden’s sex-purchase ban. Rebuilding the analysis from scratch, they find no effect once the model is fixed, prompting a wider conversation on replication, journal accountability and the small coding choices that can sway policy debates.

In this episode, we talk with Felix Holzmeister, Assistant Professor of Economics at the University of Innsbruck, about an unconventional but fascinating approach to evaluating research credibility: prediction markets.

Can prediction markets really help us assess whether a study will replicate? Are they better than peer review? We explore how these markets work, what they reveal about scientific beliefs, and why they might be a useful tool for open science.

If you’re interested in transparency, reproducibility, or just curious about how economists place bets on replication, this one’s for you.

In this episode of Allegedly Does Not Replicate, we sit down with Nate Breznau, an accomplished researcher known for his work on many-analyst projects. Nate shares his insights on the challenges and opportunities of collaborating with multiple analysts and navigating interdisciplinary research as a sociologist working alongside economists. We also dive into his paper with Borjas, “Ideological Bias in Estimates of the Impact of Immigration,” which explores how political biases can influence the way researchers write papers. Join us as we explore the dynamics of these projects and what they reveal about the reproducibility and robustness of social science research.

In this episode of Allegedly Does Not Replicate, we sit down with Misha Teplitskiy, Assistant Professor at the University of Michigan, to discuss his paper ‘Is novel research worth doing? Evidence from peer review at 49 journals.’ We explore how peer review processes assess novelty in scientific research and whether they encourage or discourage truly groundbreaking work.

Misha shares insights on the incentives that drive academic publishing, the balance between risk-taking and incremental contributions, and what his findings reveal about the broader scientific ecosystem. We also discuss the implications of these results for researchers, journal editors, and policymakers looking to foster more innovative and impactful science.

Join us for a thought-provoking conversation on the challenges and rewards of doing novel research—and why it might be more valuable than we think.

In this special episode of Allegedly Does Not Replicate, we are honored to welcome John Ioannidis, one of the most influential figures in academia and a pioneer in the field of metascience. We dive into his groundbreaking paper, Why Most Published Research Is Wrong, and discuss its lasting impact on how we think about scientific validity and reproducibility.

Beyond this, we explore the broader challenges facing the scientific community, including the role of open science, the replication crisis, and how researchers can improve the credibility of their work. Whether you’re a scientist, a policymaker, or simply curious about how scientific knowledge is built and tested, this conversation offers valuable insights into the evolving landscape of research.

Join us for a thought-provoking discussion on what it truly means to do rigorous and reliable science.

In this episode of Allegedly Does Not Replicate, we’re thrilled to welcome Bruno Barbaroli, a Research Scientist at I4R. Bruno dives into the exciting world of AI applications in replication and open science, sharing insights on I4R’s latest initiatives. We also discuss our new meta-paper, exploring the differences between human-led and AI-supported teams in replicating research papers. Tune in for a deep dive into how AI is reshaping the landscape of scientific replication. This episode launches alongside the preprint of our new study—don’t miss it!

In this week’s episode, we sit down with David Reinstein and Robert Kubinak from The Unjournal, an innovative platform reshaping academic publishing. They share insights into how The Unjournal offers an open, sustainable system for evaluating and enhancing impactful research, particularly in economics, policy, and social sciences. Tune in to learn about their approach to public, journal-independent evaluations and their vision for a more efficient and transparent research assessment process.

In this episode, we sit down with Stefano DellaVigna and Elizabeth Linos to dive into their fascinating work, “RCTs to Scale.” We explore what happens when interventions designed in the lab meet the real world—and the surprising evidence their research reveals about the role of publication bias in scaling behavioral interventions. How does this connect to the broader goals of open science? Tune in to hear how their findings challenge our understanding of what works, why it works, and how we can push the boundaries of transparency and replication in science.

In this week’s episode, we sit down with Jon Hartley, host of the “Capitalism and Freedom in the 21st Century” podcast, to delve into open science initiatives within various economics subfields. We explore how open science practices are reshaping research methodologies, enhancing transparency, and fostering collaboration among economists. Join us as we discuss the transformative impact of open science on the discipline and its potential to drive innovation and inclusivity in economic research.

In this episode, we sit down with Ted Miguel, a prominent economist and a leading voice in open science, to explore the transformative role of pre-analysis plans and other open science initiatives in research. We discuss how these practices are reshaping the social sciences by fostering transparency, reproducibility, and collaboration. Ted shares his insights on the challenges and opportunities that come with implementing open science frameworks and reflects on their broader impact on academic integrity and policy relevance.

Whether you’re a researcher, a student, or simply curious about the future of scientific inquiry, this episode offers valuable perspectives on how we can all contribute to a more transparent and reliable body of knowledge. Tune in to join the conversation!

In this episode, we sit down with Joan Llull, the Data Editor for the Econometric Society’s journals: Econometrica, Quantitative Economics, and Theoretical Economics. Joan shares insights into his role, the responsibilities it entails, and the importance of data transparency in economic research. He also discusses challenges faced, recounts memorable experiences, and offers his perspective on the future of data management in academic publishing. Join us for an engaging conversation that delves into the evolving landscape of data editing and its impact on the credibility of economic scholarship.

In this episode, Jack Fitzgerald dives into his journey of using reproductions as a foundation for proposing and testing innovative statistical methods. He discusses the ins and outs of writing academic comments, the challenges and insights he’s gained, and offers a unique perspective on the role of a reproducer in advancing research rigor.

In this episode, Professors Ryan Briggs and Vincent Arel-Bundock join us to dive into their latest research on statistical power in political science. They break down the challenges and implications of statistical power, why it matters, and how it shapes the reliability of findings in the field. Tune in to learn more about how statistical rigor can impact our understanding of political science research.

In this episode of Allegedly Does Not Replicate, we sit down with Professor Michèle Nuijten to dive deep into the world of research replication. Known for her work in psychology, Michèle introduces us to StatCheck, a powerful tool used to verify the accuracy of statistical reporting. While Stat Check is widely known in fields like psychology, it remains less familiar to other disciplines—something our host, Abel Brodeur, explores in this engaging conversation.

Together, we discuss the current state of replication efforts in academia, the importance of transparency in research, and how tools like StatCheck are shaping the future of reproducibility across disciplines. Whether you're an economist or a researcher from another field, this episode will inspire you to think more critically about how we validate research findings.

Join us for this fascinating discussion and learn why research replication is more essential than ever in today's academic landscape.

In this episode, we talk about open science initiatives with a very special guest.

We welcome Jörg Peters to discuss if economics is self-correcting.

In this episode, Abel tells us all about a paper making rounds around social media.

In this episode we introduce the podcast and our Chair, Abel Brodeur.