This paper asks whether p-hacking makes the bias in published research worse, or in some cases less bad. P-hacking means researchers change analysis choices or data handling to increase the chance their result looks statistically significant. Publication bias means journals prefer to publish results that look significant. The authors build a simple model that links these two practices and asks how they interact.

In the model, researchers start with an estimate and its standard error. They can try ‘‘fast’’ p-hacking by paying a fixed cost to redraw the result — for example, running a fresh experiment or repeating an analysis until a different draw appears. They can also do ‘‘slow’’ p-hacking by paying a cost to nudge the reported estimate or standard error a little bit. The cost of slow p-hacking rises faster the more the result is changed. The model takes the publication rule as given and assumes journals publish more often when results pass certain significance thresholds. The authors analyze two cases: two-sided selection, where both unusually large positive and negative results are favored, and one-sided selection, where only results in one direction are favored.

The main theoretical findings are simple but not obvious. Fast p-hacking — actions that can shift p-values a lot — always makes the bias from selective publication worse. Slow p-hacking — many small choices that slightly change p-values — can go either way. When journals’ preference for significant results is weak, slow p-hacking increases bias. But when selection is strong, slow p-hacking can actually reduce the bias and bring published estimates closer to the true effects.

The authors also show that, under a normality assumption, one can identify both the true distribution of effects and the average result that would have been published if there were selection but no p-hacking. Using a censored version of their one-sided model, they apply this method to two meta-analyses. In the literature on behavioral nudges (Mertens et al. 2022) they find evidence that p-hacking may have reduced publication bias by as much as about 54 percent. In studies of development aid and growth (Doucouliagos and others) they estimate p-hacking increased bias by about 27 percent. The authors describe these findings as suggestive rather than definitive.