Title: A New Semi-Analytical Model to Assess the Incidence of Data-Peeking
Practices

Authors: Francesca Freuli, Stefano Noventa, Luigi Lombardi

Abstract:
The practice of stopping data collection (i.e. “data-peeking”) as soon as a
significant result is observed is a potential threat to the overall
credibility of research findings. This practice increases the rate of false
positives as it is applied to obtain desired statistically significant results
without correcting the probability of the type-I error (alpha). Despite a
growing interest in the so-called Questionable Research Practices and their
impact on the replicability crisis, the standard methods to identify and
estimate the frequency of these bad practices in literature are based on
self-report questionnaires only (which may suffer from subjective perceptions
of the phenomenon). The present work aims at introducing a semi-analytic model
to estimate the frequency of results following data-peeking practices. The
proposed model estimates a likelihood ratio index that provides information
about the largest or lowest probability that a statistically significant
result was obtained by means of data-peeking strategy. To evaluate the
performance of our proposed model, we compared the values of the new index
with those obtained by means of Monte-Carlo simulations generated according to
different conditions involving several alpha levels (.01, .05, .025, .001),
total sample sizes (15, 20, 30, 60), and interim analyses performed (from 1 to
10). High Jaccard index values, low SRME values, and Bland Altman plots all
indicate a strong agreement between the semi-analytic and the simulated
results, suggesting the validity of the proposed new model.