05.02.2022 Publication in Psychological Methods: Waldian t tests

Waldian t tests: Sequential Bayesian t tests with controlled error probabilities

Authors: Martin Schnuerch, Daniel W. Heck, & Edgar Erdfelder

Abstract:

Bayesian t tests have become increasingly popular alternatives to nullhypothesis significance testing (NHST) in psychological research. In contrast to NHST, they allow for the quantification of evidence in favor of the null hypothesis and for optional stopping. A major drawback of Bayesian t tests, however, is that error probabilities of statistical decisions remain uncontrolled. Previous approaches in the literature to remedy this problem require time-consuming simulations to calibrate decision thresholds. In this article, we propose a sequential probability ratio test that combines Bayesian t tests with simple decision criteria developed by Abraham Wald in 1947. We discuss this sequential procedure, which we call Waldian t test, in the context of three recently proposed specifications of Bayesian t tests. We show that Waldian t tests reliably control frequentist error probabilities, with the nominal Type I and Type II error probabilities serving as upper bounds to the actual error rates. At the same time, the key idea of Bayesian t tests (i.e, assuming a prior distribution for the effect size under the alternative hypothesis) is preserved. Thus, Waldian t tests are fully justified from both a frequentist and a Bayesian point of view. We highlight the relationship between frequentist and Bayesian error probabilities and critically discuss the implications of conventional stopping criteria for sequential Bayesian t tests. Finally, we provide a user-friendly web application that implements the proposed procedure for interested researchers.

Preprint: https://psyarxiv.com/x4ybm/