Where Fisher, Neyman and Pearson went astray: On the logic (plus some history and philosophy) of Statistical Tests

Abstract

Every scientific endeavour consists of (at least) two components: A hypothesis on the one hand and data on the other. There is always a more or less abstract level - some theory, a set of concepts, certain relations of ideas - and a concrete level, i.e., empirical evidence, experiments or some observations which constitute matters of fact. The focus of this contribution is on elementary models connecting both levels that have been very popular in the social sciences - statistical tests. Going from simple to complex we will examine four paradigms of statistical testing (Fisher, Likelihood, Bayes, Neyman & Pearson) and an elegant contemporary treatment. In a nutshell, testing is an easy problem that has a straightforward mathematical solution. However, it is rather surprising that the statistical mainstream has pursued a different line of argument. The application of the latter theory in psychology and other fields has brought some progress but has also impaired scientific thinking.