softwareposted on 02.04.2019 by Patrick Groenen
Code as a research output can either be uploaded directly from your computer or through the code management system GitHub. Versioning of code repositories is supported.
These are matlab files to show the effect of p-hacking through a visual simulation. It simulates a natural experiments with two groups with the default of n = 20 each, both drawn from the standard normal distribution. Default is 1000 simulations and the first 15 are visualized. The 95% confidence intervals of the difference between the two group means are shown (green means that zero is included in the 95% confidence interval, red means that zero is not included in the 95% confidence interval). It is recorded how often in the simulation the null hypothesis is rejected at a 5% level.
The effect of the following four p-hacking cheats can be simulated:
A: Not doing the test once, but test two for two variables and their mean.
B: If no no significant effect is found, add 10 cases to each of the experimental groups
C: Add tests with a covariate (such as gender) and a test of an interaction effect of gender and the experimental condition.
D: Add a third experimental condition and repeat the tests for any pair of experimental condition.
In matlab, run
to start the GUI. Note that higher values than 5% are found because essentially multiple tests are done without correcting for the number of tests.
This simulation is the implementation of:
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.