TEXT
pvaluesim.m (20.91 kB)
.FIG
pvaluesim.fig (60.14 kB)
1/0
pvaluesim.m
These are matlab files to show the effect of p-hacking through a visual simulation. It simulates a natural experiments with two groups with the default of n = 20 each, both drawn from the standard normal distribution. Default is 1000 simulations and the first 15 are visualized. The 95% confidence intervals of the difference between the two group means are shown (green means that zero is included in the 95% confidence interval, red means that zero is not included in the 95% confidence interval). It is recorded how often in the simulation the null hypothesis is rejected at a 5% level.
The effect of the following four p-hacking cheats can be simulated:
A: Not doing the test once, but test two for two variables and their mean.
B: If no no significant effect is found, add 10 cases to each of the experimental groups
C: Add tests with a covariate (such as gender) and a test of an interaction effect of gender and the experimental condition.
D: Add a third experimental condition and repeat the tests for any pair of experimental condition.
In matlab, run
pvaluesim
to start the GUI. Note that higher values than 5% are found because essentially multiple tests are done without correcting for the number of tests.
This simulation is the implementation of:
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.
