Illustrating Statistical Powerand Decision Error Rates
Using Archives of Experimental Results

Burrton Woodruff, Butler University

A presentation at the MidAmerica Teaching of Psychology Conference, Fall 1996. (Modified for Web Delivery: March, 1998) Click here to download as a pdf document.


In planning the laboratory component of a course, the instructor tries to find projects which meet several pedagogical criteria. Among these are the following: the project should demonstrate an interesting phenomenon of general significance; for experimental psychology courses, the project should be in the form of an experiment so design and procedural considerations can be illustrated; and, the experimental effect should be large enough to be replicable with a small number of subjects yet generate results which need inferential statistics to assess reliability. Since a primary purpose of a lab project is the demonstration of a behavioral phenomenon, it follows that one should avoid experiments likely to produce Type II decision errors. After all, if the phenomenon exists and the class was unable to replicate it, few conclusions can be drawn either in class discussion or by the students should a lab report be assigned.

However, lab projects often have a second important goal, that of providing the student with opportunities for applying the hypothesis testing model of inferential statistics. In this context, lab projects which tend to produce Type-II decision errors can be very useful for teaching concepts of statistical power analysis and illustrating why failure to reject the null hypothesis does not prove the null hypothesis to be true.

By using the same experiment semester after semester and maintaining an archive of the results, the instructor of a lab course is, in effect, conducting a Monte Carlo study to determine the probabilities of correct detections and decision errors for replications of varying sample size.

Whatever the outcome of the project, the instructor can relate this semester's results to the archived results to illustrate the reality of the probabilistic nature of inferential statistical decisions and the usefulness of statistical power analysis.

I regularly use two experiments during the first part of the semester in this manner. The first project is a repeated measures experiment on the effect of materials (letters vs digits) on the size of the immediate memory span (IMS). The second is an independent-groups replication of Benussi's (1904; cited in Kling & Riggs, 1971) experiment showing the effect of instructional set on the magnitude of the Müller-Lyer Illusion.

Immediate Memory Span. I have 13 replications of this experiment of which 5 correctly rejected the null hypothesis (see handout and figure). However, 12 of the 13 found a larger mean digit span as is reported in the literature (Cavanagh, 1972). Taking all 13 replications (n = 206), the effect is highly reliable ( t (102) = 5.78, p < .001). With an mean sample size of 16, an effect size calculated from the observed means and standard deviations for the combined replications, and a 5% Type I error rate, this experiment has a calculated power of 0.28. The observed correct decision rate is slightly higher, 0.38.

Müller-Lyer Illusion. I have 16 replications of which only 2 correctly rejected the null hypothesis (see handout and figure). Thirteen of the 16 duplicate Benussi's result, but not reliably, by finding "part-perceiving" instructions produce a smaller illusion than "whole-perceiving" instructions. When all 13 replications are taken as a large experiment, the effect is highly reliable. Assuming a typical group size of 8, and estimating the effect from the observed means and standard deviations, the power of the experiment is only 0.08 which is approximated by the observed correct-decision rate of 0.125 (1 in 16).

References

Cavanagh, J. P. (1972). Relations between IMS and the memory search rate. Psychological Review, 79, 525-530.

Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press.

Kling, J. W., & Riggs, L. A. (1971). Woodworth & Schlosberg's experimental psychology (3rd ed.). New York: Hold, Rinehart, and Winston.


© 2002 by BurrtonWoodruff. All rights reserved. Modified Sunday, March 25, 2007