Power and Type-1 Error Trade-Off in ANOVA versus Linear Mixed-Effects Models with Approach-Avoidance Task Data
Keywords
No Thumbnail Available
Authors
Issue Date
2018-07-01
Language
nl
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The Approach-Avoidance Task (AAT) and other implicit measurement tasks have in common that they use samples of stimuli, e.g., pictures or words. Consequently—in addition to variation due to the sample of participants—there is also variation in the data due to the sample of stimuli. Researchers typically analyse such data using ANOVA, which requires aggregation over stimuli and therefore ignores this variation due to stimuli. Linear Mixed-Effects Models (LMEM) do not require aggregation and can simultaneously model variation due to participants and stimuli. The current study used simulations to investigate the extent to which variation due to stimuli influences power and type-1 errors of ANOVA versus different LMEM specifications. Simulations were based on the characteristics of a real AAT dataset (N = 1590). ANOVA resulted in higher power than LMEM, but ANOVA showed severely inflated type-1 error rates of up to 58%, indicating that the high power was based on false positive identification of non-existent effects. Properly specified LMEMs performed around the nominal .05 type-1 error level and, after correcting power for type-1 error rates, both ANOVA and LMEM exhibited comparable power. ANOVA's inflated type-1 error rate remained even after reducing the variation due to stimuli to 1/8th of the original value. Given the widespread use of ANOVA to analyse implicit measurement tasks, our results suggest that the respective literature likely contains many false positive results. Hopefully, the current paper raises awareness about the consequences of ignoring variation due to stimuli whilst also pointing out appropriate analysis methods.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen