The performance of multivariate methods for two-group comparisons with small samples and incomplete data

Abstract

In intervention studies having multiple outcomes, researchers often use a series of univari-ate tests (e.g., ANOVAs) to assess group mean differences. Previous research found that thisapproach properly controls Type I error and generally provides greater power compared toMANOVA, especially under realistic effect size and correlation combinations. However, whengroup differences are assessed for a specific outcome, these procedures are strictly univari-ate and do not consider the outcome correlations, which may be problematic with missingoutcome data. Linear mixed or multivariate multilevel models (MVMMs), implemented withmaximum likelihood estimation, present an alternative analysis option where outcome cor-relations are taken into account when specific group mean differences are estimated. In thisstudy, we use simulation methods to compare the performance of separate independentsamplesttests estimated with ordinary least squares and analogousttests from MVMMs toassess two-group mean differences with multiple outcomes under small sample and miss-ingness conditions. Study results indicated that a MVMM implemented with restricted maximum likelihood estimation combined with the Kenward–Roger correction had the best performance. Therefore, for intervention studies with smallNand normally distributed multi-variate outcomes, the Kenward–Roger procedure is recommended over traditional methods and conventional MVMM analyses, particularly with incomplete data.