Title: Standard Errors and Null-Hypothesis Significance Tests for Reliability
Coefficients

Author: L. Andries van der Ark

Abstract:
Reliability analysis is one of the most conducted analyses in applied
psychometrics. It entails the assessment of reliability of both item scores
and scale scores using coefficients that estimate the reliability (e.g.
Cronbach’s alpha), estimate measurement precision (e.g., estimated standard
error of measurement), or estimate the contribution of individual items to the
reliability (e.g., corrected item-total correlations). Most statistical
software packages used in the social and behavioral sciences offer these
reliability coefficients. Standard errors and null-hypothesis significance
tests (NHSTs) are generally unavailable for reliability coefficients, which is
a bit ironic for coefficients that are about measurement precision. For a
large number of coefficients, I derived standard errors and NHSTs. In this
presentation, I will discuss the dilemmas and challenges of this task. In
particular, I will discuss (1) categorical marginal models (CMMs), which I
used as a framework for finding the correct sampling distributions of
reliability statistics, (2) the challenges of estimating CMMs when a large
number of items is involved, and (3) the challenges of developing
user-friendly R software. Finally, I will show what I think
reliability-analysis computer output should look like in the future.