Content area
Abstract
Through computer simulation of data this study explored the effectiveness of various estimators of reliability in the context of trials-to-criterion testing. This method of evaluation consists of observing sequential trials of some dichotomously scored task until a pre-specified number of successes, or criterion score, is achieved.
The estimation procedures employed were coefficient alpha, parallel forms, and two methods based on estimates of error variance. The intent of this research was to determine which estimation method was more appropriate, assess the relationships among the various estimators, and provide further insights in general to trials-to-criterion testing.
Using each of the four methods, 1000 estimates of reliability were compiled. The mean and variance of these estimates were computed and compared with one another and with values of true reliability. Based on results of the computer simulation one of the error variance estimators (UEV) was recommended as the procedure to employ when a single administration of a trials-to-criterion test is planned. However, if a second test administration is feasible the parallel forms estimator was recommended for estimating reliability in the context of trials-to-criterion testing.
