Linking Generalizability Theory and Social Judgment Theory
This year I have been putting the final touches on a manuscript (with Steve Schilling) entitled "Multilevel Judgment and Reliability Analysis: Hierarchical Linear Models as a Bridge Between Generalizability Theory and the Lens Model Equation." The following is an overview of our paper:
Although generalizability (G) theory and the lens model equation of social judgment theory (SJT) are two distinct analytic models of judgment that have developed along seemingly separate paths, they share common theoretical underpinnings provided by Egon Brunswik's probabilistic functionalism.
In this paper, hierarchical linear models are shown to be a promising analytical tool for combining these two previously divergent approaches into a single coherent framework - a framework simultaneously capable of assessing the dependability (reliability) of judgments and modeling diversity in judgment models across judges. In effect, G-theory is extended to take into account the information (cues) upon which judgments are based, and the lens model equation's pairwise consideration of interjudge agreement is supplemented by an analytic framework that considers all judges simultaneously, both individually and collectively.
Scientific paradigms begin as theoretical and philosophical orientations directed toward solving particular questions of interest to their proponents. Methodological developments, including statistical methods, arise as a means for implementing a particular research paradigm. But as an approach matures, it can become limited by its methodology in the sense that the methodology becomes inadequate to address important issues. We argue that this is the case with both SJT and G-theory.
Aggregation of ideographic results and assessing the adequacy of linear models of judgment are two important issues for SJT; however, up to now, the standard SJT technique of ideographic multiple regression has provided no coherent methodology for addressing these issues. Similarly, the standard G-theory technique of random effects ANOVA has been inadequate for addressing the issue of facets nested within persons.
But just as paradigms can be limited by their methods, outside methodological developments can serve to expand individual paradigms and unite seemingly disparate paradigms. Recent advances in statistical methods have lead many researchers to consider the utility of implementing a hierarchical approach to the analysis of linear models in a variety of contexts in psychology, social policy, and education. By applying hierarchical linear models in the context of human judgment, we see potential for the expansion and unification of two heretofore disconnected approaches for the analysis of human judgment data: SJT and G-theory.
Specifically, we demonstrate how HLM analyses supplement separate analyses within the frameworks of G-theory and SJT in several important ways:
1. Judges are simultaneously described at the ideographic and nomothetic levels; thus, the tension between the classic ideographic Brunswikian emphasis and the nomothetic orientation of G-theory is resolved.
2. The HLM analysis easily deals with both linear and quadratic relationships between the cues and judges' ratings. There is no need to accept the potentially unwarranted assumption that a strictly linear model is adequate to describe the typical judge.
3. Significance tests are available to guide the nomothetic aggregation of ideographic descriptions of judges. Differences among the judges with respect to the weights they assign individual cues can be detected, and inferences can be made about cue utilization in the population represented by the judges.
4. Sources of unreliability that can be examined are extended beyond the standard G-theory approach and can inform the design of subsequent decision studies. For example, the impact of judges' differential cue utilization can be taken into account.
5. HLM analyses extend the lens model equation by permitting the decomposition of lack of agreement among judges into error variance and random effects of cues. This information subsequently can be used to guide training intended to improve interjudge agreement.
In summary, the HLM framework is a promising tool for combining SJT and G-theory into a single coherent framework that is simultaneously capable of assessing the dependability (reliability) of judgments and modeling diversity in judgment models across judges. In effect, G-theory is extended to take into account the information (cues) upon which judgments are based, and the lens model equation's pairwise consideration of interjudge agreement is supplemented by an analytic framework that considers all judges simultaneously, both individually and collectively.
Anyone desiring an electronic copy of the latest version of this manuscript may contact me at the e-mail address below.
Contact James Hogge