In the 60's early 70's "policy capturing" was universally used. In the 70's and the 80's "judgment analysis" came into widespread use, but both terms were considered synonymous. Now we have a clear view of the distinction between these two terms, and we can use the correct term when appropriate. The following comments trace this evolution and describe the current view.
From the Brunswik list archives (1995)
>>> Posting number 180, dated 1 Aug 1995 09:08:55
Date: Tue, 1 Aug 1995 09:08:55 -0400
From: Tom Stewart
Subject: Social judgment theory, judgment analysis, and policy capturing
Following up on Ken Hammond's message about the origins of social judgment theory, and also in response to a private message from Len Dalgleish, I've been doing a little investigation of the terms "policy capturing" and "judgment analysis."
First, judgment analysis (the term I strongly prefer) is not synonomous with SJT. Judgment analysis is just a method for modeling judgment. It is often used in SJT research, and it is useful in implementing some of the remedies suggested by SJT, but there is much more to SJT.
Both terms originated at about the same time, in the early 60's, in connection with work being done at Lackland AFB (see appended messages from Joe Ward and Jim Naylor). "Policy capturing" was first used by Christal and Bottenberg. Naylor and Wherry, who were doing contract work with Lackland used the acronym "JAN" (Judgment ANalysis) in 1965 to refer to a technique for clustering judges based on their judgment policies. Sam Houston (1974), also associated with Lackland, wrote a monograph with judgment analysis in the title. Classic papers in the Brunswikian tradition, such as Hammond (1955), Hoffman (1960), and Slovic and Lichtenstein (1973) do not use either term.
For a long time, I have been encouraging the use of the term "judgment analysis" instead of "policy capturing." I can remember the exact moment when my distaste for the term "policy capturing" peaked: I was sitting next to Len Adelman at a meeting when somebody mentioned policy capturing. I looked over at Len and he was doodling a stick figure carrying a butterfly net and chasing a "policy." I then realized that we don't capture anything, we analyze judgment. Furthermore only a few people (mostly psychologists) have used the word "policy" to mean "judgment policy." To everyone else, it has a different meaning. I am happy that Ray Cooksey has used "judgment analysis" in his forthcoming book.
Of course, "policy capturing" is still in widespread use. In bibliographic searches, one of the best ways to find studies that use judgment analysis is to search for the keywords "policy capturing." In the current issue of Medical Decision Making is an article with "policy capturing" in its title. Among its authors are highly respected subscribers to this list who are well acquainted with judgment analysis. Presumably, they chose "policy capturing" for a reason. They may want to respond to this message.
One unfortunate terminological hybrid must be stopped. I have seen policy capturing and judgment analysis combined into "policy analysis." We really don't need a third term to refer to the same procedure. This one is obviously unacceptable since policy analysis has long been used to refer to a completely different type of inquiry. I hope we can put a stop to it before it spreads.
Hammond, K. R. (1955). Probabilistic functioning and the clinical method. Psychological Review, 62, 255-262.
Hoffman, P.J. (1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57, 116-131.
Houston, S. R. (1974). Judgment analysis: Tool for decision makers. New York: MSS Information Corporation.
Naylor, J. C., and Wherry, R. J., Sr. (1965). The use of simulated stimuli and the "JAN" technique to capture and cluster the policies of raters. Educational and Psychological Measurement, 25(4), 969-986.
Slovic, P., and S. Lichtenstein, (1973). Comparison of Bayesian and regression approaches to the study of information processing in judgment. In L. Rappoport and D. A. Summers (Eds.), Human Judgment and Social Interaction. New York: Holt, Rinehart & Winston, pp. 16-108.
I have just talked with Bob Bottenberg about your questions. We both agree that THE ORIGIN OF TERMS "POLICY CAPTURING" and "JUDGMENT ANALYSIS (JAN)" as used at the Personnel Research Laboratory at Lackland AFB came from informal discussions among Ray Christal, Bob Bottenberg and Joe Ward. The terms may have been used earlier and may have been publicly documented earlier by others; however, Bob and I believe that we did not know of the use of these expressions by others.
The idea of "POLICY CAPTURING" using REGRESSION MODELS to "mathematically capture a policy" was stimulated from the recognition in the 1950's that the PERSONNEL ASSIGNMENT PROBLEM (assigning persons to jobs, or "person-job match") was mathematically the same as the TRANSPORTATION PROBLEM OF LINEAR PROGRAMMING.
When we talked about trying to "assign personnel to jobs to maximize "payoff" or "value" to the Air Force", the universal response was that "But we don't know the "payoff" or "value" of each person on each job"! Then our response was "Then we do not need the many counselors who are trying to put each person into the "right" job!" "So we can just assign the personnel at random"!
Of course, the counselors probably were doing something toward "maximizing" the "payoff" of the assignment process. So we decided that we might be able to "capture" the counselors "policy" with one or more regression models and then we could fill in the "payoff" array with the predicted values from the regression model(s). Then I developed the DECISION INDEX to provide counselors with information that would allow them to approximate the "optimum" assignment of personnel to jobs. The DECISION INDEX is about the best approach to the "sequential" personnel assignment problem.
Later we incorporated the idea of "clustering" regression equations to determine how many different "policies" were involved.
Several of the original Air Force Publications have been published in the Journal of Experimental Education. This happened because Jack Schmid at Univ. of Northern Colorado at Greeley was editor of the J of Exp. Ed. Sam Houston did a lot of Policy Capturing while he was at Univ. of Northern Colorado, and Sam's wife did her dissertaion (at UNC) using Policy capturing in the study of Pornography.
The most concentrated source is in the J of Exp. Ed. vol. 36, No.4, Summer'68 But the J of Exp. Ed. contains other examples.
This volume contains:
1. JAN: A TECHNIQUE FOR ANALYZING GROUP JUDGMENT by Ray Christal (Footnote #2 indicates "Dr. Joe H. Ward, Jr. is credited with suggesting use of a least-squares-weighted regression formula to capture the policy of a single rater: also see (2)." The (2) is Paul Hoffman's article, The Paramorphic Representation of Clinical Judgments", Psychological Bulletin, LVII (1960), pp. 116-131.
2. SELECTING AN HAREM - AND OTHER APPLICATIONS OF THE POLICY-CAPTURING MODEL by Ray Christal.
3. AN ITERATIVE TECHNIQUE CLUSTERING CRITERIA WHICH RETAINS OPTIMUM PREDICTIVE EFFICIENCY.
JAN - which has been used to describe both the "POLICY CAPTURING" using regression models and the "CLUSTERING" of the POLICY EQUATIONS -combines regression modeling and "CLUSTERING" that was first defined in 1961 in HIERARCHICAL GROUPING TO MAXIMIZE PAYOFF and then revised and published in the J. of the ASA in 1963. This original hierarchical clustering algorithm is now contained in many of the statistical packages.
The DECISION INDEX was first described in THE COUNSELING ASSIGNMENT
PROBLEM BY Ward in Psychometrika, 23, 55-65
Policy Specifying provides a procedure to create prediction models and allows the judge to observe the output of the models. Then the models are modified and the judge takes another look. This continues until the output of the function seems OK. This approach was developed for the Air Force PROMIS system for recruiting. The idea was created to allow for the judge to express more easily "INTERACTIONS" among variables. Our observations with "POLICY CAPTURING" is that it is not easy for judges to express interactions and non-linearities. But POLICY SPECIFYING MAKES THIS EASIER.
About Policy Capturing vs. Policy Specifying
Policy Capturing attempts to predict the judgments from a mathematical model.
Policy Specifying attempts to allow the judge to create a mathematical model that produces output values ("judgments") that are desired by the judge. Rather than make judgments and then try to allow the regression model to reproduce the judgments, the Policy Specifying approach allows the judge to define a model that hopefully produces output that expresses the judge's policy. This approach seems to allow for the expression of interactions and non-linearities.
It gets a bit murky, particulary since I cannot locate a copy of the Bottenberg and Christal Tech Report from 1961.
Bottenberg, R. A. & Christal, R. E. An iterative technique for clustering criteria which retains optimum predictive efficiency. WADD-TN-61-30, AD-261 615. Lackland AFB, Tex.: Personnel Research Laboratory, Wright Air Development Division, March, 1961.
My memory is that the first ACTUAL use of the term "policy capturing" in print was not in the above, but in the tech report
Naylor, J. C., and Wherry, R. J. Feasibility of Distinguishing Supervisor's Policies in Evaluation of Subordinates by Using Ratings of Simulated Job Encumbents. PRL-TR-64-25. Personnel Reseaarch laboratory, Aerospace Medical Division, Air Force Systems Command, Lackland Air force Base, Texas, October, 1964.
I have a copy of the above and the term policy capturing is used numerous times.
The first journal article to use both the terms "policy capturing" and "Judgment Analysis" was the 1965 Naylor and Wherry paper in EPM.
The first ACTUAL use of the Judgment Analysis term was in the tech report of Christal:
Christal, R. E. JAN: A Technique for Analyzing Individual and Group Judgment. Lackland AFB, Tex.: PRL-TDR-63-3, AD-403-813. Personnel Research Laboratory, Aerospace Medical Division, February, 1963.
Confused! Hope this helps some.
James C. Naylor, Ph.D. 614-292-3038 Office
Additional comment from Naylor:
Looks fine to me. I don't disagree with any of your comments. I'm glad
joe could confirm that the Bottenberg and Christal paper contains both
terms. My own concern about using the term Judment Analysis to replace
policy capturing is that I have always seen JAN as a very specific technicque
developed at Lackland for clustering policies. The first stage of that
process involved capturing the policies of individual raters using linear
regression. To me these two terms are quite explicit in meaning and are
>>>End of Posting number 180, dated 1 Aug 1995 09:08:55
Further comments from Tom Stewart (1998)
Policy capturing is simply the application of regression analysis to modeling judgment. It is not based on the lens model or any Brunswikian ideas. Policy capturing studies are atheoretical, generally do not include the environment side of the lens model, generally use orthogonal designs, and generally use linear regression (although ANOVA and conjoint measurement is also used), and often confront people with unfamiliar tasks.
"Judgment analysis" to refers to modeling of judgment in the Brunswik/Hammond
tradition. This requires an analysis of the environment, which policy capturing
does not include. Furthermore, judgment analysis is clearly not wedded
Comments from Ray Cooksey (1998)
(Author of Judgment Analysis: Theory, Methods, and Applications)
Policy capturing has traditionally only referred to the modeling of judgments and is almost always associated with multiple regression models. However, this is just a small aspect of judgment analysis which involves a full implementation of representative design, considerations of task ecology as well as judgment process, and need not be tied to multiple regression analysis. In the context of judgment analysis, 'policy capturing' in the Bottenberg, Ward & Naylor sense, is exclusively linked to the single system model which ignores the existence of an ecological criterion. However, it is still quite common for people who do 'judgment analysis' to refer to the process of modeling judgments as 'policy capturing'.
Thus, there is a technical use of 'policy capturing'
which refers to a stand-alone method linked only in a very minor way to
Brunswik's ideas (if one uses the single system conceptualization) and
a less formal sense which refers to the methodological exercise of modeling
judgments in the larger context of at least a double system lens model.
The former sense is synonymous with multiple regression modeling of judgments
made on a series of profiles (usually hypothetical) whereas the latter
sense potentially encompasses any statistical, mathematical or modeling
procedure that allows one to model or 'capture' judgments as well as models
of an ecological criterion (which then permits the measurement of achievement).
My view is this latter sense is the one which continues to keep 'policy
capturing' alive as a term of reference for what we do and is why the term
is so hard to extinguish. I deliberately titled my book 'Judgment Analysis'
to signal this but even I am guilty of slipping into using the term in
its less formal sense (witness chapter 4 in my book, which is entitled
'Capturing Judgment Policies'!). The debate is further confused if we throw
the label 'Social Judgment Theory' into the fray! This label is largely
synonomous with Judgment Analysis as it has evolved - but both labels are
still commonly used (witness an entire edition of Thinking & reasoning
devoted to SJT). What I think we are seeing is the slow process of evolution
at work, which for a period of time, means that many related 'species'
may co-exist before one comes to predominate. Tom is arguing for Judgment
Analysis to predominate and I agree - it signals best what we are doing.
Associate Professor Ray W. Cooksey
Home | Egon Brunswik | Sign up | Annual Meetings | Newsletters | Email list | Notes and essays | Resources | Photos | Links | Sitemap