EPSY440 - Evaluation


Chapter Five Notes (Nitko, 2001)


professional responsibility - independence and autonomy in gathering and using decision-making information appropriately.

Each decision made by an educator will have positive or negative consequences for the students.

serious decision consequences - when students benefit or lose something very valuable and it cannot be easily recovered (e.g., final grades).

less serious decision consequences - when students benefit or lose something very valuable, and can easily recover it (e.g., quiz grade).

high-quality information - highly reliable and valid for the decisions at hand.

Codes of ethics and responsibilities cover areas such as one's professional role in society, integrity, conflicts of interest, diligent and due care, confidentiality, and communication with clients (students, parents) and the public.

Code of Professional Responsibility in Educational Measurement (CPR) - found in Appendix C, this document serves the purpose of guiding the conduct of members of the National Council on Measurement in Education (NCME) who are involved in any type of assessment activity in education.

Code of Fair Testing Practices in Education - found in Appendix B, this document states the major obligations to test takers of professionals who develop or use educational tests and is directed primarily at professionally developed tests sold by commercial test publishers.

The notes in this section are based upon the two documents mentioned above.

As an educator who uses assessments, you are expected to uphold principles of professional conduct such as:

  1. protecting the safety, health, and welfare of all examinees;
  2. knowing about and behaving in compliance with state and federal laws relevant to your activities;
  3. maintaining and improving competence in assessment;
  4. providing assessment services only in your area of expertise;
  5. adhering to, and promoting high standards of professional conduct within and between educational institutions;
  6. promoting the understanding of sound assessment practices;
  7. performing your professional responsibilities with honesty, integrity, due care, and fairness.

Six categories of assessment-related activities each having their own ethical concerns include:

  1. crafting assessment procedures - craft them so they are of high quality;
  2. choosing assessment procedures - make sure they are appropriate;
  3. administering assessment procedures - assure the administration process is fair and doesn't result in uninterpretable results;
  4. scoring assessment results - evaluate responses accurately and report them in a timely manner;
  5. communicating assessment results - providing complete, useful, and correct information that will promote positive (and minimize negative) consequences.

 

Responsibilities When Crafting Assessments:

If appropriate assessments are not available, it is the educator's responsibility to craft one which provides valid and reliable results.

triangulation - increasing reliability of information by combining results from several assessments of the same learning targets (e.g., quizzes, class performance, homework, projects, and tests).

Specific responsibilities when crafting assessments include:

  1. Applying sound principles of assessment planning, design, task development, item writing, rubric development, and assessment marking;
  2. Crafting procedures that are free from irrelevant characteristics and gender, ethnic, racial, social, and religious bias;
  3. Accommodating students with disabilities or special needs;
  4. Obtaining permission to use copyrighted material;
  5. Presenting assessments in a way that encourages students and others to properly interpret results;
  6. Assuring assessment materials do not contain errors, or if errors do exist, correcting them as soon as possible.

 

Responsibilities When Choosing Assessment Procedures:

You will probably use assessments developed by others, in which case you need to ensure that they, too, are valid and reliable.

It is often necessary to edit and rewrite assessment tasks from procedures developed by others because the tasks do not match the content, emphasis, vocabulary, or other methods you may have used.

Just because an assessment has been published does not guarantee the accuracy of its content or its quality as an assessment tool.

Teachers will often serve on committees designated at district or state levels to review and select published assessments...those engaged in these activities must have considerable knowledge of both content and assessment practices and principles.

Selectors of assessment procedures also have a professional responsibility to consider potential misuses and misinterpretations of the procedures, as well as the procedures potential for fulfilling its intended purposes.

People involved in selection committees need to read widely and keep current about assessments and their educational uses.

 

Responsibilities When Administering Assessments:

For students to perform at their maximum on an assessment, they need to know when it will be given, the conditions under which it will be given, the content and abilities required, what the assessment will emphasize, the level of performance expected, how it will be scored, and what effects the results will have on any educational decisions.

"Your professional responsibility is to provide 'reasonable opportunities for individuals to ask questions about the assessment procedures or directions prior to and at appropriate times during administration.'" (quoted on p.83 in textbook).

You have professional and legal responsibility to provide accommodations for persons with disabilities.

The purpose in providing accommodations is to provide more reliable scores from the assessment procedure; these scores may be higher or lower than in the standard administration conditions. The point is, they are more valid.

Teachers have a legal responsibility to provide accommodations that are outlined in a student's Individual Educational Plan (IEP).

You may need to provide additional or different accommodations than those outlined in the IEPs, as these are usually based on common disabilities, but specific manifestations of disabilities differ across individuals.

mandated tests - those you are required to administer by your school, district, or state.

standardized tests - developed by professional agencies such that the same test and procedures can be used for all students and provide comparable results.

Added responsibilities when administering mandated or standardized tests include:

(More complete instructions for administering standardized tests will be covered in chapter 16).

 

Responsibilities When Scoring Assessments

When scoring your own assessments, your obligations are:

  1. To score students' responses accurately using scoring keys, rubrics, checklists, or rating scales;
  2. To score students fairly by removing from the process anything that would cause scores to be unfairly given (use objective items, place names on back of tests, rescore a sample, have others rescore);
  3. To provide students with appropriate feedback (show them what they did incorrectly);
  4. To explain to students the rationale for scoring;
  5. To give students the opportunity to review their evaluations individually;
  6. To correct scoring errors and make necessary adjustments as soon as possible; and
  7. To score and return results in a timely manner.

When scoring standardized or other mandated tests, you have the responsibility to follow scoring procedures given in the test manual or other materials.

Common mistakes teachers make in scoring standardized tests include improperly using scoring keys, using the wrong norm-referenced samples, and not using the correct norm-referenced samples for that time of year.

Teachers who are required to score open-ended items on standardized or state-mandated tests should receive proper training in doing so, and be provided with appropriate scoring rubrics.

In some states, schools receive negative or positive consequences (e.g., funding) for their scores on mandated tests, which may tempt some teachers or schools to improperly report scores...teachers must remain vigilant.

 

Responsibilities When Interpreting and Using Assessment Results:

Professional obligations when interpreting and using assessment results from your own tests include:

  1. Interpreting students' performance in the context of the learning targets emphasized in your teaching;
  2. Interpreting performance on one assessment in relation to another assessment;
  3. Integrating performance in relation to the limitations of the assessment procedure you used;
  4. Helping students and parents to properly interpret test scores;
  5. Helping students and parents understand the consequences of improperly interpreting results;
  6. Interpreting students' results in relation to learning targets attained rather than as a weapon for punishment; and
  7. Keeping classroom assessments confidential and protecting students' right to privacy.

When interpreting and using results from state mandated or standardized tests, obligations and responsibilities (as outlined in section 6 of the CPR) include:

  1. Conducting activities in an informed, objective, and fair manner, with the understanding of potential consequences;
  2. Providing those who receive assessment results the purposes, limitations, and uses necessary for proper interpretation;
  3. Providing those who receive score reports written descriptions of all reported scores;
  4. Communicating results in an understandable and timely manner;
  5. Evaluating and communicating the adequacy and appropriateness of any norms and standards;
  6. Informing parties involved of how assessment results affect them;
  7. Using multiple sources and and types of relevant information in making decisions;
  8. Avoiding making, and actively discouraging others from making, inaccurate reports, unsubstantiated claims, and inappropriate interpretations;
  9. Disclosing whether and how long the results will be kept on file, procedures for appeal and rescoring, and rights of the examinees to access results;
  10. Reporting any apparent misuses; and
  11. Protecting the rights to privacy.

To fulfill the responsibilities outlined in Section 6 of Appendix C, you need to understand validity, reliability, norm-referencing, criterion-referencing, and standardized tests.

Responsibilities When Communicating Assessment Results:

You need to determine the level of understanding of parents, students, school officials, and others, and then tailor your reporting of results to those levels.

Your professional responsibility is to follow a regular communication schedule to report student progress to parents.


School districts and state departments may have policies and procedures regarding students' and parents' rights concerning assessment results and school records.

Table 5.3 summarizes some rights for parents and students under the law. These include the right to inspect records, the right to privacy, the right to parental involvement, the right to fairness in testing, and the right to be accommodated in testing.

Your school should also have a written policy in terms of:

  1. maintenance and release of assessment results;
  2. release of nonconfidential information;
  3. nondiscrimination; and
  4. representational consent information.

Various test-taker rights groups are working on rights such as:

  1. being treated with dignity and respect in the testing process;
  2. being tested by a qualified person;
  3. being notified about testing schedules and fees;
  4. being tested with appropriate accommodations;
  5. being informed about the purpose of testing;
  6. being informed about how the results will be used;
  7. being informed about the quality of information obtained from the results (i.e., reliability, etc.);
  8. being asked to give informed consent;
  9. being given the right to refuse testing if legally applicable;
  10. being given an understanding of the outcome of testing;
  11. being given an understandable explanation of consequences and decisions made from test results;
  12. being afforded privacy and confidentiality; and
  13. being given the opportunity to review records and appeal interpretations and findings.

Students also have responsibilities, such as properly studying and preparing for tests, being honest and not cheating, and avoiding plagiarism.

Restricting access to information, such as test items or exact scores, used to be justified on grounds that such access could result in more harm than good (e.g., students memorizing answers to specific items versus the learning target itself), but recent years have shown abuses that arise from such secrecy (e.g., misinterpretation of results by test users).

A number of state and federal laws have arisen allowing students and parents greater access to assessment results and more involvement in the assessment-based decision-making process.

Teachers should be aware of ethical principals (e.g., students' rights as subjects) regarding the privacy and confidentiality of students' performances on assessments in regards to research conducted within and outside the school.

Teachers and schools also have a responsibility to purge old records and correct any mistakes found in existing records. Anecdotal information should be destroyed when no longer needed.

Privacy and confidentiality of students' assessment results need to be guarded, and inappropriate gossip in the teachers' lounge (or other places) should not occur.

informed consent - consent given to others to obtain and use personal information (privacy is not violated in this case).

Informed consent should contain numerous points which young children often will not understand, and in that case, a proxy (e.g., parent or teacher) must give the consent.

Accommodations for students with cognitive disabilities (e.g., learning disabilities) is usually more controversial than those for students with physical disabilities.

Validity of interpretations in accommodating situations depends on the type of test administered, the purpose of the testing, the type of accommodation, the type of disability, and the nature of the interpretation.

You will need to preface norm-referenced and criterion-referenced interpretations in accommodating situations with statements specific to the circumstances. Sometimes it is possible and preferable to report results under both the standard condition and the accommodating condition.

Questions to consider when considering accommodations include:

  1. Will format changes or alterations in testing conditions change the skill being assessed?
  2. Will scores of examinees tested under standard conditions have different meaning than those obtained under accommodating conditions?
  3. Would non-disabled students benefit if allowed the same accommodations?
  4. Does the examinee with the disability have any capacity for adjusting to standard testing conditions?
  5. Is the disability evidence or testing accommodation policy based on procedures with doubtful validity and reliability?

Phillips (1994) argues that answering yes to any of the above questions negates the use of accommodations because it would invalidate the findings.

Others argue that making changes may change the skills assessed, but this may make the results more valid (e.g., it may make conditions more authentic such as those that occur in the real world).

Another issue regarding making accommodations is whether to identify in school records students for whom accommodations have been made.

Because accommodations are required by law under acts such as the Americans with Disabilities Act of 1994, and educators must try to maintain the validity of assessment results, it is best for them to consult legal advisors under such conditions.

The most prominent legal issues that are addressed in courts involve gender and racial discrimination, tests' contributions toward segregation in schools, unfairness of particular tests, and violation of due process.

Psychometric issues (i.e., the technical aspects of testing) addressed in court include:

Nonpsychometric issues addressed in court include:

Many definitions of bias exist in the media and professional literature:

bias as mean differences - an assessment is biased against a particular group when the average (mean) score of a group is lower than that of another group (however, this could be the result of actual group differences).

bias as differential item functioning - this type of bias exist when persons of the same ability (e.g., boys and girls) perform differently on specific items (again, this could be due to actual differences and not bias).

bias as misinterpretation of scores - this occurs when someone uses the results to make inappropriate inferences about students that go beyond the content of the test (e.g., saying females have lower math ability versus saying females had difficulty solving spatial problems).

facial bias - offensive stereotype use of language and pictures in assessment tasks and materials.

role stereotypes - the extent to which assessment tasks represent male/female, white/nonwhite role stereotypes.

The goal is not to produce gender-free, racial-free, ethnic-free assessment tasks, but rather, to present a balance and fair representation that is not offensive.

bias as differential validity - an assessment is biased when it has different predictive validity for one group than it does for another (few tests actually exhibit this).

bias as content/experience differential - when the content of assessment tasks is radically different from a particular groups' life experiences but interpretations do not account for this.

Bias as the statistical model - focuses on whether the statistical procedure used for selection is fair to all persons regardless of group membership (problems occur when certain subgroups consistently score lower, and when assessments have different relationships with measures of success on the criterion).

bias as wrong criterion - when the criterion measure itself is biased, which results in the selection procedure becoming biased (e.g., performance on a job not related to reading skills, but performance on the test is).

bias stemming from testing conditions - biases relating from test anxiety, feeling unwelcome, or being tested by persons of the other sex or another race that result in differential interpretations (also, when impoverished schools are "documented" as inferior based on mandated, standardized tests).

Back to course notes.

Back to course homepage.

This Webpage designed and updated (9/30/01) by Ron Dugan, University at Albany, State University of New York.