Assessment and Evaluation

This is Unit Twelve, part one, on assessment: its purposes, validity and reliability, teacher-made tests and rubrics, and how assessment shapes motivation. Unit Eleven closed on consequences that repair. Unit Twelve turns to how teachers judge what students have learned. Carry one idea through this part: validity belongs to how a score is used, so choose each assessment for its purpose. At the entry on choosing validity evidence, the part shows one question deciding which evidence matters. By the end you will be able to tell diagnostic, formative and summative assessment apart. To choose the validity evidence a use of scores calls for. To estimate reliability and read a score as a band. To fix a flawed test item. And to choose a rubric for a purpose.

Three entries on what assessment is, when it happens, and how assessment for learning works.

Assessment, measurement and evaluation.

EDP 1101 Assessment, Measurement and Evaluation

Assessment is gathering information and judging students' progress; measurement assigns numbers, answering how much; evaluation makes judgments about students, methods or one's own teaching. Scoring a quiz eighteen of twenty is measurement; deciding the student has mastered fractions is evaluation.

Three purposes of assessment.

EDP 1102 Diagnostic, Formative and Summative

Diagnostic assessment comes before instruction; formative assessment, assessment for learning, happens during it; summative assessment, assessment of learning, comes after, for grades and certification. A pretest is diagnostic, an exit ticket formative, a final exam summative, though the same quiz can be either, depending on its use.

Assessment for learning.

EDP 1103 Assessment for Learning

Assessment for learning follows five steps: share clear goals, choose valid, reliable, unbiased and practical techniques, assess in ways that support motivation, adjust instruction, and communicate with parents, as a teacher posts the goal, checks it with a quick quiz, and reteaches what is weak. Like backward design, it starts from the goal.

Nine entries on validity, reliability and bias: what a score can mean, and how consistent it is.

Validity.

EDP 1104 Validity: Interpretations and Uses

Validity is the adequacy of the interpretations and uses made of test results for a group, not a property of the test itself, and it is a matter of degree: a reading test may support placing students in reading groups but not grading their science. Ask valid for what.

Content evidence.

EDP 1105 Content Evidence

Content evidence asks whether a test's items are relevant to and representative of what was taught; a table of specifications crosses topics with levels of thinking, so a unit spending half its time on fractions gets about half its items on fractions. A test can be relevant yet unrepresentative.

Construct evidence.

EDP 1106 Construct Evidence

Construct evidence asks whether performance reflects the intended construct, the ability the test means to measure, rather than construct-irrelevant factors such as English proficiency, anxiety or reading speed: a math word test English learners fail because of vocabulary measures English too. Such factors lower validity even when scores are consistent.

Criterion-related evidence.

EDP 1107 Criterion-Related Evidence

Criterion-related evidence compares test scores with another measure: predictive evidence when the criterion comes later, as admission scores predict first-year grades, and concurrent evidence when it is measured at the same time. Face validity, how a test looks to takers, is appearance, not validity evidence.

The payoff: which validity evidence?

EDP 1108 Which Validity Evidence?

Here the part's idea does its work. To decide which validity evidence a situation calls for, ask what the scores will be used for. To show mastery of what was taught, content evidence matters: do the items match the curriculum? To measure an ability such as reasoning, construct evidence matters: does the score reflect that ability rather than reading speed or language? To select or predict, criterion-related evidence matters. A company that tests applicants and later checks whether high scorers perform best on the job gathers predictive evidence; a teacher comparing test items with lesson plans gathers content evidence. A test that merely looks relevant has face appeal, not validity evidence.

Reliability in the classroom.

EDP 1109 Reliability in the Classroom

Reliability is consistency of results across occasions, raters and tasks, and teachers raise it with more items, clear directions and clear scoring criteria: a ten-item quiz gives a steadier picture of a skill than a two-item quiz. Reliability is necessary for validity but never sufficient.

Estimating reliability.

EDP 1110 Estimating Reliability

Reliability is estimated with coefficients: test-retest, giving the same form twice; alternate-form, giving two equivalent forms; and internal consistency from one administration, by split halves, Kuder-Richardson formula twenty or coefficient alpha, as when odd and even items are compared. Scorer consistency covers raters, and each coefficient captures a different source of error.

The standard error of measurement.

EDP 1111 The Standard Error of Measurement

No score is perfectly precise: the standard error of measurement estimates how much a person's score would vary on retesting, and a relatively large one indicates relatively low reliability, so a reported score is best read as a band, not a point. Systematic errors, pushing every score one way, reduce validity instead.

Two forms of bias.

EDP 1112 Offensiveness and Unfair Penalization

Assessment bias takes two forms: offensiveness, content that stereotypes or insults a group, and unfair penalization, content that depends on background experience not taught in class, like a reading item built on yacht-racing terms. An item can be inoffensive yet still unfairly penalize, which is the harder bias to spot.

Ten entries on teacher-made assessment: informal checks, selected and constructed responses, rubrics, performances, portfolios, and the testing effect.

Observation and questioning.

EDP 1113 Observation and Questioning

Informal assessment relies on observation and questioning: longer wait time after a question, varied and higher-level questions, and follow-up questions that probe a student's first answer. Waiting several seconds after asking brings fuller answers from more students, and wait time appeared earlier as a cultural difference in language use.

The limits of informal assessment.

EDP 1114 Limits of Informal Assessment

Informal assessment has limits: teachers may see what they expect, focus on process rather than learning, sample only some students, or misread culturally shaped behavior. Anecdotal records help by noting specific events, like Tuesday: Ana explained regrouping to her partner correctly, and formal observation uses a planned schedule.

Selected-response items.

EDP 1115 Selected-Response Items

Selected-response items, such as multiple-choice, matching and true-false, test recognition rather than recall; their objective scoring raises reliability, but a test made only of them can lower validity, since multiple-choice suits vocabulary but cannot show whether a student can write a persuasive essay. Align the format to the goal.

Fixing a flawed item.

EDP 1116 Fixing a Flawed Item

Common item-writing faults each let something other than knowledge decide the answer. Unclear wording confuses students who know the content; irrelevant cues, such as grammatical mismatches, give the answer away; negatives, especially double negatives, test reading; textbook sentences reward memorized wording; trivia tests unimportant facts; implausible distractors leave one reasonable choice, and all of the above rewards partial knowledge. An item asking which of the following is not an example of an unreliable measure, with all of the above as an option, has three faults. Repaired, it asks which measure gives the same result each time on the same object, with four plausible options. Fix the stem first.

Constructed-response items.

EDP 1117 Constructed-Response Items

Constructed-response items ask students to produce an answer: completion and short-answer items are easy to write but often allow more than one right answer, while extended responses, like explain why the Articles of Confederation failed, assess complex outcomes and writing, though they are scored unreliably without careful procedures.

Scoring extended responses.

EDP 1118 Scoring Extended Responses

Reliable scoring of extended responses uses a model answer, a reading of sample papers, rules for irrelevant content and mechanics, and a rubric; teachers grade question by question, anonymously, and avoid optional questions, which mean students take different tests. Grading every first essay before any second keeps the standard steady.

Choosing a rubric.

EDP 1119 Choosing a Rubric

Scoring guides range from simple to detailed. A checklist records whether each required element is present. A rating scale records how well each element is done. A holistic rubric assigns one overall score; it is fast but gives little feedback. An analytic rubric scores each criterion separately; it is slower but gives the most feedback. A science teacher scoring one hundred twenty lab reports in two days for a quick progress check uses a holistic rubric; for the final report, where students need to know what to improve, an analytic rubric scoring hypothesis, method, data and conclusion fits better. Shared in advance, a rubric also teaches.

Performance assessment.

EDP 1120 Performance Assessment

Performance assessment asks students to carry out complex tasks, judging process and product, as students design and run an experiment scored on procedure and report; authentic assessment resembles real-world tasks, and alternative assessment is any format beyond paper and pencil. It captures complex outcomes but costs time and is harder to score reliably.

Portfolios.

EDP 1121 Portfolios

A portfolio is a purposeful collection of student work, varying on four dimensions: for or of learning, progress or current accomplishment, showcase or documentation, and working or finished. A showcase portfolio holds best essays, a documentation portfolio drafts showing growth. Portfolios support self-reflection but cost time, and Vermont's program had low interrater reliability.

Tests that teach.

EDP 1122 The Testing Effect

Frequent tests promote learning and reduce cramming: the testing effect means retrieving information strengthens memory more than restudying does over the long run. Five minutes after study, restudying looks better; days later, students who were tested remember substantially more. Tests are learning events, not only measurements.

Seven entries on assessment, motivation and grading: beliefs about ability, feedback, self-assessment, teacher efficacy, and grading choices.

What assessment says about ability.

EDP 1123 Fixed and Incremental Views in Assessment

Teachers who hold an incremental view of ability, that it grows with effort, assess in ways that emphasize improvement, while fixed views and interpersonal competition produce few winners and shift attention to winning; posting only the top scores tells most students the contest is lost. Assessment sends messages about ability, intended or not.

Assessments that motivate.

EDP 1124 Assessments That Motivate

Motivating assessments have clear, attainable criteria, meaningful tasks, some choice and achievable challenge, as students choose an essay or a recorded talk scored on one rubric; choice supports self-determination but can lower reliability when students do different tasks. Challenge motivates only when success is within reach.

Effective feedback.

EDP 1125 Effective Feedback

Effective feedback is timely and specific about what was done well and what needs work, is based on a rubric, keeps grades private and de-emphasized, and praises the process, not the person: your evidence in paragraph two is strong; your conclusion needs a counterargument. You're so smart can backfire.

Wise feedback.

EDP 1126 Wise Feedback

Wise feedback, studied by Cohen, Steele and Ross, combines high standards, specific criticism and an assurance that the student can meet the standard, as in I hold high standards, and I know you can reach them, a strategy researched for building trust across racial lines. It describes feedback, not any group of students.

Self- and peer assessment.

EDP 1127 Self- and Peer Assessment

Self- and peer assessment give students a share of judging their work, within a climate focused on learning goals rather than grades, and build the ability to monitor their own progress, as students hold up green, yellow or red cards to show their confidence with a new skill. Clear criteria make these judgments meaningful.

Teacher efficacy.

EDP 1128 Teacher Efficacy

Teacher efficacy is a teacher's belief in their capacity to affect student learning; efficacious teachers treat assessment results as information about their own teaching, while others attribute results to fixed traits or home life, so after poor quiz results one teacher reteaches and another blames the students' background.

Grading choices.

EDP 1129 Grading Choices

Grading requires choices about weighting components and whether to include social skills, effort and improvement; mixing achievement with such factors produces hodgepodge grading, a grade that means different things for different students, as when two students earn a B, one for strong work, one for effort on weak work.

Look at what you can now do. You can tell diagnostic assessment, before instruction, from formative, during it, and summative, after it. You can choose the validity evidence a use of scores calls for: content for mastery, construct for an ability, criterion for prediction. You can estimate reliability several ways and read every score as a band around the number. You can fix a flawed item so that only knowledge decides the answer. And you can choose a rubric for a purpose: holistic for speed, analytic for feedback. Part two turns to standardized tests, statistics and scores.