WHAT
TESTS TELL US
Making Sense Out of Data
Holistic (single) test scores are useful for policymakers but are seldom
useful for teachers and administrators. For example, a reading score of
70% tells only what percentage of the questions was answered correctly;
it does not tell what skills the student has or how good the score is.
One way to make more sense of test data is to separate it into components
and convert the results to rubric scores.
Rather than rely on one global reading measure, the Pacific Communities
with High-performance In Literacy Development (Pacific CHILD) project
has developed a battery of early reading assessments in English, Chuukese,
Kosraean, Marshallese, Palauan, Pohnpeian, and Yapese. The assessment
battery includes Concepts About Print, Alphabet Sound/Symbol Recognition,
Phonemic Awareness, Listen and Retell, Sight Word Identification, Written
Story Construction, and the Open-Ended Reading Test from Harcourt Educational
Measurement. Altogether, 14 different measures are used to gauge students’
progress in learning to read.
The following four-step process explains how to make meaning out of test
data. It uses the Alphabet Sound/Symbol Recognition Assessment as an example,
but you can apply the method to a wide
range of assessment data.
1. Determine what tests measure.
The Alphabet Sound/Symbol Recognition Assessment measures how well a student
knows upper- and lowercase letters. Recognizing letters and knowing their
sounds is an important skill in learning how to read.
Administer the assessment to students individually, each on a separate
form (see Figure 1). A complete form would include all the letters of
the alphabet.
Summarize the data into two scores, upper- and lowercase, and use it to
identifythe specific letters a student knows well or has difficulty with.
This data is available and in your hands immediately after the assessment.
2. Set achievement levels.
The next step is to make the test scale – the total number of points
scored on the test – more meaningful. The English version of the
Alphabet Sound/Symbol Recognition Assessment has 26 possible points for
uppercase and 28 for lowercase. Lowercase has two more because the lowercase
“a” and “g”
have two different forms.
Now convert the scale into a rubric that has meaning. Pacific CHILD teachers and staff developed the rubric in Figure 2, setting “cut scores” that made meaning out of the data. Cut scores are the number ranges associated with each achievement level; the most important is the one that separates “Meets Standard” from “Below Standard.” For uppercase letters, a score of 24 or 25 met theachievement standard, 26 exceeded it, and 23 or lower did not meet it.
How do you determine cut scores? You need to set the “Meets Standard”
level first and then move on to the other levels. For the Alphabet Sound/Symbol
Recognition Assessment, grade level teachers came together and agreed
on the number of letters they expected 1st grade students to be able to
identify correctly by the end of the year – namely, 24 of 26 uppercase
letters and 26 of 28 lowercase letters. Next they set criteria for the
“Exceeds Standard” level, followed by the “Below Standard”
level.
Cut scores for “Below Standard” should identify students who
are catching on but still make frequent mistakes. Students in the “Well
Below Standard” category do not yet understand the concepts
being taught, which in this case means that they frequently do not associate
letters with names or sounds.
Notice that the scale is divided into four unequal achievement standards
based on the teachers’ expectations for student learning. The range
of points is not divided into four equal parts, nor is the scale segmented
by percentage scores. To do so would be to impose an arbitrary structure
that, while symmetrical, does not reflect teachers’ expectations
for student achievement.
3. Profile the class.
Step 3 creates the big picture of the class using the achievement standards.
Programs like Microsoft Excel are very handy for this step. Create a worksheet
for the class (see Figure 3) by entering each student’s name and
raw scores. Next, convert the raw scores to achievement level scores using
the rubric conversion
table (Figure 2).
These new achievement level scores range from 0-4. For these
six students, only one, Rita, met or exceeded the standard in uppercase
letters, and only Cathy met or exceeded the standard for lowercase
letters.
Next, tally the number of 4s, 3s, 2s, and 1s for upper- and lowercase.
Create a table indicating the number and percentage of students in each
achievement level, as in Figure 4.
What does Figure 4 tell you? You can see that 83.3% of the students do
not meet the standard for uppercase letters and 83.3% do not meet the
standard for lowercase letters. If you gave this assessment
early in the school year, you would know that nearly all students need
work on upper- and lowercase letters. If you gave the assessment at the
end of the year, you would know that only one student
met the standard for lowercase letters and only one student met the standard
for uppercase letters.
4. Group students for learning.
Figure 4 also shows that three students scored in the “Well Below
Standard” category for uppercase letters. These students might need
different instruction from the two students in the “Below Standard”
category. Two students also need in-depth instruction in lowercase letters,
while three students are beginning to identify them. The individual score
sheet you created in the first step pinpoints the letters students have
not yet learned. You can use this information to plan class instruction.
From the individual score sheets, you also know that Rita already knows
her uppercase letters, so there is no need for her to sit through more
instruction on this. Cathy knows her lowercase letters.
One of teachers’ most effective and least understood “power
tools” is the use of assessment information to inform instruction.
Through the four steps described above, you can quickly and easily put
assessment data to work. Both you and your students will benefit.
Don Burger, EdD, is the Program Director for Pacific Assessment Systems and Services.