Tag Archives: schools

In my last post I looked at student growth percentiles, a new kind of data that the Washington State Office of Superintendent of Public Instruction. This post follows up by going into more detail about how SGPs are calculated. Back to the simple video with cartoon Anthony!

Cartoon Anthony got an MSP reading score of 344 in 3rd grade and a score of 381 in 4th grade. His score grew in that time. To get his SGP, they took the amount of growth and compared it to a group of 3rd grade students who began with that same score. His score of 381 was higher than 80% of the students in his comparison group, so he gets an SGP of 80. That shows “high growth.”

I’m going to set aside for the moment a couple of really large questions. First, exactly what does the MSP reading score measure? Second, what would have happened to the SGP if Cartoon Anthony was instead compared to a different set of students?

I’m just going to pretend those other questions don’t exist and ask, “How accurate is the SGP?” There are different ways to look at the answer to this question, but I’ll just give the easiest one here. Hang on to your hats, because I’m going to use just an eentsy weentsy bit of algebra. I’m going to use the variable x to indicate the absolute measurement error of a single MSP score. I’d use the real number, but I don’t know it and I couldn’t find it online anywhere. 

The change in Anthony’s MSP scores is 381-344, or 37.

The absolute measurement error for each test is x.

Using the simple method (that doesn’t rely on calculating standard deviations), the absolute measurement error for the change in scores is 2x.

So let’s suppose x is 10. That would make the absolute error 20, which means the change in scores could be anything from 17 to 47. Pretty big difference there!

If the measurement error is high enough, it makes Anthony’s SGP meaningless.

Is it meaningless? I don’t know. But the question needs asking.

In early December, the Washington State Office of Superintendent of Public Instruction (OSPI) announced that it would be making student growth percentiles available to school districts. Parents will be able to see the student growth percentile for their individual children.

What’s a student growth percentile? Good question. My answer is:  magic fairy dust. The OSPI’s answer is:

A student growth percentile (SGP) describes a student’s growth compared to other students with similar prior test scores (their academic peers)

What are they talking about? Height? Weight? No. “Academic growth.”

Well, then you have to ask, “What kind of academic growth? Measured how? And how accurately?”

You’ll find answers to the first two questions, but not the third.

They’re defining “academic growth” as a change in test scores on the state tests. They’re measuring it by a rather complex, involved process. It’s worth looking under the hood here. Maybe we need to go find a mechanic to ask whether it’s a working machine or a convoluted Rube Goldberg device with a fault in the middle? Maybe we need a crash course in statistics. Or a friend who understands this stuff.

Rather than trying to explain their Rube Goldberg device, the OSPI gives us a video with a fairly simple explanation. Too simple, really. They take a cute cartoon kid, Anthony, and show us that last year his 3rd grade MSP reading assessment score was 344, and that this year his 4th grade score 381. But is that amount of growth more, less, or the same as his peers? To answer that question, they take a bunch of kids who had the same 3rd grade score and compare the growth. Turns out that although Anthony is still behind, he had more academic growth than his peers.

Fairy dust. Did he really have more academic growth? Or did he maybe forget to eat breakfast in the third grade, making that score unusually low? That would compare him with a whole different set of academic peers. The problem here is that the OSPI is pretending that those two scores are accurate measures of how well Anthony can read. But they’re not. They’re both estimates.

Student Growth Percentiles are made by mashing up two estimates in really complicated ways. What is that, twice the inaccuracy? Or worse?

Let’s dig into the details here by looking at “A Technical Overview of the Student Growth Percentile Methodology: Student Growth Percentiles and Percentile Growth Projections/Trajectories.” I’m sorry. I really am. Math can be scary. But it’s like reading the fine print in a contract: you have to do it if you want to know what you’re getting. At the very least, head on over to page 16, under “Student Growth Percentile Estimation.” It says:

“Calculation of a student’s growth percentile is based upon the estimation of the conditional density associated with a student’s score at time t using the student’s prior scores at times 1, 2, . . . , t-1 as the conditioning variables.”

It is an estimate. How come the OSPI doesn’t say that on its website? Do they think we’re too stupid to understand? Or do they not understand?

Either way, it’s a bad sign. Because they’re going to be using the SGPs to evaluate teachers. Without looking under the hood. Teachers’ jobs will be on the line, so somebody has to look under the hood! Which means we have to learn some statistics. Oh dear!
Luckily, they have comic books for that nowadays. Here we go:

The Cartoon Guide to Statistics