Student Growth is WHAT?

In early December, the Washington State Office of Superintendent of Public Instruction (OSPI) announced that it would be making student growth percentiles available to school districts. Parents will be able to see the student growth percentile for their individual children.

What’s a student growth percentile? Good question. My answer is:  magic fairy dust. The OSPI’s answer is:

A student growth percentile (SGP) describes a student’s growth compared to other students with similar prior test scores (their academic peers)

What are they talking about? Height? Weight? No. “Academic growth.”

Well, then you have to ask, “What kind of academic growth? Measured how? And how accurately?”

You’ll find answers to the first two questions, but not the third.

They’re defining “academic growth” as a change in test scores on the state tests. They’re measuring it by a rather complex, involved process. It’s worth looking under the hood here. Maybe we need to go find a mechanic to ask whether it’s a working machine or a convoluted Rube Goldberg device with a fault in the middle? Maybe we need a crash course in statistics. Or a friend who understands this stuff.

Rather than trying to explain their Rube Goldberg device, the OSPI gives us a video with a fairly simple explanation. Too simple, really. They take a cute cartoon kid, Anthony, and show us that last year his 3rd grade MSP reading assessment score was 344, and that this year his 4th grade score 381. But is that amount of growth more, less, or the same as his peers? To answer that question, they take a bunch of kids who had the same 3rd grade score and compare the growth. Turns out that although Anthony is still behind, he had more academic growth than his peers.

Fairy dust. Did he really have more academic growth? Or did he maybe forget to eat breakfast in the third grade, making that score unusually low? That would compare him with a whole different set of academic peers. The problem here is that the OSPI is pretending that those two scores are accurate measures of how well Anthony can read. But they’re not. They’re both estimates.

Student Growth Percentiles are made by mashing up two estimates in really complicated ways. What is that, twice the inaccuracy? Or worse?

Let’s dig into the details here by looking at “A Technical Overview of the Student Growth Percentile Methodology: Student Growth Percentiles and Percentile Growth Projections/Trajectories.” I’m sorry. I really am. Math can be scary. But it’s like reading the fine print in a contract: you have to do it if you want to know what you’re getting. At the very least, head on over to page 16, under “Student Growth Percentile Estimation.” It says:

“Calculation of a student’s growth percentile is based upon the estimation of the conditional density associated with a student’s score at time t using the student’s prior scores at times 1, 2, . . . , t-1 as the conditioning variables.”

It is an estimate. How come the OSPI doesn’t say that on its website? Do they think we’re too stupid to understand? Or do they not understand?

Either way, it’s a bad sign. Because they’re going to be using the SGPs to evaluate teachers. Without looking under the hood. Teachers’ jobs will be on the line, so somebody has to look under the hood! Which means we have to learn some statistics. Oh dear!
Luckily, they have comic books for that nowadays. Here we go:

The Cartoon Guide to Statistics