Yet Another Deceptive Statistic

Let me just tell you upfront that this deception is MUCH WORSE than the falsehood about "only 17% of graduates meet college entrance requirements". This is much worse than that.

There are two numbers on the school reports that do not mean what you think they mean. These numbers do not mean what the District says they mean. These numbers are COMPLETELY misleading. The more I think about them, the more convinced I become that they don't mean anything at all. Yet they are two of the most important numbers on the report and they determine half of the school's overall grade.

On the school report, the numbers are labelled "Students making gains on the state reading test" and "Students making gains on the state math test".

Look at the report for Beacon Hill Elementary. You will see that it says that 69% of students made gains on the state reading test and 67% of students made gains on the state math test. That sounds pretty good, doesn't it?

What do you think it means?

I'll tell what I thought it meant. I thought it meant that 69% of the students at Beacon Hill got a higher raw score on the MSP this year than they got last year. If a student scored 410 last year and scored 420 this year, then that student counted as one who made a gain on the state test. Is that what you thought the number meant? What else could it mean? Isn't that the percentage of the students making gains on the state test? Well that is NOT what the number means.

In fact, if you take a look further down on the page you will see that students in every category passed the state reading test in lower numbers than last year.

How can it be that the pass rates dropped when 69% of students made a gain?

And here's something weird: the district averages are 66% every year in every category at every grade. That's kinda weird, isn't it?

The answer to both mysteries lies in the derivation of the number labelled "Students making gains on the state reading test" and what - if anything - it really means.

WARNING - HEAVY MATH TALK AHEAD.
Honestly, it's probably enough for you to know that the schools' numbers just don't mean what they appear to mean and that the District number is rigged to always come out to 66%. If you need to know more, try this sentence for coherence:

The student gain number represents the percentage of students who out-performed the 33rd percentile of other students in the district who scored the same as they did in the previous year.

Here's how it works.
If a student scored 410 last year then their score this year is ranked with all of the other scores from students who scored 410 last year. If that ranking is better than the bottom third, then that student counts as having a gain. The student might actually have scored worse than he or she scored last year, but - so long as they outscore one-third of the students who got the same score as them last year - they count as having made a gain.

That's incredibly deceptive.

Here are some interesting consequences of using this as the measure for growth:

1) Since 66% of the students will ALWAYS finish above the bottom third in any ranking, the District average will ALWAYS work out to 66%. That means that the superintendent will always get to make it appear as if year after year 66% of the students in the district made gains on the state tests. Even in years such as this one when pass rates were down, it will appear that 66% of the students across all grades and subjects made gains on the state tests.

2) The choice of the 33rd percentile as the bar is completely arbitrary. The District could have chosen the 50th percentile, in which case it would appear that only 50% of district students made gains on the state tests each year. Of course they could have chosen the 17th percentile, in which case it would always appear as if 83% of the students made gains on the state tests.

3) This is a zero net-sum game. If at some school they have 70% of the students beat the 33rd percentile then some other school will have only 62% of the students beat the 33rd percentile. There will always be 33% of the students who fail to beat the 33rd percentile. A gain for one school will come at the expense of others. They cannot all do really well.

4) In a year like this, when test scores are down across the District, the 33rd percentile will have a negative delta. That means that some portion of the students who are counted as having made a gain on the state test will actually have gotten a lower score than in the previous year.

Here's what's driving me crazy: why couldn't the District have simply reported the percentage of students who got a higher score than in the previous year?

To their credit, the District does disclose the meaning of this statistic. Here's how they explain it:

% of students at or above the 33rd percentile of growth from year to year on the state test using the Colorodo Growth Model. The Colorado growth model estimates a "growth percentile" for each student with at least two years of test data by creating a peer group of all students in the same grade who had a similar test history, and then rank this group of students by their test scores in the current year.

The District says that this number is important because "All students should be making progress every year and be on their way to meeting or exceeding standards." All students SHOULD be making progress, but this calculation guarantees that 33% of them will not be counted as making progress - whether they do or not - and that 66% will be counted as making progress - whether they do or not.

Is anyone really okay with this?

Let's remember that school segmentation, whether a school is Level 1, Level 2 or Level 3 is based, in part, on this "growth" measure. Every student at Beacon Hill Elementary doesn't know it, but they are in a race with every other student who got the same score as them on last year's test. And if they don't finish ahead of the bottom third, their school will drop down in the segmentation and be subject to more District-level control and interventions.