Let me begin by saying that I would absolutely love to be able to use some objective measure of student academic growth as a measure of teacher effectiveness. I sincerely would. I am open to hearing about such a measure that can isolate the growth of student learning that is attributable to the teacher. I'm dying to hear about it. I just haven't heard it yet.
I won't pretend that the proposed measure is the student's cumulative acheivement. I will grant folks who want to use student test results as a measure of teacher effectiveness the sophistication that they are using a measure of change. I'll even grant them an additional level of sophistication that acknowledges the smaller potential for growth among students who are already near the top of the measure. The five point change from 45 to 50 is easier to coach than the five point change from 85 to 90. Just the same, the data and the conclusions have grave faults.
I review data for a living, so I have some familiarity - if not expertise - with how it can be misused. I have very little confidence in the data that I have seen that links student outcomes to teacher effectiveness. It makes a number of unsupportable assumptions. The primary one is to attribute all of the change in student outcomes to the teacher. That's crazy. The second one is to presume that all classes are statistically equivalent. That's also crazy. The third is to base the conclusions on a sample too small to be statistically significant (such as 30 students). The data is suspiciously reported without a margin of error. I find that odd and hubristically certain. Where is the standard deviation of outcomes? I haven't seen any reported. I'm always suspicious that unreported data is contradictory to the conclusion of the author. In this case I wonder if the range of outcomes isn't so great as to make the conclusion unsupportable, that the range of outcomes dwarfs the differences in outcomes.
Let's add some questions about the student assessment used. In Seattle the proposed measure is the MAP. The MAP was not intended for this purpose and is a poor choice. First is the primary function of MAP as a formative assessment. The MAP was designed to spark questions, not to provide answers. Second is the general unreliability of the MAP. We see student scores hop about, in part I suppose because the students can manipulate the experience. Finally we have the constraint that the MAP measures students along one dimension only, grade level progress.
When my eldest daughter first tested into Spectrum I had to ask about twelve people before I got a cogent description of the program. The one I finally got - and it is one that was confirmed by the program manager - was that Spectrum was about going beyond the standard curriculum in three ways: farther (to the next grade level expectations), deeper (a deeper understanding of the concepts), and broader (an understanding of the concepts in a wider variety of contexts). MAP only measures one of those dimensions, the one that I think I care least about: farther. Also, given the District's focus on vertical articulation and curricular alignment, teachers are actively discouraged from teaching students along this dimension. The District leadership actively discourages teachers from providing the students with the support that would really boost their MAP scores - instruction in the next grade level's curriculum.
Unfortunately the District has no other norm-referenced assessment of student academic achievement. The MSP and HSPE (former WASL) is criterion-referenced and therefore absolutely inappropriate for this purpose - unless the assessment is exclusively about getting under-performing students to Standard. Even then, it is inappropriate to use MSP scores for ranking so the only change that can be recorded is changes in Levels (1,2, 3, or 4) for students. This presumes, inappropriately, no progress for a student who did not change levels and loss for students who go down a level. Still, I would be more comfortable with the District using this gross tool as a measure since this test was actually designed for this purpose. It would still present the problems with an inadequate pool to form a statistically significant sample and it would still require the reporting of a margin of error. On top of that there all of the students who opt out. Of course the real problem with the MSP and the HSPE is the slow turnaround in the results and the unknown impact of summer learning (or loss).
So, in short, I'm not saying that I wouldn't love an objective measure of student growth as a measure of teacher effectiveness, but it would have to be better than the suggestions I have seen to date.
The critiques of using student test data to assess teacher effectiveness are well known. So where is there a response from the proponents of this idea to the legitimate concerns and objections? I don't see it.