Assessment Taskforce Update
Update: the Superintendent's latest message on MAP. From his letter:
In the meantime, our spring assessments will be held from April 22 to June 7. Beginning this spring, the District recommends that students enrolled in an Algebra 1 course take the NWEA Algebra End-of-Course (EOC) exam instead of Math 6+ test.
Based on a preliminary review of MAP by staff, we’ve made the following adjustment to our testing policy: For 9th grade, only students below standard based on the state reading assessment will be required to take the MAP reading test. It will be optional for 9th-graders who are at or above standard in reading.
Reader Joan Sias attended Thursday's Assessment Taskforce meeting. Here are her notes on the meeting.
The new MAP policy for 9th grade was handed out at the meeting. Not a single member of the TF asked what the process was for this policy decision. I am not a TF member, so I could not ask this question.
A handout was given that was a compilation of comments from TF members that had, in the past week, taken one or more MAP tests. TF members took these tests for purpose of learning more about the MAP test.
The comments showed that there are significant problems with this test.
I have heard many anecdotal stories of problems with individual test questions, with scope of questions (especially on the reading test), and on the adaption algorithm.
These are problems that - as far as I know - NWEA has not and will not admit to, and are very hard to document by people outside of NWEA.
This handout is the closest thing we have to cite documentation of the problems with these three aspects of the MAP test.
The TF has only three more meetings. One of these will be taken up with a visit by NWEA. Thus there are at best only two and a half more sessions for developing and finalizing recommendations.
The TF has not yet started a process for arriving a set of final recommendations. The only substantive discussion I have heard at the three meetings I have been two has been on the question of whether NWEA should or should not be invited to make a presentation. It was decided finally that NWEA would not be allowed to give a presentation to the TF, but would come to answer questions that were submitted to them (in advance) by the Task Force (via Eric Anderson).
Even in the most recent meeting, questions and comments from the TF members indicate that they still are not clear about what is the appropriate scope of their recommendations.
From Mirmac 1:
Assessment Task Force comments on MAP
SPS Spring MAP Testing Policy
Taskforce members and minutes
Principal Survey on MAP
Teachers of Tested Subject Survey Results
Thank you to both women for this information.
I will also note that at yesterday's Work Session on the Budget that there was a notation of a cost for implementing any recommendations from the Taskforce. However, no sum of money was noted and it was in a list of items that they would like to fund but are unlikely to have to the money to fund. I don't think the district is really serious about changing anything if only for lack of funds. (More on the Budget in a separate thread.)
In the meantime, our spring assessments will be held from April 22 to June 7. Beginning this spring, the District recommends that students enrolled in an Algebra 1 course take the NWEA Algebra End-of-Course (EOC) exam instead of Math 6+ test.
Based on a preliminary review of MAP by staff, we’ve made the following adjustment to our testing policy: For 9th grade, only students below standard based on the state reading assessment will be required to take the MAP reading test. It will be optional for 9th-graders who are at or above standard in reading.
Reader Joan Sias attended Thursday's Assessment Taskforce meeting. Here are her notes on the meeting.
The new MAP policy for 9th grade was handed out at the meeting. Not a single member of the TF asked what the process was for this policy decision. I am not a TF member, so I could not ask this question.
A handout was given that was a compilation of comments from TF members that had, in the past week, taken one or more MAP tests. TF members took these tests for purpose of learning more about the MAP test.
The comments showed that there are significant problems with this test.
I have heard many anecdotal stories of problems with individual test questions, with scope of questions (especially on the reading test), and on the adaption algorithm.
These are problems that - as far as I know - NWEA has not and will not admit to, and are very hard to document by people outside of NWEA.
This handout is the closest thing we have to cite documentation of the problems with these three aspects of the MAP test.
The TF has only three more meetings. One of these will be taken up with a visit by NWEA. Thus there are at best only two and a half more sessions for developing and finalizing recommendations.
The TF has not yet started a process for arriving a set of final recommendations. The only substantive discussion I have heard at the three meetings I have been two has been on the question of whether NWEA should or should not be invited to make a presentation. It was decided finally that NWEA would not be allowed to give a presentation to the TF, but would come to answer questions that were submitted to them (in advance) by the Task Force (via Eric Anderson).
Even in the most recent meeting, questions and comments from the TF members indicate that they still are not clear about what is the appropriate scope of their recommendations.
From Mirmac 1:
Assessment Task Force comments on MAP
SPS Spring MAP Testing Policy
Taskforce members and minutes
Principal Survey on MAP
Teachers of Tested Subject Survey Results
Thank you to both women for this information.
I will also note that at yesterday's Work Session on the Budget that there was a notation of a cost for implementing any recommendations from the Taskforce. However, no sum of money was noted and it was in a list of items that they would like to fund but are unlikely to have to the money to fund. I don't think the district is really serious about changing anything if only for lack of funds. (More on the Budget in a separate thread.)
Comments
I would love to hear what the TF is hearing and doing from any members.
Say goodbye to the library for another few weeks.
open ears
For you procrastinators:
Opt Out Form English
MAP Opt Out Vietnamese
Chinese Map Opt Out
MAP-Opt Out Oromo
Map Opt Out Form-somali
Please share these with family and friends who would like to see their children spending more time learning, than testing.
- concerned and watchful
Can someone on the task force do this?
I also think they need to acknowledge kids top out (especially with the reading), which makes the test of little value at some point, and especially poor for teacher evaluation purposes.
MAP skeptic
Thank you for representing the needs of our students and teachers.
Concerned and watchful
This is not what the principals survey says, in my judgment, and I don't recall hearing this as a common theme at the meetings I have attended.
I think the survey tells that the principals - particularly at middle school level - find the data useful. In contrast the teachers find relatively little value in the data.
I think this discrepancy has to do with the contrasting aggregation level of the data. Principals are generally looking at higher level of aggregation of MAP data. As aggregation level goes up, the statistical information from the MAP test becomes more powerful. Important placement decisions are made in middle school more than in elementary. I think the principals are using the data to help them make these decisions.
The teachers are mostly trying to use the data at a low level of aggregation, for formative assessment. In my opinion, based on looking into MAP deeply, I am not surprised in the least that the teachers are not finding the MAP data very useful. In fact, I would be surprised to find teachers that get much formative assessment value out of this product.
The value of the MAP data really only emerges as the data is aggregated.
The best use of the MAP data, I would argue, is for evaluating the academic effects of policy decisions, program placement, and curriculum pilots.
Unfortunately, District officials do not appear interested in making the best use of this MAP data.
Using the data for high stakes evaluation of teachers is a bad idea. HAving said that, I think there is a place for sharing among staff within a building each teacher's value added score. This data will encourage the weaker teachers to either strive to improve or to move on.
If a TF member were to ask the NWEA rep if the MAP data becomes more powerful and actionable as it is aggregated s/he would have to answer unequivocally "Yes."
For technical/statistical reasons, there is low actionability in the individual student data.
Joan
Joan -- the whole notion of "value added" measures (VAM) is bogus. It is incredibly controversial and has not been proven to be accurate. There are many stories of teachers being inaccurately damned by VAM one year, then somehow deemed excellent the next, with wide swings in evaluations in between. Please don't advocate for the junk science of "VAM."
See:
Does Value-Added Assessment Ever Work? -- Diane Ravitch
Measuring the worth of a teacher?
L.A. Unified School District's Academic Growth Over Time measurement system, based on students' progress on standardized tests, spurs debate over fairness, accuracy.-- L.A. Times
Evidence: Value-Added Assessment Is Junk Science
Linda Darling-Hammond and Edward Haertel of Stanford University explain why value-added assessment doesn’t work and how inaccurate it is.
Will John Deasy listen? Will the Gates Foundation listen?
Will the Los Angeles Times, which published their article, stop seeking names to publish inaccurate data about teacher “effectiveness”?
"Ravitch Says New Evaluation System Is 'Madness'"
Also, there is zero evidence that creating such tension and competition between teachers results in better teaching or mediocre teachers bowing out of the profession. This is another bogus notion of the corporate ed reformers, and it is another example of how the business model does not apply to a school. Teaching at its best is a cooperative, collaborative profession. Concepts like VAM, or public shaming of teachers by publicizing their so-called scores either in the newspaper (LA Times, for example) or within a school building simply poisons the profession and the work environment.
This, by the way, may be the true goal of the corporate ed reformers who are pushing these clearly damaging and failed concepts. Discouraging or harassing experienced (and unionized) teachers out of the profession would allow them to fill the field with the "young" and the malleable, non-union "teacher-leaders" they invest millions of dollars in promoting.
And lastly, perfectly mediocre teachers can be falsely credited with high student test scores, and can be shielded by this superficial "assessment" of their skills. I've seen it happen.
--Sue
I am on your side. I think I wasn't clear enough about what I was advocating.
I am not advocating high stakes uses of MAP data against teachers.
I KNOW that this is a misuse of test data. The National Sciences Academy has made this very clear.
I am NOT advocating for publically shaming teachers. I saw what happened in L.A. when teacher performance data was made public in the newspaper.
I am only saying that within the confines of a building's teaching staff, the shared knowledge about how each classroom is growing might lead to some teachers trying to improve their practice, all for the right reasons.
I am suggesting that some teachers have enough pride in what they do that they are motivated by real data that reveals a deficiency in their practice. Their pride leads them to try to get better results for their students.
I tutor a kid whose was getting no benefit from instruction at school. I could tell from working with the student.
I started seeing evidence at some point that my student was benifitting from classroom instruction. My best guess is that this teacher was embarrassed about MAP data, and decided to do something about it. She got help from the most effective teacher in the building.
This teacher wouldn't have faced any penalty had she done nothing (she was very close to retirement). I think it was her professional pride and clear evidence (from MAP) of falling short that motivated her to do something.
To me this is an example of a benefit of MAP data, and this doesn't involve high stakes.
I think a very good use of the data would be to allow the elementary teachers who are weaker at teaching math to opt out of teaching math. Wouldn't it be wonderful to not have to teach math if you are a math-phobic teachers, and leave it up to the several teachers in the building who love to teach it and teach it well?
This seems to me a positive, constructive, non-controversial, widely-embraceable beneficial use of MAP data.
I suspect that there are many teachers who would welcome the chance to teach more humanities and less math, and others who would love to teach more math and less humanities.
Why not allow elementary teachers, if THEY want it, to be able to focus their energy teaching the subjects they teach best?
I specifically mention elementary teachers, only because in elementary classroom teachers are expected to teach all subjects and not specialize.
Joan
SPS attaches high stakes for teachers to the MAP test.
This looks like an example of exactly why high stakes should NOT be attached to assessments.
I would be very happy if the Task Force recommended NO HIGH STAKES attached to Assessments. The TF was told up front, though, that they could not make any recs related to use of MAP data for Performance Management.
The example I gave in my earlier post was for a teacher for whom no high stakes existed. It was her last year before retirement when she brought in a mentor teacher and improved her practice. I call it pride, not shame, that motivated her.
Do you think if the high stakes were removed from the test the teachers in your building would stop teaching to the test?
As long as there was no financial reward or penalty associated with letting teachers specialize in your building such as I suggested, might the incentive to teach shallowly to the MAP test evaporate?
In my kids bldg, the librarian who proctors the test is now teaching poetry concepts. I figure its because she sees what's on the MAP test. Does the Reading MAP also have high stakes attached?
Joan
Would that the TF members who read this blog heed your recommendations, informed by years of in-depth analysis of MAP, and recommend to the other TF members to use this unique, if limited, opportunity to speak directly on MAP's efficacy and usage.
For those of you who may not know, Joan was my co-plaintiff in our 2009 suit against the district's unethical and improper acquisition of MAP. The question was deemed moot, because MGJ had already executed the renewal contract.
I argued the contractual and ethical issues, Joan used her specific knowledge of statistics and analysis to look at the good and bad about MAP. Thanks Joan.
Thinking more about Taxpayer's comment, I am doubting whether it is really possible to have the good (the data and its valuable uses) without the bad (highstakes, teaching to the test, cheating scandals...).
Even if the District had a policy of No High Stakes, would we still not have Principals putting pressure on teachers, leading to continued teaching-to-the-test and perhaps even worse (cheating), for the sake of the Principal's and school's recognition or good standing on the District Score Card?
I really do see some potentially wonderful benefits from adaptive test data, but alas, the district does not seem interested in making the best use of the data: To evaluate whether policy and programs are benefiting students.
It was after making such a plea at most recent Task Force Meeting - as well as advocating for the discontinuation of using MAP cut scores as basis for high stake purposes for students - that Clover informed me that I was no longer permitted to make comments/ask questions at the meeting.
Well, Clover can't keep me from writing about the Task Force (and hopefully having a constructive indirect influence on the Task Force) through this blog.
A major reason I shared technical information about the MAP over the last couple days is because I sense that there are many TF members that have little understanding of the MAP, and the organizers of the TF have not given a good orientation presentation on MAP.
I suggest that if folks have ideas for recommendations or have questions they would like answered by NWEA, share them here. Just maybe a TF member will follow up on suggestions they read about here.
Thank you Ms. M., for the compliment
Thinking more about Taxpayer's comment, I am doubting whether it is really possible to have the good (the data and its valuable uses) without the bad (highstakes, teaching to the test, cheating scandals...).
Even if the District had a policy of No High Stakes, would we still not have Principals putting pressure on teachers, leading to continued teaching-to-the-test and perhaps even worse (cheating), for the sake of the Principal's and school's recognition or good standing on the District Score Card?
I really do see some potentially wonderful benefits from adaptive test data, but alas, the district does not seem interested in making the best use of the data: To evaluate whether policy and programs are benefiting students.
It was after making such a plea at most recent Task Force Meeting - as well as advocating for the discontinuation of using MAP cut scores as basis for high stake purposes for students - that Clover informed me that I was no longer permitted to make comments/ask questions at the meeting.
Well, Clover can't keep me from writing about the Task Force (and hopefully having a constructive indirect influence on the Task Force) through this blog.
A major reason I shared technical information about the MAP over the last couple days is because I sense that there are many TF members that have little understanding of the MAP, and the organizers of the TF have not given a good orientation presentation on MAP.
I suggest that if folks have ideas for recommendations or have questions they would like answered by NWEA, share them here. Just maybe a TF member will follow up on suggestions they read about here.
Joan
You are persuasive. I hope the Smarter Balance test is more beneficial for students.
I am not a big fan of CCSS, though.
Perhaps we have a better test of poorer standards.
Would that the District would use the Smarter Balance data for evaluating programs and policy decisions.
(Example: Have Spectrum students benefited from mainstreamin? Are high-needs Special Education students benefiting from mainstreaming? Are kids benefitting from Readers/Writers Workshop...)
As with MAP, using the adaptive test data for program/policy/curriculum evaluation may turn out to be the most beneficial use of the Smarter Balance test data.
Do you like the idea - which can be implemented regardless of how/whether MAP is used - of letting teachers in a building specialize somewhat?
You are persuasive. I hope the Smarter Balance test is more beneficial for students.
I am not a big fan of CCSS, though.
Perhaps we have a better test of poorer standards.
Would that the District would use the Smarter Balance data for evaluating programs and policy decisions.
(Example: Have Spectrum students benefited from mainstreamin? Are high-needs Special Education students benefiting from mainstreaming? Are kids benefitting from Readers/Writers Workshop...)
As with MAP, using the adaptive test data for program/policy/curriculum evaluation may turn out to be the most beneficial use of the Smarter Balance test data.
Do you like the idea - which can be implemented regardless of how/whether MAP is used - of letting teachers in a building specialize somewhat?
Joan