Disqus

Thursday, March 28, 2013

Assessment Taskforce Update

 Update:  the Superintendent's latest message on MAP.   From his letter:

In the meantime, our spring assessments will be held from April 22 to June 7. Beginning this spring, the District recommends that students enrolled in an Algebra 1 course take the NWEA Algebra End-of-Course (EOC) exam instead of Math 6+ test.

Based on a preliminary review of MAP by staff, we’ve made the following adjustment to our testing policy: For 9th grade, only students below standard based on the state reading assessment will be required to take the MAP reading test. It will be optional for 9th-graders who are at or above standard in reading.


Reader Joan Sias attended Thursday's Assessment Taskforce meeting.  Here are her notes on the meeting.

The new MAP policy for 9th grade was handed out at the meeting. Not a single member of the TF asked what the process was for this policy decision. I am not a TF member, so I could not ask this question.

A handout was given that was a compilation of comments from TF members that had, in the past week, taken one or more MAP tests. TF members took these tests for purpose of learning more about the MAP test.

The comments showed that there are significant problems with this test.

I have heard many anecdotal stories of problems with individual test questions, with scope of questions (especially on the reading test), and on the adaption algorithm.

These are problems that - as far as I know - NWEA has not and will not admit to, and are very hard to document by people outside of NWEA.

This handout is the closest thing we have to cite documentation of the problems with these three aspects of the MAP test.

The TF has only three more meetings. One of these will be taken up with a visit by NWEA. Thus there are at best only two and a half more sessions for developing and finalizing recommendations.

The TF has not yet started a process for arriving a set of final recommendations. The only substantive discussion I have heard at the three meetings I have been two has been on the question of whether NWEA should or should not be invited to make a presentation. It was decided finally that NWEA would not be allowed to give a presentation to the TF, but would come to answer questions that were submitted to them (in advance) by the Task Force (via Eric Anderson).

Even in the most recent meeting, questions and comments from the TF members indicate that they still are not clear about what is the appropriate scope of their recommendations.


From Mirmac 1:

Assessment Task Force comments on MAP
SPS Spring MAP Testing Policy
Taskforce members and minutes
Principal Survey on MAP
Teachers of Tested Subject Survey Results

Thank you to both women for this information.

I will also note that at yesterday's Work Session on the Budget that there was a notation of a cost for implementing any recommendations from the Taskforce.  However, no sum of money was noted and it was in a list of items that they would like to fund but are unlikely to have to the money to fund.  I don't think the district is really serious about changing anything if only for lack of funds.   (More on the Budget in a separate thread.)

24 comments:

mirmac1 said...

The members of the TaskForce are effectively muzzled by the charter district staff/facilitator pushed on them.

I would love to hear what the TF is hearing and doing from any members.

Anonymous said...

It will soon be time for Spring MAP testing and the MSP. Are your opt-outs in place? If not, your school may assume that you only intended to opt out of the winter test.

Say goodbye to the library for another few weeks.

open ears

mirmac1 said...

Excellent Observation!

For you procrastinators:

Opt Out Form English

MAP Opt Out Vietnamese

Chinese Map Opt Out

MAP-Opt Out Oromo

Map Opt Out Form-somali

Please share these with family and friends who would like to see their children spending more time learning, than testing.

Anonymous said...

The Task Force meetings are open to the public and more observers should attend. There has not yet been a successful vetting of the MAP and it's very disheartening to see people try and make decisions on anecdotal information. The teacher and principal surveys are very informative and tell us that the test is not informing instruction or identifying students that were not previously identified as needing additional support. There is disagreement among the TF members and I don't see issues being resolved.
- concerned and watchful

uxolo said...

As a member of an advisory committee, I encourage you to be as outspoken as need be - this is not the time to be polite and wait. They'll bring in more district administrators to try to dampen any controversy, but there are undoubtedly members who think as you do, 'concerned and watchful."

topped it said...

One thing I wish the task force would do is force SPS and the NWEA to acknowledge publicly that there is a "top" to the MAP, and when the kids reach that top they should automatically be excluded from taking it from that point forward. This is not an unreasonable request in any way, and the kids' test scores are essentially guaranteed to drop at the point, so it doesn't benefit anyone for them to continue.

Can someone on the task force do this?

Anonymous said...

Has there been any discussion of the [mis]use of MAP as the sole determinant for 6th grade Algebra 1 placement in middle school?

I also think they need to acknowledge kids top out (especially with the reading), which makes the test of little value at some point, and especially poor for teacher evaluation purposes.

MAP skeptic

mirmac1 said...

concernedandwatchful,

Thank you for representing the needs of our students and teachers.

Taxpayer said...

Doesn't it disturb anyone that the test is not useful and yet (if the contract is renewed) money will continue to be wasted on it? This is our money, taxpayer money, wasted on a useless test. And we're not talking hundreds of dollars, either...

Anonymous said...

There is constant mention on the TF about the test not being used as intended but still the movement is going forward that it's the only test we have so even though it's already obsolete.... Principals are saying they know it's not valid but.... It's a very strange world in there. Please come and take note.

Concerned and watchful

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

Principals say they know it's not valid?

This is not what the principals survey says, in my judgment, and I don't recall hearing this as a common theme at the meetings I have attended.

I think the survey tells that the principals - particularly at middle school level - find the data useful. In contrast the teachers find relatively little value in the data.

I think this discrepancy has to do with the contrasting aggregation level of the data. Principals are generally looking at higher level of aggregation of MAP data. As aggregation level goes up, the statistical information from the MAP test becomes more powerful. Important placement decisions are made in middle school more than in elementary. I think the principals are using the data to help them make these decisions.

The teachers are mostly trying to use the data at a low level of aggregation, for formative assessment. In my opinion, based on looking into MAP deeply, I am not surprised in the least that the teachers are not finding the MAP data very useful. In fact, I would be surprised to find teachers that get much formative assessment value out of this product.

The value of the MAP data really only emerges as the data is aggregated.

The best use of the MAP data, I would argue, is for evaluating the academic effects of policy decisions, program placement, and curriculum pilots.

Unfortunately, District officials do not appear interested in making the best use of this MAP data.

Using the data for high stakes evaluation of teachers is a bad idea. HAving said that, I think there is a place for sharing among staff within a building each teacher's value added score. This data will encourage the weaker teachers to either strive to improve or to move on.

If a TF member were to ask the NWEA rep if the MAP data becomes more powerful and actionable as it is aggregated s/he would have to answer unequivocally "Yes."

For technical/statistical reasons, there is low actionability in the individual student data.

Joan

mirmac1 said...

Joan, if you wrote the posts at 12:08 and 12:15, please indicate so that they are not deleted! thx

suep. said...

Joan said (...) HAving said that, I think there is a place for sharing among staff within a building each teacher's value added score. This data will encourage the weaker teachers to either strive to improve or to move on."

Joan -- the whole notion of "value added" measures (VAM) is bogus. It is incredibly controversial and has not been proven to be accurate. There are many stories of teachers being inaccurately damned by VAM one year, then somehow deemed excellent the next, with wide swings in evaluations in between. Please don't advocate for the junk science of "VAM."

See:

Does Value-Added Assessment Ever Work? -- Diane Ravitch

Measuring the worth of a teacher?
L.A. Unified School District's Academic Growth Over Time measurement system, based on students' progress on standardized tests, spurs debate over fairness, accuracy.
-- L.A. Times


Evidence: Value-Added Assessment Is Junk Science
Linda Darling-Hammond and Edward Haertel of Stanford University explain why value-added assessment doesn’t work and how inaccurate it is.

Will John Deasy listen? Will the Gates Foundation listen?

Will the Los Angeles Times, which published their article, stop seeking names to publish inaccurate data about teacher “effectiveness”?


"Ravitch Says New Evaluation System Is 'Madness'"

Also, there is zero evidence that creating such tension and competition between teachers results in better teaching or mediocre teachers bowing out of the profession. This is another bogus notion of the corporate ed reformers, and it is another example of how the business model does not apply to a school. Teaching at its best is a cooperative, collaborative profession. Concepts like VAM, or public shaming of teachers by publicizing their so-called scores either in the newspaper (LA Times, for example) or within a school building simply poisons the profession and the work environment.

This, by the way, may be the true goal of the corporate ed reformers who are pushing these clearly damaging and failed concepts. Discouraging or harassing experienced (and unionized) teachers out of the profession would allow them to fill the field with the "young" and the malleable, non-union "teacher-leaders" they invest millions of dollars in promoting.

And lastly, perfectly mediocre teachers can be falsely credited with high student test scores, and can be shielded by this superficial "assessment" of their skills. I've seen it happen.

--Sue

Anonymous said...

Hi Sue,

I am on your side. I think I wasn't clear enough about what I was advocating.

I am not advocating high stakes uses of MAP data against teachers.

I KNOW that this is a misuse of test data. The National Sciences Academy has made this very clear.

I am NOT advocating for publically shaming teachers. I saw what happened in L.A. when teacher performance data was made public in the newspaper.

I am only saying that within the confines of a building's teaching staff, the shared knowledge about how each classroom is growing might lead to some teachers trying to improve their practice, all for the right reasons.


I am suggesting that some teachers have enough pride in what they do that they are motivated by real data that reveals a deficiency in their practice. Their pride leads them to try to get better results for their students.

I tutor a kid whose was getting no benefit from instruction at school. I could tell from working with the student.


I started seeing evidence at some point that my student was benifitting from classroom instruction. My best guess is that this teacher was embarrassed about MAP data, and decided to do something about it. She got help from the most effective teacher in the building.

This teacher wouldn't have faced any penalty had she done nothing (she was very close to retirement). I think it was her professional pride and clear evidence (from MAP) of falling short that motivated her to do something.

To me this is an example of a benefit of MAP data, and this doesn't involve high stakes.


I think a very good use of the data would be to allow the elementary teachers who are weaker at teaching math to opt out of teaching math. Wouldn't it be wonderful to not have to teach math if you are a math-phobic teachers, and leave it up to the several teachers in the building who love to teach it and teach it well?

This seems to me a positive, constructive, non-controversial, widely-embraceable beneficial use of MAP data.

I suspect that there are many teachers who would welcome the chance to teach more humanities and less math, and others who would love to teach more math and less humanities.

Why not allow elementary teachers, if THEY want it, to be able to focus their energy teaching the subjects they teach best?

I specifically mention elementary teachers, only because in elementary classroom teachers are expected to teach all subjects and not specialize.

Joan

Taxpayer said...

Let me speak to this idea that MAP shames teachers into performing better. It has been my experience just the opposite has happened in my building. Teachers are trying to teach to the MAP--which covers a wide, cursory range of math ideas with no in-depth assessment (what is radius but not what radius tells us about area, etc.)--and cramming a lot of surface knowledge down their students' throats. When students come to me they know all about PEMDAS but have no idea how to apply your most basic division concepts to real world application. In otherwords, they can tell you the order of operations in order to pass MAP but have no idea how to apply these operations to the world. What is happening is a quick, down and dirty memorization of math without the in-depth understanding. Nasty business, cramming for that particular test given what an awful test it is. And, really, placing my second grade child at a high-school reading level--really?! I had him try to read me Moby Dick with very little success. Really?!? Get serious!

Anonymous said...

Taxpayer - You give an important cautionary anecdote. What you describe may be very common.

SPS attaches high stakes for teachers to the MAP test.

This looks like an example of exactly why high stakes should NOT be attached to assessments.

I would be very happy if the Task Force recommended NO HIGH STAKES attached to Assessments. The TF was told up front, though, that they could not make any recs related to use of MAP data for Performance Management.

The example I gave in my earlier post was for a teacher for whom no high stakes existed. It was her last year before retirement when she brought in a mentor teacher and improved her practice. I call it pride, not shame, that motivated her.

Do you think if the high stakes were removed from the test the teachers in your building would stop teaching to the test?

As long as there was no financial reward or penalty associated with letting teachers specialize in your building such as I suggested, might the incentive to teach shallowly to the MAP test evaporate?

In my kids bldg, the librarian who proctors the test is now teaching poetry concepts. I figure its because she sees what's on the MAP test. Does the Reading MAP also have high stakes attached?

Joan

mirmac1 said...

Joan,

Would that the TF members who read this blog heed your recommendations, informed by years of in-depth analysis of MAP, and recommend to the other TF members to use this unique, if limited, opportunity to speak directly on MAP's efficacy and usage.

For those of you who may not know, Joan was my co-plaintiff in our 2009 suit against the district's unethical and improper acquisition of MAP. The question was deemed moot, because MGJ had already executed the renewal contract.

I argued the contractual and ethical issues, Joan used her specific knowledge of statistics and analysis to look at the good and bad about MAP. Thanks Joan.

Anonymous said...

Thank you Ms. M., for the compliment

Thinking more about Taxpayer's comment, I am doubting whether it is really possible to have the good (the data and its valuable uses) without the bad (highstakes, teaching to the test, cheating scandals...).

Even if the District had a policy of No High Stakes, would we still not have Principals putting pressure on teachers, leading to continued teaching-to-the-test and perhaps even worse (cheating), for the sake of the Principal's and school's recognition or good standing on the District Score Card?

I really do see some potentially wonderful benefits from adaptive test data, but alas, the district does not seem interested in making the best use of the data: To evaluate whether policy and programs are benefiting students.

It was after making such a plea at most recent Task Force Meeting - as well as advocating for the discontinuation of using MAP cut scores as basis for high stake purposes for students - that Clover informed me that I was no longer permitted to make comments/ask questions at the meeting.

Well, Clover can't keep me from writing about the Task Force (and hopefully having a constructive indirect influence on the Task Force) through this blog.

A major reason I shared technical information about the MAP over the last couple days is because I sense that there are many TF members that have little understanding of the MAP, and the organizers of the TF have not given a good orientation presentation on MAP.

I suggest that if folks have ideas for recommendations or have questions they would like answered by NWEA, share them here. Just maybe a TF member will follow up on suggestions they read about here.

Anonymous said...

Repeat of 3/31/13, 11:32 AM (forgot to sign, will be deleted)

Thank you Ms. M., for the compliment

Thinking more about Taxpayer's comment, I am doubting whether it is really possible to have the good (the data and its valuable uses) without the bad (highstakes, teaching to the test, cheating scandals...).

Even if the District had a policy of No High Stakes, would we still not have Principals putting pressure on teachers, leading to continued teaching-to-the-test and perhaps even worse (cheating), for the sake of the Principal's and school's recognition or good standing on the District Score Card?

I really do see some potentially wonderful benefits from adaptive test data, but alas, the district does not seem interested in making the best use of the data: To evaluate whether policy and programs are benefiting students.

It was after making such a plea at most recent Task Force Meeting - as well as advocating for the discontinuation of using MAP cut scores as basis for high stake purposes for students - that Clover informed me that I was no longer permitted to make comments/ask questions at the meeting.

Well, Clover can't keep me from writing about the Task Force (and hopefully having a constructive indirect influence on the Task Force) through this blog.

A major reason I shared technical information about the MAP over the last couple days is because I sense that there are many TF members that have little understanding of the MAP, and the organizers of the TF have not given a good orientation presentation on MAP.

I suggest that if folks have ideas for recommendations or have questions they would like answered by NWEA, share them here. Just maybe a TF member will follow up on suggestions they read about here.

Joan

Taxpayer said...

I see nothing wrong with an adaptive test to inform teaching practice but NOT THIS TEST. This is not a very good test. Maybe because it is cheaper? Smarter Balance is coming out with an adaptive test and it would be worth investing in. The problem with MAP is that it is not tied to the Common Core Standards and there is no way to teach effectively to what it does claim to assess. Give me a test that adapts based on the core standards by grade level strand and that would be an effective test? Why? Because the standards are clear, clearly explained, examples of what are expected are given in abundance, and each grade level builds upon the next at a developmentally appropriate pace. I am also a teacher, if you haven't guessed. And, no, before MAP became high-stakes, when it was just a test, my colleagues still taught to it because once you see those scores, you really start to believe you might actually be a bad teacher because you didn't teach about radius and PEMDAS in second grade. ! I have also fallen prey to cramming reading and math material down students' throats, seen them suffocate from too much information, had students cry from being overwhelmed with too much too soon too fast. Trust me, MAP creates bad teaching. And the two absolutely most hideous teachers in our building have the highest MAP scores because they spend all day teaching math and reading, no writing, no science, no soc. studies, and they send hours worth of homework home. THat is NOT good teaching. The one teacher stopped doing her wonderful unit on the ocean and squid because it was not on the MAP. ! Give an adaptive test tied to the Common Core Standards or give no test at all. Give a test that is transparent, that makes abundantly clear what it is assessing, that gives ample examples of what it is assessing (the way the MSP does--many released test items--very transparent) or give no test at all.

Anonymous said...

Taxpayer Teacher,

You are persuasive. I hope the Smarter Balance test is more beneficial for students.

I am not a big fan of CCSS, though.

Perhaps we have a better test of poorer standards.

Would that the District would use the Smarter Balance data for evaluating programs and policy decisions.

(Example: Have Spectrum students benefited from mainstreamin? Are high-needs Special Education students benefiting from mainstreaming? Are kids benefitting from Readers/Writers Workshop...)

As with MAP, using the adaptive test data for program/policy/curriculum evaluation may turn out to be the most beneficial use of the Smarter Balance test data.

Do you like the idea - which can be implemented regardless of how/whether MAP is used - of letting teachers in a building specialize somewhat?

Anonymous said...

Taxpayer Teacher,

You are persuasive. I hope the Smarter Balance test is more beneficial for students.

I am not a big fan of CCSS, though.

Perhaps we have a better test of poorer standards.

Would that the District would use the Smarter Balance data for evaluating programs and policy decisions.

(Example: Have Spectrum students benefited from mainstreamin? Are high-needs Special Education students benefiting from mainstreaming? Are kids benefitting from Readers/Writers Workshop...)

As with MAP, using the adaptive test data for program/policy/curriculum evaluation may turn out to be the most beneficial use of the Smarter Balance test data.

Do you like the idea - which can be implemented regardless of how/whether MAP is used - of letting teachers in a building specialize somewhat?

Joan