Sunday, November 07, 2010

Signed, Annoyed

A thread was requested (please note I didn't write this thread but I believe the info is accurate). Weigh in on the MAP testing for use in Advanced Learning placement. (They used to use the WASL.) I think on the one hand most of the money for Advanced Learning goes to testing. It's just ridiculous that more of the money can't go to programming. However, using MAP may benefit those students who can use computers well.

From Annoyed's comments:

The change this year was that MAP testing would determine if students were qualified to take the CogAT, rather than the reverse, where students first take the CogAT, then if they have qualifying scores, they take an achievement test for math and reading.

In recent years, I believe the K-2 kids took the the Woodcock-Johnson test and the older kids took the ITBS test for reading and math achievement.

Now the MAP test is being used as a gatekeeper and preventing many students from taking the CogAT. There may be some justification for this in the higher grades, if MAP scores show a strong correlation to ITBS scores, but I have not seen evidence that MAP in the primary grades is a valid measure of a student's academic performance when testing for highly capable programs.

45 comments:

Charlie Mas said...

There have been a number of changes in the advanced learning programs.

While the collection of changes is diverse, they all have one thing in common: they will all lead to further disproportionate under-representation by students from low-income homes.

The use of the MAP test as a gatekeeper is one. Because it is a computer-based test, it heavily favors students, particularly in the early grades, who have and use a computer at home and puts a heavy bias against students, particularly in the early grades, who do not.

The District has eliminated Spectrum Young Scholars, a program specifically designed to find and nuture gifted students who have been privileged with exposure to academic opportunities.

The District no longer allows students to qualify for middle school Spectrum in either math or reading. Now, to get into the Spectrum LA/SS class, a student must demonstrate high achievement in math.

The District has eliminated middle school A.L.O.s. Isn't it odd that we recognize non-Spectrum and non-APP students working beyond grade level in elementary school and provide them with access to more challenging and rigorous academic opportunity there, but suddenly, that is no longer necessary when the students reach middle school?

And, on a continuing basis, the District absolutely refuses to offer any quality assurance or even quality assessment for advanced learning programs so that low-income students can be locked into programs that exist only on paper but not in real life.

Charlie Mas said...

I should note that all of these changes were made by Bob Vaughan, with the full knowledge and approval of Susan Enfield and Dr. Goodloe-Johnson, and with the tacit approval of every member of the Board.

Meg said...

I have some scattershot thoughts, so please bear with me.

I agree with Charlie that this is likely to strongly - and adversely - decrease the participation in advanced learning of children from low income homes, as well as children whose home language is not English. This is really, really not okay.

I wrote Bob Vaughan to ask why this decision was made, and it appears to be entirely because there was a significant surge in applications for testing this year. This year there were 5,100 applications, up from a long-standing usual of 2,300-3,000. I didn't think to ask if there was an increase last year, as well.

I plan on asking some more questions, but this appears to be an unfortunate reactive decision on the district's part. I would guess that the surge in testing has a lot to do with issues and uncertainty created by the New Student Assignment Plan, and applications will, as the plan (hopefully) stabilizes and improves, drop back to more normal levels, at least in proportion to enrollment.

Other districts do not appear to use MAP as a gatekeeper to advanced learning testing.

I don't know how much it costs to test each kid. I do know, based on a letter APP families received from SPS, that for every APP kid who rides the bus, the district gets about $3500 from the state (compared to $350 for regular education students).

I realize that the testing/bus revenue issues are, in fact, separate issues, but it's hard to ignore that on the one hand, the district is asking APP families to help them out, while on the other, SPS is looking at further chopping up that very program, this time at the high school level, and reducing access to all advanced learning programs.

Skeptinonymous said...

What bothers me the most is that the District is not truly looking at multiple measures to identify students for AL opportunities.

If MAP is used as the cutoff, without knowing the CogAT scores, has the District really fulfilled the intent of the law? The burden is now on families to provide the additional test data (at great cost) and go through the appeals process.

But if you opted your child out of the MAP, the District would need to administer the CogAT, since there would be no achievement data available, yes?

The draft report on District MAP data shows very spotty testing for K students. Where is the data that supports using MAP as a qualifier for this age group? I don't find it acceptable to tell families to just wait another year. Are the students truly not capable or is the test not valid for this use and this age group?

Melissa Westbrook said...

Skep, this is a ploy to get those parents who think the WASL or MAP is not much help for their child to take the MAP. I'll bet the answer to your question is either go get private testing (very expensive) or take the MAP if you want your child in Advanced Learning.

Maureen said...

Is there an APP path for kids who are outside of SPS (homeschooled, private or out of state)? If so, couldn't people who opt out of the MAP demand that their kids follow that same pathway?

Charlie Mas said...

The MAP was presented as a formative assessment to inform instruction.

Is it suitable for use as a placement test?

MissedTheDeadline said...

I'm a new parent to SPS (my child just started K). I knew she was a bright kid, but I was poking around on the SPS website last week, saw "The Source" and found out that my 5-year-old scored 99th percentile on reading and math on the MAP test. I found this out weeks after the deadline for advanced learning testing.

It seems to me like a big argument against using MAP for advanced learning is that parents who don't know the system (like me), won't even have any idea that they should apply for their child in October (before they even see those MAP scores).

Not that it really matters in my case, since she's at a school with no Spectrum, no ALO. But I would have liked to be made aware of the possibility for her.

jp70 said...

Missed the Deadline - Not sure what school your child goes to, but my kids' school had continual reminders in their newsletter about the spectrum deadline. It is frustrating that you could be excluded due to not being aware of the program. On another note, I'm not sure how appealing works when you don't even take the Cogat, but my guess is you may be able to appeal to be in spectrum based on those MAP scores without even private testing. I could be wrong, but I know several people who used MAP scores to successfully appeal last year. If you are interested, I would definitely look further into it. Good luck!

Diane said...

i'm wondering at this point what is going to happen to the kindergartners that were supposedly "qalified candidates" based on teacher input and/or parents gut feeling/past experience with older sibs in spectrum or app, yet did not meet the 85% cut off. I asked the adv learning office serveral weeks ago and they told me something was in the works, but as yet there was no process to appeal the "unqualified candidate" based on MAP. i'm hearing all sorts of stories of K MAP score craziness.

Melissa Westbrook said...

Missed, I am really surprised your child's teacher didn't say something to you.

There are three routes to applying for Advanced Learning; a teacher alerting you to the program believing it may be good for your child, seeing information at your school (bulletin board or information in the office) or hearing about it at a PTA meeting or in the parent newsletter.

But maybe it might be worth finding out if the district, in its information for kindergarteners and/or new students, has info on Advanced Learning. I mean no one can hold your hand and feed you all the information but if it isn't available in district info handed out or at your school, then something is wrong.

Anonymous said...

We had the same experience as Missed the Deadline.

As we later learned, there was open hostility toward the program from many parents, teachers, and even the principal. Several teachers tended not to recommend students for testing and one parent was actually discouraged from doing the testing, by a teacher. If there was notification, I certainly missed it. Perhaps it was in small print somewhere in a newsletter.

So I can empathize with Missed the Deadline and wouldn't assume it was a matter of needing hand-holding.

Unfortunately, there is no option to appeal if you missed the deadline and didn't apply.

-Been there

Joan NE said...

I have a couple of statistical reports from NWEA. In these reports, the NWEA does NOT give any statistics for K-2, because, they say in their own report, that the statistics in K-2 are not good. The K-2 scores are not reliable. I can look up the exact quote, if any one is interested.

Poor reliability of K-2 MAP results means that they should NEVER be used for a screen for APP and ALO.

Lisa said...

I think Missed the Deadline has an excellent point. A parent can be aware of advanced learning options without having any idea at all their child might qualify. A child who seems "bright" but is thriving in kindergarten is not crying out for testing, especially if the child's friends are bright so the parent has little comparison to work with. The deadline ought to be AFTER fall conferences, which provide the first formal chance for a parent to learn their child is actually exceptional and not just the apple of the parent's own eye.

And/or all children who score high enough on the MAP need to be recommended for testing automatically. If MAP is a gateway it should usher children in more than keep them out!

gavroche said...

Joan NE -- Yes, please post that NWEA statement. It sounds like a lot of time, money (and grief) could be saved if SPS would discontinue the apparently useless MAP test for K-2.

uxolo said...

Missed the Deadline: send an email to EACH School Board member, the Supt, Bob Vaughan, AND the Alliance for Education.

They all want you in the still-public system. If you take your little guy private, it hurts their budget. Even if you have no intention of leaving the system now, you may find that a typical classroom is not going to meet your child's needs.

This is another example of how the supt is actively changing the structure of our options for education in this city. It should signal a letter writing campaign for anyone reading about your situation - even if it is "hearsay" (for us blog readers), it is something the people at the top should hear about,

Lori said...

Charlie and others, correct me if I'm wrong, but didn't the district increase the number of Spectrum programs in certain parts of town recently? Plus, they've required several schools to now have formal ALO programs that haven't had them in the past. So more seats, more applications, right?

My point is that having 5100 applications this year may partly reflect the fact that the district increased AL opportunities. Parents at Bryant may not have tested in the past because they were happy at the school and wouldn't have moved to a Spectrum school or Lowell anyway, so why test? But now, there is something "new" called at ALO, and if you want your child in the "new" thing, you may have to test. Voila, more applications. It would be interesting to know where the surge in appls is coming from.

David said...

Anonymous wrote, "There was open hostility toward [advanced learning] from many parents, teachers, and even the principal. Several teachers tended not to recommend students for testing and one parent was actually discouraged from doing the testing, by a teacher."

Unfortunately, since schools are compared in part on standardized test scores, there is a strong incentive for parents, teachers, and principals to oppose sending their best scoring students off to some specialized program at a different school, even if it is the best thing for the student.

In general, the idea that the school's primary goal should be maximizing the educational opportunities and pace of learning of all students seems to be lost in this district.

RPM said...

Trying to keep it real here...I took my K daughter for testing on Saturday. I assume she is being tested based on her MAP scores, but both were not above the 85th percentile - only math was.

I ran into SEVERAL parents from our school (North Beach) with older kids. They received letters from the district encouraging them to go to testing based on last Spring's WASL.

Again, I am new here, and I agree with most everything on this blog, but if the above is true, the district is making efforts to test more kids than the posters here would have you believe.

Now, we'll just have to see what happens with the results of the testing.

MidBall said...

We still haven't received our letter. Daughter scored above the 85% in both tests, and we applied for the program before the deadline. Guess I should call the school district, wish there was a way to check online.

mirmac1 said...

Let's see what the SPS assessment expert thought about possible uses for MAP, both before and after pressure was brought to bear to use this perfect instrument for everything but wax the floors.

Interesting how concerned they are that doing anything that might " seriously damage the district's communication strategy on MAP, and buy-in from teachers. Got to keep up the appearance of informing instruction and all that.

MAP can do anything!

Jessica said...

Just to toss another idea into the mix, I know parents who were pleasantly surprised to receive a letter from the district -- before the APP deadline -- saying that their children's MAP scores indicated they might do well in APP and recommended APP testing. These parents hadn't considered APP before, but they are now.

So while there are clearly downsides to the MAP approach for some families (including for my kindergartner), there are upsides for other families.

Also, if applications for APP testing are up so dramatically, we might also consider that a positive endorsement by parents of the program -- not just a negative reaction against the new student assignment plan.

My daughter is new to APP this year, and many, many parents w/children in private school tell me they wish they weren't paying $25k/year for academics that most people don't seem to think are any better than APP.

Jessica said...

MidBall -- I am in the same boat and called this morning to find out why we hadn't received a letter. Apparently, they are scheduling tests by school and our school has not had tests scheduled yet.

She said to call back in a week if we haven't heard anything.

mirmac1 said...

Jessica, at risk of stating the obvious, I can't think of many parents who would say "Hell No, no %*@%%$ APP for my kid!"

What both versions of this memo show is a broad brush attempt at using an inferior product for purposes for which it was never intended. Inferior because it was never REALLY evaluated effectively against competitors. Is using MAP in this way a new cost-saving measure? LOL! oh gawd, I'm killing myself.

It also shows SPS talking outa both ends:
Let's put forth a message that MAP is good for our kids (inform instruction), while going down a different path altogether (evaluate teacher "effectiveness," SpecEd eligibility, summer school, math courses)!

Ironic in light of MGJ's ludicrous statement at the Board meeting.

"Assessing every student and placing them according to ability sounds like tracking and discrimination…and we’ve moved light years away from that."

Right, MAP is color-blind, mind-blind, curriculum-blind, the list goes on.

mirmac1 said...

BTW, anytime I read that something can be read as " a positive endorsement by parents", I suspect that SPS is trying to make a silk purse out of a sow's ear.

P.S. my last post was to Jessica at 11:18

Skeptinonymous said...

The email posted by mirmac1 has some interesting recommendations...

Jessica de Barros states that if MAP were to be used for purposes other than "informing instruction", then "more than one point of data - preferably 3 points - [should] be used when drawing conclusions about student performance."

For use in Advanced Learning, the permission was given to use MAP for the purposes of identifying students from under-represented groups. The assumption is that "MAP would be used along with other student performance data points."

In bold, it states "at least two other types of data must be available to support the decision-making process."

So, yes, multiple points are being considered for those that have other data - WASL, etc. - when recommending students for testing. Yet, the 85% MAP cut-off for additional testing is eliminating access to multiple data points when being considered for Advanced Learning opportunities. Which makes it a high-stakes test.

What about a K student that has no other data points besides MAP? Are they out of luck? Notice that the other uses include 6th and 9th grade math placement.

The First Arnold said...

"I have a couple of statistical reports from NWEA. In these reports, the NWEA does NOT give any statistics for K-2, because, they say in their own report, that the statistics in K-2 are not good. The K-2 scores are not reliable. I can look up the exact quote, if any one is interested."

I'd love to see this information. Don't forget, the district is planning on spending tens of millions of dollars on this system. I might add-these dollars will be taken out of classrooms.

Worth analyzing.

Jessica said...

Every family can make its own decision about whether they like, hate or whatever APP. I can only speak for our family: we're happy w/APP so far and see it as an opportunity for a better-than-average, free public education.

Public school has its complications. Lowell APP seems to be oversubscribed. TM APP seems to be under-subscribed in grade 1 but well-attended in older grades. It's definitely a quandary for everyone when "success" in APP enrollment puts pressure on budgets and schools' capacities.

Diane said...

the teacher input forms are not even due until the end of this week (for K only, grades 1+ deadline was earlier). if K's are getting test dates already, does that mean that MAP scores are the sole determinant if a child should be tested? or perhaps K adv learning testing is being phased scheduled and the k's are being tested more 1:1 rather than cattle call? with my first child, K's were tested the same day/location as older kids at their elementary school.

RPM i'm trying not to take offense at "keep it real here".. an unproven (other than not ready for statistical analysis in grades K-2) test is being used as gatekeeper to adv learning for Kindergarteners. the exam has little to do with propensity to learn, and don't even get me started on how 1 teacher can manage to proctor 30 kids in a computer lab where the kids can't even keep their headphones on staight and the district expects to have accurate reflection of knowledge in scores.

yes, it's wonderful the district is taking MAP scores and letting parents know perhaps they should test their child, but it shouldn't be used the other way around as a gatekeeper.

Lori said...

I agree with others that it's wonderful that they are using MAP to identify potentially gifted children who wouldn't have been identified otherwise. But I have a big problem that they are using it to deny testing to other students who provide other "evidence" of potential giftedness (ie, letters of recommendation from a teacher), particularly at early grades.

I hope this analogy isn't too strained, but a good screening test is used for "asymptomatic" people, so in this case, you can screen for potential giftedness even when there are no "symptoms" or clues that have been detected in other ways (ie, no application filed). Great, I support that, especially if the goal is to increase diversity and rely less on self-selection for testing.

However, once someone has "symptoms," you can't just rely on the screening test and ignore other data.

Imagine if a women went to her doctor with a painful lump in her breast and the doctor said "Well, I'm not going to order a mammogram until I run another test that has never been shown to be effective at detecting breast cancer. How about we do a colonoscopy and if that's positive, well then, maybe you can get a mammogram?"

This is absurd, right? But it's exactly what SPS is doing this year. MAP is not a validated tool for identifying giftedness in young children (right? can anyone refute that with evidence?). Families who submitted the application packet provided evidence of potential giftedness but the district won't apply the appropriate "diagnostic test" (CogAT) unless the kid gets past the arbitrary screening test first.

And for those who say, hey, you can try again next year, you have probably not lived with a miserable, bored, and unchallenged child for a year. It's not fun for anyone. I am very sensitive to this issue because I've seen the transformative power of a good program fit in my own family.

lalasea said...

I have a 6th grader and we received a letter suggesting we test him for advanced learning based on MAP scores. He was placed in advanced math, but we have since moved him out. I think there are so many other factors that need to be considered when placing kids in advanced learning programs. I have no doubt my son is "smart enough" to work at an advanced level, but based on his performance in advanced math I just don't think he is mature enough and motivated enough. He was miserable and was almost failing.

Strangely his MAP percentile ranking fell almost 20 points from Spring 2010 to Fall 2010, so he would not have qualified for the Cogat testing in any case.

My 3rd grader (who did not receive a letter) scored in the 95th percentile for reading and the 84th percentile for Math. He scored the exact same way for Fall 2009 and then his scores fell for Winter and Spring cycles. I think he's a bright boy, but not a candidate for advanced learning.

I think too much weight is being given to the MAP scores and to scores for one specific testing cycle. I think parents and teachers probably know best which students are the best candidates for advanced learning no matter what test scores show.

mirmac1 said...

lalasea, this is likely because, as a "norm-based" assessment, MAP utilizes a percantile rank based on other MAP clients; in other words, only other districts that were lame enough to buy MAP.

ArchStanton said...

uxolo: They all want you in the still-public system. If you take your little guy private, it hurts their budget.

No. No they don't.

Well, I suppose if you are willing to take what they are offering with a smile and a hearty "Thank you sir, may I have another", they may want you.

But, threaten to go private and all you'll hear is, "Don't let the door hit you in the arse on the way out", followed by the chirp of crickets. If you actually go private, don't expect anyone to call and ask what caused you to leave SPS in general or the APP program in particular.

MG-J has stated before that she doesn't see private schools as competition. SPS does not see families as clients or customers that need to be wooed. They don't care if you find a better product or service because there are many more who don't have that choice.

ArchStanton said...

Lori: But now, there is something "new" called at ALO, and if you want your child in the "new" thing, you may have to test.

Unless things have changed, testing is not required to be in an ALO. My understanding is that in theory it is more of an opt-in program. So, that shouldn't drive that many more families to testing - if anything it should be fewer, since they might get their needs met locally. At both Lowell and Thurgood Marshall last year, the non-APP classes were called ALO, but to my knowledge, the ALO classes weren't any different from any other Gen. Ed. class.

If you dig through the blog you'll find plenty of posts that lament the lack of standardization among Spectrum/ALO and call out many schools for being "Spectrum/ALO in name only". One of the attractions of APP was that in being self-contained at single sites, there wasn't ever any sense that one APP school was better or somehow "more" APP than another. Of course, that foundation has been eroded, as well.

Moose said...

"Unless things have changed, testing is not required to be in an ALO. My understanding is that in theory it is more of an opt-in program."

I wish that were true at my child's school.

The kids considered for the ALO pull outs at my child's school must be District identified as Spectrum or APP. It is important for all to understand that ALO comes with little oversight (schools must submit a plan for ALO but the District doesn't have any guidelines about what the program must/should have) AND the program, once certified (what a joke!) by the District comes with no money. No money to pay for pull outs (if that is the model) or any materials, etc. Thus, our school limits ALO opportunities to APP or Spectrum kids who have chosen to remain at the neighborhood school rather than move for a program.

In theory and in its origination, ALO was to be an opt-in program. In practice, I understand from talking to other parents that my child's school is not the only one that limits access to it.

Moose said...

All of the above is to say that families are driven to test for ALO services so that they can get kids needs met locally.

ArchStanton said...

@Moose,

That's disappointing to learn. If I were suspicious; I might suspect that MG-J/SPS were trying to bring about the slow demise of APP/Spectrum.

Now we have to test into ALO? That surely indicates that an ALO school is as rigorous as Spectrum/APP. Before long they can claim that every school can meet the needs of highly gifted students and disband APP/Spectrum altogether.

Another nail in the coffin *sigh*

hschinske said...

ArchStanton, what's relatively new about ALOs is that being in one WHILE ALSO QUALIFIED for APP or Spectrum preserves your eligibility for those programs (if there is no Spectrum or APP program at your school, that is). So a student at, say, West Woodland can qualify for APP, stay at the ALO at West Woodland, and then get into APP at Hamilton, without having to retest.

The Advanced Learning site says that ALO students must be either district-identified or teacher-identified -- there doesn't seem to be a full opt-in position. It sounds as though depending on the attitude of the teachers/administration at your school, you may not be able to qualify except via formal test scores.

Helen Schinske

Meg said...

I'm with Lori - I think it's a good thing to use MAP as a possible way to identify kids who might be a good fit in advanced learning programs, but not as a way to keep them out.

Anonymous said...

My kids' school has always required ALO students to be APP or Spectrum qualified. Definitely no opting in is permitted.


Another Parent

Joan NE said...

Sorry it took me so long to reply to requests for the NWEA statistics.

Here is a quotation form the NWEA technical manual.

[page 54] "It is generally known that the assessment of second graders is inconsistent for many reasons including the mixed reactions that these students have to taking multiple-choice test items. For these resaons, NWEA test users should expect lest consistency in the test scores of second-graders."

Technical Manual for the NWEA MAP and ALT Assessment (NWEA, 2003).

I received this manual by email from Jessica DeBarros. I was not able to find a copy of it on-line, so I cannot provide a link to it.


It must be that the consistency of test scores for K and G1 is even worse than for second graders. From this technical manaul we cannot learn just how bad are the statistics are for these two lower grades, because the statistics for K-1 are not reported in this technical manual.

It is probable that a high proportion of K-1-2 children did not obtain statistically valid MAP test scores.

A student's test is not valid if the SEM (standard error of measurement) for a particular test event falls outside the range of 1.0 and 5.5.

I wonder how many children did not get flagged for CogAT testing because their SEMs fell outside this range?

Joan NE said...

The NWEA Techical Manaul (2003) is available at this URL:

http://www.scribd.com/doc/41812304/NWEA-Technical-Manual-2003

ArchStanton said...

My kids' school has always required ALO students to be APP or Spectrum qualified. Definitely no opting in is permitted.

That may be true for individual schools (I just don't know), but I have a hard time believing that all of the non-APP students at Lowell and T. Marshall last year tested into Spectrum or APP. Unless I am mistaken, there was no Gen. Ed. at either of those schools - just APP and ALO. Supposedly, Lowell got a chunk of T.T. Minor's students and T.Marshall kept a chunk of it's previous students (although I'm sure there was a lot of churn).

Maybe last year was a transition year and they started testing everyone this year? If not it just points to the variability in what can be labeled ALO.

Bird said...

Unless I am mistaken, there was no Gen. Ed. at either of those schools - just APP and ALO.

I think you are mistaken. There is APP and a general ed program where children may participate in ALO at these schools. In SPS it's never general ed or ALO. It's always general ed and ALO. It's the same at every school.

Just because you have an ALO program doesn't mean that all the students participate. Likely very few participate.

Lowell's website says

ALOs serve two primary student groups: (a) district-identified students who are academically highly gifted/gifted, and (b) teacher-identified students who demonstrate skills and readiness for participation in an accelerated and rigorous curriculum that is based on Spectrum curricular guidelines.

So it's both test based and teacher identified at Lowell. That's the district-wide expectation, I believe. Check the advanced learning page and you'll see the same definition.

In practice, however, I bet schools who say the program is only for district identified students are just getting push back from teachers. If the teachers don't want to identify any additional students, effectively it's only offered for kids who test in.

Charlie Mas said...

ALOs were supposed to be inclusive programs (not pull-outs) and available on a self-selected basis (no testing required).

If schools are doing something else, that is just further evidence of failure by the Advanced Learning office.