Check Yourself Screener for Middle School Youth; Cracks Starting to Show
I have been reporting on the issue of the middle school mental health screener, Check Yourself, being used in some SPS middle schools as well as middle school throughout King County. It comes to districts via King County's levy, Best Starts for Kids, and involves screening students for issues and then referring them, via a practice called SBIRT (Screening, Brief Intervention, and Referral to Treatment) to treatment. The screener, which longer than almost any other of its type, is also not been validated. My earlier stories are here and here.
There are new developments.
One is a story out of Oregon where Portland State University researchers doing a similar kind of screening to King County's, were outed by a whistleblower in the form of a grad student in the teaching program. Portland State University houses Reclaiming Futures and they are King County's SBIRT program contractor. Here's the story from OPB, dated March 10, 2018:
Also to note, the screener has been adjusted since my partner and I have started raising questions.
The ability to skip some questions was added (previously students could skip zero questions), a bigger notice on the home page that a student can opt out was added, a question about religious service attendance by family was taken out.
What that means, if you know anything about basic research, is that ANY conclusions they draw will be invalid. If you change the structure of the testing device AFTER you start using it, you cannot make any solid claims. And, the fact that each district is using it on different populations (by grade or by behavior) also would make any claims less than stellar.
Lastly, on the Oregon story in July 2018 via Willamette Weekly:
Finally, I attended the Curriculum & Instruction Committee meeting yesterday (primarily to report back on the Science adoption - separate thread to come) and there on the agenda was a review of Board policy#3232 on Parent & Student Rights in Administration of surveys, analysis or evaluations.
Head of Research and Evaluations, Dr. Eric Anderson, said they would like to make some changes to the policy. That the Check Yourself screening tool didn't really fall into any of these categories and they want to "shore up the policy with a framework for parents." Basically, there are differences between research, surveys and screeners especially in groups via individuals.
My spidey radar tells me that the district would not be bringing this to the Board if they were not worried. And they should be. (They said they couldn't get this work done in time for a Board meeting in April but would by May.)
What was interesting to me is that Director Pinkham - twice - asked about students and their ability to opt out. He said, "What if the counselor or teacher is telling you to do this? It's hard for a kid to say no to the teacher who is the authority." Right he is.
And, again, embedded in that policy is the right for parents to examine any survey BEFORE it is given and/or opt their student out.
But you can do neither action when you don't know of the existence of said thing.
The policy must be amended so that if the district wants to do this kind of work, parents have the opportunity - in clear ways - to know about it.
Districts, governmental entities and higher education cannot run roughshod over student data privacy rights.
There are new developments.
One is a story out of Oregon where Portland State University researchers doing a similar kind of screening to King County's, were outed by a whistleblower in the form of a grad student in the teaching program. Portland State University houses Reclaiming Futures and they are King County's SBIRT program contractor. Here's the story from OPB, dated March 10, 2018:
In allegations first reported in Willamette Week this week, grad student Ezra Whitman accused professors in Portland State University’s Graduate School of Education of requiring students to break the Family Education Rights and Privacy Act, or FERPA, by collecting students’ race, gender and other data.
PSU professors acknowledge the assignment pushed boundaries and delved into sensitive areas around race and student achievement, but contend there was nothing improper in the assignment.
The focus on “equity” called for students to document performance on the pre-test and throughout the subsequent lessons, in a chart which included student ethnicity, income and special education status — all protected classes under federal law. Faculty say they counseled grad students to remove children’s names, to maintain data security.That last paragraph? Exactly what is happening with Check Yourself. King County plans to publish in "peer-reviewed" journals and present at conferences (but SPS and King County both claim the data gathered is not "research"). As well, how districts carry out the use of Check Yourself is different from district to district.
The use of private student data is complicated by two other factors: The PSU professors plan to publish research in a national academic journal based on the collected data; and the legal permissions appear different, based on what school district in which the graduate students are teaching.
Also to note, the screener has been adjusted since my partner and I have started raising questions.
The ability to skip some questions was added (previously students could skip zero questions), a bigger notice on the home page that a student can opt out was added, a question about religious service attendance by family was taken out.
What that means, if you know anything about basic research, is that ANY conclusions they draw will be invalid. If you change the structure of the testing device AFTER you start using it, you cannot make any solid claims. And, the fact that each district is using it on different populations (by grade or by behavior) also would make any claims less than stellar.
PSU professors confirm that no notice was given to parents of students, nor was it seriously considered. PSU professors said it wasn’t necessary because the graduate students were the subjects of the research experiment, not the children in the public schools, whose data was being collected. Whitman disagrees with that decision.A year ago, the Willamette Weekly also published this:
“For the intention of research and generating new data, and doing these little experiments on a human subject for that matter, and then looking for results and taking those results elsewhere — requires consent,” Whitman told OPB.
The Portland State University researchers behind a project that asked teaching candidates in the Graduate School of Education to harvest personal data on thousands of Portland-area K-12 public-school students have cancelled a planned presentation of that research scheduled for April 16.It will be interesting to see what happens if researchers at Children's Hospital or UW Medical try to publish the results from Check Yourself usage in journals or at conferences.
Lastly, on the Oregon story in July 2018 via Willamette Weekly:
An internal review at Portland State University has found that a research project that gathered data from public K-12 school computers violated federal law.
That "proper authorization to use the data?" That's parents.
The university now concedes the project did not have proper authorization to use the data, and probably violated the Family Education Rights and Privacy Act, a federal law that protects student data.
Finally, I attended the Curriculum & Instruction Committee meeting yesterday (primarily to report back on the Science adoption - separate thread to come) and there on the agenda was a review of Board policy#3232 on Parent & Student Rights in Administration of surveys, analysis or evaluations.
Head of Research and Evaluations, Dr. Eric Anderson, said they would like to make some changes to the policy. That the Check Yourself screening tool didn't really fall into any of these categories and they want to "shore up the policy with a framework for parents." Basically, there are differences between research, surveys and screeners especially in groups via individuals.
My spidey radar tells me that the district would not be bringing this to the Board if they were not worried. And they should be. (They said they couldn't get this work done in time for a Board meeting in April but would by May.)
What was interesting to me is that Director Pinkham - twice - asked about students and their ability to opt out. He said, "What if the counselor or teacher is telling you to do this? It's hard for a kid to say no to the teacher who is the authority." Right he is.
And, again, embedded in that policy is the right for parents to examine any survey BEFORE it is given and/or opt their student out.
But you can do neither action when you don't know of the existence of said thing.
The policy must be amended so that if the district wants to do this kind of work, parents have the opportunity - in clear ways - to know about it.
Districts, governmental entities and higher education cannot run roughshod over student data privacy rights.
Comments
value privacy
"A lot if not most academic research changes measures part way through data collection, usually towards the beginning. If your sample size is large, it does not have too much of an impact on the findings. You just don't have data for a certain percentage of your sample, so you don't report findings for them. Researchers usually have rules around 'data missingness'. If someone skips a certain percentage of a measure, a subscale or scoring will not be calculated for that measure. It's typically better to change items or procedures that are not good, such as forcing individuals to answer all items, and deal with data missingness and what not, then it is to have bad measures or procedures."
Most changes partway thru? What evidence do you have of that? You might change during a piloting of the methodology but knee-deep in it? I find that hard to believe. Certainly, if you do allow skipping of questions, you note what gets skipped.
But this search changed because of pressure from me and my partner, not because they thought up.
Amazing Black Cloud South Bend Clutches Dodge