Transparency in Texas Tech Literacy for Student Assessments
Image: Based on results cited below.
A colleague recently challenged my thinking about the Texas Technology Literacy Assessment for 8th graders. As you may recall, I asked Texas technology directors to share the results of 2007-2008 8th grade students—who were assessed in 9th grade or Fall 2008) completing a technology literacy assessment.
The results are startling:
(Note: You can download the results (Excel format | OpenOffice format) and check my numbers at any time.)
The reason I asked for Texas districts to share their results was that I noticed a trend—students assessed with Infosource Learning’s SimpleAssessment.com scored SIGNIFICANTLY LOW. At first, I thought, “Wow, students in Texas don’t know much about technology use!” But considering slow adoption of technology into core content area, that’s not a surprise.
Then, someone made this observation—with slight edits on my part to anonymize the information—about their own Texas public school district’s results:
- My District’s students have not been taught the Tech Apps TEKS and yet over 60% passed the assessment.
- A passing score on this assessment was 70% but to get that 70% a student only needed to get 20 of the 40 items correct.
- A one hour discussion on the reliability and validity of the assessment with the vendor did not yield a positive reaction about the integrity of the assessment.
- We used Learning.com as our assessment since this is the same one TEA is using for its assessment pilot. Our cost was approximately $18,000.
- Infosource Learning’s SimpleAssessment.com was another alternative and it was free and their scores were much lower. I believe the lower scores are much closer to what our students really know.
So the questions that are hinted at in this data, as well as this observation, but not overtly stated…and which I’m going to try to state include the following:
-
How have each of these instruments been checked for validity/reliability?
-
Who at the commercial vendors (e.g. Learning.com) makes the decision about the weighting of certain test items over another? Have they been transparent about this?
-
How transparent is the Texas Education Agency in sharing the directions provided to the commercial vendor chosen for their technology literacy assessment pilot?
-
Does TEA plan to release results similar to those reflected in my simple survey above?
-
What are the usage statistics for Technology Applications:TEKS electronic materials?
-
There are obvious benefits to having students in Texas being perceived to score low (e.g. “Our scores are awful, we need more funding.”) but the converse is also true. The reaction might be this: “TEA, you’ve funnelled funding to schools for quite some time…and these are the results you have to show for it?”
But HIGH scores—perhaps inflated, we don’t know—might also allow TEA to say, “See? We’ve invested in technology—for TA:TEKS Electronic Curriculum, Technology Immersion—for public schools and it’s starting to pay off in higher test scores.” Which is truer or is the truth in another quadrant of reality?
Since part of this discussion has taken place on the TCEA TEC-SIG list, a list composed of primarily technology directors, it is incumbent on the TCEA Membership to ask some additional questions:
- What is TCEA doing—a la American Library Association (ALA)—to hold TEA accountable for test-assessment measures that almost all respondents to my survey found to be NOT VALID in their opinion?
- What should TCEA do to advocate on behalf of school districts—to TEA and the State Legislature—regarding the assessment protocols put into place to ascertain the technology literacy of students who were 8th graders during the 2007-2008 school year?
- What should public school district superintendents be encouraged to consider in regards to technology literacy at the upcoming TASA Midwinters’ Conference, and what is the TCEA Board of Directors doing about that?
- What partnerships has TCEA developed that impinge on its objectivity in regards to state assessments, and has it disclosed those?
Some questions for the vendors of these assessments:
- Have you published your weighting or grading scale for the assessments?
- How do your assessments match the Technology Applications:TEKS electronic materials? How about the revised ISTE National Education Technology Standards for Students?
- Will you be publishing an overview of all Texas—and perhaps other states as well—school district scores (how many 8th graders assessed, percent passing, etc.)?
Again, it is important to ask these questions of TEA, the commercial vendors and TCEA. The goal is not to put them on the spot, but rather, to ensure that everyone clearly understands what purpose of the assessments, how the assessments were implemented with what criteria, and how this has impacted the entire process of preparing children to meet NCLB Technology Literacy requirements.
If you are a Texas district and would like to participate in the survey—anonymously—please let me know.
