TNReady test faced more trouble when 10,000 tests received an incorrect score.
Two data issues were revealed in a letter sent by the State Department of Education to local school directors.
Around 8,000 tests were scored too low and another 1,400 scored too high.
There was also an issue linking teachers with the children in their classes.
Getting the results correct is important because teacher evaluations are tied in to TNReady test scores.
The Dept. of Education's letter:
Directors,
I wanted to make sure you were aware of two data issues we identified through conversations with districts during the TNReady embargo period. Both of these issues have now been resolved, and every district who was affected has been notified via EdTools. If you have not been notified already, you are not impacted.
The first issue was with “linkage data” submitted through teacher-student claiming. The vast majority of this linkage data was compiled via the student schedules recorded in the SIS, and about 97% of all linkages were completed that way. Some districts did not have complete scheduling data available in their SIS, and because of Questar’s scanning delay earlier this summer, roster information was not available from the scanned test records. As you may recall, as a work-around, we advised districts to submit any additional teacher-student claiming data not in the SIS through supplemental Excel files, which accounted for about 3% of records.
However, one of our data processing vendors, RANDA Solutions, did not fully process the supplemental Excel files. This meant that some educators did not have all of their linkages fully applied. In many cases, the number of students that were missing from a teacher’s linkage file was so small that it likely will not impact their individual TVAAS composite. The total number of teachers who were missing at least one student is about 1,700, but about half of those had five or fewer linkage records included in the supplemental Excel files – a number that is unlikely to change their current growth measure. There are about 900 teachers with more than five records in the supplemental files, andonly 9 districts have more than 10 teachers who are in this second category. District and school TVAAS composites are not affected by this situation given that all students are included in those metrics, regardless of which teacher is linked to them.
We are working to generate updated TVAAS growth measures for these teachers, and we sent a letter to all impacted districts last week to share with their affected educators. We are also contacting all educators through TNCompass when they access the site. It is possible that the new data ultimately will notchange an educator’s growth measure.
As you know, we changed the teacher-student claiming process this year to begin much earlier, instead of asking you to complete it at the end of the year, which does not leave much room for adjustments in the case of unexpected delays. The teacher-student claiming application opens next week for teachers to verify their rosters, and all scheduling data must be reported in the SIS to adhere with our process improvements this year.
The second issue we identified was due to Questar incorrectly programming some of their scanners for the English I & II and Integrated Math II end-of-course exams. This coding error in Questar’s scanning software resulted in a small number of the forms for the EOC tests having incorrect scores – exactly 5 out of 275 versions of the EOC tests across all 11 content areas.
This means there are about 8,000 tests that were initially reported with a lower scale score than was actually earned. Another 1,400 tests had scale scores that were reported with a higher scale score than was earned. Most of these scale score changes were very small, so just under 1,700 of nearly 600,000 EOC tests (or 0.3%) had any change in overall performance level (e.g. moving from approaching to on track). Questar has now correctly re-scored these tests, and they are processing new score reports for those students, which we will distribute to districts. Additionally, we will update the individual TVAAS growth measures for those 230 teachers who may be impacted by these student score changes. We are reaching out to those educators to let them know about this issue through TNCompass as well.
Impacted districts already received a notification from us earlier today. The EOC scoring issue impacted about 70 schools in 33 districts. As we noted above, because relatively few – 0.3% – of tests had performance level changes, there is no impact to the statewide results. All of the updated information will be reflected in the public TVAAS site and our State Report Card releases later this fall. Final TVAAS composites for educators will be reflected in TNCompass and our TVAAS restricted site.
In grades 3-8, we completed secondary checks and confirmed with Questar that there were no issueswith the scanning program and resulting scores for any of those tests. All student score results for grades 3-8 are correct and final. Additionally, all of the other EOC content areas – biology, chemistry, algebra I, algebra II, geometry, integrated math I, integrated math III, and English III – are also final.As we have shared on the virtual meeting and in the Commissioner’s Update for Directors, the embargo on data for grades 3-8 will lift next week on Thursday, Oct. 19.
The department goes through multiple quality checks to ensure the assessment and linkage data that is used to calculate TVAAS is accurate. The two data issues are not related to the TVAAS model, but stem from scanning program changes and resulting delays that Questar issued a statement about earlier this year – issues that have now been addressed. Our team checks the vendors’ work on everything that we possibly can, such as test forms, item statistics, and equating results, but we do not have visibility into the vendor’s scanning program. We have worked with Questar to ensure that the quality processes outlined in our contract are being followed for scanning and scoring programs. For context on the total scope of what we have been reviewing, the department has processed about 1.5 million linkage records to generate individual TVAAS scores for nearly 19,000 educators based on the assessment results from over 1.9 million student tests in grades 2-8 and high school.
Though we know that we have reported over 99% of grade 3-8 and EOC score data correctly, we need to be at 100% accuracy. We hold our vendor and ourselves to the highest standard of delivery because that is what educators, students, and families in Tennessee expect and deserve. Please let us know if you have additional questions or concerns regarding this information by sending an email to tned.assessment@tn.gov.
Thank you for your continued partnership as we finalize 2016-17 reporting and move forward with a successful administration in 2017-18.
Best,
Candice
