How Does NAATE Participation Connect to Student Achievement Test Data?
A. Introduction: Analyzing student achievement gains is another strand of researching performed by NAATE. Some organizations question the merits of evaluating student achievement gains, when the NAATE target audience is comprised exclusively of experienced classroom teachers, already identified as having strong performance results and categorized as "top 10% performers." NAATE is exploring whether meaningful information related to program efficacy can be gleaned from these analyses. While past attempts at systematically collecting raw student achievement scores over multiple years from all teachers and students in school partner organizations has proved challenging, recent efforts have begun to bear fruit in terms of data collection.
B. Overview: NAATE, working closely with statisticians at the Harvard Graduate School of Education, is currently studying student achievement data before, during and after NAATE programming for the students of NAATE and non-NAATE teachers (where data is available). We are using R statistical software and visualize data. The effort includes:
Univariable descriptive statistics on all variables
Correlational analysis of all variables
Average student performance on tests of NAATE vs. non-NAATE teachers
C. Limited Validity of Student Achievement Analysis: Our HGSE statistical support has indicated that results should be interpreted cautiously, considering:
The number of factors influencing student achievement unrelated to teacher quality, especially when comparing results from within large districts with diverse school environments, or CMOs with both new and more established schools
The number of NAATE teachers who teach special education or non-tested subjects or grades
The statistical invalidity of connecting any changes in student achievement data with the NAATE program specifically
Variability in cut scores for "proficient" from state to state (each state has different standards for proficiency)
The limited sample size, which ultimately makes any statistical analysis insignificant
D. Timetable for Data Acquisition and Analysis: NAATE has been working to rebuild relationships with CMO and district data analysts one-by-one, after a series of less successful rounds of data collection between 2012-2013. Identifying and working with data managers, building trust, and transferring data is a lengthy process, as privacy concerns surrounding data abound and extracting data can be a time-consuming process for many partners. We're currently working to gather data from a number of its major school partners where testing data is currently available.
E. Summary of Urban District Analysis: NAATE acquired and completed its first round of student achievement data analysis in November 2014. Data came from a large, urban public district. All teachers included in the analysis were Cohort III or IV teachers (teachers who began NAATE in 2013). Of a total of 13 NAATE participants, five had valid math test data, and four had valid reading test data. 2012-13 data reflects their students' results in the year prior to their participation at NAATE. 2013-14 data reflects their students' performance after partial completion of the NAATE program.