Charters Improving at Faster Pace in Urban Areas

Our results have stimulated a good deal of conversation among policy analysts, particularly among those who support school charters. We have been surprised at the modest amount of criticism the study has received from charter critics. However, a report on the study in The 74 includes a critical comment by one observer, who suggests the sample we used was “skewed” in favor of urban charters, a locale where charter students are improving at a more rapid pace than those in district schools.

We were puzzled by the criticism because the sample used for our study comes directly from NAEP, often referred to as the nation’s report card, which, in compliance with federal law, seeks to draw a representative sample for each state and for the United States as a whole. Admittedly, NAEP draws its sample to be representative of all public school students, not to be representative of the district and charter sectors separately. It is conceivable, though not likely, that NAEP somehow drew its charter sample in such a way as to over-represent urban students.

For such a skew to bias our results, the skew would need to increase between 2005 and 2017, because our study shows only that cohorts of students at charter schools, on average, are improving at a more rapid rate over this period than those at district schools, not that the average charter school student is performing at a higher level than the average district school student.

Fortunately, we can check to see whether the composition of students tested by NAEP has become increasingly urban in a way that is inconsistent with trends reported by the National Center for Education Statistics (NCES) between 2007 and 2017. (The year 2007 is the first year for which the definition of urban is the same for the two data sets.) The comparison between the two data sets must be done cautiously because they are not measuring the same population and they are both subject to error. NAEP samples only students in 4th and 8th grade, because those are the two grades it tests (we excluded grade 12 due to the low number of observations for charters). NCES asks all public schools in the United States to report enrollment for all students in elementary and secondary schools.  Further, NAEP tests in the spring, whereas NCES reports fall enrollments. And reporting error can bedevil any national data collection effort.

Even with these caveats, an inspection of the urban composition reported by both NAEP and NCES is instructive. If the skewness hypothesis is correct, it implies that the urban trend is steeper in the NAEP than the NCES data set. But in fact, the trend is just the opposite of what the skew hypothesis implies (see the table below). In 2007, and even more in 2011, urban charters were over-represented in the NAEP data set (as compared to the NCES one), but that shifts to under-representation of urban charters in NAEP by 2017. In this final year of the available data, the urban share in the NAEP sample is 3.5 percentage points less than percentage given in the NCES data set. If the NCES data correctly identify the percentage of 4th and 8th grade students in urban charters, students in urban charters are under-represented in NAEP in 2017, after having been over-represented previously.

Source link