I’m proud to teach high school math at The Soulsville Charter School. Every day I come to work confident that my students are receiving the best education that I and my colleagues can give them. I see students and teachers working hard in a collaborative and constructive environment that has resulted in 100 percent of our graduating seniors being accepted to 4 year colleges three years running. But we can’t just rely on anecdotes to prove our effectiveness. Soulsville, and indeed all charter schools, need to be able to prove their effectiveness through data, not just hearsay. This is more important than ever given the likely growth in the charter school movement in our state. Its also important to take a critical look at charters using data because some people tell charter stories that aren’t as positive as mine. Many detractors use their own anecdotes to claim that charters should be abolished all together. They claim that charters aren’t any better than traditional public schools and that their claims to overcome poverty should be equated to charlatans selling fools gold or snake oil. Even when individual charters are found to be effective, detractors claim that these effects should be discounted because charters serve lower percentages of impoverished students than do traditional public schools. They also claim that charters “skim” the best students out of the general school population, that is, take the best and leave the rest, and that their achievement can therefore be explained by their students starting at a higher academic level than their peers. If these accusations are true, they would and should call into question the entire charter movement. The problem is that they each fall apart when we look at the hard data behind charter schools here in Tennessee. I’ll go deeper into each of these claims throughout this piece using data from two sources, the CREDO National Charter School Study and Tennessee’s Charter Schools Annual Report for 2012-2013, but here’s a brief summary of what we’ll find;
- Data shows charters in Tennessee serve a more impoverished and higher minority population than traditional public schools
- Data shows that charter school students in Tennessee start at a lower academic level than their peers, refuting the “skimming” claim
- Data shows that Tennessee charters outpace traditional public schools on both growth and academic achievement measures
- This all suggests that charters are making meaningful progress towards overcoming poverty and closing the achievement gap
Who Are the Charters? Before digging into the data, lets get our bearings as to who charter schools serve here in Tennessee. According to the Center for Research on Education Outcomes (CREDO) Study of charter schools, our nation contains approximately 6,000 charter schools serving 2.3 million students per year nationally (as of 2012-2013). Tennessee contains 49 of these charter schools (again, as of 2012-2013) serving a population of 11,698 students. The table below shows the concentration of charter schools in Tennessee by district.
Tennessee’s Charter School Annual Report, Page 5 http://www.tennessee.gov/education/schools/docs/Annual_CS_report_2013.pdf
Exhibit A: Who Do Tennessee Charters Serve?
Tennessee’s Charter School Annual Report, Page 5 http://www.tennessee.gov/education/schools/docs/Annual_CS_report_2013.pdf
Who do these 49 charter schools serve? Is their population really different than traditional public schools as detractors claim? When we examine the data, we do in fact see a difference, though not the one they would like to see. Here in Tennessee, charters serve a higher percentage of students in poverty (measured with %Free or Reduced Price Lunch), a higher percentage of minority students and a comparable percentage of students receiving special education to traditional public schools. This is evident in the table to the right, pulled from Tennessee’s Charter Schools Annual Report from 2012-2013. In summary, the demographics claims made by charter detractors can be discarded at the aggregate. There may be individual exceptions but on the whole Tennessee charters are in fact educating a more impoverished and higher minority population in every district in which they operate. Exhibit B: Are Tennessee Charters Skimming the Best?
Stanford CREDO Study, Page 13 http://credo.stanford.edu/documents/NCSS%202013%20Final%20Draft.pdf
Next, lets examine the validity of the anecdotal “skimming the best students” claim made by charter detractors. If skimming were taking place, we would expect the average charter school student to start at an academic level near the average public school student. What we find is quite different. The CREDO study found that Tennessee charter students typically start at a much lower academic level than their peers across the state. Tennessee students began 0.41 standard deviations below the average in reading and 0.42 standard deviations below in math. Using the table to the right (from the CREDO study), this equates to starting 288 days behind in both reading and math. To put that in perspective, the average charter school student would need a year and a half’s worth of growth just to catch the average Tennessee student statewide. On the whole this data suggests that charters are not, in fact, skimming the best and leaving the rest. Exhibit C: Are Tennessee Charters Closing the Achievement Gap? We’ve now established that Tennessee charter schools serve a population similar to traditional public schools and that their students start off well behind the average Tennessee public school student. But are charters helping students overcome these disadvantages and make meaningful progress towards closing the achievement gap? Or are the detractors correct when they claim that poverty is deterministic when it comes to student achievement? Here we have some great news. The CREDO Study found that charter school students in Tennessee gained 86 days more in reading and 72 days more in math than their peers in traditional public schools. When we compare this to charters in other states, our reading gains tied with Rhode island for the highest in the country and our math gains are fifth behind only Rhode Island, Washington DC and New York City and New York State. With those numbers, it would take the average student starting in a charter in Tennessee three and a half years in reading and four years in math to overcome the starting gap highlighted in Exhibit B between the average charter student and the average non charter student in Tennessee. That won’t fix the achievement gap in a year, but if charter gains are sustained over 3-4 years, the gap would theoretically disappear. Over time this data does suggest that Tennessee charters at the aggregate have the potential to show that poverty doesn’t have to be destiny. Exhibit D (bonus): Are Tennessee Charters Making a Dent in Absolute Achievement? Tennessee charters don’t just perform above average when it comes to growth. They’re also doing better than the average public school on measures of absolute student achievement. Consider the performance of charters vs traditional public schools in the state’s two largest districts, Shelby County and Metro Nashville Public Schools (MNPS). You can see a comparison between charters and non charters in the two tables below for these districts (note: I’ve only included 3-8 scores for Nashville because they only have one charter high school at present). In both districts charter achievement ties or exceeds absolute district achievement in all subjects. In Shelby County, charter students do even better in high school, exceeding the absolute achievement of traditional public schools by 5 percent or more in every subject. And lets keep in mind that we’re also comparing charters to every non charter in the district, including some that don’t share the same student profiles as the charters in question. If we could remove these schools from the traditional population, charters would do even better by comparison. This is pretty impressive given the population being served by charters. Given that the charter student population is higher poverty, we would likely expect that their average achievement would be lower than the district average if poverty is indeed destiny. However, this is not the case. Higher scores in charter high schools should give credence to the idea that sustained charter growth can make a meaningful impact on overcoming the achievement gap if those gains are sustained over time.
Table created with Microsoft Excel, data from the Tennessee Charter Schools Annual Report http://www.tennessee.gov/education/schools/docs/Annual_CS_report_2013.pdf
Table created with Microsoft Excel, created with data from Tennessee’s Charter Schools Annual Report http://www.tennessee.gov/education/schools/docs/Annual_CS_report_2013.pdf
A Quick Summary Here’s a quick recap of what we’ve learned from examining the data around charter schools in Tennessee. On average, charters statewide serve a more impoverished, higher minority population than traditional public schools. Their students start at a lower point academically, but they achieve more growth on a yearly basis than traditional public schools. Within the state’s two largest districts they also show equal or higher achievement on an absolute scale than traditional district schools, and this gets larger once students reach high school. This all suggests that charters are in fact making a measurable impact on closing the achievement gap. I’d say the claims of charter detractors have been pretty well refuted here in Tennessee. Closing: How’s My School Doing? So how does my school, Soulsville, come out when we look at our hard data? Turns our success isn’t just anecdotal. Our average ACT score in 2013 was a 20.1, compared to 16.1 for our school district. Our high school’s value added scores in all subjects were well above projected levels and 2 of the 4 TCAP subjects saw above predicted growth. And this sustained effort over time led to our 2014 graduating class being offered over $10 million in scholarship offers, along with another year of 100 percent acceptance to four year colleges. Are we perfect? No, but we never claim to be. We’re doing some amazing things and along with the other charter schools of Tennessee, working hard every day to make a measurable dent in the achievement gap. We’re a part of the solution and its time we are recognized as such by all parties. After all, we have the data to prove it. All data in this piece was drawn from the CREDO National Charter School Study, the Tennessee Department of Education’s Charter Schools Annual Report 2012-2013 and the Tennessee State Report card from 2013. All of this data is available publicly and can be accessed by anyone. By Jon Alfuth Follow Bluff City Education on Twitter @bluffcityed and look for the hashtag #iteachiam and #TNedu to find more of our stories. Please also like our page on facebook. The views expressed in this piece are solely those of the author and do not represent those of any affiliated organizations.
Hi, Jon. I’ve been quietly following this conversation and many others on ed reform for a while now. I’d like to add my two cents about your claim that the “data don’t lie”. They may not “lie” but they can be incorrectly or partially interpreted. A few points on what I mean:
A. Yes, charter schools have higher percentages of disadvantaged students than the state public school average. That’s no surprise. However, they do not have higher percentages of these students when looking at comparable public schools in the same communities or from the communities in which charter school students live. In fact, they appear to have lower percentages of students with these characteristics compared to other Memphis-specific public schools. This is exactly what Jersey Jazzman was trying to demonstrate in his June 8th post. Even looking at the graph you provide in “Exhibit A”- when you look at the Memphis specific demographics, charters are actually comparable to public schools. These lower-level comparisons of school groups by city and neighborhood do matter. From what I understand, this is the comparison that “detractors” are trying to discuss. Aggregates never tell the whole story and to present them as such can be misleading or careless.
B. Your data doesn’t convincingly refute the skimming or creaming claim. Again in Exhibit B you use charters and state-wide public school data, which isn’t the only (or most) meaningful comparison to make. How do they compare (by neighborhood and student characteristic) to public schools in Memphis? This is also something that has been brought up by other social scientists. There may well be important differences in the kinds of students who enroll at charters than those who don’t (not to mention stay without dropping out or transferring out, which I’ll get to later). These differences are unobservable given the data. You simply cannot definitively claim that skimming isn’t happening based on the data you provide.
C. The CREDO researchers themselves say using the “day of learning” to explain the standard deviation differences are only an estimation and shouldn’t be used as an empirical transformation. Researchers have questioned the meaningfulness of this measure as well. I’ve honestly never really understood how people could throw it around like that, since we’re talking about number of correct answers on a test equated to days spent in a classroom. It just doesn’t seem like a sound or truly meaningful translation to me. The difference between the TN charters and selected comparison schools in this study was .1 of a standard deviation. It is statistically significant, but it’s not a huge quantitative difference. In addition, no study is perfect, and other social scientists who reviewed the study had questions about the choice in methodology that they say likely exaggerate that .1 standard deviation difference. What do other studies say?
D. Exhibit D: Are these differences statistically significant when controlling for everything one should when trying to make a comparison? That’s an additional piece that’s essential in making and confirming your comparison. But I do appreciate visuals, and your blog has lots of those.
E. If we’re going to talk data, then student retention is a necessary part of that discussion. Many charters report low drop-out rates, but considerably high “transfer-out” rates. Indeed, the TN Charter Study reported most charters having double-digit transfer-out rates. Some even as high as 30%. Surely, some of these are due to families moving to other neighborhoods, but getting the “full picture” would also mean having comparison data for Memphis public schools, which the charter report doesn’t have (at least not that I’ve seen). It would also mean providing breakdowns of why students leave each school, as it seems there’s a lot of cooling-out happening based on student behavior and other “discipline-related” issues which, again, gets at the issue of “creaming”. You say that a lot of “detractors” point to anecdotal evidence about these types of issues, but considering the lawsuits being brought against some charters in New Orleans, this isn’t an anecdotal issue. To me, it raises larger questions that I think many in the ed reform and charter community sincerely care about. Who is served by charters and who are the ones left out (or politely asked to leave)? How does this also potentially undermine efforts for equality? I ask these questions not as someone trying to “tear down” the charter community, as you might suspect, but as someone who thinks the answers here matter.
There really hasn’t been overwhelming evidence yet that charters are significantly different than public schools in outcomes. All I ever see cited is the CREDO study, which, if I understand correctly, wasn’t peer-reviewed. It would be much more convincing if there were multiple sources of evidence comparing these schools, but this doesn’t seem to be the case.
You can look at people trying to point out these things as “detractors”, but I think they’re trying to make sure the full picture is there. I don’t think they’re trying to diminish the hard work of your students and colleagues, nor are they saying it’s all for naught. However, I think it should be a point well-taken that this data doesn’t mean what you think it means. If we’re going to invoke the almighty power of data, we have a responsibility to understand how our data can address certain questions and how it cannot. If we’re trying to make comparisons, we are responsible in knowing what’s accounted for, what’s missing, how terms are defined, and what’s unobserved.
Thanks for the thoughtful reply Colleen. I think that the important point that is often raised by anti-reformers is that data will always contain some flaws. For example, JJ’s data on charter poverty in Memphis doesn’t include students in poverty that don’t receive free and reduced lunch, so those numbers are actually off (I’m working on getting a more accurate table together). But the question is then what we do with the data, and all too often the reaction from detractors is to completely discard it, even when we might glean some useful information from it.
I chose to do an analysis in the aggregate to make the point that from the data we have it appears that collectively charters are achieving something important in Tennessee. Once we have that down we can look at individual schools and data and determine who’s for real and who’s not.
I also appreciate your point about transfer rates from the TN Charter school report. I was originally going to include that in my post, but I could not find attrition rates for SCS/MCS schools online for a valid comparison, hence why it was left off.
If you want to drill down to the individual school level, I’d check out the Tennessee Charter Schools Annual Report 2013, where they tons of visuals of the data break down by school. Very useful, but again, I wanted to do a meta analysis to make the point that collectively at least, charters are doing something special.
On the issue of skimming, I would like some more in depth information on student populations that no one seems to report. For one, what is the retention rates in charters? There’s three important milestones for this. 1) from beginning to the school year to its end; 2) from one school year to the next; 3) and since this is a high school, from freshman year to graduation. This should be available for all schools and, honestly, should be reported on the report card. Of course, there will always be transitions. but if the numbers in charters are higher than that in traditional public schools, and statistically significantly so, then that would point to skimming. Which, in the case of number 1, it would point to “counseling out” (KIPP was found to have done this in Nashville a couple years ago right before TCAP). Furthermore, in many cases charters do not have to take in children past the application deadline (unlike traditional public schools), which is also a barrier to equitable services provided by charters.
If your school has this data on retention (and I’m sure if it doesn’t have it available, it could very quickly calculate it), then I would love to hear your report on it.
Next, you have to look at access to special services, that is, Special Education and English as a Second Language. This is information reported by the state’s report card. MCS pre-merger had 7.6% English Language Learners and 12.6% of students with disabilities. To compare, Soulsvillle had no English Language Learners and 9.8% of students with disabilities. This is fairly typical of charters across the Memphis, Nashville, and the ASD. If charters are not providing equal services to all students, that’s a form of skimming.
You also has to look at how charters establish themselves, particularly in the case of turn-around models (like the ASD). Very rarely do they take on the whole school. They take the younger, often non-tested grades and leaving the older and more difficult students with the traditional public schools system. This is a form of skimming.
There are also softer issues like not providing transportation, requiring parental involvement, expulsion for truancy or lack of attendance, etc.
So it’s misleading to look at only Free and Reduced Lunch. It doesn’t tell the whole story.
Thanks for the comment Bert. In researching this piece I spent a pretty substantial amount of time searching the web for yearly retention data for both charter and public schools in Tennessee but was unable to find it, hence why it was not included in the piece. If you know where I might be able to find it please let me know.
I don’t, the state doesn’t report it. However, you could break ground and present your own school’s retention rate and inviting other charters to do the same. I would also be interested in knowing the rate of acceptance, how many apply vs. how many are accepted.
But I find it interesting that you don’t meet the concern about ESL or SPED. Just taking a quick look at neighboring High Schools. I notice there are some definite differences. Central has a lower rate of students with disabilities with 6.4%, but has an ELL population of 4.7% (TVAAS composite of 5). Booker T has no ELLs but has 17.5% of students with disabilities (TVAAS composite of 5). Hamilton High School has 16.3% of students with disabilities (TVAAS of 1). Carver High School has 18.2% of students with disabilities (TVAAS of 1). Even MASE, the first charter in Tennessee, has 10.4% of students with disabilities (TVAAS composite of 5, though has had issues with achievement and has been on the verge of closing). All of these schools are within 3 miles of Soulsville and are providing a higher level of services to students with special needs. This points to skimming pure and simple. So congratulations on the success of Soulsville, but you can hardly say that it’s better than the traditional, or even the other charters, in the same area of town.
I would bet that if I did this sort of comparison to most charters I would find the same sort of of information, significantly lower rates of ELLs and students with disabilities when compared to like schools in the same area.
I do find it interesting that you left out the percentage of students with disabilities served by Soulsville in your comment, which is 9.8% or almost the same percentage as is served by MASE. Lower than surrounding neighborhood schools, yes, I’ll give you that one.
I’d like to offer two alternative explanations other than blatant skimming for the difference. First, my understanding from talking to special education teachers at other charters is that the way the district funds special education in Shelby County deters charters from taking on students with special education. Where traditional schools receive extra money for SPED students, my understanding is that charters do not. Rather the district sends specialists around occasionally that may or may not visit the schools. in this way charters are at a disadvantage in terms of resources compared to traditional public schools, thus offering them an incentive not to serve the SPED population at a higher rate.
Second, the discrepancy may be because the public school system is honestly using the SPED designation inappropriately. I used to teach at a district school before coming to Soulsville, and I can tell you anecdotally that we had a large number of students who had IEPs but quite honestly I couldn’t understand why they still had them. At some point they had struggled learning wise and someone had assigned that IEP to them, but they hadn’t been tested since to see if they still needed it or not simply because we didn’t have to time and resources to do so on a regular basis. I can’t say what percent of all IEPs this encompasses, but I would suspect that it would be a noticeable number. On the flip side, Soulsville for one has gone out of its way to keep on top of who needs IEPs and who doesn’t. We don’t just leave a student with an IEP if they don’t need it. Our SPED teachers spend months going through every student and testing them, rewriting IEPs for ones who’ve had theirs changed and closing those who don’t need them anymore. I can think of several students this year who had this done. This likely lowers our overall “students with disabilities percentage” by a noticeable amount given that we are a smaller school.
Again, this is not to excuse the difference or say it would disappear if these factors were changed, just to offer to scenarios that may explain the large difference we’re seeing other than overt skimming.
[this comment has been edited to remove specific identification information as of 7/1/2014]
I realize I forgot the ELL aspect. My theory for this is that the explosion of ELL students is fairly recent and concentrated in a few neighborhoods around Memphis. As such serving a high ELL population wasn’t part of many charter’s design. Why haven’t we shifted? I think that it could be due in large part to the simple fact that ELL service requires more time and effort and individual schools simply may not be equipped to put together an effective program. Instead of hitting existing charters for not serving high ELL populations, I think a better solution is to recruit charter operators that specifically target the ELL population and have a proven track record of doing so effectively. Specializing as it were.
I didn’t leave it out, I mentioned it in the first comment. I followed up with everyone else.
You may or may not be right about the funding issue, but it’s my understanding that the money follows the students. I’ve never heard that “the money follows the students, except in the case of…” I’d happily accept your answer if you provided proof. I’ll do my own research, but really this should be on you as you’re the one claiming charters are doing so much better than traditional public schools and that there are no true differences between the student populations. But there are real differences within the populations.
As for your claims about mislabeling students. I am sure this is happening in some cases, even at Soulsville. It’s a matter of statistics, there’s bound to be some errors. But let’s be clear: you’re claiming negligence at best and fraud at worst. Both of which are inflammatory claims. So you’ll have to excuse me if I take anecdotal evidence against hard data with a grain of salt. What may or may not be happening at your old school, I find it highly suspect that every traditional public school within a three mile radius is fudging their SPED numbers. Because I am certain the teachers at traditional public schools are every bit as dedicated to their students as those at Soulsville to their students with special needs.
Also, I’m not suggesting overt skimming. That information would be more obvious with retention data. These are systemic issues, they’re under the surface and quiet. I think your being a charter apologist in this case. Particularly with the lack of ESL in charters. It should be a responsibility of existing charters to take on the ELLs of the students that apply to their schools. Particularly if there is a large percentage of ELLs in the area in which the charter educates. Especially if they, like yourself, are trying to make the claim that charters are doing better than traditional public schools even though they are not providing the same level of service (which you have yet to truly refute). If we’re talking about true choice in public education, it should be the responsibility of every school to educate every child that comes through their doors. Otherwise it’s all just hot air and marketing.