At the beginning of September we ran a piece titled “making the grade: 2014 Memphis Charter High School Performance.” In this piece we examined absolute proficiency levels as an effort to identify which charters are doing well, which are worth of replication and which need additional support. The data showed that several schools were doing well in both math and English and that two schools, MBA and City University, have seen dramatic turnaround in their proficiency rates. However, I did not analyze this data using another common measure, Value Added Data, or student growth compared to what was predicted. This piece attempts to fill that gap.
Why Compare High Schools
I believe that analyzing charter performance in this way is important for several reasons (full disclosure: I work in a charter high school).
First, I do believe in the idea of charter schools as laboratories of innovation. If something is working, why not replicate it?
Second, it’s important for parents and community members to have this information in one place to be able to make informed decisions about their children’s education. It’s also important for community members to hear this information to hold charter schools accountable for the outcomes that they seek.
Third, and maybe most important, both the iZone and ASD have indicated that they want to continue growing the charter school population as a part of their turnaround efforts. If we are to do this, we must study which charters are most effective and use what we find to create more schools that use similar strategies. It seems foolish not to.
Proficiency vs Growth
However, proficiency rates only tell one side of the story. What is absent from this first piece is information about the starting point of students in each school and how achievement compared to what could be expected from each school’s student population.
Such an analysis is easy because here in Tennessee we have the TVAAS data system. This system uses data from prior years to “predict” where each student is likely to land on their end of year tests. Furthermore, we can then compare this “predicted” value to the actual value to see how charter schools are actually doing with the students they have. Aggregated together this information can tell us a lot about the school and whether or not it should truly be held up as a model for replication.
This piece aims to answer three questions. First, where do charter high schools start in terms of student ability? Second, how does this compare to the SCS average to determine whether students are truly being skimmed? Third, how does their growth compare to where they started? And fourth, how does their growth rank compared to schools across the state?
Data
Data for this analysis was pulled from the tvaas.sas.com public site. It is all accessible to the public. In the interest of transparency the spreadsheet used to calculate this information has been included here: Charter School Value Added.
Questions 1 and 2: Who is starting off where and how does it compare to SCS?
[update 9/14/14: Freedom Prep’s Algebra I numbers are all from their 8th grade students. All 9th grade students take geometry. I’ve left the Algebra I scores in just for the sake of comparison.]
To make comparison easier, I first averaged the predicted scores for all charter high schools predicted score and then for the entire SCS district and then ranked them accordingly from highest to lowest predicted. The result can be seen above. You can look at each subject by clicking on the appropriate dot next to the subject.
[update 9/15/14: it was brought to my attention by a reader that the comparison of charters to SCS might be confusing and not entirely appropriate given that most charters have less than 100 kids take each test while SCS has over 11,000. Using SCS as a benchmark is not a commentary on the district or its performance; rather, it was done to give readers a sense of the average across the entire district and how charters fit in as a part of that average. The SCS comparison should only be viewed through that lens, as the primary purpose of this piece is to examine individual charter performance, not that of SCS].
In general, most charter high school achievement expectations start near to or below the districts. The only charter high school that starts higher than the SCS average in every subject is Freedom Preparatory Academy (Freedom Prep). On average, Freedom Prep, Power Center Academy (PCA) and KIPP High start of at a higher level than SCS, while Soulsville, Memphis Academy of Health Sciences (MAHS), Memphis Academy of Science and Engineering (MASE), Memphis Business Academy (MBA) and City University School of Liberal Arts (City University) start below the SCS average. MBA and City University are notable in that they start off almost 10 scale score points behind the next closest school.
Question 3: How does their growth compare to where they started?
Again, these schools are ranked in order of highest to lowest starting point from the first table. A score of zero indicates that the actual EOC scores were equal to the predicted value. A score of lower than zero indicates lower than predicted performance while a score of above zero indicates a higher than predicted performance.
For this comparison, I’ve divided the eight charter schools into three different groups; low (bottom 3), middle (the next two) and top (the top three).
In general, charter schools show consistently positive growth over their predicted score in English, with more mixed results in math. In English the only school with negative growth in either English I or II is MASE. All other charter high schools saw positive growth. Mathematics is a different story. Some schools consistently outperformed their expectations in Algebra I, notably PCA, Soulsville, MBA and City University. Other schools who may have had high achievement underperformed, including Freedom Pre, KIPP High, MAHS and MASE, who did especially poorly in Algebra I. Only PCA and Soulsville were able to continue their record of math performance into Algebra II. Biology has a similar mixed record, with six schools seeing positive growth while two (KIPP and MASE) see negative growth.
Comparing Groups
The actual achievement compared to predicted achievement of schools with similar prediction scores is also worthy of note. Some schools do remarkably well with similar students compared to other schools in this comparison.
The Bottom
Let’s take the bottom three schools, MASE, MBA and City University. MASE students start off on average lower than all but two other schools, but they still see consistently negative results compared to what is predicted. By comparison, MBA and City University, who start off at a lower level than MASE, show consistently positive growth in all subjects except for Algebra II. Clearly City University and MBA did something right this past year, while MASE continued to regress.
The Top
Let’s also consider the high starting schools, PCA, KIPP and Freedom Prep. Of these, PCA has the most consistent positive growth relative to its starting point. The other two, KIPP and Freedom Prep, underperform in at least one subject, two in the case of KIPP (algebra I and biology).
The Middle
For the middle level starting high schools (Soulsville and MAHS) both do very well with their students, with only MAHS showing negative growth relative to its predicted growth in Algebra I. Soulsville does especially well, with its lowest growth coming in English I. However that growth is still 10 scale points above what was projected.
Question 4: How does this growth rank compared to schools across the state?
This is a great comparison because it allows us to compare the growth of charter schools in Memphis to every school in the state using a percentile ranking. The results of this comparison can be seen below:
For the sake of comparison, I’ll do a quick analysis of which schools fall within the top 25th percentile (75 and above) in terms of growth in the state. Since the stated goal of charter schools is to dramatically improve student outcomes, this seems like a reasonable goal.
In general, almost all SCS charter high schools ranked in the top 25th percentile in English. Six of eight schools make it in English I and six of seven make it in English II. MASE is the only school to not make it into the top 25 percent in both subjects, while KIPP High fails to make it in English I.
For Algebra I, half of the schools studied (four total) made it into the top 75th percentile: PCA, Soulsville, MBA and City University. Only three schools made it for Algebra II; MAHS, Soulsville and PCA.
For biology, five high schools made the top 25 percent; Freedom Prep, PCA, Soulsville, MAHS and MBA.
Analysis
First, let’s acknowledge the great work being done by the majority of charter high schools in Memphis with the students they have. In general we see consistently high growth across all schools. In fact, MASE is the only school that has no bright spots. Their best subject was English II, but their growth compared to what was predicted still fell in the bottom 25 percent of all schools in the state.
Second, where the district average fell right around predicted, most charters greatly surpassed that predicted level, indicating that they are truly bending the curve at their schools by helping their students achieve at a higher than predicted level. This occurred for almost all schools in English and for a majority of schools in mathematics.
Third, Memphis charters are doing very well by state level standards in English, Math and Biology, where we find the majority of Memphis charter high schools ranked in the top 25th percentile for growth in all subjects except for Algebra II.
Conclusions
First, this data suggests that almost all Memphis charter schools are worthy of study to find out what’s happening inside their walls. Schools starting off with low, mid-level and higher student ability levels are generally performing well. City University, MBA, MAHS, Soulsville and PCA should all be noted individually for consistently positive growth above and beyond predicted levels across the board in all subjects for Soulsville and PCA and all but one subject in the case of MAHS, MBA and City University.
Second, this carries broad implications for what we should be expecting out of education in Memphis. We have schools across the board demonstrating that they can bend the achievement curve beyond what is predicted, regardless of their starting point. This achievement cannot all be attributed to skimming by any reasonable analysis, especially for the lower and middle groups of schools that show tremendous growth. Those organizations such as the ASD and iZone looking to growth the charter school population in Memphis should be taking deliberate steps to study our existing charter high schools to find out what is working and why as they solicit applications for new schools.
This is not to say that charters are perfect or that this data should be the end of the discussion. So much more needs to come out of education than test scores. Education needs to be about creating future leaders and truly preparing students for college. I think that most of us would acknowledge that the current TCAP assessments don’t necessarily measure up to those standards. However, they do serve as a reasonable starting point for comparison and by the three comparisons used here, charter high schools in Memphis are doing remarkably well. They are certainly worth of study to find out what is going on and what, if anything, can be learned from them and implemented in the general school population as well as for the continued growth of charter schools in Memphis. After all, what good is innovation if we don’t learn from it?
By Jon Alfuth
Like this piece? Check out Ezra Howard’s analysis of the ASD or Jon’s previous piece on charter school performance in Memphis.
Follow Bluff City Education on Twitter @bluffcityed and look for the hashtag #iteachiam and #TNedu to find more of our stories. Please also like our page on facebook. The views expressed in this piece are solely those of the author and do not represent those of any affiliated organizations or Bluff City Ed writers. Inflammatory or defamatory comments wi