Just the Facts: Charter High School Performance in Memphis

My school has a strong data driven culture. We use data not just to identify grade level or class wide trends, but trends with individual students. Because it’s not enough for me to know that the entire class scored, say, an 80 percent on a specific assessment. I need to be able to identify high, middle and low performers on each assessment and reflect on what has made them successful. This is a wonderful practice because it helps me identify what the strong performers are doing well and consider how I can use this information to help the weaker performers improve.

I think this offers a perfect analogy for what we need to be doing with charter schools here in Tennessee.

Last week I wrote a profile of charter schools in Tennessee with the purpose of establishing that collectively charters in Tennessee are in fact the real deal. They are doing very well in the aggregate compared to traditional public schools. From a public policy perspective this is a vital question to answer because it informs our view about where best to spend our limited public resources.

But its not enough just to know that collectively we’re doing well. Just like in my classroom, it’s important to be able to identify who is doing well so that we can study and potentially replicate their success, especially when it comes to serving students from a low income background.

This piece looks at charter schools compared to three different groups of schools to identify which individual schools are best serving this student population. When we look closely at charters, particularly high schools here in Memphis, it turns out that we have a lot to be excited about.

Summary of Methods (a more detailed explanation can be found at the end)

To create a valid measure, I first decided to compare individual Memphis charter high schools to traditional high public schools using data from the Tennessee Department of Education’s 2013 Report Card. The report card allows anyone to separate out achievement data by category, in this case of economically disadvantaged students only (economically disadvantaged is defined as students living at 185 percent or below of the poverty level).

I can then compared each charter school performance to that of traditional public schools using three measures:

1) A group of peer schools with similar poverty rates average for each subject

2) The legacy MCS average for each subject

3) The average of a group of the best optional schools in Memphis (schools with our version of an honors program) for each subject

To determine how each charter high school performs in relation to each measure I simply subtracted the average of each comparison group from each charters achievement data. The resulting number is the difference. Lists of the schools in each group can be found at the end of this piece.

Before going into the data, I want to add a couple of notes. First, I want to fully acknowledge that teach in one of the schools in question, The Soulsville Charter School, but that this post is in no way affiliated with the school nor has it been commissioned by them. It is entirely my independent creation.

Second, I chose achievement data rather than growth data for this first comparison because a charge often levied against charters is that while they grow students they don’t cut it when compared on an absolute achievement scale. As such I think this is an important point to examine and to my knowledge it hasn’t been undertaken by any official studies here in Tennessee.

Third, my goal is not to advocate for an expansion of charter schools nor call for shutting down any individual ones based on this data, especially given that I’ve only used achievement data for one year. All I’m trying to identify is who is doing well educating students in poverty using one measure from one year to identify whom we should be studying.

The Data

You can see all data shown in the table below (green cells = higher than average achievement, red = lower) followed in the next section by analysis:

Individual Achievement Averages TABLE

The Analysis by Comparison Group:

In analyzing the data I decided to break it up further by comparison measure. To achieve this I placed all six schools together on a graph for each measure. For each graph, a bar above zero indicates higher than average scores while a bar below zero indicates lower than average scores for the comparison group.

Comparison #1, Peer School: this compared each charter school to the average of a group of schools serving a similar percentage of economically disadvantaged students. Three schools stand out as having higher than average scores; Memphis Academy of Health Sciences (MAHS), Power Center Academy (PCA) and The Soulsville Charter School. These schools displayed above average achievement in all three accountability subjects. Notably all three of these schools scored 18 percentage points higher than the peer average in Algebra II (a very difficult test), Biology and English I. Two other schools perform at or near the peer school average, Memphis Academy of Science and Engineering (MASE) and Memphis Business Academy (MBA). In both cases these schools scored higher than the comparison group average in at least two subjects.  One school, City University, scored below the peer average in all five EOC subjects.

I should note that there has been some discussion about the most appropriate measure for comparison, free and reduced price lunch or just free lunch. Some people contend that the reduced lunch population can vary enough within schools that they produce completely different environments. Just to be safe I also ran the same analysis using schools with only similar populations of free lunch students, but the results were negligibly different, only 2-3% in most cases. I decided to keep free and reduced lunch peer comparisons rather than just free because of this and because the sample size was larger for all comparison groups.

Created using data from the Tennessee Education Report Card from 2013

Created using data from the Tennessee Education Report Card from 2013

Comparison #2, MCS Average: this compared each charter school to the entire MCS system’s achievement average for its economically disadvantaged students. The data looks almost identical to the peer school comparison in its trends but with different orders of magnitude. Soulsville and PCA continue to perform well, achieving even higher outcomes than they did in the peer school comparison. MAHS and MBA perform at a similar level to their peer school comparison levels with MBA doing slightly worse on English I. City University still performs below the average on three of five subjects, but the difference shrinks noticeably from the peer average to the MCS average.

Charter Performance MCS Comp

Created with data from the Tennessee State Report Card, 2013

Comparison #3, Optional Schools: 

Beyond looking at the comparison to peer schools and MCS in general, I wanted to see how charter schools in Memphis fare when compared to the best our city high schools have to offer. In Memphis, this means our optional program. Optional programs are essentially honors programs nested within traditional schools for which students can apply. They do not encompass all students in the schools, but the very presence of the optional program creates a different educational climate in those schools. This shows up in that these schools typically have higher achievement than the general city-wide average.

With that in mind, I wanted to see how our charter schools fare when stacked up against the best our city has to offer. Keep in mind that this isn’t compared to students in the optional program, but to economically disadvantaged students in general, some of which do participate in the optional program and some of which do not. You can find a list of the schools used in the comparison at the end of the piece.

Optional Program Comparison: Two charter high schools in the city still perform at a higher level than the optional average: PCA and Soulsville with PCA doing better than Soulsville overall across subjects but Soulsville doing better in mathematics than either PCA or the optional programs. MAHS performs at a slightly higher level for three subjects, but slightly lower in Algebra I and English II. MBA performs at a similar or higher level to optional programs for Algebra II and Biology but notably lower in all other subjects.  MASE and City University perform significantly lower than the optional average in all subjects with the exception of English II for MASE, where it performs only slightly below the average.

Charter Performance Optional Comp

Created using data from the Tennessee Education Report Card from 2013

Summary of Findings:

Soulsville, MAHS, PCA and MBA all meet or exceed the average achievement score for economically disadvantaged students when compared to peer schools and the city average. Charter high school’s lowest score comes when compared to the city’s optional high schools, but two schools, Soulsville and PCA, still have above average achievement levels, while achievement at MAHS and MBA drop off slightly but still perform well in comparison in some subjects.

MASE does about average when compared to its peer schools and the MCS average achievement in terms of poverty but we see a noticeable drop off when compared to optional schools.

City University performs lower than the average on all three comparison measures except for English I when compared to the MCS average and notably lower when compared to the city’s optional programs.

The Interpretation/Take-Away:

When we break down this data we have four Memphis charter high schools out of six that are doing a significantly above average job when it comes to student achievement with economically disadvantaged students and two that perform at or above the optional program schools. Soulsville and PCA do the best, followed by MAHS and MBA. We also have two charter high schools that, despite the overall positive performance of charters, perform as well or worse than traditional public schools by almost all measures in MASE and City University.

So what can we take away from this?

First, we have some great schools that are serving our economically disadvantaged student population at the high school level when it comes to absolute achievement, and those schools should be lauded for their outcomes. Three of the six schools in question perform significantly above the average of their peer schools and two more perform at or above in individual subjects. Only one school (City University) performs lower than the average for its peer and MCS comparison.

Second, three of the six are delivering outcomes on par or better than with our city’s best high schools. We can debate whether or not these schools are fully closing the national achievement gap, but at the very least they’ve closed the gap with those students that they do serve between themselves and similar students at optional schools.

Third, achievement on five test subjects is impressive, but its simply one measure of a school’s performance, not everything. We should also be studying these schools to learn how students are doing when it comes to areas such as critical thinking, problem solving and imparting a life long love of learning. We should also be looking at other measures such as ACT, graduation rate, AP scores and, perhaps most importantly, success in college after high school. The data I’ve presented here shows us something valuable about these schools, but its only one piece of the puzzle that must be examined.

Fourth, it’s important to note that not all charters are equal in this comparison. Some do better than traditional public schools, some meet the bar and some fall below it. This should inform our decisions as we identify what charters we should be studying and whose methods we should potentially be replicating. This is just one measure and more work needs to be done to determine exactly what each school in question is doing to achieve these outcomes, positive or negative. But by first identifying the achievement level of our city charter schools compared to their peers, my hope is that it will lead to improvements of both charter schools and our traditional public schools here in Memphis and Shelby County.

Additionally, my goal is to do a similar comparison analysis with growth data for high schools and then also middle schools. Any constructive suggestions for future iterations is welcome and appreciated.

By Jon Alfuth

Follow Bluff City Education on Twitter @bluffcityed and look for the hashtag #iteachiam and #TNedu to find more of our stories.  Please also like our page on facebook. The views expressed in this piece are solely those of the author and do not represent those of any affiliated organizations.

 

 

 

 

Detailed Summary of background and methods

Why do this Comparison?

First and foremost, why do this comparison? I’ve decided to undertake it because to my knowledge nobody has ever done a straight up comparison of charters vs traditional public schools and how they do serving economically disadvantaged students using straight up achievement data. The Stanford CREDO study looked at overall achievement in 2012, but I can’t find comparative data to see the magnitude of the difference. Hence this comparison.

Once again, I should note that I work in one of the charters in question (Soulsville).

What charters are Included (and excluded)?

We have eight charter high schools operating in Memphis: Power Center Academy (PCA), City University School of Liberal Arts, Soulsville Charter School, Memphis Business Academy, Memphis Academy of Science and Engineering, Memphis Academy of Health Sciences, KIPP Collegiate High and Freedom Prep. KIPP’s data was not on the TN state report card and Freedom Prep only goes through 9th, so I’ve left those two out of the analysis. Comparison data uses all other high schools operating in the legacy Memphis City Schools system in 2013.

Why did you use achievement data for just MCS and not Shelby county?

In 2013, Memphis City Schools was still that, Memphis City Schools. The merger had not yet happened and we were not yet Shelby County Schools, hence using this data only.

Why did you only use one year’s worth of data (2013)?

The state report card this year is set up in such a way that they’ve made it easy to do this type of comparison with economically disadvantaged students for all five comparison tests (Algebra I, Algebra II, Bio, English I and English I). The website also has a large amount of data available for download from previous years, but the data is incomplete for the comparison I want. For example, the aggregate data file only includes Algebra I and English II. In the base accountability file, which does include all five tests, they do not include scores for economically disadvantaged students only for Biology, English I or Algebra II. The only year that allowed me to do that comparison I wanted with all five subjects was 2013.

What are the actual levels of economically disadvantaged students in each school in 2013?

You can see the graph below for numbers with charters highlighted and the legacy MCS average noted in red:

Percent Economically Disadvantaged Graph

Which Schools Were Used as Poverty Peer Schools?

I selected all schools for each charter within 10% of a similar population of economically disadvantaged students both above and below the measure. For example, if a school had a 70% economically disadvantaged all schools at or between 60% and 80% were included in the peer group.

PCA – Central High, Ridgeway High, Power Center (included) Middle College High, City University, Sheffield, Whitehaven

City University – Central high, Ridgway High, Middle College High, City University (included), Sheffield, Whitehaven

Soulsville – City University, Sheffield, Whitehaven, Hollis F. Price Middle College, Overton, Soulsville (included), East, Kirby, Craigmont, MBA, Wooddale, Carver, MASE, Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS

Memphis Business Academy – Sheffield, Whitehaven, Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Memphis Academy of Science and Engineering – Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Memphis Academy of Health Sciences – Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Comparison schools by zip code:

Which Schools were use as optional schools?

Central, Middle College, Whitehaven, white station, Overton, Ridgeway, Craigmont

[update July 1st, 9:51 AM]

Why No KIPP Collegiate High?

for some reason KIPP has achievement data available through the report card but you can’t pull out achievement within the economically disadvantaged subgroup the way you can with the other six schools, except for world history.  Here’s what we see when we look at their economically disadvantaged data (highlighting added by me):

Why No KIPP

 

Given the interest nationally in KIPP, however, I may go ahead and do a straight comparison regardless in a future post. Thanks to commentor Meg for asking this question!

8 comments for “Just the Facts: Charter High School Performance in Memphis

  1. Meg
    June 30, 2014 at 12:49 pm

    Just a note – I was able to find KIPP’s report card data on the Dept. of Ed website, it’s under KIPP Memphis Collegiate High School. If you can find it, I think it would be great to add it in given the amount of attention KIPP receives nationally.

    • July 1, 2014 at 9:36 am

      Thanks Meg! I could swear that wasn’t there two weeks ago when I started doing this analysis. I will have to go back and do a separate KIPP one.

    • July 1, 2014 at 9:47 am

      Now that I’m looking back I recall the problem I ran into. KIPP has their data online but the only subject that has it broken down by %economically disadvantaged on the state’s report card is World History. Since they didn’t have those particular numbers I didn’t use them in my comparison. I’ll post a picture at the bottom highlighting why I didn’t use KIPP so you can see it

  2. Bert
    June 30, 2014 at 3:02 pm

    Looking solely at socioeconomic status of students is a very narrow view of student populations. I’ve pressed you on this issue before but have been left wanting, specifically with looking at rates with students with special needs. I believe you (and others arguing that charters are outperforming traditional public schools) are purposefully ignoring the differences between the student population of charter schools and traditional public schools. It’s hard to compare and contrast the achievement of two populations when they are not comprised of the same types of students in terms of SPED or ESL. Also, there are softer forms of exclusion like not providing transportation, requiring an essay in an application (and other admission requirements outside of residency), and not allowing for rolling admission throughout the school year.
    It’s a result of an idea I recently came across: Ockham’s broom. To quote, “Ockham’s broom is an implement conceived by Sydney Brenner as the device whereby inconvenient facts are swept under the carpet. This is common practice in biological research where the facts often cannot be explained all at once; but in due course the edge of the carpet must be lifted and the untidy reality confronted” (http://jbiol.com/series/ockhams_broom). Certain facts may be present, but others are being purposefully ignored in favor of a foregone conclusion.

    • July 1, 2014 at 9:34 am

      A couple thoughts Bert. One, a big disagreement that people had with my analysis of the charter sector in Tennessee was that I didn’t look at populations within schools, particularly the percentage economically disadvantaged, which a major focus of the charter school movement. Additionally, while SPED and ESL is important, economically disadvantaged percentages make up the biggest difference between schools, which is why I chose to focus there.

      Second, just for fun I went back and sorted schools for comparison by %SPED population. If you want a comparison by that measure, the Optional school graph offers a pretty close approximation as most charter schools have similar SPED populations to those schools.

      Third, if you have any specific comparisons that you feel are not being represented, feel free to suggest them here or make your own comparison and submit a written piece for review.

      Fourth, I take strong offense to your insinuation that I am purposefully sweeping any facts under the rug. In writing blog posts I am limited by several things; the availability of data with which to do analysis meaning i cannot run every conceivable comparison, the limited tools I have available (microsoft excel) and making the blog post a readable length (nobody wants a 5000 word post with 10 different comparisons). Additionally, this particular piece was in response to discussion I had with Gary Rubenstein and Jersey Jazzman about comparing schools to those serving similar percentages of socio economic students, so it is trying to answer a specific pointed charge often leveled at charters. A blog post cannot be everything to everyone all at once.

      For future comments please consult our comment policy regarding inflammatory and defamatory comments under “submissions.” This is not the Commercial Appeal’s comment section, we want to keep it constructive and avoid unfounded accusations to encourage writers to share their thoughts. I have chosen not to post comments in the past for including similar language.

      • Bert
        July 1, 2014 at 1:18 pm

        I’m sorry you took offense. I understand that there is a limited amount of information and tools at your disposal, but this emphasizes the need for clarity. You should use say things like “Just the Facts” or “Charters are the Real Deal” (and that’s just in the title) if your only analyzing information from a limited point of view. It ends up looking like you’re trying to sell something rather than trying to present a thoughtful presentation of a particular set of data. As we know, there are many ways to sort information.

        As for your reason for the post being a response in an ongoing conversation with others, why would anyone be aware of that other than yourself and them? If it’s an answer to a pointed charge, be clear. You present this as a statement of blanket fact, it’s in your title. If it was named “A Socioeconomic View of Charter Performance in Memphis” then I’d understand. To me, this seems to be a continuation of your last post on charters, which seemed to define being disadvantaged by socio-economic status. Which the state may do, but we know is simply not practically the case.

        As for the nature of my comments, I believe they’re pretty tame. Saying a person is falling into a common problem with analyzing data is not a form of defamation. It’s closer to pointing out a logical fallacy with an argument. To counter your charge of defamation, I’d point you to your own comments with “the discrepancy [in SPED rates between charters and traditional public schools] may be because the public school system is honestly using the SPED designation inappropriately.” Claiming negligence and/or fraud within a school or, worse, an entire school system is far more damning than criticizing one man’s presentation and interpretation of data.

        As for you invitation to contribute, I appreciate the offer. I may take you up on it.

        • July 1, 2014 at 3:35 pm

          Thanks Bert. I’ve had trouble in the past with commentors making personal attacks on individual writers rather than providing a critique of the ideas or information itself. They started off pretty tame but escalated quickly. It actually drove one of my contributors to not want to write again, so I’m a bit sensitive on that issue. I’ll accept that your comments were intended in that spirit of logical critique rather than personal. I do always appreciate your thoughtful commentary, it keeps me on my toes as a writer.

          As far as the continuation of a conversation, I thought that I did provide a reference as to why I chose to use economically disadvantaged data (because its used in one of the common critiques leveled at charters) but I suppose I could have made that clearer.

Leave a Reply

Your email address will not be published. Required fields are marked *