I found this video created by the American Federation of Teachers (AFT) via facebook.  It gives a critical breakdown of the PISA scores (PISA: an international test administered by OECD countries used to compare student learning across those countries) in an attempt to discount some of the more frequent claims made by education reforms related to failing schools, teacher quality, etc.

It’s colorful, creative, easy to watch and on the surface seems to dispel many of the reformers claims about American public education. However, I’m a big believer in using only verifiable facts.  If we are going to advocate for student centered policies, its very important that we have the truth on our side to effectively combat our detractors.  Misrepresenting facts is the quickest way to lose support.  In that spirit I’ve gone through the actual PISA report on the US and broken down the claims one by one to see how they stack up to detailed analysis.  My critique of this video should not be taken as support for or against any specific policy. Rather, it’s simply an effort to ensure proper representation of the facts as they are.

AFT Claim 1: “it’s not true what reformers say that performance is dropping like a rock.  Our scores have been remarkably consistent across categories and doesn’t warrant the crisis mode we often attach to these scores.”

Fact check: PARTIALLY PROVEN. Though I haven’t heard this specific claim, on the surface it would seem to be correct.  However, one thing that they fail to note from the report is that scores vary WIDELY across states.  For example, in Math, Massachusetts scored 514 (high above the mean and comparable to Germany) while Florida scored 467 (far below the mean and comparable to Isreal).  Note: only three states were tested extensively enough to get state-specific data.

So while their claim that scores have not been dropping would seem to be mostly true country wide, that may not be the case if we look at it on a state by state basis. And while as a country we might not need to go into crisis mode, the case across individual states is likely highly varied and some could still be backsliding.

Claim 2: “poverty’s effect on educational equity disappears when US schools are broken down into brackets (example: schools with 0-10% poverty).  In reality, when we break it down like this we actually lead all other countries in all categories.”

Fact Check: QUESTIONS REMAINING; UNPROVEN.  This data doesn’t come directly from the PISA report, but rather  from an analysis done by the National Association of Secondary School Principals blog.  The data isn’t shared, but holistically there is a big problem with the analysis, namely that the blogger only  compares schools in the United States in each bracket with THE ENTIRE COUNTRY, rather than schools in each individual category.  I’m certain that performance in countries say, like Canada (10-24% poverty) Poland (25-49.9% poverty) could probably be broken down in a similar fashion, would probably yield trends similar to that of the US.  However, since I don’t have the data they used to create these trends and break down other countries in a similar fashion, we will have to label this one as unknowable.

One thing we can be certain of from the PISA report is that poverty accounts for 15% of student outcomes in the US compared to an OECD average of 10%, so we can be confident that poverty is an important variable that must be included in education policy decisions.  But the PISA report notes that there is hope even for high poverty countries:

Some countries succeed, even under difficult conditions, to moderate the impact of socio-economic background on educational success

Claim 3: “the US only ranks slightly above average when it comes to per pupil expenditures when we remove the cost of college from the equation.”

Fact Check: UNPROVEN. AFT uses spending as a percent of GOD.  However, I feel a more accurate measure can be found in the following charts from the OECD Education at a Glance 2013.  Here we see US primary education spending (roughly ages 5-11) and secondary (11 through 16 years) compared using each country’s national currency and its corresponding purchasing power (PPP). Using this metric, the US still trends towards the high end of the spending scale even when college education is removed.  It’s also well above the national average.  

*Disclaimer: I will be the first to say that we need to increase both the level of spending and the intelligence behind our spending here in the US, but we clearly aren’t near the OECD median.  The main difference is in the methods used (OECD uses what the money can by whereas this video uses percentage of GDP).

Claim 4: “The problem is we don’t provide students in poverty with enough educational materials and they also have larger class sizes.”

Fact Check: UNPROVEN.  I can’t find any information on educational materials, but with regards to class sizes, our class sizes in the US from the same report are right around the OECD average.  When looked at in terms of student teacher ratios from the PISA key findings, the report stated that  “there is no significant difference between advantaged and disadvantaged schools in terms of student-teacher ratios” (page 27).

Claim 5: “US Education spending has gone down since the economic crisis with a 1% overall cut.”

Fact Check: MISLEADING/UNPROVEN.  The numbers used in this video again come from the OECD education at a glance 2013 report.  This video claims US K-12 Education spending has fallen by 1 percent since the economic crisis (uses numbers from 2005-2010).  However, that number represents spending as a TOTAL PERCENTAGE OF ALL PUBLIC SPENDING.  If we look one column over in the table, public expenditures as a percentage of GPD, spending has increased by 0.4%.  And if we measure it by total expenditures per student, spending has gone up by about $700 since 2005.  

*A second disclaimer: I am a strong advocate for increased public spending on K-12 education, especially in Tennessee where I work.  But we shouldn’t cite education statistics that can be disproven with a 10 minute internet search in support of this push for more education spending.  Misrepresentation like this discredits any such pushes.

Claim 6: “We also enroll less than 51% of our students in pre-K education, which is very low compared to other high performing countries.”

Fact Check: PROVEN.  No matter how you stack this one up, using the same report as before the US clearly falls near the bottom of the pack. OECD median in this report is 67%, we’re right at 50.  Only 7 countries rank below us while 22 rank above.

Fact Check: PROVEN. Pulled from the National Commission on Teaching and America’s Future (via the NEA).  Additionally, one-third of all new teachers leave after three years.

Claim 8: “US teachers in primary schools spend the most hours teaching in the classroom of any country.”

Fact Check: ALMOST PROVEN.  In reality we rank second but the difference is almost negligible.  In primary education, our teachers are not first, but in fact second to chile, who’s primary teachers spend 1,120 hours teaching per year vs 1,097 in the US.  In Lower Secondary our teachers rank second with 1,068 hours and in Upper Secondary our teachers rank second with 1,051 hours.

Claim 9: “US teacher pay after 15 years in the profession ranks near the bottom internationally.”

Fact Check: MISLEADING/UNPROVEN. The data I found included two measures.  I went ahead and posted both charts below.  The first looks at teacher salary in what it can actually buy.  Seen through this lense, we’re closer to the middle of the pack rather than closer to the bottom.  The measure that the AFT uses is the ratio of experience of salary to GDP per capita.  In laymans terms, this says that teachers have essentially a 1:1 ratio in that they are paid on par with the average worker in the US.  Do I personally think it should be higher? Absolutely.  But the claim itself, again, doesn’t hold water when examined in the context of all the data available.

Claim 10: “OECD says that the claim that other countries only recruit from the top 3rd of college graduates is not supported by evidence.”

Fact Check: PROVEN. Drawn from an OECD report on building a high quality teaching profession, the authors did indeed make this statement and backed it up with evidence.  They also went on to define the policies required to make teaching an attractive profession: supporting continuous learning, developing career structure to give new role to teachers and engaging strong teachers as active agents in school reform, not just implementers of plans designed by others highlight by me).

Claim 11: “The Higher a country ranks the more likely that country is working constructively with its unions.”

Fact Check: MISLEADING/MOSTLY PROVEN. The OECD report on the US does state the claim made by the AFT in their video:

“the higher a country is on the world’s education league tables, the more likely that country is working constructively with it’s unions and treating it’s teachers as trusted professional partners.”

However, the report also contains this nuance:  there appears to be no correlation between the presence of unions and student performance. The key factor is that where unions exist, higher performing countries treat them as valued partners.  Canada and Finland are cited as examples.

Claim 12 (summary): other countries have demonstrated “proven reforms” we need to implement here in the US:

1. Invest in early childhood education
2. Target resources to children that need them the most
3. Give teachers time to collaborate
4. Properly implement a robust curriculum
5. Test to help teachers, not punish them
6. Treat communities, unions and teachers as partners

Fact Check: Here’s a quick run down to analyze this final “proven” reforms and their effectiveness: 

1. Invest in early childhood education (PROVEN)
2. Target resources to children that need them the most (PARTIALLY PROVEN)
3. Give teachers time to collaborate (PROVEN)
4. Properly implement a robust curriculum (NOT PROVEN)
5. Test to help teachers, not punish them (NOT PROVEN)
6. Treat communities, unions and teachers as partners (PROVEN)

Claims 1 and 6 can be demonstrated to be true from the evidence used in the video.  Couldn’t find any information on resources (#2), though one of the OECD papers does make reference to the fact that the quality of resources matter more than quantity in more effective countries.  For #3 I did find a paper by Linda Darling-Hammond on how high-achieving countries develop great teachers that cites some evidence for countries like Japan, Singapore, Denmark, Finland, Hungary and others that provide substantial collaboration time, so I’ll list it as proven.  

While I may agree with claim 4 and 5 in spirit, this sounds like more of an editorial by the AFT than a claim backed by evidence.  

Verdict: while this video does contain some genuine gems of information about best international practices, it also contains a number of misrepresented facts and bits of data taken out of context. Information is useful in developing a narrative to support a specific agenda.  However, if we in the profession want to see change, we need to ensure that our claims are actually backed by data and that above all, we do not misrepresent that data.  This, more than anything, will lead to a more effective push for sound education policies in the US.  When we misrepresent data even in the name of children, we only give ammunition to our critics to shoot down our voice.

By Jon Alfuth

Follow Bluff City Education on Twitter @bluffcityed and look for the hashtag #iteachiam and #TNedu to find more of our stories.  Please also like our page on facebook.