Trust in Standardized Test Scores

If you are like me, you have been following the Chicago Teacher's strike over the past week.  Last Friday, it seemed as though the labor dispute was about to be resolved and schools would re-open on Monday, but that turned out to be wrong.  Union delegates met on Sunday and voted against the city's proposed contract and it now looks like the strike will continue until Wednesday at least.

A close friend of mine belongs to the Chicago Teachers Union.  I've been struck by how different her take on the situation is from the one we get in the local and national media.  Over the weekend, for example, the Chicago Tribune ran an editorial under the headline Don't Cave, Mr. Mayor whose opening paragraph read:

"Over the weekend, Chicago Public Schools leaders offered teachers a sweet deal that would make most workers in the city envious. Teachers stood to reap a remarkably generous 16 percent raise over four years in a new contract. Guaranteed."

Despite the fact that Chicago is broke, the editorial continued, "Everyone knows the Chicago Teachers Union response to the CPS offer: No way."


If you talk to union members, however, the overwhelming concern is not about salary issues at all.  Rather, it's charter schools and standardized testing.

Due to dropping enrollment, Chicago plans to close a large number of failing schools in the coming years -- many of which will be replaced by private, charter schools.  The argument for charter schools is that they provide parents with a choice, which means public schools have to compete.  This, in turn, gives teachers an incentive to improve, or face the threat of school closure.  But from the union's perspective, there is something more sinister going on.  That's because charter schools can hire non-union teachers, whom they tend to pay a lot less (and who also tend to be a lot younger).

Since charter schools get to pick and choose which students are allowed to enroll, they can turn away the most difficult (read: expensive) students to educate.  The latter will then be overrepresented in public schools, which all but ensures that unionized teachers will have lower standardized test scores.  These low scores would then provide the city with a convincing argument -- couched in the numerical language of statistics -- to lay off unionized teachers, close public schools, and replace them with cheaper, private, charter schools.  The fear, then, is that charter schools are basically a form of union-busting, kind of like a 21st century version of the Pinkertons!

I have my own views on both charter schools and standardized tests, but let's leave those aside for now.  One thing that I've been very struck by in the recent debate is the lack of data to support various policy proposals.  Everyone, it seems, likes to shroud themselves in the mantle of pragmatism and say that they will support whatever policy actually works best for the children, but nobody seems to agree on what works and what doesn't.

If you buy the pro-union argument that I've articulated above, you will most likely harbor an a priori suspicion about data-driven policy decisions.  This is a healthy suspicion -- I fully agree -- but, at the same time, one wonders why all those in favor of using statistics from standardized tests to gauge teacher effectiveness so rarely have good numbers about the effectiveness of standardized tests.  Why would The New York Times publish an editorial entitled Chicago Teachers Folly that roundly criticizes the strike without giving any evidence that the city's proposals for school reform are likely to succeed.

Then I came across this open letter by some 88 education professors from the Chicago area (including the infamous ex-weatherman Bill Ayers!) that addressees precisely this point.  Apparently, it's even worse than I thought!  In fact, the kind of data on the effectiveness of charter schools or the reliability of standardized tests that I'm looking for mostly don't yet exist.  For this reason, the open letter pleads with the city to implement its reform efforts slowly, beginning with targeted pilot studies, thereby providing a window of opportunity in which to make sure these efforts do not do more harm than good before they are implemented across the board.

To my mind, the most damning claim the authors of this letter make is about the use of standardized tests as a measure of teacher effectiveness.  Not only does their implementation invariable incentive teachers to teach the test, the letter also cites a number of studies that show existing tests are actually a highly unreliable measure of student improvement.  A teacher's rating based on such tests can significantly change from year to year, class to class, and even from test to test!  Thus, although there may be some correlation between a student's real educational achievements and their performance on standardized tests, the latter is simply too noisy to serve as a reliable indicator for policy decisions.  The letter-writers deploy the following analogy to make their point: "Using [standardized test scores] to measure [teacher effectiveness] is akin to using a meter stick to weigh a person: you might be able to develop a formula that links height and weight, but there will be plenty of error in your calculations."

If that's right, it's hard not to be cynical about Rahm Emmanuel's motivations here.  But then again, he is rumored to listen to Nickelback!