Sophmoric application of readability tests

A story on NPR’s Morning Edition suggests, “the sophistication of congressional speech-making is on the decline,” citing a study by the Sunlight Foundation, a nonpartisan political organization. NPR and Sunlight both present the finding in a way that appears to validate the conventional wisdom that American politics has taken an anti-intellectual turn of late. However, both should remember to beware of intellectual bias and the danger of finding what you were looking for.

Lee Drutman, a senior fellow at Sunlight and a professor of political science at Johns Hopkins University, calculated the Flesch-Kinkaid Readability score for the Congressional Record from 1996-2012. According to his calculation, text in the Congressional Record from 2005 scored 11.5, while text from 2011 scored 10.6.

According to NPR’s Tamara Keith, “In other words, Congress dropped from talking like juniors to talking like sophomores.”

Nonsense. The Flesch-Kinkaid Readability Test purports to be a test of readability, not speech style. Furthermore, although the grade-level score is intended to map reading-ease scores to US grade levels, both scores need to be taken with a proverbial grain of salt.

As Gabe Doyle at Motivated Grammar argued in 2008, Flesch-Kinkaid scores do not correlate significantly with listening comprehension, even for edited prose. Transcription of extemporaneous speech is likely even less well correlated, since things like corrections, false starts, and fillers do not appear in most written texts, and there is often no obvious and uncontroversial way to segment a spoken utterance into sentences. (Flesch-Kinkaid essentially measures the number of words per sentence, and the number of syllables per word. See below for an example of how arbitrary such measurements can be.)

Of course, what lies behind Drutman’s analysis and NPR and Keith’s reporting of it is the impression that recent political movements, especially the Tea Party, have dumbed down the level of political discourse in the United States. New York Times columnist David Brooks has written of the Tea Party, “The members of this movement do not accept the legitimacy of scholars and intellectual authorities.” Conservative Republicans, Tea Party members, even presidential candidate Mitt Romney are accused of either being or acting dumb.

It is against this background that Drutman notes, “[The] complexity of speech in the Congressional Record has declined steadily since 2005, with the drop among Republicans slightly outpacing that for Democrats,” and NPR finds it newsworthy.

I leave it to more competent statisticians to decide whether the variation in grade level scores Drutman found is statistically significant. Given the vagaries involved in deciding how to represent speech in written form, and the irrelevance of reading ease to listening comprehension, though, I suggest that the findings are linguistically insignificant.

Just for fun — punctuation and F-K scores

Tamara Keith cites the following as an example of a sentence from the law maker with the highest grade rating, Republican California Representative Dan Lungren:

This Justice Department, in my judgment, based on the experience I’ve had here in this Congress, 18 years, my years as the chief legal officer of the state of California and 35 or 40 years as a practicing attorney tells me that this administration has fundamentally failed in its obligation to attempt to faithfully carry out the laws of the United States.

By my calculation, that sentence has a Flesh-Kinkaid Grade Level score of 27.63. (According to Drutman, Lungren has a “Career grade level” of 16.01.)

But the phrase “This justice department” does not appear to have a clear grammatical role in any of the clauses that follow it. “This justice department” looks like a topic initiation, and it is co-referential with “this administration”, the subject of an embedded clause. But it is not an argument in the clauses “my years… tells me” [sic] or “this administration has fundamentally failed”, nor in the non-finite clause “obligation to attempt to faithfully carry out the laws”, and it’s certainly not an argument of “I’ve had here”.

Likewise, the phrases “the experience”, “18 years”, and “my years” all appear to refer to the same thing. Lungren thus re-starts the utterance several times. Such restarts are not uncommon in extemporaneous speech.

What if we change the punctuation to make the abandoned and re-started topics in Lungren’s utterance free-standing sentence fragments?

This Justice Department. In my judgment, based on the experience I’ve had here in this Congress. 18 years. My years as the chief legal officer of the state of California and 35 or 40 years as a practicing attorney tells me that this administration has fundamentally failed in its obligation to attempt to faithfully carry out the laws of the United States.

Suddenly grade level drops to 9.68. Is this a more sensible way of calculating grade level? Probably not. Both transcriptions make somewhat arbitrary choices to force the extemporaneous speech into something resembling written prose. Neither tells us anything particularly interesting about Representative Lungren or his speech style. Any analysis of either should probably remain where Drutman says the idea came from: “We just kind of did it for fun.”

Afterword

I anticipate that some commenter may refer to Wisconsin Republican Governor Scott Walker’s position on public education, or erstwhile presidential candidates Rick Perry, who promised to eliminate the US Department of Education, or Rick Santorum, who seemed to suggest that college attendance is snobbery, as evidence of an anti-intellectual bent among Republicans. I’ll stake no strong position on such an argument at this time, but will assert that simple-minded word counting is a poor gauge of trends in political ideology under the best of circumstances.

I also recommend Mark Liberman’s 2010 critique on Language Log of a conversely simple-minded conclusion, a suggestion that a speech by President Obama was “too professorial”. Amusingly, in 2010 Obama’s ninth-grade reading level speech was called too complex, while Congress’s average tenth-grade reading level speeches are now called unsophisticated.