read

This post first appeared on aud.life

I didn’t plan to write about Michael Petrilli’s latest list of “Top K–12 Education Policy People on Social Media 2015,” in no small part because I said something last year. Last year was the second in a row I made the list; I didn’t make the list this year. Petrilli’s qualifications for “Top” changed. Ell. Oh. Ell.

To revisit, here’s what I wrote this time last year:

Renouncing My Klout


For the second year in the row, I’m on (Thomas B. Fordham Institute president) Michael Petrilli’s list of “The Top Twitter Feeds in Education Policy.”


Truth be told, it’s only the second year because last year my friend José Vilson asked why there were so few women and people of color on Petrilli’s list and volunteered my name as someone who might be missing. Thus I was added to the list after it was initially published.


And I’m only on the list this year because I didn’t delete my Klout account – one of the metrics Petrilli uses to determine eligibility – soon enough.


Confession: I noticed Petrilli tweet a week or so ago that he was in the middle of prepping this year’s list; and it reminded me that I needed to delete my Klout account. I’ve never cared about my Klout score and I’ve never used the account, but Klout has, without my consent, created an account and a score for me. Thanks, technology industry!


You actually have to log in – even if you’ve never signed up for Klout – to request the company delete your account. I just did this last night for Hack Education’s Twitter account – an account that is, for all intents and purposes, an RSS bot. And I did this for the Klout account linked to @audreywatters. But apparently not in time to disqualify me from Petrilli’s list.


The Fault in our Algorithms


Naming “the top” is a power play, no doubt. But Klout is an incredibly flawed way to rank the “Top Twitter Feeds in Education Policy” in part because the score doesn’t simply reflect Twitter “influence.” (Whatever “influence” might be. More on that below.) The company encourages users to link their Facebook, Google+, LinkedIn, Foursquare, and Instagram accounts as well as their Twitter accounts and uses data from all these services to calculate and to boost Klout scores. (It also uses Wikipedia and Bing search result data to determine the score.)


I don’t have a Facebook or LinkedIn account. There’s no Wikipedia entry for “Audrey Watters.” So my Klout score, I imagine, is lower for it.


I say “I imagine” because it’s not clear how the Klout score is actually derived. The company says it uses “more than 400 signals from eight different networks to update your Klout Score every day” and uses “machine learning models” to make sense of all the social media data it sucks up. For what it’s worth, however, several years ago someone reverse-engineered the Klout score and argued that about 94% of the differences in people’s scores could be accounted for by the number of their Twitter followers. Surely, it’s tweaked the algorithm since then. Surely.


But we don’t know. It’s a black box, the company’s “secret sauce.”


Of course, complaints about Klout aren’t new. Science fiction author John Scalzi has said that he quit Klout because “I suspect the service is in fact a little bit socially evil.” Fellow SF writer Charles Stross has also described Klout as “evil” – and quite possibly illegal (as data collection without consent violates UK privacy laws).


Ideology and Ranking


But even if we did know the algorithm that drives the Klout score, I’d still want to ask questions about the meaning of the measurement and the weight that the number – any ranking system, really – carries. Why, if nothing else, are we so obsessed with ranking?


What purposes does Klout serve? Whose purposes does Klout serve? Why is Michael Petrilli or Forbes or Rick Hess or any of the other popular list-makers interested in a ranking or rating system for those in education?


See, this isn’t simply about “influence”; it’s about ideology.


I’m in the middle of writing a chapter for Teaching Machines that examines the histories of “intelligence” and ed-tech – intelligence testing, artificial intelligence, “intelligent tutoring systems.” Much like “influence,” “intelligence” is something difficult to define let alone quantify. And yet we do.


We can debate, as philosophers have for ages, the meaning of these terms – “intelligence,” “influence.” But more importantly, we should ask: why do these characteristics matter? To whom do they matter? And once there’s a practice in place that has defined these terms and has designed measurement tools to assess them and a scale to rank them, we should ask what purposes these designations serve. I don’t mean what sorts of perks do you get with your Klout score or your IQ; I mean for us to consider how might these ranking systems reinscribe hierarchy and inequality, all the while purporting to offer an “objective” tool that reflects ability.


Sorta like “science,” but not.


So yes, I’ve been thinking a lot lately about the power that comes with crafting definitions, with promoting standards, with devising measurement systems – and the role that technology and algorithms will increasingly play here.


Whose interests do these definitions and standards and measurements and algorithms serve? What sorts of (often unexamined) legacies do these practices carry forward?


From the OED:


   psychometry: from the Greek ψῡχο- psycho- + -µετρια measuring – literally “soul-” or “mind-measuring.”


   1. The (alleged) faculty of divining, from physical contact or proximity only, the qualities or properties of an object, or of persons or things that have been in contact with it.


   The first reported use of this word was 1854 – J. R. Buchanan’s “lectures on the neurological system of anthropology” in which he wrote “The influence of Psychometry will be highly valuable ‥. in the selection from candidates for appointments to important offices.”


   2. The measurement of the duration and intensity of mental states or processes.


   The first reported use for this definition was 1879 – Frances Galton who wrote “Psychometry ‥. means the art of imposing measurement and number upon operations of the mind, as in the practice of determining the reaction-time of different persons.”


As Mark Garrison writes in his book A Measure of Failure: The Political Origins of Standardized Testing, “Standardized testing – or the theory and practice known as ‘psychometrics’ – … is not a form of measurement. Psychometrics is best understood as the development of tools for vertical classification and the production of social value.”


Psychometry claims to measure the mind. Klout claims to measure online influence. But look at the OED. Look at those definitions: influence and intelligence. Psychometry and Klout. I’m fascinated how they seem to dovetail so neatly in today’s education politics and how readily they become a sort of “disciplinary power” that maintains the functioning of schools, economies, and other hierarchical systems. Who “measures up”?

I stand by what I wrote then (which always feels like a bit of a triumph for a blogger). But I want to add a few more thoughts, particularly as Petrilli has changed his “formula” for awarding his “top” honors:

Klout remains a ridiculously flawed metric.

The startup was acquired (for $200 million) by Lithium Technologies last year, and it really hasn’t been heard from much since. Oh sure, it continues to push out PR boasting on how it “measures influence” in social media marketing but it’s no clearer today about what that actually means. And if someone is going to adopt it as their metric in education, they should probably grok that. As I noted in my blog post last year, there are a lot of signals that purportedly feed Klout’s algorithm, but there’s no transparency on what those are.

So here I must push back on some of what Petrilli writes about Klout, things that strike me as underscoring that he doesn’t care about the algorithm; he’s only obsessed with its output. (Things that, I confess, I find analogous to education reformers’ larger obsession with “what the data says," only insofar as it confirms their political beliefs.) Petrilli doesn’t seem to have explored what Klout measures or how – hence his surprise that people with a lot of followers on Twitter don’t necessarily have high Klout scores. Here’s his explanation:

Perhaps they are powerhouses on Facebook or other social media platforms, or are particularly effective at stirring “engagement” on Twitter (such as getting prominent folks to re-tweet their posts).

“Engagement” is in quotations here, and Petrilli indicates with punctuation that this might a questionable metric. But then he doesn’t follow up with a question – he follows up with a statement which is really an assumption on his part about what Klout might count as engagement. We simply don’t know. Folks might “engage” with Diane Ravitch’s tweets, but rarely does she engage. Rarely does Arne Duncan’s Twitter account "engage" either, unless a Department of Education employee is sitting at the keyboard fielding questions under his name, often under the guise of some department-sanctioned hashtag-sponsored chat. And do folks respond? LOL yes, he’s the Secretary of Education. Does Klout include any sentiment analysis? We don’t know. (How does being “verified” on Twitter – a service made available to politicians, celebrities, and journalists who work at mainstream news organizations – feed one’s Klout score? Again, we don’t know.) Now Xian F’znger Barrett, on the other hand, engages on Twitter. Oh my does he. There have been times I’ve clicked on one of his tweets, and I see that the thread is over 300 Tweets long of Xian going back and forth over and over and over and over and over with some corporate education reformer. Perhaps that’s “engagement” according to Klout. Certainly that’s Xian’s high tolerance for bullshit in 140 characters, according to Audrey.

Petrilli says that folks should sign up for Klout. Again, he doesn't seem to understand that Klout signs you up without your permission. He says that several people on his list of "top" policy people based on Twitter followers don't have Klout accounts. Funny, I was able to find Klout accounts for most of them. I'm pretty sure that Alfie Kohn (who does not really Tweet) never actually signed up for Klout. (His Klout score nonetheless: 64.) In fact, I'm sure that if someone told Alfie Kohn how to log in and delete his Klout account (Alfie: call me), he'd do just that.

So Petrilli's call to "sign up, folks!" is just silly. And honestly, it's just wrong. Klout signs you up without your consent -- wow, that's a lot like how education data collection works. Funny, right?

Who’s eligible for “the list”?

So, I didn’t make Petrilli’s list this year – not because my Klout score was non-existent. (Dammit.) It’s because he decided that “ed-tech” is not “ed policy,” and as such, I – along with a whole crew of ed-tech folks – was deemed ineligible.

Cool story, bro.

“Who counts” is another gate-keeping method to keep prestige metrics “prestigious” – simply eliminate the riff-raff from the outset. It’s likely that the list you create only includes the people you see and the people you see is colored by prestige and privilege and in-group politics. Again, I’ve written about this before as it pertains to another education reformer’s ranking for education scholars. If you make a list of “the top,” you should be honest that it’s nothing more than your viewpoint of “who counts.” It’s your top. In Petrilli’s case, it’s white-guy-conservative-think-tank-top. So golf clap for those who score.

Petrilli, to his credit, did make an effort this year – particularly after being chastised in years past for having no women or people of color on the list – to expand the pool of candidates he rated. That is, he asked on Twitter for nominations, so if you 1) use Twitter, 2) follow him on Twitter, 3) follow people who retweet Petrilli, 4) don’t have Petrilli blocked, you might have seen a call to participate. (Sampling error? Cronyism? I dunno. I’m a folklorist. Not an K–12 education policy expert. Clearly.)

But even with input from his public, Petrilli’s list of 500 overlooks some folks that I think are pretty damn influential when it comes to K–12 education policy. Oh say, Rahm Emanuel – he’s got 119,000 followers on Twitter, for what it’s worth, and can boast a Klout score of 84. What about Scott Walker – 198,000 followers on Twitter and a Klout score of 87. Dude, that means he’s more influential than Arne Duncan! You know who’s even more influential than Scott Walker? John Legend (Klout score: 88; Twitter follower count: 7,260,000).

Sure, you can say “oh no, Scott Walker and John Legend and Rahm Emanuel and Audrey Watters aren’t 100% talking about K–12 education policy therefore they aren’t influential in K–12 education policy like how I mean you should be focused on and influential in K–12 education policy” – but then I call bullshit on your listing and your choosing and your metrics from the start. Who is on Petrilli's list, with the exception of those who simply auto-tweet announcements from their blogs/departments/magazines, only tweets about education policy?

And, what counts as “education policy”? Probably only the messages and the voices that powerful people want to gesture towards including at their table. Gesture. And Klout is a usefully opaque metric for them to decide to whom that gesture “looks like.” Then they can throw up their hands and be all like "man... that's what the algorithm said. And the algorithm is science."

Audrey Watters


Published

Audrey Watters

Writer

Back to Archives