Is the U.S. News Ranking Methodology Too Simple?

[WSJ Law Blog]

The U.S. News & World Report annual ranking of U.S. law schools has become the legal academy’s favorite punching bag. Every year, about this time, folks start criticizing the survey’s methodologies, reiterating how easy it is for schools to game the system. (At the same time, few can question the survey’s importance.)

In recent years, one method some schools used to boost the average test scores of their students was to cut the number of full-time students and add more part-time students. (Click here for a 2008 WSJ story by Amir Efrati, touching on this phenomenon.)

But we recently came across a different, perhaps more fundamental critique of the survey over at the Concurring Opinions blog.

The post, written by George Washington law professor Daniel Solove, opens with this explanation:

Every year, US News compiles its law school rankings by relying heavily on reputation ratings by law professors (mainly deans and associate deans) and practitioners and judges. They are asked to assign a score (from 1 to 5) for the roughly 200 law schools on the form. A 5 is the highest score and a 1 is the lowest. While many factors that go into the US News ranking have been criticized, the reputation ratings by and large are considered one of the best components in the ranking system. But should it be?

And how, exactly, might one go about critiquing so straighforward a methodology? A school has a reputation; deans rank what they feel those reputations are, on a scale of 1-5. No complicated formulas, just an easy (and admittedly subjective) reputational ranking. What could be simpler?

But in Solove’s mind, it might be too simple. He continues:

Here’s my hypothetical dean’s stream of consciousness:

Okay, I think Yale is the top law school, so I’ll give it a 5.

What about Michigan? Great school, but not quite as high as Yale. I’ll give it a 4.

Cornell is an excellent school too, one of the best. But it’s not Yale or Harvard, so I can’t give it a 5. It’s not as good as Michigan in my view, so I can’t give it a 4. I gave Penn and Berkeley 4’s too, and I think Cornell isn’t quite at the same level. So it’s a 3.

What about USC? Another excellent school, but it’s not as high as Cornell. So it’s a 2.

Ruh-roh! I’m not even out of the top 20, and I have 160+ law schools to assign scores to, and I only have one number left. But I must go on!

How about Emory? That’s a bit lower than USC in my view, so I’ll give it a 1.

What about American? Another terrific school, but I think Emory’s better. I can’t give American a 0. What do I do? Okay, I guess I’ll give it a 1 as well.

But I’m not even out of the top 50. Yikes! I’ve run out of numbers. Maybe I’ll call Robert Morse and ask him if I can start assigning negative numbers. What do I do?

Solove tries to solve it. He writes:

Time to try some math. To make things easy, I’ll assume there are roughly 200 law schools. And I have 5 numbers to assign — 1, 2, 3, 4, and 5. Assuming an equal number of schools assigned to each number, that’s 40 schools for each number. OMG! So I need to give schools I rank 1-40 a score of 5, schools I rank 41-80 a 4, schools I rank 81-120 a 3, schools I rank 121-160 a 2, and every other school a 1. But that’s ridiculous. The law school I think ranks #40 isn’t anywhere near the law school ranked #1. This system is impossible.

He leaves the post feeling, yes, a bit flummoxed, it seems. He asks: “How are people to fill out the US News ranking forms in good faith to reflect accurately their sense of law school reputations?”

We certainly don’t have the answer. We recently left a message with Morse, but have yet to hear back. We’ll of course let you know if we do.