Law School Innovation

Rankings leakage and other buzz-worthy stuff around the blogosphere

The US News law school rankings are officially due out later this week, but there is already much buzzing in the blogosphere about the leakage of the (perhaps official) results.  Here is just a sample of all the blog talk on the rankings from various different law professor blogs:

• Early Release of 2010 U.S. Law School Rankings
• Extensive 2010 US News Law Rankings
• 2009 Reputational Scores from U.S. News Surveys of Academics and Practitioners
• U.S. News and "assessments"

I am, of course, eager to have comments about all the comments about rankings here.  But I would also like to recommend other law school blog reading that, in my opinion, deserves more attention from law professors than the latest round of the usual ranking wrangling:

• The Worthlessness of American Legal Education
• Pre-Law Courses Would Help

Amazing law school insights from an innovative source

I was at first amused and then amazed by the terrific law-school-selection advice to be found in this great column by Clay Travis at CBS Sports.  Though the first 7 points are cheeky, the final 10 are remarkably insightful about so many aspects of law school.  Here are extended snippets from a must-read piece for everyone around a law school or aspiring to be around a law school:

Judging by the e-mails to ClayNation the overlap between lawyers, aspiring lawyers, and sports fans is substantial.... So without further ado, here's ClayNation advice on how to pick a law school.

1. Visit the school when there's good weather, if at least half of the guys aren't wearing shorts, flip-flops, and t-shirts, then you don't want to go there.  Law school should be fun because being a lawyer isn't fun....

2. Don't be completely seduced by law school rankings....

4. Go sit in the law school library for a half-hour. Pretend to read a newspaper and check to see how often the students smile or laugh when interacting with each other. If no one ever smiles or laughs it's a horrible sign.

5. Think about the size of the law school....

8. Assuming the law school is above 80 percent, comparing bar passage rates tells you nothing about the quality of a law school. Don't be moved by the trumpeting of these stats. By the time you're studying for the bar exam you realize that most intelligent people could spend three months studying the Barbri course outlines, memorizing the absurd MBE fact patterns, and pass the bar exam. But by that time you've spent three years learning how to be a lawyer. Congratulations. Once you've practiced law for a couple of years you wouldn't be able to pass the bar exam. It's a great system.

9. The better the school you attend the more it costs but the less hard you have to work while you're there....

10. If you destroy the LSAT and have a good GPA you may have a decision to make regarding whether to take a scholarship to a lesser school or pay more to go to a more selective school.  It's hard to give advice in this situation because no matter what you think now, you have no real idea whether you'll actually like practicing law....

15. Where should I go to make the most money?  If money is your ultimate goal you shouldn't be a lawyer. There are thousands of ways you can make much more money. Plaintiff's lawyers notwithstanding, as a lawyer you're ultimately hamstrung by how many billable hours you can crank out. And every hour you bill is one less hour you get to have a life. Be careful chasing those big firm golden handcuffs, be careful.

16. Keep in mind that the law is completely unbalanced when it comes to career search.  The most competitive jobs in the legal profession are either the highest-paying or the lowest paying....

And if you're miserable [in law school] keep in mind that after six months of practicing there's a 100 percent chance you'd rather be a law student again. Law school is one of the few places on earth where students compete to one day be able to go back to law school. If you doubt this pick up the resumes of your favorite professors who are under 45. Look at how few years they actually spent practicing law. Yep, in the circle of law-school life, as soon as you leave, you start working on finding a way to get back.

Isn't bar passage a terrible law school ranking metric?

Responding to Mark's post on US News rankings, co-blogger Anupam comments that a useful ranking metric "might be Bar Passage, adjusted to reflect the jurisdiction's overall bar passage rate."  I could not disagree more, in part because I think bar passage is a very harmful aspect of the US News ranking system.  Let me explain:

Bar passage rates tell us what percentage of a law school class has passed the — silly? culturally biased? poorly graded? — timed high-pressured test that many jurisdictions use as one barrier to being a licensed lawyer.  I have long been troubled by bar exams for lots of reasons (too numerous to detail here), and I am especially troubled that US News gives these exams extra legitimacy through its ranking criteria.  Let me (too) quickly explain my anti-bar bias:

1.  I do not think the sole or chief goal of law schools is to help a student pass the timed high-pressured bar exam.  Notably, major law schools clearly don't think this should be their sole or chief goal: if they did, law school classes would look and sound and operate much more like Bar-Bri classes.

2.  Because law school is obviously about a lot more than bar passage, every rational law student (with sufficient resources) takes a bar prep course.  Consequently, it seems fair to assume that bar passage rates reflect the quality of a bar prep course more than the quality of law school instruction. 

3.  Bar passage rates also, obviously, reflect the quality of the student body that a law school admits.  But US News and other rankings already use a variety of other metrics to directly assess/reward the quality of the student body that a school admits.

4.  In my view, students and faculty at most schools — at least those outside the top 10 — already obsess way too much about bar passage rates (in part because US News has used this as a metric).  I do not want there to be even more energy focused on a timed high-pressured test that seems, in my view, to be pernicious in many ways.

Of course, I may be wailing on Anupam's comment principally because I have long wanted to wail on the craziness I see in bar exam realities.  So, because I realize I may be blinded by my anti-bar biases, I would like to hear Anupam and others explain why bar passage might be a useful and valuable law school ranking metric.

A new law school ranking system ... and no blog metric?!?!?!

According to this article online at Insider Higer Ed, The Green Bag has a new law school ranking system in the works.  Here is how the article describes the plan:

The Green Bag ... announces in an editorial in its forthcoming issue [that] this spring, it will begin work on the “Deadwood Report,” which it envisions being an annual assessment of “whether faculty members do the work that the law schools say they do.”  The journal acknowledges that the ranking will provide “rough and admittedly partial” measures of law school faculty quality, but posits that by being transparent (it will disclose the sources of its data and how it derives its numbers and rankings from those data), and by bringing more information into public view, “it will help law school applicants make better decisions about where to study or work.... We are trying to do some good here.”

The Green Bag editorial kicking this all off is a very interesting read, though I am not sure it (yet) provides the supposedly transparent reality it aspires to create.  Nevertheless, anyone with a web-based inlking will be interested to see this aspect of the proposed methodology:

An up-to-date web site is a wonderful thing.  That is where we will gather all of our information.  This seems reasonable to us because your web site is surely where most applicants and other inquisitive people go for information about your law school.  If a school cannot be bothered to provide accurate information about the teaching, scholarship, and service of its own faculty on its own web site, it deserves to be haunted by any inaccuracies.

This is no indication, however, that law faculty blogs or blogging activities will be a significant part of the endeavor, even though faculty blogging and/or other types of faculty web dissemination would seem to be a valid and significant way to assess and measure whether faculty do what law schools say they do.  Needless to say, blogging issues aside, this Deadwood Report is a project worth watching for all those interested in the law school universe.

Posted by DAB

Understanding US News Law School Rankings

Link: SSRN-Understanding the U.S. News Law School Rankings by Theodore Seto. Loyola (Los Angeles) prof Theodore Seto has written an important study of the US News methodology, which should raise many cautions among those who rely upon the study.

This Article explores in detail the U.S. News & World Report law school rankings. Its descriptions, analyses, and conclusions are based primarily on U.S. News' published descriptions of its 2006 computations, telephone conversations with U.S. News' staff clarifying those descriptions, and a spreadsheet I have written that approximately replicates those computations. The Article's goals are relatively modest: to help prospective students, employers, and other law school stakeholders read the U.S. News rankings more critically and to help law school administrators get a better handle on how to manage their schools' rankings. In addition, the Article suggests ways in which U.S. News methodology might be improved. It does not, however, purport to offer a systematic critique of either the U.S. News rankings or ranking in general.

Part I describes both U.S. News' 2006 methodology and problems involved in replicating it. Part II is intended to help prospective students, employers, and other law school stakeholders read U.S. News' results intelligently. Prospective students and others trying to understand how to use U.S. News' rankings in their decision-making may wish to focus on this part, although a reading of Part I may also be necessary to understand some of the technical details. Part III addresses the problem of managing rankings. Part IV, finally, suggests ways in which the rankings might be improved.

Anupam Chander

Stanford Law Students Rank Law Firm Diversity: Will Clients Notice?

A bunch of law students at Stanford have started assigning letter grades to their prospective employers...

The students are handing out “diversity report cards” to the big law firms, ranking them by how many female, minority and gay lawyers they have.

“Many of the firms have atrocious, appalling records on diversity,” said Michele Landis Dauber, a law professor at Stanford and the adviser for the project, called Building a Better Legal Profession. The rankings are at www.betterlegalprofession.org.

In New York, Cleary Gottlieb Steen & Hamilton got the top grade, an A-minus. At Cleary, the project says, 48.8 percent of the associates are women, 8.7 percent are black, 8.3 percent are Hispanic and 4.5 percent are openly gay.

Herrick, Feinstein, by contrast, got an F. Its numbers: 37.7 percent women, 4.9 percent black, 1.6 percent Hispanics, and no openly gay people.

... The numbers were provided to a central clearinghouse by the firms themselves. “Our process is simple,” the student group said in explaining its methodology. “Cut, paste and rank.”

Firms in the top fifth received A’s, in the second fifth B’s, and so on. Overall grades were arrived at by averaging grades for partners and associates in five categories: women, blacks, Hispanics, Asians and gay people.

The diversity breakdowns have been available to students for at least 15 years, through the NALP guide, which carefully catalogs the number of associates and partners by gender and ethnicity.

I suspect that minorities and women contemplating offers have long scrutinized the NALP reports, especially to see whether there is any viable partnership track for them, as indicated by history.

The Stanford Law Students leading this project have simply offered one kind of aggregation of those detailed numbers, simplifying the analysis for students. This strikes me as a valuable service. Firms that lag seriously in hiring women and minorities are likely to be unattractive for many.

One interesting question is whether the Better Legal Profession grades will become popular among clients. My own anecdotally-derived view is that businesses are ahead of law firms in diversity (is there a law firm equivalent of McKinsey's Rajat Kumar Gupta, Stan O'Neal, or Meg Whitman?), and thus they may be concerned when one of their major vendors seems indifferent to it. They are perhaps less likely to have used the NALP resource previously. But they may become concerned that their law firm receives low marks for diversity.

Cleary, Gottlieb, my old firm, has long sought to improve the diversity of its lawyers, driven in part by enlightened leadership and in part by the fact that a large part of its clientele is foreign.

Anupam Chander

Would it be unethical (or even illegal) to put my US News vote up for sale?

For the second consecutive year, I have received US News' survey asking me to help identify "law schools having the top programs in intellectual property law."  My receipt of this survey highlights just how flawed some aspects of US News' rankings can be. 

I was an IP litigator a decade ago and I taught a few IP courses early in my career.  But, especially with my primary field so active, I cannot even hope to keep up with all the IP doings in law schools.  Nevertheless, US News seeks my opinion on which 15 schools have "the highest-quality intellectual property law courses or programs."  Candidly, I have absolutely no idea.

I suppose I could try to make educated guesses about the best IP programs based on who sent around the hottest "law porn" covering IP topics this year.  But I also could throw darts at the survey form and probably not do much worse. 

Consequently, I am now wondering if I could and should simply offer my US News survey to the highest bidder.  Helpfully, US News promises that survey responses are kept confidential, so nobody would know whether or to whom I sold my vote.

Of course, I do not want to do anything unethical or illegal, so I am genuinely seeking an answer to the question posed in this post's title.  I know vote selling in some contexts can be illegal, but I don't think a survey by a private magazine garners too much public protection.  As for ethics, well, what I am proposing seems no less savory than what some schools have reportedly done to game the US News rankings system.  Plus, some recent research suggests that open vote buying/selling may be efficient in this kind of setting.

Posted by DAB

P.S.  Though I started this post in jest, the free-market libertarian lurking inside me is really getting jazzed about this US News vote-selling idea.  Perhaps some law-and-econ folks need to talk some sense into me.

Innovation and Law School Rankings

Doug asks in an earlier post whether it might be useful to rank schools based on innovation. I suspect that the principal uses of law school rankings are (1) to signal quality of the education to students; and (2) to signal quality of students to employers. An innovation-meter would not have a significant relationship to either, I think. Schools that innovate may not necessarily be great in teaching--some of the best teachers at UC Davis have been teaching using the same methods that they've used for decades. Yes, innovation can improve teaching, student experience, employment prospects, and scholarship--but it isn't the best indicator for the quality of the education offered, or for the quality of the student. Take two schools, Blackacre Law School and Whiteacre Law School, both exactly equal in T0. If Blackacre innovates--say by offering an array of teaching styles other than the traditional Socratic method, offering allowing its students to fulfill a substantial percentage of requirements by self-directed study or non-law related internship--does that make Blackacre the better school for either students or employers? This blog, of course, does not promote innovation-for-innovation's-sake, but innovation likely to improve legal education and scholarship, so perhaps my concern could be met by any innovation metric somehow accounting for the quality of the innovation. (Congrats, Doug, on the incisive, top of the fold, NY Times front page quote on the Libby commutation, by the way!) Anupam Chander

A lot of new buzz about rankings, though little talk of innovation

Perhaps in part because the summer is the time for admitted law students to decide where they are going, there lately has been lots of new buzz about law school rankings.  For example, the Wall Street Journal has this new article, entitled "Law Schools Also Ranked By Blogs Now," and the WSJ Law Blog has this follow-up post that has generated interesting comments.  Meanwhile, over at the National Law Journal there is this new article, entitled "Law Schools Unlikely to Boycott Magazine Rankings: Fallout from liberal arts boycott of 'U.S. News & World Report' survey minimal."  Disappointingly, I see little discussion of innovation in this coverage of rankings.

Perhaps this blog should start its own ranking of the most innovative law school.  I think my own school  has a number of innovative programs, though I am not sure how exactly I would develop a metric for ranking law school innovations.  Readers are highly encouraged in the comments to (1) suggest ways to assess/measure law school innovation, and (2) nominate particular school for top innovators.

Posted by DAB

How Do U.S. News Rankings Affect the Financial Resources of Public Colleges?

A recent study's findings suggest that "in addition to a consumer response, the publication of quality rankings may influence the provision of quality through a political channel." Details at Law Librarian Blog. -- Joe Hodnicki

Thoughts on US News rankings?

This year's US News rankings are out, and they are already being discussed broadly around the blogosphere as detailed here and here and here.  Commentary, rants and even praise(?) concerning the latest rankings are encouraged in the comments.

I'll start with a sports observation based on my own cynicism of the US News methodology.  I have long thought that success of a school's sports teams might raise US News rankings because of the positive buzz and name recognition that comes with football and basketball success.  But, providing anecdotal disproof of this hypothesis, the two schools that have dominated the college sports headlines this year, Florida and my own Ohio State, fared very differently in this year's rankings.   

Posted by DAB

Debating SSRN downloads and exclusions

Though Brian Leiter call himself a "skeptic about SSRN downloads," he created and posted here two lists of SSRN download rankings under the heading "Most Downloaded Law Faculties, 2006."  A fully accurate heading for Brian's rankings would be "Most Downlaoded Law Faculties, 2006 (excluding one outlier)" because Brian decided and explained that it "was necessary to exclude Ohio State and Emory, whose presence in the top 15 was due entirely to one provocatively titled article by Christopher Fairman."

Professors Ann Bartow at Feminist Law Professors and Neil Buchanan at Dorf on Law and Ruth Colker here at LSI criticized Brian's decision to exclude one paper from his rankings.  Ruth's expressed criticisms, which was sent via e-mail to Brian, begat an interesting e-mail discussion with Brian and others (on which I was cc:d).  I learned a great deal from the series of back-and-forth e-mails and I asked the group if I could post parts of the exchange on LSI, but not everyone was interested in a full public airing of the e-mail dialogue that Ruth started.

Interestingly, Brian has now discussed aspects of this exchange in this new post that appears to be his defense of excluding one paper from his rankings.  Brian's new post suggests that he believes that SSRN download rankings would be transformed from "a pretty weak measure of scholarly performance" to "just a joke" had he not excluded Chris's paper from his rankings.

As I explained on my home blog and as Neil lamented at Dorf on Law, there is a real risk that undue emphasis on SSRN downloads — which already seems to be a reality as evidenced by blog buzz here and here — could significantly distort how law professors produce, disseminate and market their scholarship.  I am thus especially interested how folks are responding to Brian's SSRN rankings and to his exclusion decision.

Any further thoughts on SSRN download rankings or on Brian's defense of his rankings?  (Comments need to be here because Brian does not enable comments on his blog.)

Posted by DAB

SSRN rankings and Leiter's (rank?) omission

Aided by this post at Feminist Law Professors by Ann Bartow, I see a possible controversy brewing over Brian Leiter's decision, in his most recent SSRN download calculations, to exclude my Ohio State colleague Chris Fairman's heavily downloaded paper, Fuck.  Ann's post assails the decision, and my colleague Ruth Colker sent Leiter this e-mail (which she authorized my posting):

Brian, I was shocked that, after you acknowledged the bias in favor of individuals who write in the corporate law area with respect to download counts, you then excluded from your download list the only piece that is on a civil rights related topic, and included all the corporate pieces. I have read and learned significantly from Chris Fairman's piece entitled Fuck, which will soon be published in the Cardozo Law Review.  A criteria-less exclusion of one article is empirical scholarship at its worst. Ironically, a reader of Chris' piece would have expected your actions, because law and society often overreact to the mere use of the term "fuck" without consideration of its meaning in a particular context.  I encourage you to reverse your actions and republish the list without implicitly editorializing about whether one article "deserves" its downloads.

Joyfully, Chris Fairman has himself jumped into the fray with an intriguing essay discussing this situation available here (via SSRN, of course).  Chris' new piece is entitled "Fuck and Law Faculty Rankings."  Because sequels rarely do better than the original, I doubt that this new fuck-related meta-scholarship will perform as well as Chris' first piece.  But perhaps it will still help folks figure out what the fuck to make of SSRN download counts.

Posted by DAB

UPDATE: In this post entitled Quantifying Scholarship, Neil Buchanan weighs in on the problems of relying on cite counts and/or SSRN downloads in law faculty rankings.  Especially notable is this (unsurprising?) report Neil provides in the comment section to his post: 

[A]n acquaintance from another law school called me recently and asked if I would please download all of his papers from SSRN.  His dean is making a big push to pump up their numbers.

Posner on the Economics of College and University Rankings

Asked:

The rankings raise several interesting economic questions: the effect of rankings on information costs, in general and with particular reference to higher education; the manipulability of rankings by the colleges themselves; the effect of the rankings on education; and why U.S. News & World Report's annual rankings, though fiercely criticized by prominent universities (such as Stanford), face little competition.

And answered by Judge Posner on The Becker-Posner Blog. -- Joe Hodnicki

How Do U.S. News Rankings Affect the Financial Resources of Public Colleges?

In The Power of Information: How Do U.S. News Rankings Affect the Financial Resources of Public Colleges? (NBER Working Paper No. 12941) Ginger Zhe Jin and Alex Whalley reach three main conclusions:

1. USNWR coverage causes colleges to increase educational and general expenditures per student.
2. These expenditure responses are funded by a 6.5% increase in state appropriations per student, but tuition revenue does not respond.
3. The state appropriation response to USNWR exposure is larger the larger the pre-college age population, voter turnout and USNWR newsstand sales are in a state.

The authors write "our results suggest that, in addition to a consumer response, the publication of quality rankings may influence the provision of quality through a political channel."  -- Joe Hodnicki

Are the U.S. News rankings a force for or against innovation?

Back in November, Doug asked essentially this question, and didn't get much response. Perhaps we should have offered "free coffie." However, as April approaches, law deans and profs around the country no doubt are feeling somewhat more anxious about rankings, even without a big cuppa coffie. There is probably nothing more important to the image of many schools than their ranking in the U.S. News survey each year, which will be released about five or six weeks from now.

Doug's observation was that the rankings probably spur innovation in lower-ranked schools (as they try to improve) and discourage them in higher-ranked ones (as they stay with a formula that got them there). Since he made that observation, however, at least two institutions (Harvard and Stanford) have announced curriculum changes that were described as major shifts in emphasis.

In the end, though, my own suspicion is that the rankings very generally suppress innovation at both high and low-ranked schools. Over time, the two proven methods of boosting (or maintaining) ranking seem to be (1) using scholarships to attract more-qualified students and (2) increasing production of scholarship by the faculty. Notably, for the most part the type of scholarship that is rewarded is the most traditional kind (long law review articles), and there isn't much innovative about enticing students with cash. So long as resources are focused on these two things, innovative teaching and scholarship (which is inherently riskier) probably are not going to be valued as highly.

This is not to say that innovation doesn't have some positive effect on rankings. For example, if a well-qualified prospective student sees innovative teaching, she may be drawn to that school. Similarly, some of the most successful scholarship is also innovative in its approach, and at times is successful in large part because it is innovative. However, this is just a fraction of the innovation people are bringing to their teaching and scholarship. Shouldn't it count for more?

-- Mark Osler

Does US News promote good or bad innovation?

Thanks to Paul Caron at TaxProf Blog here, I see that the cover story in the January 2007 National Jurist is entitled "The Rankings Race: How Far Will Law Schools Go to Win?"  Here is the teaser:

Misreporting or selectively reporting statistics, firing deans, spending thousands on marketing; the law school rumor mill runs rampant with stories of law schools going to great lengths to boost their U.S. News & World Report ranks.  How much is true, and why are schools willing to put it all on the line just to get a few points higher on the list?

As Paul notes, the story does not break any new ground (though the technology supporting web access to the National Jurist article seemed impressively innovative).  Intriguingly, two high-profile law professor bloggers get their pictures and pull-out quotes in the piece.

Perhaps to provoke some discussion, let me make this somewhat provocative point about the impact of US News rankings on law school programming and activities: though I am sometimes troubled by how US News can impact law school programming and activities, I am far more troubled by how inertia and status quo biases impact law school programming and activities.  As but one example, I suspect US News concerns about reputation scores has helpfully fueled interest in work-related blogging and on-line activities by law professors, while inertia and status quo biases have harmfully retarded these valuable activities.

Posted by DAB.

Does US News spur or thwart innovation?

This weekend, this fascinating blog post at MoneyLaw reminded me of Dale Whitman's account, when he was AALS president, of some of the sneaky techniques and ugly consequences of undue concern with US News rankings.  In turn, I got to thinking about the relationship between US News rankings on law school innovation (which, arguably, is what MoneyLaw is all about).

My first-cut instinct is that great concern for rankings might spur innovations within schools that are ranked low: the dean and faculty at a school ranked low might conclude that the best way to try to change their fortunes would be though innovative programming that garners (positive) attention to improve the school's reputation (and thus its scores in the reputation criteria than make up 40% of the ranking metric).  As a corollary, highly-ranked schools might resist innovations for fear that anything new risks generating negative attention to move reputation scores in the wrong direction.

I am sure that the relationship between rankings and innovation is much more nuanced than this simple account may suggest.  Thus, I encourage readers (and other contributors to this blog) to chime in on this topic.

Posted by DAB