March 25, 2007
Debating SSRN downloads and exclusions
Though Brian Leiter call himself a "skeptic about SSRN downloads," he created and posted here two lists of SSRN download rankings under the heading "Most Downloaded Law Faculties, 2006." A fully accurate heading for Brian's rankings would be "Most Downlaoded Law Faculties, 2006 (excluding one outlier)" because Brian decided and explained that it "was necessary to exclude Ohio State and Emory, whose presence in the top 15 was due entirely to one provocatively titled article by Christopher Fairman."
Professors Ann Bartow at Feminist Law Professors and Neil Buchanan at Dorf on Law and Ruth Colker here at LSI criticized Brian's decision to exclude one paper from his rankings. Ruth's expressed criticisms, which was sent via e-mail to Brian, begat an interesting e-mail discussion with Brian and others (on which I was cc:d). I learned a great deal from the series of back-and-forth e-mails and I asked the group if I could post parts of the exchange on LSI, but not everyone was interested in a full public airing of the e-mail dialogue that Ruth started.
Interestingly, Brian has now discussed aspects of this exchange in this new post that appears to be his defense of excluding one paper from his rankings. Brian's new post suggests that he believes that SSRN download rankings would be transformed from "a pretty weak measure of scholarly performance" to "just a joke" had he not excluded Chris's paper from his rankings.
As I explained on my home blog and as Neil lamented at Dorf on Law, there is a real risk that undue emphasis on SSRN downloads — which already seems to be a reality as evidenced by blog buzz here and here — could significantly distort how law professors produce, disseminate and market their scholarship. I am thus especially interested how folks are responding to Brian's SSRN rankings and to his exclusion decision.
Any further thoughts on SSRN download rankings or on Brian's defense of his rankings? (Comments need to be here because Brian does not enable comments on his blog.)
Posted by DAB
TrackBack URL for this entry:
Listed below are links to weblogs that reference Debating SSRN downloads and exclusions:
Guess what? I am the professor who Brian Leiter complains “went so far as to imply that [he] might be prejudiced against gays and feminists.” How exciting! Except that the truth, as so often happens, is rather dull. The truth involves social science methodology and discourse, not Brian’s personal views. I’ve explained that to Brian; his post persists in confusing methodological concerns with personal prejudice. Here's the full story on an exchange that Brian first demanded be kept confidential but then posted (in part) and criticized on his blog:
In an email to Brian, I noted that Chris Fairman had written a thoughtful new essay discussing whether Brian’s recent ranking discounted the downloads to Chris's “Fuck” article because many of the downloads came from people outside the academy. Was Brian’s ranking, Chris asked, limited to downloads that came from the “right people,” particularly those in the legal academy?
I thought Chris raised an important issue about different audiences for legal scholarship and the ways in which ranking methods can marginalize certain work. I noted this to Brian, urging him to read Chris’s essay, and continued: “[B]ased on both Chris's comments in his essay and a quick look at google links referencing him, the "Fuck" article has been read by a very diverse audience. Ironically, one of the first links you find if you google ‘Chris Fairman Fuck’ is a blog discussion of Chris's paper by Brad Feld, managing director of the Foundry Group and Mobius Venture Capital. Based on your ranking method, it seems that it's legit for Feld to download a paper on venture capital (even if he later discards it as irrelevant or wrong) but not to show an interest in the first amendment--even if he writes publicly [as he did] that he has learned from the paper and thinks it's brilliant. Based on a quick look through google links, some of the other ‘wrong people’ you discount are feminists, gays, and linguistics professors--are these people less worthy as readers than men, straight people, and philosophy or economics professors?”
In other words, Brian counted all recent downloads for most papers in SSRN, including large numbers of downloads for the papers in corporate law and other subjects that appear disproportionately in that database (a concern Brian himself noted), no matter who downloaded them, whether they read the papers, or what they thought of them. But he excluded this one paper because he found it “obvious” that the “unusually high download count was due to its provocative title, not its scholarly content.” In making that assumption, Brian “discounted” in social science terms the audiences who had expressed interest in the paper. This is a garden variety critique of social science research; indeed, it’s the kind of question social scientists routinely ask themselves before publishing the results of a study.
Brian replied to my email by saying that he did not have time to read Chris’s new essay beyond the first few pages because he was “just too busy with real work.” He then expressed outrage, as he does in his blog, that I had accused him of “prejudice against gays, feminists, etc.”
In several emails, I attempted to explain to Brian why I found his rather summary conclusion about Chris’s paper disconcerting, and why I think it’s important to ask whether a methodological decision discounts any relevant interests—in this case, potential audiences for a scholarly work. Bias and discounting in social science don’t refer to personal prejudice; the concepts refer to factors that may interfere with the work accomplishing the task it set out to perform. Brian himself has understood this when discussing other rankings. His discussion of U.S. News on his rankings site repeatedly refers to ways in which the U.S. News method “favors” some schools and “skews” results against others. Another scholar might have referred to “discounting” or “bias,” because “skew” has a specialized statistical meaning, but it’s the same general idea.
From my perspective, and I think from the perspective of others who do empirical work, Brian’s statements about the people who downloaded Chris’s paper are startling. Scientists rarely pronounce the motives of almost 18,000 people “obvious,” especially when those individuals come from many different cultures, countries, and occupations. It is notable, in fact, how many of the internet discussions of Chris’s paper are in languages other than English—a fact I also pointed out to Brian. It seems unlikely that these people downloaded a paper from a scholarly database just because its title was a curse word in another language.
It happens that some of the audiences who found Chris’s paper interesting and discussed it publicly are linguistics professors, feminist scholars, gay scholars, and venture capitalists who like to curse—all groups I noted to Brian. Brian’s method discounts those audiences; that’s simply a statement of social science concern. It has nothing to do with whether Brian personally is prejudiced against gays, feminists, linguistics professors, or foul-mouthed venture capitalists. Indeed, as I suggested to Brian, the very notion of prejudice against linguistics professors and cursing capitalists is rather odd and tends to defeat the implication he claims.
All methodological decisions have the potential to discount interests; social scientists talk about this every day. Scholars who “do” social science by publishing rankings on the web have to be willing to “talk” social science as well. Legal academics, including Brian, regularly criticize the methodological decisions behind the U.S. News rankings, pointing to ways in which they “discount” certain perspectives (e.g., the reputational judgments of lawyers and judges outside the Northeast) and are “biased” against particular schools (e.g., those with a larger number of students). If we’re going to critique methodological decisions in rankings done by groups outside the academy, we have to be willing to ask questions about our own methods as well.
Posted by: Deborah Merritt | Mar 26, 2007 5:35:05 AM
Brian takes a swipe at me as well on his blog. Because I can't respond on his forum, I would like to respond here. He says about my previous comment: "Another Ohio State professor, Ruth Colker, says (falsely) that I "excluded from [the] download list the only piece that is on a civil rights related topic, and included all the corporate pieces." In fact, many civil rights articles accounted for many downloads at many different schools; Fairman's article was, again obviously, not excluded because of its subject-matter. (SSRN is utilized disproportionately by scholars working in corporate law, law and economics, and intellectual property, but that is not the same, contra Professor Colker, as a "bias.")"
Of course, I know the SSRN download count includes civil rights articles, since it includes my own. I simply meant that he excluded the only high impact article that happened to be on a civil rights article but retained all the law and economics papers that had high download counts despite the fact that he knew that, for methodological reasons, there was reason to believe that their download counts overstated their scholarly impact (the criterion that Brian purported to want to measure).
Like Debby, I wasn't using the word "bias" to suggest that Brian was personally biased. I was simply using it in a statistical sense to note that his exclusion decision further biased his already flawed statistical results. Brian, himself, noted some of those flaws; I noted that his exclusion decision contributed to, rather than reduced, those flaws.
I have learned a lot from this exchange with Brian and others about SSRN downloads. I wasn't previously aware that some people or schools were so obsessed with those figures that they were trying to artificially increase those numbers. But now that I am aware of those efforts, it seems even more important for Brian to post download figures in a completely unadulterated fashion. One can now use those pure numbers not to measure the quality of faculties but, instead, to ask what factors produce high download counts. With respect to Chris' article, I think it's obvious that his high download count is caused by the interest in his article outside the narrow world of law professors. Other articles apparently get comparatively high download counts because professors require their students to download them or even ask their friends, faculties, and maybe relatives, to download them.
But I stand by my initial statement that the criteria-less exclusion of one article is flawed methodologically. I am not the one who suggested that Brian should even report download counts by faculties; that was his decision. If he is going to do so, and exclude one article from his count, then he should expect criticism of his methodology.
Posted by: Ruth Colker | Mar 26, 2007 6:30:48 AM
Maybe it's because I'm new to the academy, but I'm not understanding the hullabaloo. Anyone can look at the aggregate numbers on SSRN. So, if someone decides to "massage" the aggregate numbers for their own view of faculty scholarship, it shouldn't be surprising, but I guess it is.
Was there this much uproar about only counting the top 3? That seems much more problematic than excluding one article.
Also, if the goal is to show how faculties perform as a whole, then it isn't all that unreasonable to me to add an asterisk that one provacatively titled paper obtained such a disproportionate amount of downloads that it skews the other numbers.
I guess another way to achieve the same end would have been to make three lists. Top poster to top poster, second to second, third to third, and then the precipitous drop-off would be visible and people could make up their own minds.
An even better solution would be like they do in figure skating - exclude the top and bottom and take an average of the rest.
Just my two cents from the outside looking in.
Posted by: Michael Risch | Mar 26, 2007 11:13:30 AM
FYI Mike: Ted Seot at MoneyLaw was so troubled by only counting the top 3 that he re-ran the numbers exclusing the top 3:
For me the broader point is that there is great value in broad discussion of ranking methods and empirical methodology. I especially think there ought not be exclusions without thoughtful explanations of their justifications.
Here's a useful thought experiment in this contaxt: Do you think there would have been a hullabaloo if Brian decided to exclude all corporate papers because SSRN use clearly skews toward corporate users/topics? (Indeed, given Brian's apparrent political commitments, I was hoping he might consider running the numbers this way to see what kind of results this would produce.)
Posted by: Doug B. | Mar 26, 2007 11:36:29 AM
I read Ted Seto's post and thought it was very good. Didn't seem to spark the same debate.
I think it would be really interesting to see the comparative rankings of corporate, L&E, IP, Civil Rights, First Amendment, etc. That would be really useful information. If those were excluded for a non-corporate ranking, I suspect no one would complain. I guess the umbrage here is the notion that this is a "faculty" ranking when the whole faculty isn't being considered.
As for a thoughtful explanation, I think that "the download results are extremely skewed by a provacatively titled article" is thoughtful enough for me. If I were doing this rankings, I would have excluded any article with the kind of download skew at issue here, regardless of title or subject area.
I did a lot of empirical research as an undergrad both on my own and for professors, and it was quite common to exclude extreme outliers (with no justification other than extremity) in order to see if the rest of the data could otherwise be explained. I don't see this exercise as any different.
Mind you, I don't think SSRN downloads are a terribly helpful count of quality - otherwise, there would be a lot more downloads of my articles! :)
Posted by: Michael Risch | Mar 26, 2007 11:47:56 AM