We recently broke the news about U.S. News’ decision to reform its disclosure of surveyed employment information. This was important in three significant ways. First, while only a small victory for those of us calling for more transparency, it’s a meaningful step towards reducing the number of prospectives who make uninformed decisions. These additional data will shed light on the meaning of the employment figures reported on U.S. News’ widely-used website, even though the ABA already publishes answers to some (but not all) of the same questions.
Second, once U.S. News begins publishing all of the important employment data it collects from schools, it will show the tendency schools have to revise their employment outcomes over time due to what NALP has termed the “data drift.” Third, and perhaps most promising, Bob Morse (the U.S. News guru) went on the record in favor of the transparency efforts, making an official pledge on behalf of his organization to publish more employment data.
From Morse Code, Mr. Morse’s U.S. News blog:
Law school students need as much information as possible to help them realistically understand the employment prospects from their school and the economic value of their degree in terms of their ability to pay back loans and earning power. U.S. News believes the information we will be publishing will help current students in those efforts. However, disclosure of employment data by law schools is still woefully lacking given the cost of attendance and poor job market. U.S. News strongly backs all ongoing efforts to require law schools to report even more detailed data on the how recent grads have fared in the job market. We would collect and publish those statistics if they were available.
Improving Visibility and Revealing the Data Drift
U.S. News was very receptive to our suggested changes in all communications, and Mr. Morse’s public comments are a clear sign that the organization behind the notorious rankings is aware of the problems that prospective law students face. It is worth noting once again that U.S. News actually collects more information about employment outcomes (thanks to its market power) than the ABA does using its regulatory power.
Some commenters have rightfully pointed out that a lot of our suggestions to U.S. News do not include data that aren’t already available from the ABA’s Official Guide to Law Schools. We only asked U.S. News to disclose data it already has, rather than lobbying for U.S. News to expand or refine its survey. But as it turns out, schools are providing different answers to the same questions over time.
According to Jim Leipold, Executive Director of NALP, schools are actually revising the data they submit over time for the same graduating class. At last week’s Questionnaire Committee hearing, Mr. Leipold spoke of the “data drift” among school-reported employment information, which make datasets submitted to the ABA and to U.S. News incompatible. For a given year, schools report data to NALP in February, U.S. News in March, and the ABA in October. For some schools, these figures are all different, even though they all reflect the same graduates’ post-graduation outcomes as of February 15 in the relevant year.
This is not because schools should answer questions differently when reporting to these three organizations, as each organization uses the same definitions and denominators. Rather, schools refine their answers over time as more information comes available. For example, a career services dean may learn in June that one of the students she could not track down was actually (un)employed in February, thus report one more (un)employed graduate to the ABA as compared to NALP. This phenomenon means that using the ABA data to better understand the U.S. News information is unreliable because the data sets are incompatible. As such, the new data U.S. News has pledged to share only appears to be redundant.
It is important to recognize that U.S. News plays a significant role in how prospectives choose where to go, and that they share responsibility with the law schools and the ABA for making sure prospectives have the information they need. The U.S. News rankings are particularly important in decisionmaking, for better or worse. Many have argued, including Law School Transparency, that prospectives put too much weight on a school’s rank. These arguments stem largely from the belief that the rankings suffer from a number of methodological flaws (law review citations at the end of this post), especially the attempt to rank local and regional schools on a national scale, the reliance on incongruent components, and the lack of a meaningful answer to “what do these rankings measure?”
We know that many prospectives use U.S. News rankings as a proxy for employment opportunities, given the lack of meaningful information from the schools or the ABA. But is that what these rankings actually show? If we look at placement in the largest national law firms, based on the 2005 NLJ 250 chart (2007, 2008), the rankings do a pretty good job at sorting the most national law schools. But even then, they only have meaning on a national scale that misses the importance that regionality plays in legal hiring. For the tens of thousands of prospectives who will end up attending a different law school that doesn’t place into the NLJ250 or who aren’t interested in working for large law firms, the rankings do not effectively distinguish schools by placement ability.
All of this is beside the point. Regardless of the many concerns about how the rankings distort the application process, there is no doubt that U.S. News’ data collection fills part of the gap that the ABA has historically left wide open. U.S. News’ commitment to disclosing more information suggests that the organization is not only paying attention to this debate but siding with those of us who are arguing for improved disclosure.
U.S. News is also showing a willingness to listen as it considers what steps to take next. Mr. Morse and his staff acknowledged that they could provide more assistance by actively seeking input on how they can help more effectively. In all of this their dedication to helping prospective students is especially important, particularly as the ABA considers making changes to what schools must report to maintain accreditation. One of the biggest differences between what the ABA requires and what U.S. News asks for (and usually gets) is salary information, which the ABA has not yet deemed to be basic consumer information. A call by U.S. News for the ABA to collect better consumer information has the potential to make a significant impact, and we will be keeping an eye on any changes over at the ABA. The Standards Review Committee is having its next meeting in a few weeks, when we look forward to continuing the discussion about the need for better regulation.
Some U.S. News rankings criticisms:
Brian Leiter, How to Rank Law Schools, 81 IND. L.J. 47–50 (2006); Andrew P. Morriss & William D. Henderson, Measuring Outcomes: Post-graduation Measures of Success in the U.S. News & World Report Law School Rankings, 83 IND. L.J. 791 (2008); Richard A. Posner, Law School Rankings, 81 IND. L.J. 13, 13 (2006); David A. Thomas, The Law School Rankings Are Harmful Deceptions: A Response to Those Who Praise the Rankings and Suggestions for a Better Approach to Evaluating Law Schools, 40 HOUS. L. REV. 419 (2003). But cf., e.g., Paul D. Carrington, On Ranking: A Response to Mitchell Berger, 53 J. LEGAL EDUC. 301 (2003); Russell Korobkin, In Praise of Law School Rankings: Solutions to Coordination and Collective Action Problems, 77 TEX. L. REV. 403 (1998).