The 509 Subcommittee’s Draft Proposal: An Explanation and Evaluation

This is our third post in a series of posts (see the first and the second) where we contemplate the 509 Subcommittee’s draft proposal and the facts needed to understand how it would advance transparency at ABA-approved law schools. This post will explain the new proposal and evaluate it using the three criteria we set out in the second post.

The Subcommittee’s Proposal

On March 14th, the Subcommittee released its first draft proposal for a revised standard for the reporting of employment data. David Yellen, dean of Loyola University Chicago School of Law and chair of the Standard 509 Subcommittee, will present this proposal to the Standards Review Committee on Saturday, April 2, 2011 in Chicago. We will provide updates on any changes that come out of the meeting.

The draft proposal has three parts: a memorandum explaining the subcommittee’s operating assumptions and goals, a new Standard 509(b), and a chart that each law school would be required to fill out and post on its website each year.

The Memorandum

In the memorandum, the Subcommittee states that the goal is to “provide more meaningful and consistent employment information to prospective law students . . . [that will] greatly assist prospective students in making informed decisions about whether to go to law school or which law school to attend.” Right away the Subcommittee recognizes that schools already gather a great deal of data, and that it follows that sharing more information with prospective students will require only a small (and, implicitly, justified) burden.

The Subcommittee describes the consumer protection standard, Standard 509, as “a vague standard” that enables schools to provide limited and hard-to-compare information. The fact that reporting practices vary so widely among schools makes it very difficult for prospectives to understand the employment outcomes of a particular set of graduates. What’s more, the Subcommittee continues to recognize that the presentation of information is occasionally misleading. This reflects previous comments made by Dean Yellen.

The memorandum then cabins the problems with the current information into two categories: employment rates and salary information. The Subcommittee establishes two principles regarding the the first category. First, “the percentages disclosed should be based on the entire graduating class, with only those known to be employed being counted as such.” The second principle regards the variety of jobs graduates take, and the problem of providing misleading impressions about the true successes of a school’s graduates. “[T]he best approach is to require schools to disclose more disaggregated data about . . . categories of jobs.” These categories include nonprofessional jobs, part-times jobs, temporary jobs, and jobs funded in part by the school.

Regarding the second category, the Subcommittee recognizes the limited utility of salary medians and the likelihood that readers will misunderstand what the medians refer to and how they are calculated. The Subcommittee proposes that “all salary information clearly indicate the number of respondents and percentage of all graduates included.” This is an important revision that will change the manner in which many schools currently portray salary statistics. For examples of how problematic this can be, check out LST’s data clearinghouse. (The linked example shows a school that reported a median salary of $160,000, despite it being the median for only about 16% of the entire class.)

Proposed Standard 509(b)

The first proposal made by the Subcommittee is as follows:

Standard 509. BASIC CONSUMER INFORMATION
(b) A law school must publicly disclose the employment outcomes of its graduates by preparing and posting on its website the attached chart.
(1) The employment information must be accurate as of February 15th for persons who graduated with a JD or equivalent degree between September 1 two calendar years prior and August 31 one calendar year prior.
(2) The information must be posted on the school’s website by March 31 each year.
(3) The information posted must remain on the school’s website for at least three years, so that at any time, at least three graduating classes’ data is posted.
(4) The information must be gathered and disclosed in accordance with the instructions and definitions issued by the Section’s Questionnaire Committee.
(5) Any additional employment information the law school discloses must be fair, accurate and not misleading.
(A) Any publicly disclosed statistics regarding graduates’ salaries must clearly identify the number of salaries and the percentage of graduating students included.

The proposed Standard 509(b) requires that schools publicly disclose the employment outcomes of the most recent graduating class as true on the first February 15th following graduation. Schools must disclose these outcomes, at minimum, on the “attached chart” by the first March 31st following graduation. It also requires schools to keep the chart on their websites for at least three years. Finally, it adds a catch-all in 509(b)(5) to protect against predatory, opportunistic practices. This specifically includes a solution to misleading median salary practices that some law schools currently use.

The Chart

[View the chart]

The proposed Standard 509(b) “attached chart” aims to exhibit the outcomes of the entire graduating class as of the first February 15th following graduation. The chart disaggregates the current information into smaller categories to illuminate the outcomes graduates achieve at a particular school. The chart is also the first official recognition by an arm of the Section of Legal Education that salary information is in fact “basic consumer information.”

There are two classes of categories on this chart: employment status and employment type. For each category and subcategory, schools must report the percentage of all graduates, rather than of only employed graduates, as well as the raw number of graduates included in the calculation. This decision aims to limit the impact of creative accounting and less than forthright attempts at collecting employment data from graduates.

The employment status class places all graduates into four exhaustive categories: employed, pursuing a graduate degree full-time, unemployed, and employment status unknown.

The chart breaks “employed graduates” into two subcategories. First, this category breaks all employed graduates into four exhaustive kinds of employment: full-time long-term, full-time short-term, part-time long-term, and part-time short-term.

Second, it breaks all employed graduates into exhaustive categories based on the credentials required (or preferred) to do the job: bar passage required, J.D. preferred, other professional, or non-professional. It then further breaks each of those categories into (the same) four exhaustive kinds of employment: full-time long-term, full-time short-term, part-time long-term, and part-time short-term.

The employment type class breaks all employed graduates into six exhaustive categories based on the type of employer: law firms, business & industry, government, public interest, judicial clerkships, and academic. Of those categories, the law firm and judicial clerkships categories are further broken down by type. The law firms are disaggregated by size and the clerkships are disaggregated by level of government (state or federal).

Finally, full time salaries will accompany each category (except solo practitioners) of full-time, employed graduates whenever there are at least five salaries reported in a given category. These salaries will be reported with a 25th, 50th, and 75th percentile, as well as the number of salaries used to create these salary quartiles. There is also a space for schools to report the total number of jobs they funded.

A Good Start, But More To Be Done

The 509 Subcommittee is off to a really strong start in reforming how schools report employment information. It was made clear to us that this is only a preliminary draft, and that the Subcommittee expects more changes will be made. We hope this is the case.

The principles guiding the Subcommittee are sound. It is true that the information must be meaningful, consistent, and help prospectives make informed decisions about whether to, and where to, attend law school. But the execution of these principles still leaves something to be desired. If approved as a new accreditation standard in its current form, the proposal would certainly help prospective students and drastically cut down on misleading statistics. At the same time, it runs the risk of only providing superficial comfort, because it would not help match students to the schools that best meet their career objectives as efficiently as legal education needs.

As we previously outlined, we will use three criteria to assess the draft proposal.

(1) Does it disaggregate the current information?
(2) Does it demonstrate the economic value of a school’s J.D.?
(3) Does disclosure operate on an accelerated schedule?

Does it disaggregate the current information?

This proposal does disaggregate the current information. It helps show the nature of the jobs graduates obtained and with whom the graduates were employed. But as evidenced by comparing this draft proposal to the LST Standard, the vague “employed at 9 months” standard, where “a job is a job,” can be disaggregated to varying degrees. We’ve concluded that this draft does not disaggregate the current information to an adequate degree.

The more disaggregated employment information is, and the more data provided at that degree, the more likely it is that there will be privacy norm concerns. With these norms in mind, there is a legitimate interest in not disclosing all of the employment data that law schools already collect. On the other hand, law schools already collect all of the data needed to help prospectives make informed decisions, so cost concerns are greatly overblown (as the Subcommittee recognizes). As such, the appropriate level of disaggregation must balance privacy norms against the usefulness of additional disaggregation to anybody trying to understand the entry-level market for a school’s graduates.

It is the job of the Section of Legal Education to use its regulatory power to enforce the right balance. The Section must force schools to share the appropriate level of disaggregated information and must not opt to require less useful information because law schools have competitive concerns. The important question thus becomes how much weight the Section of Legal Education should give to schools that believe that more disaggregated information could (i) hurt their recruiting efforts, (ii) cause prospectives to focus too much on the first job in making their law school decision (as opposed to something else the schools think prospectives should focus on), and (iii) cause confusion through information overload.

Among the opportunities for improvement is how well the proposal connects job outcome features together. It does not disaggregate the locations of these jobs and does not show how the job, employer, and location connect for individual graduates. For example, we might be able to tell that 60% of a school’s graduates are working at jobs that require bar passage, but we do not know what percentage of those are working in business & industry. Likewise, we might know that 15% of a school’s graduates work in 2-10 attorney law firms, but we cannot tell what percentage of those graduates are working there as attorneys. This is not merely a theoretical concern– a sizeable percentage of law school graduates work in non-attorney positions in law firms. The decision to disaggregate further directly contravenes the Subcommittee’s principle against providing misleading impressions about the true successes of a school’s graduates.

Part of the reason additional disaggregation is so important is that it would minimize the effect of national rankings on student decision-making by offering a window directly into what graduates shortly after graduating. With this proposal, a prospective’s choice might still hinge on what a school ranks each year in U.S. News rather than on how well a school can help a student achieve her goals. Prospectives need clarity about how a school fits into the legal hiring market.

After all, the Subcommittee’s stated goal is to help prospectives make “informed decisions about whether to go to law school or which law school to attend.” The proposed solution is only satisfactory insofar that the goal is to differentiate between schools using percentage differences in broad, albeit more disaggregated, categories. It will still be too difficult to know the challenges graduates face for achieving their career objectives, which usually include a combination of location, employer type, and required credentials. Without sufficient granularity, neither will prospectives as easily understand a school’s placement niches. All together, prospectives will still struggle to understand schools’ unique placement abilities.

Another issue with the Subcommittee’s method of disaggregation is that it actually creates new gaps in the information (though not to a debilitating extent) and thus an incentive for creative accounting. One of the purposes of disaggregating the nine-month employment rate is to limit how much schools hide employment outcomes. Unnecessary gaps undermine this purpose.

The total number of graduates in each subcategory, taken together, should equal the total number in the parent category. For example, the total number of graduates who are employed, unemployed, pursuing a graduate degree, or whose employment statues are unknown should equal the total number of graduates in the graduating class because the categories are exhaustive.

The unknown status category is very important for identifying gaps in the employment status data. However, an unknown category is missing from all other exhaustive groups except the group for type of law firms. The employment type category, required credentials subcategory, judicial clerkships subcategory, and the full time and part time (and corresponding long and short term) subcategories all need an unknown field so that the numbers in the subcategories all equal the parent category’s total number.

Helping prospectives understand where data gaps exist encourages them to ask the right questions and serves to limit false impressions due to extrapolating outcomes from unrepresentative segments of the graduating class. Unfortunately, allowing schools to report graduates as “unknown” in any category incentivizes schools to avoid learning or researching employment outcomes. However, it is more important that the gaps created by non-reporting graduates are readily identifiable. As such, all exhaustive categories and subcategories need to account for each graduate.

Does it demonstrate the economic value of a school’s J.D.?

It is a huge step forward for the Subcommittee to recognize salary information as “basic consumer information.” As of right now, the only standardized, school-specific salary information is courtesy of U.S. News. Until this year, even U.S. News salary information was too opaque.

The Subcommittee’s proposal does a decent job with highlighting what new graduates make and, accordingly, demonstrates some of the economic value of each school’s J.D. This new salary information would allow prospective students to roughly understand how well graduates can service their debts immediately after law school. For the Class of 2009, the average graduate had $98,055 of law school debt, which translates to about a $1200/month loan payment.

While the Subcommittee’s approach is useful and likely the best way for schools to report school-specific salary outcomes without using job-specific salary data, it is not the approach we think the Subcommittee should take. A better way would be to leverage the reported salary data of all law schools together the way NALP does in its annual Job’s and J.D.’s. Certainly, if prospectives knew about this publication, which costs non-members $90, they could use it to have a better understanding of entry-level salaries for law school graduates. But there is currently no way to bridge the gap between this salary information and an individual school’s graduates, and the Subcommittee’s proposal does not help on that front, so it is limitedly useful for those trying to decide which law school to attend.

The aforementioned lack of connectivity between employers, job credentials, and job location makes understanding how the new salary information impacts them – particularly for loan payments – very difficult. For example, a $160,000 starting salary for a new associate grows differently in New York City compared to Houston due to salary compression in years two through seven. Additionally, $70,000 in New York City does not go as far as $70,000 in Philadelphia, Raleigh, or Nashville. The geographic impact on the ease of loan repayment cannot be understated. Even if a prospective has the Job’s and J.D.’s book, that information can only take them so far because its salary breakdowns are very specific (e.g., attorneys in 2-10 person law firms in X city). Nothing in the new standard or chart helps answer these important questions.

There is a separate concern about whether each category would have meaningful salary information associated with it. For example, 10 may work at small firms, with only four reporting. In this case, the four salaries do not get reported and thus do not serve any use. They are simply swept away. However, if these four salaries were added to a national salary database, those four become 40 or even 400, and the result is meaningful salary information about jobs that wouldn’t otherwise have salary information. Unfortunately, this resource cannot be utilized on a school-by-school basis without more disaggregation. In our next post we will explain our proposal for doing this in depth.

Does disclosure operate on an accelerated schedule?

Yes. In striking this balance between cost concerns and the need for timely information about the most recent graduating class, the Subcommittee has paved the way for significant improvements beginning as early as next year. At the Questionnaire Committee hearing in December, law school administrators expressed concern that requiring schools to report information too soon would be too high of a burden given cost constraints. But by limiting the Standard 509 requirements to only data that schools submit to NALP in February/March, the Subcommittee erases these concerns. Even small career services staff will be able to comply with the standard provided they already report to NALP, which nearly every ABA-approved law school does. Given that collection methods are now mostly electronic (through Symplicity or other user-entry databases), assembling and posting the data according to the proposed Standard 509(b) would take very few work hours and limited financial resources beyond what schools already allocate voluntarily.

Concluding Thoughts

The goal of a revised Standard 509(b) must be to help students make informed decisions about which (if any) school best meets their career objectives. While a good start, we think that, as currently conceived, the Subcommittee’s proposal will fail to adequately achieve this basic goal.

We ask that each member of the Committee imagine herself as a prospective student trying to choose a school to invest thousands of hours and dollars into. Each member must then think about how soundly she can act after analyzing employment information reported according to the new standard, and consider how well she actually understands the school’s ability to help her achieve her career objectives. We suspect that this thought experiment would leave each member uncomfortably uncertain. This uncertainty, at a minimum, should be addressed through a non-theoretical exploration of the standard’s implications. Before accepting a new standard, the Standards Review Committee should compare a few schools using real employment information presented as it would be under the proposed revisions.

An improved Standard 509 has the ability to wage an important battle against the influence of U.S. News on the decision-making of prospective law students. But without sufficient disaggregation of the current employment information, the effects can only be minimal. Under the current proposal, it is still too easy to imagine a prospective student choosing the #55 ranked school located on the east coast over the #81 ranked school on the west coast because she does not know, for example, what to make of the schools’ minute differences in percentage employed in mid-sized firms as it pertains to her goals of working out west in a mid-sized firm. Without adequate information to dissuade her, she might come to the head-scratching conclusion that #55 must be better because it is ranked higher. This is bound to worsen now that there are 45 more schools ranked on a national scale.

Each year, the Section of Legal Education makes an effort to minimize the effect of national rankings. We are sure that almost every law school administrator would agree with the Section’s sentiments, and revising Standard 509 is the chance to show that these are not empty words. We look forward to working with the Subcommittee to improve this first draft.