Shocked about Kaplan’s Survey Results? New Information Comes to Light

Many articles have covered Kaplan Test Prep’s recent press release, in which Kaplan declared that “[i]n deciding where to apply, pre-law students consider a law school’s place in the rankings more important than affordability, geographic location, its academic program – and even more important than its job placement statistics.” The same release also reemphasized the results of an earlier survey, which showed more than half of all prospective law students are ‘very confident’ they will get a legal job after graduation, while only 16% were ‘very confident’ about their peers.

Based in part on these two statements, people are questioning whether prospective law students actually deserve better information about job prospects, and whether disclosing more information would even impact their decision-making process. In other words, if prospectives really don’t care about job prospects, why should we care about law school transparency?

The survey results seemed odd to us, so we decided to dig a little deeper by contacting Kaplan Test Prep and taking a closer look at the survey. Kaplan was kind enough to not only answer our questions promptly, but to also send the full survey and results.

After reading the survey, digesting the results, and learning more about Kaplan Test Prep, it turns out that the press release and ensuing coverage did not tell the whole story. Instead, the results reflect an application landscape where important information is scarce and application decisions are complex. LST has been arguing these two things for months now; many others have been doing so for years.

You can check out the survey’s full results and our comments after the jump.

Kaplan’s survey asked four multiple choice questions to 1,383 students who were enrolled in Kaplan Test Prep’s LSAT program and took the October LSAT.

1) Which of the following describes what you hope to do with your law degree?
2) How important a factor is a law school’s ranking in determining where you will apply?
3) Which is most important to you when picking a law school to apply to?
4) If a law school admissions officer were to have full access to your current Facebook profile (status updates, wall comments, photos, etc) and looked at its content, how do you think this would affect your admissions chances?

Much of the coverage wrongly took at face value Kaplan’s conclusion that ‘prospectives care more about rankings than job statistics.’ They may be right (we doubt it), but that conclusion would have to rely on something other than this survey.

Kaplan’s conclusion is based on respondents’ answers to question #3:

3. Which is most important to you when picking a law school to apply to?
• Affordability/tuition: 12%
• Academic programming: 19%
• Availability of clinics, internships, extracurriculars: 5%
• Geographic location: 24%
• School ranking: 30%
Job placement rates: 8%
• Other: 2%

The red text is the cause for concern among people covering Kaplan’s survey. But these answers can’t be taken out of context: many respondents were at the beginning stages of the application process (taking the LSAT). The question was not about choosing a law school to attend, but choosing law schools to apply to. This means coming up with a list of schools that might accept you and deciding which ones are worth applying to. And while a prospective can’t attend a law school that she didn’t apply to, it should be obvious that these application decisions are often complex. The diversity of answers to question #3 demonstrates this, and should have been a major takeaway from this survey.

That said, we’ve been arguing for a while that job placement rates are a horrible way to sort law schools; nearly every ABA-approved school boasts impressive 9-month employment rates at between 85% to 95%. Given the uniformity of placement rates and the lack of meaningful information they include, it should come as no surprise that applicants trying to put together a list of schools start with other criteria. Although the number of applications per person is on the rise, there is a ceiling caused by time, effort, and cost. Cuts have to come from somewhere.

The U.S. News rankings are deeply flawed but may serve as a rough proxy for biglaw employment prospects, which is important for the 38% of respondents who hope to work for a large law firm (see answers to question #1). The rankings also serve as a good selectivity proxy, allowing an applicant to take her undergraduate GPA and LSAT scores and see the range of schools that may accept her. Moreover, where certain information about job prospects is scarce and difficult to obtain, it’s no wonder that would-be applicants seek rankings to make a complicated process easier.

Meanwhile, more applicants actually selected uncontroversial answers. For a person who knows that they want to work in a particular market, narrowing schools primarily by geographic location (24%) is actually a great way to start. One of the negative impacts of the U.S. News rankings is that it purports to measure national reputation for schools whose strengths are largely regional, which causes some applicants to inadvisably look outside their market of choice even when those schools cost more and lack reach to their desired market. Seeing a quarter of respondents focus on geographic location should be reassuring. And the 12% who focused on affordability/tuition show that at least some applicants are taking their expected debt loads seriously, which may also indicate that they are cautious about anticipated salaries after graduation.

The answers to question #2 are even more reassuring, but were unfortunately distorted in Kaplan’s press release. Media attention focused on question #3s answer, in which 30% of applicants found the rankings most important for determining which schools to apply to, even though the survey results indicate some clear positives. But almost half of the respondents to question #2 (46%) said that a law school’s ranking was only ‘somewhat important’ in determining to apply; a full 60% said that ranking was either ‘somewhat important,’ ‘not too important,’ or ‘not at all important.’ None of this was apparent in Kaplan’s press release because of their decision to group people who thought rankings were very important with people who only thought rankings were somewhat important.

Contrary to the message conveyed in the press release, at least 60% of survey respondents understood that rankings are not the whole picture. While the other 40% who think rankings are very important in choosing where to apply needs to be addressed, it certainly doesn’t justify the level of disdain that has been levied against prospectives. The appropriate response is to ask why this happened, and to the extent that prospectives do rely disproportionately on the rankings, how to go about fixing it.

Kaplan’s implicit message in the press release is clear. Kaplan wants prospectives to believe that they need help during the application process. They are correct; though fortunately there are many free, online resources for those who cannot afford professional help with this investment. But Kaplan might want to reevaluate what they have to say about where to apply. Interestingly, the results of this survey (including a reliance on rankings) sound a lot like what Kaplan itself actually teaches.

At Kaplan, while we tell our students that rankings can play an important part in determining what law school is right for them, we also say that it’s only one piece of the important research they have to do. We also encourage them to look at the overall strength of academic programming, internship and clinical opportunities, the culture of the school and employment stats for its graduates. It’s also a good idea to speak with recent graduates of the law school to learn about their experiences as both a student and job seeker.
– Jeff Thomas, director of pre-law programs, Kaplan Test Prep (emphasis added)

We think Kaplan can play an important part in improving the process, by getting prospectives to ask the right questions (both to themselves and to the schools). The organization is well-situated to provide guidance, as they already help tens of thousands of applicants each year. On the other hand, their ability to exacerbate the problem should not be understated. Whenever Kaplan fails to explain that current job statistics provide incomplete portrayals of entry-level hiring, even in a good economy, and that students need to demand better employment information if they want to know about the various opportunities each school offers, they only reinforce the mindset we all so quickly ridicule: the focus on a deeply flawed national ranking system.

In short, the Kaplan Test Prep survey didn’t ask the types of questions that really let us know whether applicants care about job prospects. If prospectives are to have any chance at making informed decisions about which schools to apply to, and which school to eventually attend, organizations like Kaplan, Testmasters, and Princeton Review need to motivate their clients to look deeper. Otherwise, it seems that these companies don’t really care about “determining what law school is right for [each client].”

The October 2010 Kaplan Test Prep Survey:

Which of the following describes what you hope to do with your law degree?
• Private practice – large firm: 38%
• Private practice – boutique firm: 22%
• Private practice – sole practitioner: 10%
• Public interest: 31%
• Criminal prosecution: 14%
• The judiciary: 13%
• Politics: 22%
• Education: 10%
• Business: 23%
• Other: 10%

How important a factor is a law school’s ranking in determining where you will apply?
• Very important: 40%
• Somewhat important: 46%
• Not too important: 12%
• Not at all important: 2%

Which is most important to you when picking a law school to apply to?
• Affordability/tuition: 12%
• Academic programming: 19%
• Availability of clinics, internships, extracurriculars: 5%
• Geographic location: 24%
• School ranking: 30%
• Job placement rates: 8%
• Other: 2%

If a law school admissions officer were to have full access to your current Facebook profile (status updates, wall comments, photos, etc) and looked at its content, how do you think this would affect your admissions chances?
• Would significantly improve my admissions chances 9%
• Would somewhat improve my admissions chances: 53%
• Would somewhat hurt my admissions chances: 24%
• Would significantly hurt my admission chances: 2%
• I do not have a Facebook profile: 12%