Rating Trial Lawyers - A Hit and Miss Process at Best
Today’s newspaper reports that a new company, AVVO, with thirteen million in venture capital has started a new web service to assist people looking for a lawyer. The site rates lawyers on a scale from one to ten in various fields of legal work. This service is in addition to the many rankings of lawyers already available. The oldest of these is Martindale Hubble, but it is a limited evaluation. Best Lawyers in America has been around for a long time and ranks lawyers by selecting them for inclusion under a speciality. It uses surveys filled out by lawyers to make the selection. A number of magazines rank lawyers as well. Law and Politics conducts a written survey each year and selects what it calls "super lawyers (R)" from the state. Seattle Magazine and other publications publish an annual listing as well. In addition, national legal publications like the National Law Journal, Lawdragon and others publish their selections of outstanding lawyers as well. The tendency among ego driven trial lawyers is to assume the results are accurate when they favor you and inaccurate when they don’t, but the problem is clients looking for a lawyer often rely upon these reports.
Lawyers have a natural concern and interest about these selections, but the truth is that all of these selections are not very accurate for a lot of reasons. One primary reason is the criteria used for the selection. It’s my view that if one is looking for the most qualified trial lawyer in a field the two primary factors should be ethical conduct and results obtained. Clients need a lawyer who gets results. Personal injury lawyers with a proven record in settlements and verdicts, who are respected by their opponents and recognized by insurance companies are quality lawyers. Lawyers with a great personality, are high profile can be mediocre in performance. We all know lawyers like that. It is one’s track record and trial skill reputation that is the real litmus test of a lawyer’s skill. If the wrong criteria is used in the evaluation it can and does result in outstanding lawyers falling below the radar screen or being rated lower then they should be rated.
Another, defect in these evaluations is the statistical analysis error inherent in the process. The systems which use surveys returned by lawyers are subject to the error of the size and quality of the sampling. The number involved must be large enough to be statistically significant. More important the lawyers responding have to be qualified as evaluators. The problem is that the surveys are not limited to the most knowledgeable, but are sent as a mass mailing. The most knowledable group are those in the same field of speciality, for example, medical malpractice plaintiff and defense lawyers plus their clients. One might be the most qualified microbiologist in America, but only those in the same field of speciality would know that. The others would only be aware of people in the field if they had a media profile. The result is that lawyers with a high profile receive votes even if they are not the most qualified. In addition, there are politics in the voting. Friends voting for friends or colleagues and the like. I believe if any of the evaluations relying upon surveys were subjected to expert analysis they would be found statistically unreliable with a very large margin of error.
I don’t fully know the criteria used by AVVO because they say it is a trade secret, but their website says it considers "experience, trustworthiness and industry recognition." I’m not at all sure those are sufficient or even appropriate criteria, but I can tell you from personal knowledge their conclusions are in many cases simply wrong. The reason is because they use a mathematical model and do not use any personal knowledge or experience with the lawyers. As the saying goes "garbage in, garbage out." The result is a shot gun approach in which the conclusions in some cases are accurate and in other cases are totally inaccurate. If this approach was used by a physician in treating a patient it would be medical malpractice. But, I suspect there is no perfect system and this is a tool, albeit a very imperfect one.
Well, personally I don't know any lawyers and if I needed one in a hurry I'd trust a halfway-accurate rating before I'd pick one at random from the yellow pages.
Reply to this
Hi Paul:
I write "Mullen on Law 2.0," and I also worked with the folks at Avvo for 6 months.
I am an attorney (Michigan), but my law practice is focused on ADR services (mediation and arbitration). I stumbled upon the work at Avvo after moving to Seattle, and had a blast working there. I've blogged about Avvo for a few months now and am glad that they have launched so the discussion can become more detailed.
Basically, I believe that the site has to be given the chance to evolve.
Yes, the "rating" system is opaque, and yes, most lawyers are not going to be happy about it.
However, Avvo's focus is on the consumer and in providing information in the aggregate with the objective of helping consumers in make better decisions. Hype aside, there is no doubt they are using public information and fashioning value out of highly-dispersed and/or hard to locate information. That's the Zillow influence, but it's not necessarily a negative.
While my lawyer side does have concerns, my frosted mediator side thinks that legal consumers should have as much and as varied information as possible. To be sure, rating systems are dangerous for a host of reasons : bar ethical duties with respect to comparative advertising, problems of comparing apples and oranges and the difficulty of relating paper "superiority" to excellence in practice. Yet, the site provides a lot more than a linear scale. As it evolves, it will become a community site, and one would expect the site to be somewhat "intelligent" in the way that Amazon's service got more "intelligent" over time.
Frankly, we all know that sometimes all a client needs is a mediocre lawyer. Certainly, the $500 an hour lawyer has his use, but chances are it's more a function of clout than legal ability. And, as others have pointed out, sometimes it's better to have the nephew of the judge's best friend for $75 an hour, than the best big city lawyer who went to Harvard three times over.
There are gross disparities in terms of resources (which largely distinguishes big firm lawyers from the small firm lawyer), but certainly many solos can provide just as good service in terms of what a particular client needs as an attorney with a thousand other attorneys standing beside him.
Law is most emphatically NOT rocket science,--it's the science of human relationships. The web is where many clients now turn for advice in finding resources.
So, whether Avvo sinks or swims, whether lawyers like it or not, it is poised to provide a valuable service. Other sites are going to refactor and improve upon the service, and Avvo is going to learn and evolve.
Much is made of Abraham Lincoln being, you know, dead everywhere but on Avvo, but you know, why not have dead people in there? I can easily foresee students doing book reports on Lincoln and then adding comments about his service to the country based upon what they learned in history. Maybe that's far-fetched, but then again...
Reply to this
Your points of concern are all valid, inline with most lawyers. The underlining problem, if AVVO is overwhelming accepted by the public as truth, wrong or other-wise, Godzilla has been hatched. Do you stand firm on ethics or sur-come to the potential new business an take advantage of the system?
Reply to this
I seem to keep being reduced to fewer and fewer numbers. First I received a nine digit social security number, then an eight digit driver's license number, then various six digit or less student ids. Now at long last, AVVO reduces to three alphanumeric digits (one is a ".") the sum of my upbringing, Ivy League education, law school training, professional experience, personality, and whatever other gobbledygook AVVO purports to examine. I'm a 6.4 (yay - on par with Supreme Court justices!!), and that's without manipulating my score by listing every speaking engagement, publication or organization to which I have belonged (ever). Without AVVO.com to rate me, I wouldn't have any idea whether I was a good attorney or not (and neither would the hundreds of clients I've represented). The best part of AVVO is that my score will go up every year even if no new information is posted to the site. I knew I was getting better with age!
Reply to this