Today’s newspaper reports that a new company, AVVO, with thirteen million in venture capital has started a new web service to assist people looking for a lawyer. The site rates lawyers on a scale from one to ten in various fields of legal work. This service is in addition to the many rankings of lawyers already available. The oldest of these is Martindale Hubble, but it is a limited evaluation. Best Lawyers in America has been around for a long time and ranks lawyers by selecting them for inclusion under a speciality. It uses surveys filled out by lawyers to make the selection. A number of magazines rank lawyers as well. Law and Politics conducts a written survey each year and selects what it calls "super lawyers (R)" from the state. Seattle Magazine and other publications publish an annual listing as well. In addition, national legal publications like the National Law Journal, Lawdragon and others publish their selections of outstanding lawyers as well. The tendency among ego driven trial lawyers is to assume the results are accurate when they favor you and inaccurate when they don’t, but the problem is clients looking for a lawyer often rely upon these reports.

Lawyers have a natural concern and interest about these selections, but the truth is that all of these selections are not very accurate for a lot of reasons. One primary reason is the criteria used for the selection. It’s my view that if one is looking for the most qualified trial lawyer in a field the two primary factors should be ethical conduct and results obtained. Clients need a lawyer who gets results. Personal injury lawyers with a proven record in settlements and verdicts, who are respected by their opponents and recognized by insurance companies are quality lawyers. Lawyers with a great personality, are high profile can be mediocre in performance. We all know lawyers like that. It is one’s track record and trial skill reputation that is the real litmus test of a lawyer’s skill. If the wrong criteria is used in the evaluation it can and does result in outstanding lawyers falling below the radar screen or being rated lower then they should be rated.

Another, defect in these evaluations is the statistical analysis error inherent in the process. The systems which use surveys returned by lawyers are subject to the error of the size and quality of the sampling. The number involved must be large enough to be statistically significant. More important the lawyers responding have to be qualified as evaluators. The problem is that the surveys are not limited to the most knowledgeable, but are sent as a mass mailing. The most knowledable group are those in the same field of speciality, for example, medical malpractice plaintiff and defense lawyers plus their clients. One might be the most qualified microbiologist in America, but only those in the same field of speciality would know that. The others would only be aware of people in the field if they had a media profile. The result is that lawyers with a high profile receive votes even if they are not the most qualified. In addition, there are politics in the voting. Friends voting for friends or colleagues and the like. I believe if any of the evaluations relying upon surveys were subjected to expert analysis they would be found statistically unreliable with a very large margin of error.

I don’t fully know the criteria used by AVVO because they say it is a trade secret, but their website says it considers "experience, trustworthiness and industry recognition." I’m not at all sure those are sufficient or even appropriate criteria, but I can tell you from personal knowledge their conclusions are in many cases simply wrong. The reason is because they use a mathematical model and do not use any personal knowledge or experience with the lawyers. As the saying goes "garbage in, garbage out." The result is a shot gun approach in which the conclusions in some cases are accurate and in othercases are totally inaccurate. If this approach was used by a physician in treating a patient it would be medical malpractice. But, I suspect there is no perfect system and this is a tool, albeit a very imperfect one.