ORDER
HOLLAND, District Judge.
Motion in Limine to Exclude Evidence
Defendant moves to exclude hand printing comparison evidence at trial.
This motion is opposed.
A
Daubert
hearing was held on August 10, 2001 to determine the admissibility of the evidence in question.
Facts
Defendant Chan Ian Saelee was indicted on three counts of violating federal drug laws. Count III charges defendant with importing opium from Thailand in 1998 in violation of 21 U.S.C. §§ 952 and 960(b)(3). The opium in question was concealed in Butterfinger candy bars which appeared to have been express mailed from the United States but then returned to the sender after delivery was unsuccessful. The Government had John W. Cawley, III, a forensic document analyst with the United States Postal Inspection Service National Forensic Laboratory, compare hand printing exemplars provided by defendant with the hand printing on the address labels on the packages in question. Mr. Cawley concluded that defendant was the writer of one of the questioned writings and was probably the writer of another. The Government originally proposed to have Mr. Cawley testify as to his conclusions at trial, and defendant filed the instant motion to exclude all testimony by Mr. Cawley at trial.
In its first response to defendant’s motion, the Government recognized that a
Daubert
hearing might be necessary to determine if Mr. Cawley’s testimony was admissible but opposed excluding all of Mr. Cawley’s testimony.
The court tentatively scheduled a
Daubert
hearing for August 10, 2001.
In the same order, the court directed the parties to file supplemental briefing on defendant’s motion.
In its timely filed supplemental brief, the Government changed course and proposed to have Mr. Cawley testify only about the similarities and differences between the known writing and the questioned documents and not to have Mr. Cawley testify about his ultimate conclusions as to' whether defendant authored the questioned documents.
The Government argues that such comparison evidence is admissible pursuant to Rule 701, Federal Rules of Evidence, which governs the admissibility of lay opinion testimony. At first, defendant agreed that Mr. Cawley could testify under Rule 701.
However, following a status conference, defendant filed an unopposed corrected response to the Government’s Rule 701 proposal
in which he disputed that Mr. Cawley’s testimony would be admissible under Rule 701
and argued that Mr. Cawley’s testimony, if admissible at all, could only be admissible under Rule 702, Federal Rules of Evidence, which governs the admissibility of expert testimony. Defendant continued to assert that a
Daubert
hearing was necessary to determine the admissibility of the comparison testimony.
The court agreed, and a
Daubert
hearing was held on August 10, 2001. At the
Daubert
hearing, the Government changed course once again and argued that Mr. Cawley’s testimony was admissible under Federal Rule of Evidence 901, which deals with the authentication and identification of evidence. At the close of the
Daubert
hearing, the court granted defendant’s motion
in limine
to exclude the hand printing comparison evidence at trial and advised the parties that a written order would follow.
Discussion
The question presented by defendant’s motion
in limine
was whether the forensic document analyst’s testimony as to whether defendant was the author of the questioned documents is admissible under the Federal Rules of Evidence. This question was narrowed when the Government decided not to have Mr. Cawley testify about his ultimate conclusions as to authorship of the questioned documents. The question facing the court at the
Daubert
hearing was whether the forensic document analyst’s testimony about the similarities and differences between the known documents and the questioned documents was admissible under either Rule 701 or 702. On the basis of the record made at the
Daubert
hearing, the court answers both parts of this question in the negative.
Rule 701 governs the admissibility of opinion testimony by lay witnesses and provides:
If the witness is not testifying as an expert, the witness’ testimony in the form of opinions or inferences is limited to those opinions or inferences which are (a) rationally based on the perception of the witness, (b) helpful to a clear understanding of the witness’ testimony or the determination of a fact in issue, and (c) not based on scientific, technical, or other specialized knowledge within the scope of Rule 702.
In making its proposal to have Mr. Caw-ley testify under Rule 701, the Government relies on an unpublished case,
United States v. Santillan,
No. CR-96-40169 DLJ, 1999 WL 1201765 (N.D.Ca. Dec.3, 1999). In
Santillan,
the court allowed a forensic document examiner to offer comparison testimony under Rule 701, after finding that the testimony was inadmissible under Rule 702. The court noted that allowing a handwriting expert to testify under Rule 701 might raise a Rule 403 problem, but still found the testimony admissible.
Ignoring for the moment that
Santillan
is an unpublished case,
Santillan
was decided prior to the 2000 amendments to Rule 701; presumably the court would reach a much different conclusion if faced with the same issue today. In 2000, Rule 701 was amended to expressly limit lay opinion testimony to that which is
not
based on scientific, technical, or other specialized knowledge within the scope of Rule 702. The Advisory Committee Notes to the 2000 Amendments explain that the amendment was made “to eliminate the risk that the reliability requirements set forth in Rule 702 will be evaded through the simple expedient of proffering an expert in lay witness clothing,” which is exactly what the Government is proposing to do here and what the Government was allowed to do in
Santillan.
The court required that the parties pre-file in writing their witnesses’ direct testimony for the
Daubert
hearing. The Gov-
eminent offered only the testimony of its forensic document analyst, Mr. Cawley.
Mr. Cawley’s pre-filed testimony is preceded by his
cumculum vitae.
There, Mr. Cawley sets out his employment as a “forensic document analyst” for the United States Postal Inspection Service. He sets forth his education, including training as a questioned documents examiner, the professional organizations of which he is a member (related to “forensic sciences” and document examination), and other “relevant” information including testifying as an expert in the field of questioned documents in various courts. In his report (which the Government treats as the written testimony), Mr. Cawley summarizes the examinations he has undertaken. He offers to provide for the Government testimony demonstrative of (1) his “scientific examination” of the defendant’s known handwriting, (2) his “study of handwriting habits as the basis of handwriting identification principles,” and (3) his “comparative evaluation” of the two writings (presumably the similarities and differences between known and questioned writings as now proposed by the Government).
While Mr. Cawley’s written testimony discloses very little which is relevant to
Daubert
issues, it does disclose much about whether he is a lay witness or a purported expert. Plainly, Mr. Cawley is not a lay person who proposes to offer testimony based on sensory perception. Rather, he makes reference to handwriting identification
principles
and
scientific
examinations. Mr. Cawley would testify that he is a trained, forensic document analyst.
Mr. Cawley’s testimony is plainly inadmissible under Rule 701 because of subsection (c) which precludes lay witness opinion testimony based on specialized knowledge. Any testimony Mr. Cawley would offer would be based on undisclosed handwriting principles, scientific examination of documents, and specialized knowledge gained through training and 24 years of experience as a document examiner. Such testimony must be analyzed under Rule 702 standards.
Rule 702 of the Federal Rules of Evidence governs the admission of expert testimony and provides:
If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case.
Rule 702 was amended in 2000 in response to
Daubert v. Merrell Dow Pharmaceuticals, Inc.,
509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993), and
Kumho Tire Co. v. Carmichael,
526 U.S. 137, 119 S.Ct. 1167, 143 L.Ed.2d 238 (1999).
Daubert
assigned a gatekeeping function to trial judges to exclude unreliable scientific expert testimony.
Kumho
extended this gatekeeping function not only to scientific testimony but also to all expert testimony.
As the Advisory Committee Notes to the 2000 Amendments indicate, “[c]on-sistently with
Kumho,
the Rule as amended provides that all types of expert testimony present questions of admissibility for the trial court in deciding whether the evidence is reliable and helpful.” The
Court in
Daubert
set forth a non-exclusive list of factors for courts to consider in determining whether expert evidence is reliable. These factors are:
(1) whether the expert’s theory or technique can be or has been tested;
(2) whether the expert’s theory or technique has been subjected to peer review;
(3) the known or potential rate of error;
(4) the existence and maintenance of standards controlling the technique’s operation; and
(5) general acceptance in the scientific community.
Daubert,
509 U.S. at 593-94, 113 S.Ct. 2786.
Kumho
held that these factors may apply in a case dealing with non-scientific expert testimony as well.
Kumho,
526 U.S. at 150, 119 S.Ct. 1167. However, the Court in both cases emphasized that this is not an exclusive list and the factors that a court considers must “fit” the facts of the particular case.
Factors that “fit” the instant case are whether the theories and techniques of handwriting comparison have been tested, whether they have been subjected to peer review, the known or potential error rate of forensic document examiners, the existence of standards in making comparisons between known writings and questioned documents, and the general acceptance by the forensic evidence community.
The Government bears the burden of proving that its expert’s proffered testimony is sufficiently reliable to be admissible under Rule 702.
United States v. Starzecpyzel,
880 F.Supp. 1027, 1031 (S.D.N.Y.1995). As already noted, the parties were ordered to file their direct testimony for the
Daubert
hearing prior to the hearing itself. Defendant filed an affidavit from Professor Michael J. Saks,
a professor of both law and psychology who has done extensive work in the area of scientific evidence in general, and handwriting analysis in particular. The Government pre-filed a July 2, 2001, “Summary of Forensic Testimony”.
In fact, the Government’s pre-filed testimony was Mr. Cawley’s expert report. It contains Mr. Cawley’s formal conclusions (which the Government has abandoned), the basis for the conclusions (which is itself conclusory), and a summary of examinations. The basis for Mr. Cawley’s conclusions is stated to be “significant identifying characteristics.” Mr. Cawley did not explain in his report nor at the
Daubert
hearing how one identifies such characteristics, what they mean, or how many are necessary for a positive opinion.
Mr. Cawley testified at the
Daubert
hearing that his pre-filed testimony did not include any information about empirical testing of handwriting analysis, the known error rate in the field, peer review, or the standards that control in the field.
At the
Daubert
hearing, both Professor Saks and Mr. Cawley were cross-examined. Having heard the evidence, and applying the relevant
Daubert/Kumho
factors to that evidence, the only conclusion the court can reach is that the Government has failed to meet its burden of establishing that Mr. Cawley’s testimony is sufficiently reliable to be admissible under Rule 702.
It is evident that the theories and techniques used in handwriting comparison could be empirically tested but, as Professor Saks’ testimony illustrates, there has been an overall lack of such testing.
At the
Daubert
hearing, Mr. Cawley testified that there has been at least one empirical study concerning the reliability of handwriting comparison, conducted for the FBI by Moshe Kam. According to Mr. Cawley, the Kam study determined that a document examiner could do what he said he could do,
ie.,
compare a known document and a questioned document and offer an opinion about whether the writer of the known document was the writer of the questioned document. Mr. Cawley testified that the Kam study concluded that document examiners had an error rate of 6.5% and laypersons had an error rate of 38.6%. Cawley’s testimony is in large part contradicted by Saks’ testimony on the Kam studies. As detailed in Saks’ affidavit, Kam actually conducted four studies,
which Saks avers suffered from their own methodological problems. Ignoring that issue for now, the first study found that experts outperformed nonexperts but that the best nonexperts did about as well as the experts. The second study found the rate of true positives to be almost identical for experts and non-experts, while experts committed fewer false positives. The third study found an unexplained 41% improvement in laypersons’ ability to avoid false positives. The fourth study tested the ability to determine whether a signature is genuine or not. Experts and laypersons had similar ability to detect forgeries; experts were slightly better at discerning genuine signatures than laypersons; experts incorrectly found forgeries to be genuine less often than laypersons, and experts incorrectly found genuine signatures to be forgeries less often than laypersons. From Saks’ testimony it appears that, contrary to Mr. Cawley’s testimony, the Kam studies did not conclusively establish that forensic document examiners can reliably do what they say they can do.
In addition to there being a lack of empirical evidence on the proficiency of document examiners, there has been little empirical testing done on the basic theories upon which the field is based. For instance, the basic premise on which document examiners work — namely, that no two persons write alike and that no one person writes the same all the time — has not even been adequately tested. In fact, Saks testified that one study suggested that this basic premise is not entirely true. A study done in 1958 found that the signatures of numerous people named “John Harris” were so similar that examiners could not distinguish one writer from another.
Also, there has never been any empirical research done on the theory of probability on which handwriting analysis is based. According to Professor Saks, the field of handwriting analysis is based on a theory of probability that “if we know the individual probabilities of a set of independent attributes, we can calculate the probability of their joint occurrence by multiplying them together.”
But, according to Saks, document examiners have never gone beyond stating this general theory. “They make no measurements, they make no calculations, and they report no probability of coincidental matches. In other words, they do not apply the theory to the prac
tice at all.”
Mr. Cawley did not refute this testimony.
At the
Daubert
hearing, Mr. Cawley did attempt to refute the general notion that there has been little empirical research done in his field. He offered a four-page list of published and unpublished articles dealing with handwriting analysis. According to Mr. Cawley, these articles are written by forensic document examiners, and the published articles are presented at professional meetings for peer review. However, Mr. Cawley also testified that he did not know whether any of the articles discussed error rates, empirical testing, or coincidental matches, although he claimed to have read the articles. The list, without analysis of the substance of the articles, is of little use to the court. The court infers that most of the listed articles were written by proponents of the guild style (apprentice training) process of training handwriting examiners. This is Mr. Cawley’s background. It is a training process little used today in professional and scientific callings. The list of articles gives the court no insight into whether there has been independent, empirical testing of the theories and techniques involved in handwriting comparison. The field of handwriting comparison also suffers from a lack of meaningful peer review. As Mr. Caw-ley testified, some articles are presented at professional meetings for review; nonethless, there is no evidence that any of these articles are subjected to peer review by disinterested parties, such as academics.
There is little known about the error rates of forensic document examiners. The little testing that has been done raises serious questions about the reliability of methods currently in use. As to some tasks, there is a high rate of error and forensic document examiners may not be any better at analyzing handwriting than laypersons. This is illustrated not only in the Kam studies relied on by Mr. Cawley, but also in a series of proficiency tests carried out by Collaborative Testing Service under the supervision of the Forensic Sciences Foundation.
In two such proficiency studies, the task for the examiners was to compare written letters in the natural hand of the writers with known exemplars of several suspects. In one study, the examiners were correct 89% of the time, but in the other only 52% of the time. In tests measuring their ability to determine if a signature was genuine, examiners were correct anywhere from 35% of the time to 100% of the time. A test involving hand printing produced only 13% correct answers. In a test asking examiners to identify the author of a forgery, the examiners were wrong 100% of the time.
In a study conducted by Galbraith, laypersons were given the same material as experts were given in the 1987 proficiency study.
The true positive accuracy rate of laypersons was the same as that of handwriting examiners; both groups were correct 52% of the time.
Mr. Cawley testified that he has experience in analyzing hand printing, which is what is involved here, and that hand printing has individual characteristics that can be compared. Mr. Cawley offered no evidence that this claim has been tested in any manner. It is not known whether printing is defendant’s normal, natural manner of writing. Mr. Cawley may have been asked to compare an abnormal, unnatural writing style to the questioned address labels. The court has no insight into
what requiring an abnormal, unnatural manner of writing would do to either the analysis or error rates. Professor Saks avers that any asserted special expertise in pointing out similarities or differences in hand printing has no empirical foundation of any kind.
Nor did Mr. Cawley offer any specific evidence as to his proficiency in making hand printing comparisons, other than to state generally that he has examined thousands of documents. In fact, Mr. Cawley testified that his ability to make comparisons of hand printing was based on there being a sufficient quality and quantity of printing to compare. The quantity of printing in this case is printing on an address label, which the court presumes would be considered a very small quantity of printing ,
The court infers that the very limited amount of questioned printing would diminish Mr. Cawley’s ability to render an accurate opinion or make accurate comparisons irrespective of whether printing is defendant’s normal, natural manner of writing.
The field of handwriting comparison also suffers from a lack of controlling standards. This was highlighted by Mr. Caw-ley’s testimony at the
Daubert
hearing when he explained how he conducts a comparison of known handwriting with a questioned document. Generally using microscopic enlargements, he stated that he first determines whether the handwriting is normal, natural handwriting. Next, Mr. Cawley stated that he determines the characteristics of the writing, such as the flow, smoothness, pen impulse movements, etc. After having discerned the characteristics of the writing, Mr. Cawley said he makes side-by-side comparisons. Based on these comparisons, Mr. Cawley stated that he then can render an opinion about whether the author of the known writing was the author of the questioned document.
Missing from Mr. Cawley’s description of how he makes comparisons and reaches ultimate conclusions is any testimony about the controlling standards used to make each of these determinations. The closest Mr. Cawley came to discussing any standards he employed was when he indicated that 25 samples of known writing were necessary for this process. Even then, he did not indicate what the effect would be on his ability to make comparisons or to reach an ultimate conclusion if he had a different number of samples or why 25 was the magic number. Above, the court infers that a limited quantity of questioned writing would detract from the reliability of the analysis; but in reality, we do not
know
whether there should be some minimal amount of questioned writing in order to arrive at a reliable opinion. The technique of comparing known writings with questioned documents appears to be entirely subjective and entirely lacking in controlling standards.
Finally, the evidence does indicate that there is general acceptance of the theories and techniques involved in the field of handwriting analysis among the closed universe of forensic document examiners. This proves nothing. Testimony from these experts has, until recently, been un
critically accepted as reliable in the courts. Having previously testified somewhere as an expert document examiner was usually sufficient qualification. “Courts have long received handwriting analysis testimony as admissible evidence.”
United States v. Paxil,
175 F.3d 906, 910 n. 2 (11th Cir.1999). However, the fact that this type of evidence has been generally accepted in the past by courts does not mean that it should be generally accepted now, after
Daubert
and
Kumho.
Several other district courts that have recently considered whether testimony by a handwriting expert is admissible under Rule 702 have likewise found numerous problems with handwriting comparison testimony and severely limited it at trial.
See, United States v. Hines, 55
F.Supp.2d 62, 70-71 (D.Mass.1999) (document examiner limited to testifying about the similarities between the known handwriting and the questioned document);
United States v. Van Wyk,
83 F.Supp.2d 515, 524 (D.N.J.2000) (forensic stylistic expert limited to testifying about “the specific similarities and idiosyncracies between the known writings and the questioned writings, as well as testimony regarding, for example, how frequently or infrequently in his experience, he has seen a particular idiosyncrasy”);
United States v. Rutherford,
104 F.Supp.2d 1190, 1194 (D.Neb.2000) (document examiner’s testimony limited “to identifying and explaining the similarities and dissimilarities between the known exemplars and the questioned documents”).
This court has gone one step further than other courts by excluding all of Mr. Cawley’s testimony, including his comparison testimony. The Government has failed to prove that Mr. Cawley’s proposed testimony as to similarity between known and questioned documents would be “the product of reliable principles and methods.” Fed.R.Evid. 702(2). We do not know what principles of comparison Mr. Cawley would employ. We do not know what is required to have a valid similarity' — one tending to prove that one person wrote both documents — nor do we know how many valid similarities are necessary to rule out the possibility that two people have essentially the same manner of writing the words in question. The Government would offer a witness as to similarities in handwriting who would speak with apparent authority on a subject in which he would appear to have special knowledge which the court is unable to evaluate on the present record. In the absence of tested principles for making comparisons, Mr. Cawley’s testimony as to similarities would itself be nothing more than a set of subjective observations and little different from an unsupported opinion as to the fact of authorship of a document. Mr. Caw-ley’s testimony is as likely to mislead a jury as to assist it in determining the facts of this case. It is therefore excluded entirely-
Contrary to the Government’s argument at the
Daubert
hearing, nothing in Federal Rule of Evidence 901 demands a different result. At the
Daubert
hearing, the Government argued that Mr. Cawley’s testimony must be admissible under Federal Rule of Evidence 901. Rule 901(a) provides that “[t]he requirement of authentication or identification as a condition precedent to admissibility is satisfied by evidence sufficient to support a finding that the matter in question is what its proponent claims.” Rule 901(b) lists several examples of authentication or identification that conform with the rule, including one which allows for the “[c]omparison by the trier of fact or by expert witnesses with specimens which have been authenticated.” Fed.R.Evid. 901(b)(3). In discussing Rule 901(b)(3), the Advisory Committee Notes specifically mention handwriting comparison testimony and
state that “[ejxample (3) sets no higher standard for handwriting specimens and treats all comparison situations alike.... ”
The Government argues that Rule 901(b)(3) plainly contemplates the admission of comparison evidence by forensic document analysts and that'to hold otherwise would render 901(b)(3) meaningless. The Government relies on
United States v. Jones,
107 F.3d 1147 (6th Cir.1997), in which the court considered the question of admissibility of a handwriting expert’s testimony. The court concluded that handwriting analysis was not scientific evidence; but because this was
pre-Kumho,
the court did not apply the
Dattbert
standards. Rather, the court considered whether handwriting analysis constituted “technical or other specialized knowledge” under the Federal Rules of Evidence and concluded that it did. In doing so, the court noted that:
if we were to hold that handwriting analysis is not a field of expertise under the Federal Rules of Evidence, there would be no place for expert witnesses to compare writing on one document with that on another in order to authenticate a document. In other words, appellant’s suggested approach would render Rule 901(b)(3) meaningless.
Id.
at 1159.
First of all, the court would point out that it is not holding that handwriting analysis can never be a field of expertise under the Federal Rules of Evidence. The court is merely holding that the Government has failed to meet its burden of establishing that the proffered expert testimony in this case is admissible under Rule 702. Second, even if the court were to hold that handwriting analysis is not a field of expertise under the rules, that would not render Rule 901(b)(3) meaningless. Rule 901(b)(3) does not deal exclusively with handwriting comparison, despite the fact that the Advisory Committee Notes for the rule discuss handwriting comparison testimony. Other types of comparison testimony are encompassed within the rule. Last, and most important, Rule 702 and Rule 901 must be read together. Rule 901(b)(3) contemplates testimony by an expert — but before an expert’s testimony can be admitted, it must past through the gates of Rule 702. In this case, Mr. Cawley’s testimony did not make it through the Rule 702 gate and, therefore, Rule 901 is irrelevant to the question of whether his testimony is admissible.
Conclusion
Defendant’s motion
in limine
to exclude testimony of hand printing comparison evidence at trial is granted.