301 F.Supp.2d 1030 (2004)
Karen YAPP, et al., Plaintiffs,
v.
UNION PACIFIC RAILROAD COMPANY, Defendant.
No. 4:02 CV 615 SNL.
United States District Court, E.D. Missouri, Eastern Division.
February 4, 2004.
Deborah A. Mattison, Robert F. Childs, Jr., Robert L. Wiggins, Jr., Rocco Calamusa, Jr., Gordon and Silberman, Herman N. Johnson, Jr., Wiggins and Childs, Birmingham, AL, Jon A. Ray, Mary Anne O. Sedey, Sedey and Ray, P.C., St. Louis, MO, for Plaintiffs.
Dennis C. Donnelly, Lisa Demet Martin, Michael P. Burke, Timothy C. Mooney, Jr., Bryan Cave LLP, St. Louis, MO, Douglas C. Herbert, Jr., Herbert Law Office, Washington, DC, for Defendant.
MEMORANDUM AND ORDER
LIMBAUGH, Senior District J.
This matter is before the Court on Plaintiffs' Motion to Strike Defendant's Experts (# 45). Plaintiffs brought this putative class action seeking inter alia declaratory and injunctive relief for alleged systemic racial discrimination in Defendant Union Pacific Railroad Company's ("UPRR") employee selection, training and compensation policies, practices and procedures. Plaintiffs' Motion for Class Certification is now pending before the Court, and is supported by the expert report of statistician, Dr. Edwin L. Bradley. Dr. Bradley's statistical analyses were based on computer data provided by the Defendant and purport to demonstrate systemic, statistically significant, adverse impacts against African-Americans in various aspects of the Defendant's job selection processes. Defendant responded in opposition to the Plaintiffs' Motion for Class Certification, relying on the expert report of Dr. Michael P. Ward ("Ward") and Mr. Nathan D. Woods ("Woods") (collectively "Ward and Woods Report"). The Ward and Woods Report was based on statistical analyses which differed markedly from Dr. Bradley's both in mechanism and outcome because Ward and Woods conducted a survey of UPRR employees to assess the differences between the various departments at UPRR, and the hiring practices used therein. Based on their survey, Ward and Woods determined that the proper method for analyzing the UPRR computer data was on a department-by-department basis, with extra variables beyond minimum job qualifications. Plaintiffs now move to strike the Ward and Woods Report because the survey of UPRR employees was scientifically invalid and thus unreliable, and any statistical analysis based on the unreliable survey is inadmissable under Fed.R.Evid. 702 and Daubert v. Merrell Dow Pharm., 509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993). Defendants have responded in opposition, and Plaintiffs have replied. The matter is now properly before the Court[1].
Background
Plaintiffs filed this suit on April 29, 2002, against UPRR, and subsequently filed their Amended Complaint on June 4, 2002. On August 29, 2003, Plaintiffs filed a motion seeking certification of a class made up of all African-Americans, who, from July 18, 2000, to the present, have been adversely impacted by one or more aspects of the Defendant's policies, practices and procedures related to selections for non-agreement[2] job vacancies. Plaintiffs' motion for class certification relies heavily on the testimony and reports of their expert witness, Dr. Edwin L. Bradley. Dr. Bradley conducted a statistical analysis using computer records containing personnel data provided by UPRR. See Expert Report of Edwin L. Bradley on Class Certification Issues, Ex. A to Plaintiffs' Motion for Class Certification, Doc. No. 41 ("Bradley Report I"), p. 2. Dr. Bradley tabulated and analyzed personnel data from January of 1996 to September of 2002, for all non-agreement positions. Id. at p. 5. Although UPRR has twenty-seven departments, Dr. Bradley's analysis was predicated on his "understanding that all non-agreement positions are filled using a common method of recruitment and posting, regardless of which department is seeking to fill the position." Id. Dr. Bradley's `understanding' was based on the Fed.R.Civ.P. 30(b)(6) deposition testimony of Bill Behrendt, a UPRR human resources manager, who testified regarding UPRR's human resources procedures during the relevant time periods[3]. Id. Dr. Bradley's statistical analyses eliminated applicants who were clearly unsuitable for consideration such as those who withdrew their application, were below age eighteen, reported a felony or misdemeanor conviction, or were reported as "below minimum standards". Id. at p. 6, n. 9. Dr. Bradley determined that "[b]ased on the racial distribution of qualified applicants for the non-agreement positions under consideration, I would have expected 8.95%, or approximately 367 of the selectees to have been African-American." Bradley Report I, p. 6. In fact, only 255 were African-American, and Dr. Bradley opined that "[t]his disparity of 112 fewer African-American selections than expected produces a statistically significant -7.53 standard of difference and an adverse impact ratio of 67.4%...." Id.
The Ward and Woods Report criticizes Dr. Bradley's analysis because it is "premised on the notion that the selection policies and practices utilized in all of the twenty-seven departments at UPRR to fill job vacancies are actually a single `procedure.'" Ward and Woods Report, p. 4. Ward and Woods opine that Dr. Bradley's assessment of the UPRR selection process "is made without the benefit of any empirical evidence included in Dr. Bradley's report, and ... does not emanate from any factual evidence provided by employees or representatives of UPRR who are knowledgeable about how selections occur." Id. In a footnote, Ward and Woods acknowledge Dr. Bradley's reliance on Bill Behrendt's deposition testimony, but discount this reliance as a misinterpretation of that testimony. Id. at n. 4.
In an effort to ascertain the nature of UPRR's non-agreement employee selection process, Ward and Woods "conducted sixteen different interviews with UPRR representatives knowledgeable about how selections have been made in eighteen departments." Ward and Woods Report at p. 13. These representatives were not hiring decision makers, but were instead chosen by Defendant's counsel as "people who could answer [the survey] questions." Deposition of Nathan Woods, excerpts attached as Pltfs. Ex. D to Pltfs. Motion to Strike ("Woods Depo."), p. 155. The survey questions[4] were first drawn up by Dr. Ward and Mr. Woods, but were then subject to editing by Defendant's Counsel. Woods Depo., p. 17; Deposition of Dr. Michael P. Ward, excerpts attached as Pltfs. Ex. C to Pltfs. Motion to Strike ("Ward Depo."), p. 329. Next, the interviewees were provided the questionnaire some time in advance, and the interviews were conducted in the presence of Mr. Woods, Defendant's outside counsel, and UPRR house counsel. Woods Depo., p. 137; Ward Depo., p. 328. Defendant's counsel usually asked questions during the interviews. Id.
From the survey and interview data, Ward and Woods concluded that a proper statistical analysis of the hiring and selection of non-agreement employees at UPRR could only be conducted on a department-by-department basis, and only by considering several variables beyond the minimum qualifications set for each position. Ward and Woods Report, at 17-18, 25-26. Ward and Woods thus concluded that a proper statistical analysis showed that the difference between expected and actual selections of African-American applicants is not statistically significant. Id. p. 27. When Ward and Woods aggregated across all departments, while still controlling for the variables they determined were necessary to properly screen applicants, they found that African-American applicants were subject to an adverse impact ratio of 82% with a statistically significant disparity of slightly more than two standard deviations. Ward and Woods Report at p. 27 & Table 1. Ward and Woods specifically concluded that, "based upon our interviews and our review deposition testimony," an accurate statistical model of the UPRR non-agreement selection process is that reported on a department-by-department basis. Id., p. 29.
Plaintiffs' primary complaint about the Ward and Woods Report is that the interview and survey methodology used to shape the statistical analyses is scientifically invalid. They argue in turn that this invalidity rendered the entirety of Defendant's experts' testimony unreliable, and therefore inadmissible as the flawed survey data was used to (1) determine whether selections for all departments should be aggregated, and (2) select additional control variables Ward and Woods elected to layer over the minimum job qualifications. Plaintiffs' expert, Dr. Bradley, prepared a Rebuttal Expert Statistical Report ("Bradley Report II"), in which he took issue with several aspects of the Ward and Woods Report. Bradley Report II, p. 3 (shorter analysis period leads to zero selections in some departments); Id. at p. 6 (additional variables to screen applicants were not validated). However, Dr. Bradley's principal criticism was that Ward and Woods interview procedure "exhibits serious scientific, methodological flaws that call into question the reliability of information that was obtained through these interviews." Bradley Report II, p. 8-9. Dr. Bradley found the following aspects of the Ward and Woods interview protocol problematic:
1. Not all UPRR departments were included in the interviews. The departments chosen were selected by Defendant's counsel, rather than through a random selection process.
2. Only 3-5 jobs were reviewed in each department, and these were chosen by the interviewees, not by the researchers based on analysis of UPRR selection records.
3. Defense counsel chose the interview subjects, indicating a possible selection bias.
4. Questionnaires were sent out prior to interviews, eliminating contemporaneous responses.
5. Defense counsel was present during interviews and asked questions of interviewees. The interviewees may have felt pressure to respond in certain ways, to `champion' the interests of UPRR.
Bradley Report II, p. 9 (paraphrased).
Based on what was known about the interview protocol, Dr. Bradley concluded that "[t]he interviews... lack the necessary independence required of a valid scientific survey. Defense counsel appears to have had a major involvement in the initiation and design of the interview questions and participated directly in the interviews themselves." Id.
Dr. John Veres, Plaintiffs' expert, prepared an expert report in which he opined that the analyses presented in the Ward and Woods Report are "not helpful in determining whether adverse impact occurred as the result of the selection process employed by [UPRR]." John G. Veres, III, A Critique of "Statement of Qualifications and Report of Findings in Karen Yapp, et al. v. Union Pacific Railroad Company" by Michael P. Ward and Nathan D. Woods ("Veres Report"), attached as Pltf. Ex. E to Pltf. Motion to Strike, p. 2 (emphasis in original). Dr. Veres specifically found several methodological deficiencies with the Ward and Woods Report, "the most striking being a lack of independence in conducting the survey." Id.
Defendant responds by arguing that the Plaintiffs have erected a classic strawman by arguing that Ward and Woods were required to conduct a "scientific survey". Rather, Defendant argues that Ward and Woods, "in a wholly proper and accepted fashion... interviewed knowledgeable people as part of their effort to gather appropriate `facts' and `data' ... upon which to conduct their expert statistical analyses." Defendant's Memorandum in Opposition to Plaintiffs' Motion to Strike ("Deft.Opp."), p. 4. The balance of Defendant's argument consists largely of comparisons between the methodology Dr. Ward and Mr. Woods used and that employed by Dr. Bradley. Id. at 8. The Court declines to scrutinize Dr. Bradley's methodology because the issue before the Court is the admissibility of the Ward and Woods Report, and not Dr. Bradley's study.
Standard of Admissibility for Scientific Evidence
Fed.R.Evid. 702 provides that:
If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case.
Fed.R.Evid. 702 (West 2003).
In order to be admissible under this Rule, the "subject of an expert's testimony must be `scientific ... knowledge.' The adjective `scientific' implies a grounding in the methods and procedures of science." Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 589-90, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993). It is further required that "in order to qualify as `scientific knowledge,' an inference or assertion must be derived by the scientific method. Proposed testimony must be supported by appropriate validation i.e., `good grounds,' based on what is known." Id. at 590, 113 S.Ct. 2786. The Daubert Court provided a number of factors for courts to apply in determining the relevance and reliability of expert testimony including, but not limited to, "(1) whether the theory or technique `can be (and has been) tested'; (2) `whether the theory or technique has been subjected to peer review and publication'; (3) `the known or potential rate of error'; and (4) whether the theory has been generally accepted." Lauzon v. Senco Products, Inc., 270 F.3d 681, 687 (8th Cir.2001) (citations omitted). Later cases have developed "additional factors such as: whether the expertise was developed for litigation or naturally flowed from the expert's research; whether the proposed expert ruled out other alternative explanations; and whether the proposed expert sufficiently connected the proposed testimony with the facts of the case." Id. (citing exemplar cases). Despite the numerous factors available to courts for determination of the admissibility of expert testimony, "the polestar, however, must always be scientific validity and thus the evidentiary relevance and reliability of the principles that underlie a proposed submission." Jaurequi v. Carter Mfg. Co., Inc., 173 F.3d 1076, 1082 (8th Cir.1999). Finally, the "proponent of the expert testimony must prove its admissibility by a preponderance of the evidence." Lauzon, 270 F.3d at 686.
This Court has previously considered the use of survey data used to support statistical analysis. Metropolitan St. Louis Equal Housing Opportunity Council v. Gordon A. Gundaker Real Estate Co., Inc., 130 F.Supp.2d 1074 (E.D.Mo.2001) (hereinafter "MEHOC"). In MEHOC, the Court rejected scientific evidence based on a series of tests designed to investigate and study possible housing discrimination in Florissant, Missouri. Id. at 1083. The decision in MEHOC was based in part on the fact that the plaintiffs failed to "articulate a set of standards which govern the validity" of their testing protocol, the lack of a standardized protocol, and a failure to conduct appropriate statistical analyses on the resultant data. Id. at 1084, 1088. Where current, standardized social science research methodologies are adhered to closely in conducting a survey, a court may find the results admissible. MEHOC, at 1093.
Discusssion
Defendant's response to the Plaintiffs' challenge to the methodological flaws in the survey conducted by Ward and Woods is simple: "Ward and Woods never attempted to do a `scientific survey' because they saw absolutely no reason to do so." Deft. Memo. in Opp., p. 5. Although Ward and Woods may have seen no reason for their statistical analyses to be guided by reliable, scientific survey data, the Court concludes that a valid scientific survey was necessary to support their conclusions regarding additional hiring qualifications, and the purportedly unique hiring processes used by various UPRR departments. Ward and Woods predicated their entire analysis on their understanding of the hiring and selection processes used at UPRR, and a failure to ground that analysis in a reliable method for understanding those processes undercuts the validity of their entire effort. Defendant recognizes that in conducting a scientific inquiry "[s]urveys are used to describe or enumerate objects or the beliefs, attitudes, or behavior of persons or other social units." Deft. Memo. in Opp., n. 2 (quoting Federal Judicial Center, Reference Manual on Scientific Evidence, p. 231). However, Defendant argues that "[g]aining an understanding of the promotion selection process utilized at UPRR would hardly seem amenable to use of such a `survey.'" Id. The Court disagrees. In fact, the only scientific way to determine that the otherwise uniform aspects of the UPRR hiring and selection process are controlled by specific departmental practices would be a scientific survey of the behavior of those departments. Although Ward and Woods conducted a survey, after a fashion, it was inherently unreliable because of the methodology employed.
The presumption guiding the analyses in the Ward and Woods Report is that "the electronic data given to [them] and the information provided by individuals knowledgeable about the selection processes at UPRR made it possible for [Ward and Woods] to evaluate the extent to which it is appropriate to analyze UPRR's selection process as a single procedure... or as many separate and departmental procedures." Ward and Woods Report, p. 12. The basis of their decision to analyze each department separately was the interview process conducted in close concert with UPRR's inside and outside counsel. Id. at 29. Counsel for Defendant selected the interviewees, screened the interview questions, and participated in each of the interviews. The study also included a non-random sample of UPRR departments, and accounted for a limited number of job types in each department. The Court finds little need to further scrutinize the Ward and Woods survey, as Defendant admits that Ward and Woods never attempted to conduct a scientifically valid survey. Further, although the Court declines to adopt a rigid bar against the participation of counsel in a study of employment procedures, under the facts of this case, the heavy involvement of defense counsel in the design and conduct of a survey used to guide expert statistical analyses indicates a lack of independence and thus a lack of scientific validity. See generally, Federal Judicial Center, Reference Manual on Scientific Evidence, p. 231-279 (2d ed.2000) (discussing important protocol features designed to eliminate biases in survey results).
UPRR argues strenuously that the survey procedure was merely designed to familiarize Ward and Woods with UPRR's selection procedure, and to derive background information. This argument is unpersuasive because Ward and Woods used their survey data to justify the two major methodological differences between their analyses and Dr. Bradley's. First, Ward and Woods determined that it was inappropriate to aggregate selection data across all UPRR departments when examining racial differences in selection. Second, Ward and Woods determined that criteria other than minimum qualifications should be used to eliminate African-American applicants from the selection pool when comparing selection and hiring process outcomes. Thus, the survey provided Ward and Woods with a basis for conducting a department-by-department statistical analysis, and for controlling for additional variables beyond each job's minimum qualifications in all of their statistical calculations. For those analyses to be reliable, the underlying survey data used to support and shape those analyses must also be the product of reliable scientific methods. MEHOC, supra; Kumho Tire, 526 U.S. at 156-57, 119 S.Ct. 1167. As the Court has concluded that the survey methodology was flawed, and Defendant concedes that the survey was not scientific or inherently reliable, the Ward and Woods Report is not admissible scientific evidence.
Accordingly,
IT IS HEREBY ORDERED that Plaintiffs' Motion to Strike Defendant's Experts (# 45) be and is GRANTED. The report and testimony of Defendant's experts Dr. Michael P. Ward and Mr. Nathan D. Woods are not admissible under Fed.R.Evid. 702.
NOTES
[1] The Court wrote to counsel in this case suggesting that a hearing on the Motion to Strike would be held if either party so desired, and that certain dates were available. See Doc. No. 57. Counsel for all parties declined a hearing, and thus the Court will proceed to decide the Motion based on the parties' submissions.
[2] The parties agree that non-agreement employees are those who are not party to a collective bargaining agreement.
[3] The Court declines the parties' tacit invitation to make factual determinations about the employee selection process at times relevant to this cause. The only issue before the Court is whether Mr. Ward and Dr. Woods' testimony and reports are admissible under Fed.R.Evid. 702, and only factual determinations essential to resolving that issue will be made at this time.
[4] The Court has not been provided with a copy of the survey questions.