OPINION
KETHLEDGE, Circuit Judge.
In this case the EEOC sued the defendants for using the same type of background check that the EEOC itself uses. The EEOC’s personnel handbook recites that “[ojverdue just debts increase temptation to commit illegal or unethical acts as a means of gaining funds to meet financial obligations.” Because of that concern, the EEOC runs credit checks on applicants for 84 of the agency’s 97 positions. The defendants (collectively, “Kaplan”) have the same concern; and thus Kaplan runs credit checks on applicants for positions that provide access to students’ financial-loan information, among other positions. For that practice, the EEOC sued Kaplan.
Specifically, the EEOC alleges that Kap-lan’s use of credit checks causes it to screen out more African-American applicants than white applicants, creating a disparate impact in violation of Title VII of the federal Civil Rights Act.
See
42 U.S.C. § 2000e-2(a)(l), (a)(2), (k). Proof of disparate impact is usually statistical proof in the form of expert testimony; and here the EEOC relied solely on statistical data compiled by Kevin Murphy, who holds a doctorate in industrial and organizational psychology. For two reasons, however, the district court excluded Murphy’s testimony on grounds that it was unreliable. First, the EEOC presented “no evidence” that Murphy’s methodology satisfied any of the factors that courts typically consider in determining reliability under Federal Rule of Evidence 702; and second, as Murphy himself admitted, his sample was not representative of Kaplan’s applicant pool as a whole. The district court therefore granted summary judgment to Kaplan. The EEOC now argues that the district court “erred” — a telling, oft-repeated, and mistaken choice of word here — when it excluded Murphy’s testimony. We reject the EEOC’s arguments and affirm.
Kaplan offers undergraduate and graduate degrees to students across the country.
Some of Kaplan’s students obtain financial aid through programs operated by the United States Department of Education; and consequently, some of Kaplan’s employees have access to those students’ financial information. The Department has regulations that circumscribe the manner in which Kaplan can access and use students’ information. Violations of those regulations can bring severe penalties.
Kaplan’s concerns became reality about a decade ago, when it discovered that some of its financial-aid officers had stolen payments that belonged to students. Kaplan also learned that some of its executives had engaged in self-dealing, by hiring relatives as vendors. In response, Kaplan implemented a number of measures to prevent these abuses. One of those measures was to run credit checks on applicants for senior-executive positions, accounting and other positions with access to company financials or cash, and positions with access to student financial-aid information. The credit checks are performed by a third-party vendor, which reports, among other things, whether the applicant has ever filed for bankruptcy, is delinquent on child-support payments, has any garnishments on earnings, has outstanding civil judgments exceeding $2,000, or has a social-security number that does not match the number the credit bureau has on file. If an applicant’s credit history includes any of the enumerated items, the vendor flags the applicant’s file for “review.” At that point, Kaplan typically reviews the file and makes an ad hoc decision as to whether to move forward with the application. The credit-check process is racially blind: the vendor does not report the applicant’s race with her other information.
Kaplan has used several vendors for its credit checks, but Murphy focused upon applications screened by one vendor, General Information Services (“GIS”). Murphy obtained GIS data for 4,670 applicants. That data, as discussed above, did not include the applicant’s race, so the EEOC subpoenaed records from the departments of motor vehicles. Eleven states provided records that identified an applicant’s race. Thirty-six states and the District of Columbia provided color copies of drivers’ license photos for approximately 900 applicants.
The dispute in this case concerns the reliability, or lack thereof, of the process — which the EEOC calls “race rating” — by which Murphy purported to identify the race of each person in those drivers’ license photos. The process was crafted by Murphy himself, specifically for purposes of litigation — though the record contains no indication that Murphy has any particular expertise in constructing methodologies to identify race by visual means. In any event, Murphy assembled a team of five “race raters,” each of whom has experience in what the EEOC calls “multicultural, multiracial, treatment outcome research” — a term undefined by the EEOC here. But that term assuredly does not refer to the raters’ experience with methodologies to identify race by visual means — since, undisputedly, they have none. Murphy directed each rater separately to review each applicant’s drivers’ license photograph and then classify the person’s race in one of five ways: “African-American,” “Asian,” “Hispanic,” “White,” or “Other.” If four of five raters agreed upon a particular applicant’s race, the applicant was so classified for purposes of Murphy’s statistics. For 11.7% of the photographs, the raters failed to reach that consensus. For some reason Murphy also provided the raters with each applicant’s name — which, the EEOC concedes, the raters were supposed to disregard when classifying an applicant’s race.
Murphy filed his expert report on May 1, 2012 and then a revised report on August 17, 2012, both per the district court’s scheduling order. The revised report included the putative race and credit-check results for a total of 1,090 applicants, of whom 803 had been racially classified per Murphy’s “rating” process. In that sample of 1,090 applicants (out of a total of 4,670 applicants for whom GIS provided data), the percentage of black applicants who were flagged for review, based upon their credit histories, was higher than the percentage of white applicants who were flagged. (That is essentially the basis upon which the EEOC claims disparate impact here.) But Murphy’s sample overrepresented “fails” generally: 23.8% of the applicants in his sample of 1,090 were rejected because of their credit history, whereas only 13.3% of the total GIS pool of 4,670 were.
Murphy then proceeded to file additional reports, contrary to the terms of the district court’s scheduling order. On September 5, 2012 — in response to a critical analysis of his work by Kaplan’s expert— Murphy submitted a third report, which Kaplan moved to strike, but which the district court reluctantly permitted in an October 5 order, with the admonition that “[n]o further expert reports are allowed.” Yet Murphy filed another report on November 8, 2012, two weeks before summary-judgment briefing was due. In that report, Murphy provided what the EEOC describes as “anecdotal corroboration” of the reliability of his race-rating process. Murphy filed yet another report on December 21, 2012, this time in response to Kaplan’s motion specifically to exclude his testimony as unreliable under Rule 702.
Free access — add to your briefcase to read the full text and ask questions with AI
OPINION
KETHLEDGE, Circuit Judge.
In this case the EEOC sued the defendants for using the same type of background check that the EEOC itself uses. The EEOC’s personnel handbook recites that “[ojverdue just debts increase temptation to commit illegal or unethical acts as a means of gaining funds to meet financial obligations.” Because of that concern, the EEOC runs credit checks on applicants for 84 of the agency’s 97 positions. The defendants (collectively, “Kaplan”) have the same concern; and thus Kaplan runs credit checks on applicants for positions that provide access to students’ financial-loan information, among other positions. For that practice, the EEOC sued Kaplan.
Specifically, the EEOC alleges that Kap-lan’s use of credit checks causes it to screen out more African-American applicants than white applicants, creating a disparate impact in violation of Title VII of the federal Civil Rights Act.
See
42 U.S.C. § 2000e-2(a)(l), (a)(2), (k). Proof of disparate impact is usually statistical proof in the form of expert testimony; and here the EEOC relied solely on statistical data compiled by Kevin Murphy, who holds a doctorate in industrial and organizational psychology. For two reasons, however, the district court excluded Murphy’s testimony on grounds that it was unreliable. First, the EEOC presented “no evidence” that Murphy’s methodology satisfied any of the factors that courts typically consider in determining reliability under Federal Rule of Evidence 702; and second, as Murphy himself admitted, his sample was not representative of Kaplan’s applicant pool as a whole. The district court therefore granted summary judgment to Kaplan. The EEOC now argues that the district court “erred” — a telling, oft-repeated, and mistaken choice of word here — when it excluded Murphy’s testimony. We reject the EEOC’s arguments and affirm.
Kaplan offers undergraduate and graduate degrees to students across the country.
Some of Kaplan’s students obtain financial aid through programs operated by the United States Department of Education; and consequently, some of Kaplan’s employees have access to those students’ financial information. The Department has regulations that circumscribe the manner in which Kaplan can access and use students’ information. Violations of those regulations can bring severe penalties.
Kaplan’s concerns became reality about a decade ago, when it discovered that some of its financial-aid officers had stolen payments that belonged to students. Kaplan also learned that some of its executives had engaged in self-dealing, by hiring relatives as vendors. In response, Kaplan implemented a number of measures to prevent these abuses. One of those measures was to run credit checks on applicants for senior-executive positions, accounting and other positions with access to company financials or cash, and positions with access to student financial-aid information. The credit checks are performed by a third-party vendor, which reports, among other things, whether the applicant has ever filed for bankruptcy, is delinquent on child-support payments, has any garnishments on earnings, has outstanding civil judgments exceeding $2,000, or has a social-security number that does not match the number the credit bureau has on file. If an applicant’s credit history includes any of the enumerated items, the vendor flags the applicant’s file for “review.” At that point, Kaplan typically reviews the file and makes an ad hoc decision as to whether to move forward with the application. The credit-check process is racially blind: the vendor does not report the applicant’s race with her other information.
Kaplan has used several vendors for its credit checks, but Murphy focused upon applications screened by one vendor, General Information Services (“GIS”). Murphy obtained GIS data for 4,670 applicants. That data, as discussed above, did not include the applicant’s race, so the EEOC subpoenaed records from the departments of motor vehicles. Eleven states provided records that identified an applicant’s race. Thirty-six states and the District of Columbia provided color copies of drivers’ license photos for approximately 900 applicants.
The dispute in this case concerns the reliability, or lack thereof, of the process — which the EEOC calls “race rating” — by which Murphy purported to identify the race of each person in those drivers’ license photos. The process was crafted by Murphy himself, specifically for purposes of litigation — though the record contains no indication that Murphy has any particular expertise in constructing methodologies to identify race by visual means. In any event, Murphy assembled a team of five “race raters,” each of whom has experience in what the EEOC calls “multicultural, multiracial, treatment outcome research” — a term undefined by the EEOC here. But that term assuredly does not refer to the raters’ experience with methodologies to identify race by visual means — since, undisputedly, they have none. Murphy directed each rater separately to review each applicant’s drivers’ license photograph and then classify the person’s race in one of five ways: “African-American,” “Asian,” “Hispanic,” “White,” or “Other.” If four of five raters agreed upon a particular applicant’s race, the applicant was so classified for purposes of Murphy’s statistics. For 11.7% of the photographs, the raters failed to reach that consensus. For some reason Murphy also provided the raters with each applicant’s name — which, the EEOC concedes, the raters were supposed to disregard when classifying an applicant’s race.
Murphy filed his expert report on May 1, 2012 and then a revised report on August 17, 2012, both per the district court’s scheduling order. The revised report included the putative race and credit-check results for a total of 1,090 applicants, of whom 803 had been racially classified per Murphy’s “rating” process. In that sample of 1,090 applicants (out of a total of 4,670 applicants for whom GIS provided data), the percentage of black applicants who were flagged for review, based upon their credit histories, was higher than the percentage of white applicants who were flagged. (That is essentially the basis upon which the EEOC claims disparate impact here.) But Murphy’s sample overrepresented “fails” generally: 23.8% of the applicants in his sample of 1,090 were rejected because of their credit history, whereas only 13.3% of the total GIS pool of 4,670 were.
Murphy then proceeded to file additional reports, contrary to the terms of the district court’s scheduling order. On September 5, 2012 — in response to a critical analysis of his work by Kaplan’s expert— Murphy submitted a third report, which Kaplan moved to strike, but which the district court reluctantly permitted in an October 5 order, with the admonition that “[n]o further expert reports are allowed.” Yet Murphy filed another report on November 8, 2012, two weeks before summary-judgment briefing was due. In that report, Murphy provided what the EEOC describes as “anecdotal corroboration” of the reliability of his race-rating process. Murphy filed yet another report on December 21, 2012, this time in response to Kaplan’s motion specifically to exclude his testimony as unreliable under Rule 702.
The district court thereafter excluded Murphy’s testimony in a meticulously reasoned opinion. We review that exclusion deferentially, for an abuse of discretion.
Gen. Elec. Co. v. Joiner,
522 U.S. 136, 139, 118 S.Ct. 512, 139 L.Ed.2d 508 (1997).
Rule 702 provides that “[a] witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if[,]” among other requirements, the testimony “is based on sufficient facts and data” and “is the product of reliable principles and methods[.]” Fed.R.Evid. 702(b), (c). As the proponent of expert testimony, the EEOC bears the burden of proving its admissibility.
Nelson v. Tenn. Gas Pipeline Co.,
243 F.3d 244, 251 (6th Cir.2001).
In determining whether an expert’s methodology is reliable, courts frequently consider the factors set forth by the Supreme Court in
Daubert v. Merrell Dow Pharmaceuticals, Inc.,
509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993). But the district court has “broad latitude” as to which factors to consider in a particular case.
Kumho Tire Co. v. Carmichael,
526 U.S. 137, 142, 119 S.Ct. 1167, 143 L.Ed.2d 238 (1999).
Here, the district court considered every one of the
Daubert
factors— and found that Murphy’s methodology flunked them all. Two factors we consider together. “Ordinarily, a key question to be answered” in determining whether a technique is reliable is “whether it can be (and has been) tested.”
Daubert,
509 U.S. at 593, 113 S.Ct. 2786. Similarly, “in the case of a particular scientific technique, the court ordinarily should consider the known or potential rate of error.”
Id.
at 594, 113 S.Ct. 2786. The district court found that the EEOC “wholly fail[ed]” to provide evidence in support of either of these factors. Op. at 13. In response, the EEOC argues that we can find that support in Murphy’s “anecdotal corroboration” — set forth in his November 8 report, submitted a month after the district court said there would be no more reports. We
consider it now only because the district court did. Murphy says that, as to 47 applicants, he cross-checked his raters’ classifications with racial identifications provided by a DMV, finding 95.7% agreement between the two. Murphy also says that, for another 10 applicants, he crosschecked his raters’ classifications with racial information provided by PeopleSoft, a personnel-software program used internally by Kaplan. That cross-check yielded an 80% match — an unimpressive correlation in case where a few percentage points (in credit-check fail rates for blacks and whites) might make the difference between significant liability and none. But more to the point, as Murphy himself candidly conceded, a mere 57 instances of anecdotal corroboration is not enough “to establish the reliability of my photo rating methodology.” The district court was well within its “broad latitude!,]”
Kumho,
526 U.S. at 142, 119 S.Ct. 1167, to find these factors unmet.
The EEOC’s case goes downhill from there. In determining reliability, “[ajnother pertinent consideration is whether the theory or technique has been subjected to peer review and publication.”
Daubert,
509 U.S. at 593, 113 S.Ct. 2786. Again the district court found no evidence in support of this factor, and undisputedly there is none. Before us, the EEOC simply argues that the district court should not have considered the factor. The argument is meritless: “submission to the scrutiny of the scientific community is a component of ‘good science,’ in part because it increases the likelihood that substantive flaws in methodology will be detected!,]”
id.;
and the district court had good reason to think that such scrutiny might have detected flaws here.
The district court also found that Murphy’s methodology lacked “standards controlling the technique’s operation.”
Id.
at 594, 113 S.Ct. 2786. The EEOC responds that the relevant standard was Murphy’s requirement that four of five raters agree on an applicant’s race. But that response overlooks Murphy’s own concession that the raters themselves had no particular standard in classifying each applicant; instead, they just eyeballed the DMV photos. The district court was also troubled by an affirmative breakdown in Murphy’s controls: “the ‘race raters’ were provided the names of the applicants,” which — for a methodology purportedly based exclusively on visual identification of race — might “create an unintended bias on the part of the panel.” Op. at 16. The EEOC countered at oral argument that there is “no evidence” that the raters considered any of the applicants’ names in classifying them by race. But that argument merely illustrates a fallacy that pervades the agency’s entire argument on appeal, to wit: that it was Kaplan’s (or the district court’s) burden to show that Murphy’s testimony was
in
admissible, rather than the EEOC’s burden to show that his testimony was admissible. The law says the contrary.
See Nelson,
243 F.3d at 251; Fed.R.Evid. 702 Advisory Comm. Notes, 2000 Amendments (“the proponent has the burden of establishing that the pertinent admissibility requirements are met by a preponderance of the evidence”).
The district court also found that the EEOC “presented] no evidence” that Murphy’s race-rating methodology “is generally accepted in the scientific community.” Op. at 16;
see generally Daubert,
509 U.S. at 593, 113 S.Ct. 2786. On this point the EEOC offers no response — perhaps because, as the district court observed, “the EEOC itself discourages employers from visually identifying an individual by race and indicates that visual identification is appropriate ‘only if an employee refuses to self identify.’ ” Op. at 16.
Finally, as an independent ground for excluding Murphy’s testimony, the district
court found that “[t]here is no indication” that Murphy’s group of 1,090 applicants is in any “way ‘representative’ of the applicant pool as a whole.” Op. at 18. Instead there is a strong indication to the contrary: Murphy’s group had a fail rate of 23.8%, whereas the GIS applicant pool had a fail rate of only 13.3%. On this point, suffice it to say that an unrepresentative sample by definition might not be representative of the respective fail rates of black and white applicants in the larger pool — and thus is not a reliable means to demonstrate disparate impact.
We need not belabor the issue further. The EEOC brought this case on the basis of a homemade methodology, crafted by a witness with no particular expertise to craft it, administered by persons with no particular expertise to administer it, tested by no one, and accepted only by the witness himself. The district court did not abuse its discretion in excluding Murphy’s testimony.
The district court’s judgment is affirmed.