The Association Of Mexican-American Educators v. State Of California

231 F.3d 572

Opinion

231 F.3d 572 (9th Cir. 2000)

THE ASSOCIATION OF MEXICAN-AMERICAN EDUCATORS ("AMAE"); CALIFORNIA ASSOCIATION FOR ASIAN-PACIFIC BILINGUAL EDUCATION ("CAFABE"), on behalf of themselves, their members, and all others similarly situated; OAKLAND ALLIANCE OF BLACK EDUCATORS ("OABE"), on behalf of themselves, their members, and all others similarly situated; SARA MACNEIL BOYD;SAM GENIS; TOUA YANG; BOB WILLIAMS; MARTA LECLAIRE; ANTOINETTE WILLIAMS; DIANA KWAN; and AGNES HAYNES, on behalf of themselves and all others similarly situated, Plaintiffs-Appellants/ Cross-Appellees,
v.
STATE OF CALIFORNIA; THE CALIFORNIA COMMISSION ON TEACHER CREDENTIALING, Defendants-Appellees/ Cross-Appellants.

Nos. 96-17131, 97-15422

UNITED STATES COURT OF APPEALS FOR THE NINTH CIRCUIT

Argued and Submitted June 20, 2000
Filed October 30, 2000

[Copyrighted Material Omitted][Copyrighted Material Omitted][Copyrighted Material Omitted][Copyrighted Material Omitted]

John T. Affeldt, Public Advocates, Inc., San Francisco, California, for the plaintiffs-appellants/appellees.

R. Lawrence Ashe, Jr., and Eric J. Taylor, Paul, Hastings, Janofsky & Walker, LLP, Atlanta, Georgia; and William Lockyer, Attorney General, and Stephanie Wald, Deputy Attorney General, San Francisco, California, for the defendants-appellees/appellants.

Stewart Weinberg, Van Bourg, Weinberg, Roger & Rosenfeld, Oakland, California; Paul J. Dostart, Dostart Clapp Sterrett & Coveney, LLP, San Diego, California; Martin Bienstock, Assistant Attorney General, New York, New York; Peter Roos, San Francisco, California; and Gregory B. Friel and Jennifer Levin, United States Department of Justice, Washington, D.C., for the amici.

Appeals from the United States District Court for the Northern District of California William H. Orrick, Jr., District Judge, Presiding. D.C. No.CV-92-03874-WHO

Before: Procter Hug, Jr., Chief Judge, and Mary M. Schroeder, Stephen Reinhardt, Diarmuid F. O'Scannlain, Ferdinand F. Fernandez, Pamela Ann Rymer, Andrew J. Kleinfeld, A. Wallace Tashima, Sidney R. Thomas, Susan P. Graber, and Ronald M. Gould, Circuit Judges.

Opinion by Judge Graber; Partial Concurrence and Partial Dissent by Judge Reinhardt; Partial Concurrence and Partial Dissent by Judge Fernandez; Partial Concurrence and Partial Dissent by Judge Kleinfeld; Partial Concurrence and Partial Dissent by Judge Gould; Dissent by Judge Tashima

GRABER, Circuit Judge:

Plaintiffs are a class of Mexican-American, Asian American, and African-American educators and would-be educators in California. They appeal from an adverse judgment in their action against the State of California and its agency, challenging (1) the district court's holding that the California Basic Education Skills Test ("CBEST"), which is a prerequisite to employment in a variety of positions in the California public schools, violates neither Title VI nor Title VII of the Civil Rights Act of 1964, and (2) the district court's use of a technical advisor at trial. Defendants cross-appeal from the district court's rulings, on summary judgment, that Title VI and Title VII apply in this case. Defendants also appeal from the district court's order denying their request for costs. For the reasons that follow, we hold that Title VII applies to the CBEST; that the CBEST was validated properly; that the district court permissibly used a technical advisor; and that the district court did not abuse its discretion by refusing to award costs to Defendants. Accordingly, we affirm both the judgment in Defendants' favor and the order denying them costs.

FACTUAL AND PROCEDURAL BACKGROUND

Effective February 1, 1983, the California legislature amended the California Education Code to prohibit the California Commission on Teacher Credentialing ("CCTC") from issuing "any credential, permit, certificate, or renewal of an emergency credential to any person to serve in the public schools unless the person has demonstrated proficiency in basic reading, writing, and mathematics skills." Cal. Educ. Code S 44252(b). At the same time, the legislature authorized the state's Superintendent of Public Instruction to "adopt an appropriate state test to measure proficiency in these basic skills." Cal. Educ. Code S 44252(c). The Superintendent adopted the CBEST and, in May 1983, CCTC assumed responsibility for administering and revising the test.

The CBEST is a pass-fail examination consisting of three sections: reading, writing, and mathematics.1 The reading and mathematics sections each contain 50 multiple-choice questions, 40 of which are scored. The writing section consists of two essays. The CBEST was revised in 1995. At that time, questions that tested "higher order" mathematical skills, such as geometry, were eliminated from the mathematics section of the test.

To pass the CBEST, a candidate must receive a "scaled" score of 123. Accordingly, a candidate passes by averaging 41 points on each of the three sections (out of a score range of 20 to 80). A scaled score of 41 on the reading section translates into a raw score of 28 out of 40 questions correct; on the mathematics section, a scaled score of 41 equates to a raw score of 26 out of 40 correct. Each of the two essays is graded by two readers, who give raw scores of between one and four points per essay. Thus, the range of possible scores for the writing section is between four and 16 points. A raw score of 12 points translates into a scaled score of 41 points. The CBEST employs a "compensatory scoring" model, under which a candidate passes the test with a scaled score lower than 41 on a particular section, so long as his or her total scaled score is at least 123.

A passing score on the CBEST is required for all public elementary and secondary school teachers in California. See Cal. Educ. Code SS 44256, 44257, 44259. A passing score also is required for many non teaching employees of the California public schools, including administrators, see id. S 44270, school counselors, see id.S 44266, and school librarians, see id. S 44269.

Since the CBEST's inception, minority candidates have disproportionately received failing scores. The named Plaintiffs are three nonprofit organizations that represent the interests of minority educators, and eight individual minority candidates. They brought this action against the State of California and the CCTC to challenge the validity of the test under Title VI and Title VII, on behalf of themselves and all others similarly situated. The district court certified the following class:

All Latinos, African-Americans and Asians who have sought or are seeking California public school credentials and certificated positions who have been, are being, or will be adversely affected in their abil ity to obtain credentials and certificated positions by [CBEST] results.

In their complaint, Plaintiffs sought to enjoin the use of the CBEST, alleging that the test has a disproportionate, adverse impact on minority candidates and that Defendants have failed to adopt screening procedures with a less adverse impact.

The parties filed cross-motions for summary judgment, addressing the applicability of Titles VI and VII. In August 1993, the district court granted partial summary judgment to Plaintiffs, concluding that both Titles VI and VII apply to the CBEST. See Association of Mexican-American Educators v. California, 836 F. Supp. 1534 (N.D. Cal. 1993) ("AMAE I"). Following a bench trial, the district court held that: (1) Plaintiffs had demonstrated that the CBEST has a disparate impact on minorities; (2) the studies that were submitted at trial demonstrated that the test was a valid measure of job-related skills; (3) the level at which the passing scores were set reflected reasonable professional judgments about minimum levels of basic knowledge; and (4) Plaintiffs had failed to demonstrate that other, equally effective screening devices existed. The court entered a judgment in Defendants' favor. See Association of Mexican-American Educators v. California, 937 F. Supp. 1397 (N.D. Cal. 1996) (" AMAE II").

Defendants then presented the district court with a bill for taxable costs pursuant to Federal Rule of Civil Procedure 54(d)(1). That cost bill totaled $216,443.67. In an order dated February 12, 1997, the district court denied the cost bill in its entirety.

On appeal, Plaintiffs argue (1) that the district court erred in concluding, after trial, that the CBEST was validated properly and (2) that the court violated Federal Rule of Evidence 706 by relying on the advice of an expert who was not subject to cross-examination and did not prepare a report. Defendants cross-appeal with respect to the district court's conclusions, on summary judgment, that Titles VI and VII apply. Defendants also appeal from the district court's order denying costs.2

DISCUSSION

I. Title VII

A. Title VII Applies to the CBEST.3

Defendants appeal from the district court's summary judgment in favor of Plaintiffs on the issue of the applicability of Titles VI and VII. We review de novo the district court's grant of summary judgment. See Robi v. Reed, 173 F.3d 736, 739 (9th Cir.), cert. denied, 120 S. Ct. 375 (1999).

As a threshold matter, we note that, because we ultimately hold that the CBEST was validated properly (see Part I.B., below), we could decline to decide whether Titles VI and VII apply. Were we to do so, we simply would assume for the sake of argument that the statutes apply and move immediately to the question of validation. Although that might appear to be an expedient approach, we decline to follow it for three reasons. First, as a matter of logic, the applicability of Title VI or Title VII is a predicate to any discussion of validation. Validation would not be required, and indeed would not even be relevant, if neither Title VI nor Title VII applies. Second, as a matter of fairness, these parties deserve an answer not only to the bare question of who wins this case, but also to the underlying question of the applicability of federal civil rights law to the CBEST. The state, in particular, has proceeded for years on the assumption that those laws apply to its administration of the CBEST and has expended considerable effort and expense in attempting to comply with federal law in this area. If that effort was unnecessary, the state deserves to know, so that it may act accordingly in the future. Third, as a matter of judicial economy, our answer to the statutory question can avoid future litigation by other parties. We turn, then, to a discussion of Title VII's application.

Title VII of the Civil Rights Act of 1964 provides:

It shall be an unlawful employment practice for an employer (1) to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment, because of such individual's race, color, religion, sex, or national origin[.]

42 U.S.C. S 2000e-2(a)(1). Title VII applies "to governmental and private employers alike." Dothard v. Rawlinson, 433 U.S. 321, 332 n.14 (1977).

Plaintiffs and Defendants do not have a direct employment relationship. Rather, Plaintiffs are employees and potential employees of individual school districts in California. That fact does not end our inquiry, however. A direct employment relationship is not a prerequisite to Title VII liability. Although "there must be some connection with an employment relationship for Title VII protections to apply," that "connection with employment need not necessarily be direct." Lutcher v. Musicians Union Local 47, 633 F.2d 880, 883 (9th Cir. 1980).

Among other things, we have held that an entity that is not the direct employer of a Title VII plaintiff nevertheless may be liable if it " `interferes with an individual's employment opportunities with another employer.' " Gomez v. Alexian Bros. Hosp., 698 F.2d 1019, 1021 (9th Cir. 1983) (quoting Lutcher, 633 F.2d at 883 n.3). In Gomez, we held that the defendant hospital could be held liable under Title VII for its discriminatory treatment of the plaintiff, notwithstanding the fact that the plaintiff was employed by a third party, if the defendant had interfered with the plaintiff's employment by that third party. See id. at 1021.

In so holding, we followed the opinion of the District of Columbia Circuit in Sibley Memorial Hospital v. Wilson, 488 F.2d 1338, 1340-41 (D.C. Cir. 1973).4 In Sibley, the plaintiff was a male private-duty nurse. When a patient in the defendant hospital requested a private nurse, the hospital arranged through a registry service to have a private nurse provided. That nurse attended the patient at the hospital, but was paid directly by the patient. The plaintiff alleged that the defendant allowed male nurses like him to attend male patients only, but allowed female nurses to attend both male and female patients. See id. at 1339-40.

The D.C. Circuit concluded that the defendant could be held liable under Title VII even though it was not the plaintiff's direct employer. The court reasoned that, although the defendant did not employ the plaintiff, it exercised considerable power over his ability to form employment relationships with third parties. The court noted that Congress intended, through Title VII, to prohibit entities that possessed such power from "foreclos[ing], on invidious grounds, access by any individual to employment opportunities otherwise available to him." Id. at 1341. The court further stated:

To permit a covered employer to exploit circumstances particularly affording it the capability of discriminatorily interfering with an individual's employment opportunities with another employer, while it could not do so with respect to employment in its own service, would be to condone continued use of the very criteria for employment that Congress has prohibited.

Id. Finally, the court held that the defendant's control over the premises on which the plaintiff provided his services, as well as its control over the plaintiff's access to patients, created a "highly visible nexus with the creation and continuance of direct employment relationships between third parties" that brought the defendant's actions within the scope of Title VII. Id. at 1342.

The D.C. Circuit's holding in Sibley was rooted in the text of Title VII. The Court reasoned that, although Title VII applies to "employees," Congress extended the protections of the statute to "any individual" who suffers discrimination: "nowhere are there words of limitation that restrict references in the Act to `any individual' as comprehending only an employee of an employer." Id. at 1341. As we did in Gomez, we agree that the D.C. Circuit's interpretation of the statutory text is the proper one in view of "Congress' directive to read Title VII broadly so as to best effectuate its remedial purposes." Duffield v. Robertson Stephens & Co. , 144 F.3d 1182, 1192 (9th Cir.), cert. denied, 525 U.S. 982 (1998).

We also note, as did the Sibley court, see 488 F.2d at 1342, that Congress explicitly made Title VII applicable outside the"direct employment" context by including employment agencies and labor organizations in the statute's coverage. See 42 U.S.C. S 2000e-2(b), (c). In particular, employment agencies -defined as parties "regularly undertaking with or without compensation to procure employees for an employer or to procure for employees opportunities to work for an employer," 42 U.S.C. S 2000e(c) -may not "refuse to refer . . . or otherwise discriminate against" any individual under Title VII. 42 U.S.C. S 2000e-2(b). This provision of Title VII, like the others, applies to states. See Dothard, 433 U.S. at 332 n.14; Dumas v. Town of Mt. Vernon, 612 F.2d 974, 980 (5th Cir. 1980), overruled on other grounds by Larkin v. Pullman Standard Div., Pullman Inc., 854 F.2d 1549 (11th Cir. 1988), rev'd sub nom. Pullman-Standard, Inc. v. Swint, 493 U.S. 929 (1989). We do not suggest that Defendants are, strictly speaking, an "employment agency" under Title VII, although they perform an analogous function. Rather, we mention this statutory provision as evidence that Congress intended to close any loopholes in Title VII's coverage and to extend the statute's coverage to entities with actual "[c]ontrol over access to the job market," Sibley, 488 F.2d at 1341, whether or not they are direct employers.

In concluding that Title VII applies in this case, the district court held that Defendants "interfere[d] " with Plaintiffs' employment opportunities with local school districts in California by requiring, implementing, and administering the CBEST. See AMAE I, 836 F. Supp. at 1551. We agree.

Our conclusion is dictated by the peculiar degree of control that the State of California exercises over local school districts. In California, public schools are "a matter of statewide rather than local or municipal concern; their establishment, regulation and operation are covered by the[state] constitution and the state Legislature is given comprehensive powers in relation thereto." Hall v. City of Taft, 302 P.2d 574, 576 (Cal. 1956). The California legislature "has plenary authority over the education of California's youth. " San Francisco NAACP v. San Francisco Unified Sch. Dist., 484 F. Supp. 657, 662 (N.D. Cal. 1979). It is "well settled that the California Constitution makes public education uniquely a fundamental concern of the state" and that "the degree of supervision . . . retained by the State over the common school system is high indeed." Butt v. California , 842 P.2d 1240, 1251, 1254 (Cal. 1992).

The state's involvement is not limited to general legislative oversight but, rather, affects the day-to-day operations of local public schools. "Unlike most states, California school districts have budgets that are controlled and funded by the state government rather than the local districts. " Belanger v. Madera Unified Sch. Dist., 963 F.2d 248, 251 (9th Cir. 1992). As the California Supreme Court noted in Butt , California statutes regulate "district organization, elections, and governance; educational programs, instructional materials, and proficiency testing; sex discrimination and affirmative action; admission standards; compulsory attendance; school facilities; rights and responsibilities of students and parents; holidays; school health, safety, and nutrition; teacher credentialing and certification; rights and duties of public school employees; and the pension system for public school teachers. " Butt, 842 P.2d at 1254 (citations omitted). The state also "dictates when students may be expelled or suspended, and . . . exerts control over the textbooks that are used in public schools. " Belanger, 963 F.2d at 253 (citations omitted).

Indeed, the state is so entangled with the operation of California's local school districts that individual districts are treated as "state agencies" for purposes of the Eleventh Amendment. See Freeman v. Oakland Unified Sch. Dist., 179 F.3d 846, 846 (9th Cir. 1999). The fact that the districts are entitled to assert Eleventh Amendment immunity underscores the state's unusually high degree of involvement in the operation of local schools.

Against that background of "plenary" state control, we have no difficulty concluding that the State of California is in a theoretical and practical position to "interfere" with the employment decisions of local school districts. And by requiring, formulating, and administering the CBEST, the state has "interfered" to a degree sufficient to bring it within the reach of Title VII. Through the CBEST, the state has created a limited list of candidates from which local public school districts may hire. Private schools may hire candidates who have not passed the CBEST; but California's public schools, which are under the state's control in almost every aspect of their operations, may not. Thus, in addition to controlling local districts' budgets and textbooks and regulating the duties of public school employees, the state dictates whom the districts may and may not hire. That degree of control over districts' hiring decisions subjects Defendants to the coverage of Title VII in this case.

The relationship between the State of California and California's local school districts is analogous to the relationship between a corporate parent and its wholly owned subsidiaries. "In the absence of special circumstances, a parent corporation is not liable for the Title VII violations of its wholly owned subsidiary." Watson v. Gulf & W. Indus., 650 F.2d 990, 993 (9th Cir. 1981). In Watson, this court held that the parent corporation was not subject to Title VII because the case presented no "special circumstances." Id. But the court went on to explain that, "[i]f there was any evidence that [the parent] participated in or influenced the employment policies of [the subsidiary], . . . then we would be presented with a very different case." Id. Ours is that "very different case." The "parent" state has participated extensively in, and influenced, the employment policies and practices of the "subsidiary" local school districts; therefore, the state is covered by Title VII.

Defendants contend, however, that they are not subject to Title VII because the CBEST is merely a licensing examination. The administration of such examinations, they argue, is not covered by Title VII. As support for that argument, they cite several cases that have held that governmental agencies are not subject to Title VII with regard to their licensing activities. See Haddock v. Board of Dental Exam'rs , 777 F.2d 462 (9th Cir. 1985); Fields v. Hallsville Indep. Sch. Dist., 906 F.2d 1017 (5th Cir. 1990); George v. New Jersey Bd. of Veterinary Med. Exam'rs, 794 F.2d 113 (3d Cir. 1986).5

The cases on which Defendants rely are not controlling for two reasons. First, the state's high level of involvement in the operation of local public schools distinguishes this case from those that Defendants cite. In those cases, licensing was the entire connection between the plaintiffs and the defendants; here, the CBEST is but one aspect of pervasive state control. Second, the CBEST is not merely an ordinary licensing examination; it applies only to public school employees. In other words, the State of California is acting pursuant to its proprietary, as well as its police, power.

There is no overarching "licensing" exception to Title VII. The cases that Defendants cite stand for a related but narrower proposition -that Title VII does not apply when the only connection among the licensing agency, the plaintiff, and the universe of prospective employers is the agency's implementation of a general licensing examination. In such cases, to borrow the words of the Sibley court, the agency does not have a "highly visible nexus with the creation and continuance of direct employment relationships between third parties," such as would subject it to Title VII under an "interference" theory. Sibley, 488 F.2d at 1342.

In Haddock, the plaintiff conceded that his only connection to the defendant Board of Dental Examiners was that the Board had given him an examination that he failed. This court concluded that such a connection, by itself, was insufficient to subject the Board to Title VII liability. See 777 F.2d at 464. The plaintiff apparently did not argue that the Board had "interfered" with his employment under the principle adopted in Gomez and Sibley; if he did, the opinion does not mention it.

In Fields, the plaintiffs argued that the defendants had violated Title VII through their administration of the Texas Examination for Current Administrators and Teachers (TECAT), a compulsory certification examination. See 906 F.2d at 1019. The district court granted summary judgment in favor of the state defendants, concluding that they did not have an employment relationship with the plaintiffs. On appeal, the plaintiffs challenged that conclusion, arguing that the state defendants actually controlled their employment even though the plaintiffs nominally were employed by local districts. In rejecting that argument, the Fifth Circuit noted that "[t]he only evidence presented by [the plaintiffs] suggesting control is the Texas State Board of Education's administration of the TECAT exam and its ability to decertify teachers who fail the exam." Id. (emphasis in original). Of particular relevance to this case is footnote three of the opinion, which reads:

In a footnote in [the plaintiffs'] brief on appeal, they present evidence regarding state funding of facilities, payment of salaries and selection of text books. As this evidence was not before the district court, it is not part of the summary judgment record on appeal.

Id. at 1019 n.3. The court suggested that the outcome might be different if there were such evidence of the state's right to control the work of the teachers. See id. at 1019-20.

Finally, in George, the plaintiff alleged that the defendant Board of Veterinary Medical Examiners had violated Title VII by administering a licensing examination that discriminated against him on the basis of national origin. See 794 F.2d at 114. The Third Circuit affirmed the district court's dismissal of the plaintiff's action. The court distinguished Sibley, on which the plaintiff had relied, stating:

In the Sibley Memorial Hospital case the relationship of the hospital to the employment by its patients of private duty nurses secured for them by the hospital was very close, whereas in the present case there was nothing even remotely resembling an employer employee relationship between the Board and the plaintiff.

Id.

To summarize, the circumstances here demonstrate a level of control and interference far greater than that in the "mere licensing" cases on which Defendants rely. The State of California exerts a high degree of control over the operation of local public school districts. That control is evidenced both by the record and by California law.

Defendants cite George for the further proposition that state licensing examinations are acts of state police power, to which Title VII does not apply. But in George , the Board was acting only pursuant to the state's police power to protect the public from incompetent veterinarians and was not attempting to control the hiring practices of, or the performance of work for, any specific employer. By contrast, the CBEST does not apply across-the-board to all who wish to teach in California, as (for example) a veterinary licensing examination applies to all who wish to practice the profession of veterinary medicine within the state's borders. Rather, the CBEST applies only to those who wish to teach for the public school system -a system over which the State of California exerts plenary control, including regulation of employees' duties.

We conclude, therefore, that administration of the CBEST is not solely an exercise of the state's police power. Rather, it is an exercise of both the state's police power and its proprietary power; and it is the exercise of proprietary power that subjects the state to the coverage of Title VII in this case.

We hold that the CBEST examination is subject to the provisions of Title VII. We turn next to the question whether the CBEST violates the provisions of that Act.

B. The District Court Did Not Clearly Err in Concluding that the CBEST Was Properly Validated.6

"[D]iscriminatory tests are impermissible unless shown, by professionally acceptable methods, to be predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job or jobs for which candidates are being evaluated." Albemarle Paper Co. v. Moody, 422 U.S. 405, 431 (1975). In evaluating employment tests that are alleged to have a racially disparate impact, we first consider whether the plaintiff has established a prima facie case by demonstrating that the test causes a disparate impact on the basis of race. Here, the district court concluded that Plaintiffs had established a prima facie case. See AMAE II, 937 F. Supp. at 1403. Defendants do not challenge that conclusion on appeal.

Because Plaintiffs have established a prima facie case, the burden shifts to Defendants to demonstrate that the CBEST was validated properly.7 See Albemarle Paper, 422 U.S. at 425. In its detailed and careful opinion, the district court concluded that Defendants had met their burden and that the test had been validated properly based on three studies: (1) the 1982 Wheeler and Elias study; (2) the 1985 Practitioners' Review; and (3) the 1995 Lundquist study. See AMAE II, 937 F. Supp. at 1411. Plaintiffs challenge that conclusion.

Although this court has not discussed in detail the appropriate standard of review for a district court's ruling on test validation, we have applied the "clearly erroneous " standard. Clady v. County of Los Angeles, 770 F.2d 1421, 1434 (9th Cir. 1985). The other circuits that have addressed this issue likewise have applied the "clearly erroneous" standard. See, e.g., Melendez v. Illinois Bell Tel. Co., 79 F.3d 661, 669 (7th Cir. 1996); Bernard v. Gulf Oil Corp., 890 F.2d 735, 743 (5th Cir. 1989); Hamer v. City of Atlanta, 872 F.2d 1521, 1526 (11th Cir. 1989). The question whether a test has been validated properly is primarily a factual question, which depends on underlying factual determinations regarding the content and reliability of the validation studies that a defendant utilized. Consistent with Clady, we review for clear error the district court's determination in this case that the CBEST was validated properly.

To demonstrate that the CBEST was validated properly, Defendants are required to "show that it has `a manifest relationship to the employment in question.' " Clady, 770 F.2d at 1427 (quoting Griggs v. Duke Power Co. , 401 U.S. 424, 432 (1971)). In cases in which a scored test, like this one, is challenged, we require that the test be "job related" -that is, "that it actually measures skills, knowledge, or ability required for successful performance of the job." Contreras v. City of Los Angeles, 656 F.2d 1267, 1271 (9th Cir. 1981). In making a determination about job-relatedness, we follow a three-step approach:

The employer must first specify the particular trait or characteristic which the selection device is being used to identify or measure. The employer must then determine that the particular trait or characteristic is an important element of work behavior. Finally, the employer must demonstrate by "professionally acceptable methods" that the selection device is "predictive of or significantly correlated" with the element of work behavior identified in the second step.

Craig v. County of Los Angeles, 626 F.2d 659, 662 (9th Cir. 1980) (quoting Albemarle Paper, 422 U.S. at 431).8

We will analyze each of those three steps in turn. In addition, we will consider Plaintiffs' argument that the passing score on the writing component of the CBEST is set too high.9

1. Specific Traits or Characteristics

The first step of our inquiry is to identify the trait or characteristic that the test is designed to measure. See Craig, 626 F.2d at 662. Here, the district court found that the test was being used to measure "basic skills in reading, writing, and mathematics," AMAE II, 937 F. Supp. at 1411, and Plaintiffs do not dispute that finding.

2. Important Elements of Work Behavior

Next, we consider whether basic skills in reading, writing, and mathematics are "important element[s] of work behavior," Craig, 626 F.2d at 662, for the public school jobs for which the test is required. The district court found that the tested skills were important to the jobs at issue. See AMAE II, 937 F. Supp. at 1419. Plaintiffs challenge that finding on three grounds. First, they argue that the 1985 Practitioners' Review failed to identify any particular work behaviors or job duties and thus could not be used to assess whether the CBEST measured important elements of work behavior. Second, they argue that Lundquist's 1995 study failed to distinguish "important" skills from skills that are less important. Third, they argue that Defendants failed to demonstrate that the CBEST is job-related for the particular positions for which it is required. We address each of those arguments in turn.

Plaintiffs first argue that the 1985 Practitioners' Review, conducted by Dr. Richard Watkins, was inadequate because it failed to identify specific job duties to which the CBEST skills could be correlated. We conclude that the district court did not clearly err in finding that the 1985 study adequately identified the "element[s] of work behavior," Craig, 626 F.2d at 662, that the CBEST is designed to measure.

The district court found that the Review comprised the "pooled judgments" of knowledgeable persons, such as incumbents in the jobs, "about the relevance of the skills tested on the CBEST to the jobs for which it is required, an appropriate form of a job analysis under the professional standards of the time." AMAE II, 937 F. Supp. at 1419. Specifically, the Practitioners' Review consulted 234 teachers, administrators, and other public school employees, 36 percent of whom were members of minority groups. See id. at 1413. "The participants took part in nine review panels, in which they judged the relevance of both the skills assessed by the CBEST and the test items themselves." Id. They were asked to rate how relevant each of the CBEST skills would be to the work of four groups: (1) elementary school teachers; (2) secondary school teachers; (3) librarians, counselors, and attendance officers; and (4) school administrators. See id. The possible ratings ranged from "not relevant" to "very relevant." Id.

Thus, the Practitioners' Review was designed to learn from teachers, administrators, and other school employees the categories of skills that they considered relevant to their own jobs. The skills measured by the study tracked the categories of skills measured by the CBEST, and the skills were described in some detail on the rating forms used by the panel members. For example, the broad skill category "Mathematical concepts and relationships" was further described as follows:

Questions in this category test the understanding of basic concepts, such as the meaning of certain terms (area, for example), order among numbers, relation ships shown by graphs, elementary probability, and the like. Questions in this category may be from arithmetic, algebra, or elementary geometry.

The study's participants were guided by detailed instructions relating to each skill category and were told to rank the importance of each skill for both teaching and nonteaching jobs. The study therefore satisfies the requirement from Craig that the employer determine whether a "specific trait or characteristic is an important element of work behavior. " Craig, 626 F.2d at 662. The district court did not clearly err in concluding that the 1985 Practitioners' Review was "an appropriate form of a job analysis under the professional standards of the time." AMAE II, 937 F. Supp. at 1419.

We next consider Plaintiff's second challenge under the "important elements" prong of Craig. Plaintiffs do not challenge the 1995 Lundquist study's methodology for identifying job-related skills. They do, however, challenge that study's method for determining which skills are "important" to particular jobs.

Dr. Lundquist polled experts and interviewed and observed educators in order to develop a list of activities and skills used by educators. See AMAE II, 937 F. Supp. at 1414. She then polled 1,330 teachers and administrators, asking them to rate the importance of those activities and skills on a four-point scale from 0 ("not applicable") to 3 ("critical"). Activities and skills were retained only if "at least 80 percent of the survey respondents rated the activity or skill as applicable to the job and the mean importance rating was 1.5 or higher." Id. at 1414 (emphasis in original). Applying those standards led to elimination of a number of the activities and skills from Dr. Lundquist's list. After conducting additional studies, which are described in detail in the district court's opinion, see id. at 1415-17, Dr. Lundquist then formulated new specifications for all three sections of the test. In response to those specifications, Defendants revised the CBEST before they administered the August 1995 test.

Plaintiffs focus on the fact that Dr. Lundquist retained activities and skills on her list if they received a "mean importance rating" of 1.5 on a scale that designated 2 as "important" and 1 as "minor." By using a mean rating of 1.5, Plaintiffs argue, Dr. Lundquist retained skills and activities that were rated as "less than important" by the study's partici-pants. Therefore, their argument proceeds, the study violated the requirement from Craig that only "important" work skills be measured.

The district court rejected Plaintiffs' argument, finding that "Dr. Lundquist's decisions reflect manifestly reasonable professional judgments . . . . With respect to the 1.5 mean, as Dr. Lundquist testified at trial, a 1.5 rounds up to 2.0. It must be remembered that the mean rating of 1.5 was coupled with an 80 percent endorsement criterion, which is quite stringent." AMAE II, 937 F. Supp. at 1418 n. 35.

We agree that it is theoretically possible to imagine a circumstance that illustrates Plaintiffs' concerns on this point. For example, suppose that 80 percent of the study's participants agreed that a particular skill was relevant, but 75 percent of them rated that skill's importance as "minor." If the remaining 25 percent rated the skill as "critical," then the skill would be retained despite the fact that a majority of the study's participants rated its importance as "minor." Although that scenario is possible, such a skewed distribution of responses is unlikely. Plaintiffs present only a theoretical possibility that such "highly relevant but unimportant" skills remained on Dr. Lundquist's list. Further, as the district court noted, Dr. Lundquist conducted additional "importance" reviews of the mathematics section of the test.

Validation studies "are by their nature difficult, expensive, time consuming and rarely, if ever, free of error. " Cleghorn v. Herrington, 813 F.2d 992, 996 (9th Cir. 1987). Plaintiffs' argument demonstrates, at most, that Dr. Lundquist's study may not be totally free of error. But the argument does not persuade us that the district court clearly erred in relying on Dr. Lundquist's study.

Finally, Plaintiffs argue that Defendants failed to conduct job-specific studies to determine that the CBEST is "job related for the position[s] in question." 42 U.S.C. S 2000e2(k)(1)(A)(i). The CBEST is not intended to measure all the skills that are relevant to all the jobs for which it is required. (Indeed, it does not purport to measure all the skills of any of the jobs for which it is required.) Rather, the CBEST is intended to establish only a minimum level of competence in three areas of basic educational skills. The question is whether the validation studies in this case have satisfied the requirement that those skills be "job related" for all the positions in question. The district court found that the validation studies adequately analyzed the CBEST in terms of both the teaching and nonteaching jobs for which the test is required. See AMAE II, 937 F. Supp. at 1418-19. The district court did not clearly err in so finding.

Both the 1985 and the 1995 validation studies containedadequate consideration of the specific positions for which the CBEST is required. The 1985 Practitioners' Review defined the positions that it analyzed as (1) elementary school teachers, (2) secondary school teachers, (3) librarians, counselors, and attendance officers, and (4) school administrators. All participants in the study were asked to judge the relevance of the CBEST skills by category for those jobs. Because the study's participants were asked to determine the relevance of the basic skills measured by the CBEST to the disparate groups of positions for which the test is required, we cannot say that the district court clearly erred in finding the job analysis in the Practitioners' Review to be sufficiently specific and particularized.

The 1995 Lundquist study, as noted, identified job activities through observation, interviews, and reviews of specialized literature. See id. at 1414. Dr. Lundquist then pared her list of job skills and activities through surveys of educators and arrived at a list of "common skill requirements" that were relevant for both teachers and administrators. See id. at 141415. Her study reports:

Basic skill ratings were examined for administrators to determine if the same skill sets applied to both teacher and administrator jobs. Results showed all but one skill item (a math item) retained for teachers also applied to the administrator group. Thus, the basic skill requirements identified for teachers were found to be job-related for administrators as well, and the same test specifications may be used to test basic skills for teachers and administrators.

Dr. Lundquist's study classified jobs for which the CBEST is required as either "teacher" or "administrator" and determined that the CBEST was valid for both groups of positions. Accordingly, the 1995 study considered the validity of the CBEST across the range of jobs for which the test is required. The district court accepted the study's conclusions and found that the CBEST had been validated adequately "with respect to teaching and non-teaching jobs." Id. at 1418. On this record, that finding is not clearly erroneous.

In sum, we hold that the district court did not clearly err in finding that the skills measured by the CBEST are "important element[s] of work behavior" with regard to the jobs for which the test is required. Craig, 626 F.2d at 662.

3. Actual Measurement of Skills

The final step in this court's three-step analysis from Craigis to determine whether Defendants have demonstrated by "professionally acceptable methods that the selection device is predictive of or significantly correlated with the element of work behavior" that it is designed to measure. See id. (internal quotation marks omitted). The district court concluded that "the CBEST actually measures . . . basic skills[in reading, writing, and mathematics]." AMAE II, 937 F. Supp. at 1411. Plaintiffs claim that the district court simply accepted the "facial" validity of the CBEST without any evidence that it actually measures the basic skills that it purports to test.

This court held in Contreras that "a key requirement of [the] third step [from Craig], a requirement essential to proof of job relatedness generally, is that the validation method be professionally acceptable." 656 F.2d at 1282. Here, there is evidence in the record from an expert, Dr. William A. Mehrens, that supports the district court's findings on this issue. Dr. Mehrens reported:

ETS [Educational Testing Service] personnel wrote some of the original items and assisted the test development committees in writing other items. ETS is well known and respected as a developer of stan dardized tests. They have well trained item writers and an impressive internal set of guidelines they fol low with respect to item writing.

When asked whether "the CBEST development [was ] appropriate with respect to writing and evaluating the items," he reported:It has been. Many of the items came from an existing ETS pool. Others were written specifically for CBEST by members of the test development com mittee in concert with ETS test development special ists. The individuals on the committees worked with specialists from ETS to further develop and define the content specifications, to review an existing ETS pool test item, to write new test items, and to review the items submitted by fellow committee members. In addition, the committees studied all of the data from the field testing, made recommendations for revisions as they felt necessary, and reviewed all final test results.

There also is additional evidence in the record that the test questions were "matched" to the skills that they were intended to measure. The district court referred to the "Curriculum Matching Project, in which two ETS employees . . . matched CBEST test specifications to material found in textbooks purportedly used in the California public schools." AMAE II, 937 F. Supp. at 1412 n.21. The district court was somewhat critical of this study, but noted that "the study did support theoverall conclusion that the kinds of skills tested on the CBEST can be found in elementary and secondary school textbooks." Id.

In short, there is evidence -even if not overwhelming evidence -that the development and evaluation of the CBEST were appropriate and that the test measures the types of skills that it was designed to measure. We therefore hold that the district court did not clearly err in concluding that the test questions had been shown by professionally acceptable methods to be "predictive of or significantly correlated with the element of work behavior" that they were designed to measure. Craig, 626 F.2d at 662 (internal quotation marks omitted).

In sum, we hold that the district court did not clearly err in concluding that the CBEST was validated properly.10

4. Standards for Passing Scores

Plaintiffs also argue that the 12-out-of-16 passing score on the writing section of the CBEST is too high. According to Plaintiffs, the 1982 Wheeler and Elias study demonstrates that the proper passing score is 9 or 10 out of 16.

An employer is not required to validate separately the selection of particular passing scores on an employment test. See id. at 665. Rather, the EEOC's Guidelines more generally provide: "Where cutoff scores are used, they should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force." 29 C.F.R. S 1607.5(H). This court previously has applied that standard. See, e.g., Craig, 626 F.2d at 665. In analyzing the Guidelines' scoring requirement, the Second Circuit has stated that an employer "might establish a valid cutoff score by using a professional estimate of the requisite ability levels, or, at the very least, by analyzing the test results to locate a logical `break-point' in the distribution of scores." Guardians Ass'n of New York City Police Dep't, Inc. v. Civil Serv. Comm'n of New York, 630 F.2d 79, 105 (2d Cir. 1980).

Here, the district court found that "the passing scores on the CBEST reflect reasonable judgments about the minimum level of basic skills competence that should be required of teachers." AMAE II, 937 F. Supp. at 1420. The evidence before the court revealed that the California Superintendent of Public Instruction, who was responsible for establishing the cutoff scores, relied on polling data created as part of the Wheeler and Elias study in setting the cutoff for the writing section of the test. As part of that study, 44 readers reviewed approximately 6,800 CBEST essays and made recommendations regarding the cutoff between passing and failing scores. The readers unanimously agreed that a raw score of 12 out of 16 was a "passing" score. Approximately 80 percent of the readers agreed that a score of 11 out of 16 could be a "passing" score. On that basis, the Superintendent established a passing score of 12 out of 16, with an absolute minimum of 11 out of 16 under the "compensatory scoring" system.

Those cutoff scores represent a "logical breakpoint" between passing and failing scores. Plaintiffs argue that the breakpoint should have been set at 9 or 10 out of 16, because a majority of the readers opined that 10 out of 16 was a "passing" score. But the Superintendent was not required to set the score at the lowest level that a majority of the readers considered to be "passing." Rather, he was required to set a cutoff that was logical, reasonable, and consistent with the data before him. He chose to set the cutoff at a level that all the readers agreed was "passing," and to set an absolute minimum at a level that 80 percent of the readers thought was "passing." The district court found that the Superintendent's decision to set the cutoff score at that level was consistent with the EEOC's Guidelines. We conclude that the district court did not clearly err in so finding.

II. Title VI11

Because we have concluded that Title VII applies to the CBEST, we need not consider whether Title VI also applies. See 42 U.S.C. S 2000d. Plaintiffs proceed on a disparate impact theory under both Title VI and Title VII. Thus, our discussion of the merits under Title VII, and of the validation of the CBEST, would resolve Plaintiffs' claims on the merits under Title VI as well. Accordingly, we decline to issue an advisory opinion on the applicability of Title VI.

III. The District Court's Appointment of a Technical Advisor12

Plaintiffs also argue that the proceedings were tainted by the influence of Dr. Stephen Klein. Dr. Klein was appointed by the district court as a technical advisor, but was not called as an expert witness, was not subject to cross-examination, and did not furnish an expert's report.

In those rare cases in which outside technical expertise would be helpful to a district court, the court may appoint a technical advisor like Dr. Klein. See Reilly v. United States, 863 F.2d 149, 156 (1st Cir. 1988); see also General Elec. Co. v. Joiner, 522 U.S. 136, 149 (1997) (Breyer, J., concurring) (endorsing the appointment of special masters and specially trained law clerks to assist district courts with scientific or technical evidence). The court appointed Dr. Klein in an order that specifically identified him as a technical advisor.

Plaintiffs argue that the court committed legal error under Federal Rule of Evidence 706(a) by neither requiring Dr. Klein to submit a report nor allowing him to be cross-examined. The short answer to Plaintiffs' argument is that Rule 706 applies to court-appointed expert witnesses, but not to technical advisors like Dr. Klein. See Reilly , 863 F.2d at 155.

At one point in the proceedings, the district court told the parties that it intended to call Dr. Klein to testify near the conclusion of the trial and that the court would permit cross examination at that time. Had the court called him, Dr. Klein would have testified as an expert witness, and Federal Rule of Evidence 706 would have applied. But the court never called Dr. Klein, and there is no indication in the record that the court relied on Dr. Klein as a source of evidence. On this record, Dr. Klein acted only as a court-appointed technical advisor, and the district court did not err in refusing to allow cross-examination or to require an expert's report.

In his dissent, Judge Tashima agrees with us that district courts retain inherent authority to appoint technical advisors in appropriate cases; that this was an appropriate case for a technical advisor; that Rule 706(a) does not apply; and that we are reviewing for an abuse of discretion.

His disagreement rests on his analysis of how we should respond to the relative paucity of information in the record about Dr. Klein's interaction with the district court. In our view, the absence of any evidence even suggesting an impropriety on the part of the district court militates against a conclusion that the court abused its discretion. Although it is at least possible, as Judge Tashima suggests, that "Dr. Klein may have impermissibly influenced the court's ultimate finding," diss. op. at 13686, we instead assume that the district court did its job properly when we lack evidence to the contrary.13

Judge Tashima also proposes a list of procedures for district courts to follow when appointing technical advisors. Even assuming that those procedures are appropriate, the district court did not have the benefit of Judge Tashima's dissent before this trial, and we will not fault the court for failing to foresee his recommendations. We are not willing to find an abuse of discretion and to undo this entire trial because the district court did not follow a set of guidelines that are required nowhere in the rules or relevant case law.

IV. The District Court's Refusal to Award Costs14

Defendants appeal from the district court's order denying their cost bill in the amount of $216,443.67. Federal Rule of Civil Procedure 54(d)(1) provides that "costs other than attorneys' fees shall be allowed as of course to the prevailing party unless the court otherwise directs." By its terms, the rule creates a presumption in favor of awarding costs to a prevailing party, but vests in the district court discretion to refuse to award costs. See National Info. Servs., Inc. v. TRW, Inc., 51 F.3d 1470, 1471 (9th Cir. 1995).

That discretion is not unlimited. A district court must "specify reasons" for its refusal to award costs. Subscription Television, Inc. v. Southern Cal. Theatre Owners Ass'n, 576 F.2d 230, 234 (9th Cir. 1978). On appeal, we determine whether the reasons that the district court has specified are appropriate and whether, considering those reasons, the court abused its discretion in denying costs. See National Info. Servs., 51 F.3d at 1471-72.

In past cases, this court has approved the following reasons for refusing to award costs to a prevailing party: the losing party's limited financial resources, see National Org. for Women v. Bank of Cal., 680 F.2d 1291, 1294 (9th Cir. 1982); see also Wrighten v. Metropolitan Hosps., Inc., 726 F.2d 1346, 1358 (9th Cir. 1984); Moore v. Hughes Helicopters, Inc., 708 F.2d 475, 486 (9th Cir. 1983); and misconduct on the part of the prevailing party, see National Info. Servs., 51 F.3d at 1472. Further, in Stanley v. University of Southern California, 178 F.3d 1069, 1079-80 (9th Cir.), cert. denied, 120 S. Ct. 533 (1999), we held that the district court abused its discretion in denying a losing civil rights plaintiff's motion to re-tax costs without considering (1) the plaintiff's limited financial resources; and (2) "the chilling effect of imposing such high costs on future civil rights litigants."

Here, the district court gave four reasons for denying costs to Defendants: (1) the case "involve[s] issues of substantial public importance," specifically "educational quality, interracial disparities in economic opportunity, and access to positions of social influence"; (2) there is great economic disparity between Plaintiffs, who are individuals and "small nonprofit educational organizations," and the State of California; (3) the issues in the case are close and difficult;15 and (4) Plaintiffs' case, although unsuccessful, had some merit, as evidenced by the 1995 modification of the CBEST to eliminate "higher order" mathematics questions.

Defendants argue that the district court's reasons for denying costs were improper. According to Defendants, this court's opinion in National Information Services establishes that the only proper reason for denying costs to a prevailing party is to punish misconduct by that party. We disagree.

As noted, this court previously has held that district courts may consider other, nonpunitive reasons for denying costs to a prevailing party. National Information Services does appear to suggest that such a denial is proper only as a means of punishing a prevailing but undeserving litigant. See also Zenith Ins. Co. v. Breslaw, 108 F.3d 205, 207-08 (9th Cir. 1997) (following National Information Services). But that suggestion is inconsistent with earlier opinions of this court, see, e.g., National Org. for Women, 680 F.2d at 1294, opinions that National Information Services, as a panel's opinion, could not (and did not purport to) overrule.

More importantly, we see no basis for limiting district courts' discretion in the manner that Defendants suggest. The rule itself contains no such limitation; it provides simply that costs shall be allowed to the prevailing party unless the district court "otherwise directs." The requirement that district courts give reasons for denying costs flows logically from the presumption in favor of costs that is embodied in the text of the rule; if a district court wishes to depart from that presumption, it must explain why "so that the appellate court will be able to determine whether or not the trial court abused its discretion." Subscription Television, 576 F.2d at 234. But the limitation on district courts' discretion that Defendants advocate curtails that discretion in a manner and to a degree that are inappropriate in view of the broad wording of the rule. We now overrule National Information Services to the extent that it held that only misconduct may support the denial of costs to a prevailing party.

Federal Rule of Civil Procedure 54(d)(1) establishes that costs are to be awarded as a matter of course in the ordinary case. Our requirement that a district court give reasons for denying costs is, in essence, a requirement that the court explain why a case is not "ordinary" and why, in the circumstances, it would be inappropriate or inequitable to award costs. Misconduct on the part of the prevailing party is one factor that might render a case "extraordinary. " But it is not the only such factor. Here, the reasons that the district court gave for denying costs reflect what is clear at a glance: This is an extraordinary, and extraordinarily important, case. Plaintiffs are a group of individuals and nonprofit organizations. The record demonstrates that their resources are limited. They have brought an action that presents issues of the gravest public importance, and the action affects tens of thousands of Californians and the state's public school system as a whole. The issues in the case are close and complex. Although Plaintiffs have not prevailed in this action, their claims are not without merit. Indeed, as the district court pointed out, Defendants substantially altered the CBEST during the pendency of this litigation. Finally, costs in this case are extraordinarily high. In keeping with our decision in Stanley , we note that divesting district courts of discretion to limit or to refuse such overwhelming costs in important, close, but ultimately unsuccessful civil rights cases like this one might have the regrettable effect of discouraging potential plaintiffs from bringing such cases at all.

We do not mean to suggest that the presumption in favor of awarding costs to prevailing parties does not apply to defendants in civil rights actions. Nor are we attempting to create an exhaustive list of "good reasons" for declining to award costs. We simply hold that the reasons that the district court gave for refusing to award costs in this case were appropriate under Rule 54(d)(1) and that, considering those reasons, the court did not abuse its discretion in refusing to award costs to Defendants.

CONCLUSION

For the reasons stated, we hold that Title VII applies to the CBEST; that the CBEST was validated properly; that the district court permissibly used a technical advisor; and that the district court did not abuse its discretion by refusing to award costs to Defendants. Accordingly, we affirm both the judgment in Defendants' favor and the order denying them costs.

AFFIRMED.

Notes:

1 The CBEST is described in more detail in the district court's opinion. See Association of Mexican-American Educators v. California, 937 F. Supp. 1397 (N.D. Cal. 1996).

2 This appeal originally was heard by a three-judge panel of this court; the court later agreed to rehear the case en banc and withdrew the panel's opinion. See Association of Mexican-American Educators v. California, 195 F.3d 465 (9th Cir. 1999), withdrawn, 208 F.3d 786 (9th Cir. 2000).

3 Chief Judge Hug, and Judges Schroeder, Reinhardt, Fernandez, Rymer, and Thomas join in this part of the majority opinion.

4 Other cases following the "interference" model from Sibley include: Charlton v. Paramus Bd. of Educ., 25 F.3d 194, 202 (3d Cir. 1994); Christopher v. Stouder Mem'l Hosp., 936 F.2d 870, 876-77 (6th Cir. 1991); Pardazi v. Cullman Med. Ctr., 838 F.2d 1155, 1156 (11th Cir. 1988); and Shehadeh v. Chesapeake & Potomac Tel. Co., 595 F.2d 711, 722 (D.C. Cir. 1978).

5 To the same effect, see also Woodard v. Virginia Bd. of Bar Exam'rs, 598 F.2d 1345 (4th Cir. 1979).

6 Chief Judge Hug, and Judges O'Scannlain, Fernandez, Rymer, and Kleinfeld join in this part of the majority opinion.

7 In cases in which a defendant establishes that a test is validated properly, the burden shifts back to the plaintiff to show the existence of other selection devices that also would "serve the employer's legitimate interest in efficient and trustworthy workmanship," but that are not discriminatory. Albemarle Paper, 422 U.S. at 425 (internal quotation marks omitted). Here, Plaintiffs' challenge is limited to the validation of the test; they do not attempt on appeal to meet their burden of demonstrating the existence of preferable selection devices, assuming that the CBEST is validated.

8 Also relevant to our inquiry are the Equal Employment Opportunity Commission's ("EEOC") Uniform Guidelines on Employee Selection Procedures ("Guidelines"), which are codified at 29 C.F.R. pt. 1607. Although the Guidelines are not legally binding, they are "entitled to great deference." Albemarle Paper, 422 U.S. at 431 (internal quotation marks omitted). Failure to comply with the Guidelines, although not automatically fatal to an employment test, "diminishes the probative value of the defendants' validation study." Clady, 770 F.2d at 1430 (internal quotation marks omitted). The studies on which Defendants rely were content valid-ity studies. See 29 C.F.R. S 1607.5. Such studies establish whether the content of a test approximates the knowledge, skills, or abilities that an applicant will use on the job. See 29 C.F.R.S 1607.14. "Evidence of the validity of a test or other selection procedure by a content validity study should consist of data showing that the content of the selection procedure is representative of important aspects of performance on the job for which the candidates are to be evaluated." 29 C.F.R.S 1607.5(B).

9 Our discussion of validation owes much to the original panel majority's excellent treatment of these issues.

10 We affirm the district court's conclusion that the CBEST was validated properly based on the second and third studies: the 1985 Practitioners' Review and the 1995 Lundquist study. Because we conclude that those tests adequately support the district court's finding of job relatedness, we do not discuss in detail the 1982 Wheeler and Elias study, on which the district court also relied. All three studies reviewed the same basic version of the CBEST: the version that was given between 1983 and August 1995, when the test was revised. There is no allegation that the test changed significantly during that period or that the test was invalid until 1985 but valid thereafter. Accordingly, it is sufficient for us to conclude that the test was validated properly by the 1985 and 1995 studies and, because we so conclude, we do not decide whether the 1982 study provides an additional source of validation.

11 Chief Judge Hug, and Judges Schroeder, Reinhardt, O'Scannlain, Fernandez, Rymer, Kleinfeld, Thomas, and Gould join in this part of the majority opinion.

12 Chief Judge Hug, and Judges Schroeder, Reinhardt, O'Scannlain, Fernandez, Rymer, Kleinfeld, Thomas, and Gould join in this part of the majority opinion.

13 The present case is distinguishable from those that Judge Tashima cites, see dissenting op. at 13692-93 n.9. In all but the first and last of the cited cases, the lower tribunal failed to make findings of fact that were required by existing law. The first cited case was a death-penalty habeas corpus case in which the court remanded for an evidentiary hearing on the merits of several claims that had been dismissed, improperly, on procedural grounds. The panel's decision to remand was dictated by its application of the specific rules and procedures that govern habeas corpus in capital cases. In the last case, the district court wholly failed to address a potentially dispositive legal issue.

By contrast, here the district court had before it a nonhabeas civil case, in which it decided all issues presented and made all required findings of fact. The only question before us is whether the district court's use of a technical advisor was an abuse of discretion.

14 Chief Judge Hug, and Judges Schroeder, Reinhardt, Thomas, and Gould join in this part of the majority opinion.

15 Although we have not previously approved that reason for denying costs, other circuits have. See Teague v. Bakker , 35 F.3d 978, 997 (4th Cir. 1994); White & White, Inc. v. American Hosp. Supply Corp., 786 F.2d 728, 733 (6th Cir. 1986). Other reasons for denying costs that circuit courts have approved include: the "nominal" or partial nature of the prevailing party's recovery, see Richmond v. Southwire Co., 980 F.2d 518, 520 (8th Cir. 1992); Howell Petroleum Corp. v. Samson Resources Co., 903 F.2d 778, 783 (10th Cir. 1990); and the good faith of the losing party, see Teague, 35 F.3d at 997; White, 786 F.2d at 730. The Seventh Circuit also has suggested, in dictum, that the denial of costs might be appropriate in cases that present "landmark issues of national importance." Popeil Bros. v. Schick Elec., Inc., 516 F.2d 772, 776 (7th Cir. 1975); see also Delta Air Lines, Inc. v. Colbert, 692 F.2d 489, 490 (7th Cir. 1982). That dictum is similar to the district court's first reason for denying costs in this case, namely, the exceptional public importance of the issues presented.

REINHARDT, Circuit Judge, with whom Circuit Judges Schroeder and Thomas join, concurring in part and dissenting in part:

The result we reach is most unfortunate. Not only is it wrong as a matter of law, but it may lead some to conclude that we are insensitive to the needs and problems of minority educators and minority students. While that may not be so, one thing is certain. The loser in the case before us is clearly the already beleaguered public education system of the state of California.

Although I concur in Section IA and Parts II, III, and IV of the majority opinion, I cannot agree with the majority's holding that the CBEST was properly validated under Title VII. As a result of this ruling, qualified minority educators -teachers, administrators, librarians, and other officials -will be denied the opportunity to work in California's severely under-staffed public schools, simply because they failed to pass a test that concededly has a disparate impact on minority group members. Despite Title VII's clear requirement that a test that has a disparate impact on minorities must be validated by reference to a particular job, the majority accepts validation studies that fail to differentiate among different school jobs, ranging from bilingual education teachers to mathematics teachers to physical education teachers. Further, even though Title VII demands that the defendants demonstrate "by professionally acceptable methods" that the test items are predictive of or significantly correlated to job related skills, the majority finds that the CBEST met this requirement through a study that the district court found to be "unscientific" and "not particularly helpful."

A sampling of the plaintiff class illustrates the disservice the majority decision does not only to minority educators but also to the students in the California public school system. The plaintiff class includes a Cambodian-born applicant with a post-graduate teacher preparation certificate and a Bachelor's degree in Liberal Studies from a California State University, who is seeking to teach bilingual elementary school classes to address the needs of over 24,000 Khmer-speaking students in California public schools. He is qualified but for the fact that he is four points short of obtaining the required score on the reading section of the CBEST. The class also includes a Latino resident of Los Angeles County, in which Latinos comprise the largest ethnic group. The man seeks a position as a counselor in the public schools, and has an undergraduate degree from University of California at Irvine and a Master's degree in Counseling from California State University at Los Angeles, but has failed the reading section of the CBEST. He too is qualified, but for the test. Another class member is an African-American Assistant Principal and physical education teacher with a Bachelor's degree and a Master's degree in Physical Education from Stanford, who was promoted from his position as a teacher to Assistant Principal, and who received excellent performance reviews as an Assistant Principal. He has been denied the opportunity to return to his post as Assistant Principal because he is six points short of passing the math portion of the CBEST. The result of the failure to validate the CBEST in a lawful manner is to perpetuate discrimination against minority teachers who could be of particular assistance to minority students and serve as role models for them. It is significant in this regard that minority students outnumber white students in many of California's largest school districts, in some cases by substantial amounts.

As the majority notes, we follow the three-step approach set forth in Craig v. County of Los Angeles, 626 F.2d 659, 662 (9th Cir. 1980) to determine whether a test is job related. I believe that the district court erred in applying the second and third steps of the Craig test, because (1) the studies relied on by the CTC (Commission on Teacher Credentialing) do not differentiate among different school jobs in determining the skills to be tested; and (2) the studies do not demonstrate by professionally acceptable methods that the test items actually measure the job-related skills identified by the CTC. Either of these errors in the method of validation would be sufficient to render the CBEST invalid. Finally, although I believe that the district court's determination of job-relatedness should be reversed even under the clear error standard adopted by the majority, the proper standard for reviewing the determinations at issue in this case is de novo.

I. Failure to validate for separate jobs

The use of a test that has a disparate impact under Title VIIis impermissible unless it is proven that the test is "job related for the position in question." 42 U.S.C.S 2000e-2(k)(1)(A)(i) (emphasis added). The Uniform Guidelines provide that "[a]ny validity study should be based upon a review of information about the job for which the selection procedure is to be used." 29 C.F.R. S 1607.14(A). The CBEST is invalid because it was not validated with specific reference to the various particular jobs for which the CBEST is used.

The Supreme Court clarified the job-specific validation requirement in Albemarle Paper Co. v. Moody, holding that "[a] test may be used in jobs other than those for which it has been professionally validated only if there are no significant differences between the studied and unstudied jobs. " 422 U.S. 405, 432 (1980). In Albemarle, employment tests designed to measure general intelligence were administered to ten different job groups, which were primarily grouped together on the basis of their proximity to each other in the line of progression, rather than in terms of the specific skills that the jobs required. Id. at 429-30. The Court held the tests invalid in part because the validation study focused on higher-skilled positions on the production lines without adequately examining whether the tests were valid for workers entering lower level jobs. Id. at 433-34. Further, the Court noted that one of the tests required for the skilled lines of progression correlated to job performance in only three of the eight lines at issue. Id. at 431. Because the study had not shown that there were "no significant differences" between the various jobs this test purported to evaluate, the test was held invalid. Id. at 432.

We required a similar level of job specificity in Craig, which involved, in relevant part, a height requirement for prospective sheriffs. Craig, 626 F.2d at 666-68. Although the height requirement was justified by reference to the need for officers to "command respect," we rejected the generalization that "all police officers expend a significant portion of their time and effort in confrontation and control conditions" because "[n]o study proved the generalization to be a fact." Craig, 626 F.2d at 667. We therefore required the defendants to validate the requirement by reference to more than a general description of a police officer's functions, because different police officers have different duties. Other circuits have also rejected attempts to use a single test to measure skills for jobs that were generally similar but had appreciably different specific duties.1

The CBEST was validated without regard to the important differences among the various jobs at issue, which range from twelfth grade mathematics teachers to seventh grade bilingual education teachers to guidance counselors, librarians, and administrators. The 1982 Wheeler and Elias study encompassed skills relevant to the job of "teaching at any grade level;" the 1985 Practioner's Review study divided the "particular jobs" into four groups, specifically (1) elementary school teachers, (2) secondary school teachers, (3) pupil personnel and librarians, and (4) administrators; and the Lundquist study divided the pool into teachers and administrators. Thus, even the study that provided the most specific analysis of job categories did not differentiate among the job duties of bilingual education teachers, mathematics teachers, and physical education teachers, for example.

The majority states that the CBEST establishes only "a minimum level of competence" in skills that are job related for all the positions in question. While it may be possible to construct a test that measures skills relevant for all teachers, Albemarle holds that such a test may only be used if it has been validated for each "particular job" involved or if there are "no significant differences" among the jobs at issue. Albemarle, 422 U.S. at 432. The positions at issue in AMAE bear less resemblance to each other than those in Craig, Albemarle, or the cases from other circuits. If a validation study must differentiate between different officers in the sheriff's department or workers on different lines at the paper production factory, it is certainly improper for the CBEST to be validated by reference to undifferentiated but significantly different teaching jobs. A validation study that focuses on the "particular job" of "a teacher" or even "a secondary school teacher" fails to satisfy the legal requirement that the examination predict job performance for each particular job position in question. Teachers' jobs are too diverse to be lumped into one or two categories for validation purposes. There is no legal or logical basis to support the majority's abandonment of the job-specific validation requirement employed in Abermarle and Craig. In doing so, it clearly erred.

II. Failure to match test items to skills important to the job

As the majority notes, the third step of the Craig test requires employers to "demonstrate by `professionally acceptable methods' that the selection device is `predictive of or significantly correlated' with the element of work behavior identified [by the job analysis]." Craig, 626 F.2d at 662 (quoting Albemarle Paper, 442 U.S. at 431). In other words, the employer must demonstrate not only that the skills identified in the job analysis are job related, but also that the test items actually measure those skills. In Contreras v. City of Los Angeles, for example, this requirement was satisfied by an independent "examination review phase" in which a new group of job experts reviewed each test question and decided whether it tested one of the critical elements identified in the preceding "job analysis" phase. 656 F.2d 1267, 1282 (9th Cir. 1981); cf. Craig, 626 F.2d at 667 (finding that testimony from the sheriff and a physical training instructor that height was important in commanding respect and applying restraint was insufficient to show by professionally acceptable methods that the height requirements were predictive of the ability to carry out the "control functions" of the job). As the majority acknowledges, "a key requirement of [the] third step, a requirement essential to the proof of job relatedness, is that the validation method be professionally acceptable. " Contreras, 626 F.2d at 1282. Here, even assuming that the job analysis was sufficiently specific to generate skills important for all the positions in question, the CBEST is invalid because the CTC failed to demonstrate in a professionally acceptable manner that the test actually measures the skills identified in the job analysis.

The majority appears to concede that neither the Practitioner's Review study nor the Lundquist study attempted to match the test items to the job skills as required under Craig. Maj. op. at 13641. To support the district court's finding of validity, the majority cites evidence describing the process of test development employed by ETS. However, this evidence merely purports to show that the test items were expertly developed; it does not demonstrate, as the third step of the Craig test requires, that the final version of the CBEST administered to job applicants is "predictive of or significantly correlated" with the skills identified in the job analysis as important for the job. Where an employer seeks to use a test that has a disparate impact on minorities, it must present evidence that the test actually measures the skills that have been identified as important for the job, and may not rely solely on the fact that it used a "well known and respected as a developer of standardized tests." Maj. op. at 13640 (quoting the report of Dr. William A. Mehrens).

As evidence that the test questions were "matched " to the skills they were intended to measure, the majority points only to the Curriculum Matching Project, in which two ETS employees matched CBEST test specifications to material found in textbooks purportedly used in California public schools. The district court criticized this study as "unscientific" and "not particularly helpful," but nevertheless upheld the test. AMAE v. State of California, 937 F.Supp. 1397, 1412 n.21 (N.D. Cal. 1996). We have explicitly held that it is "essential" that the item validation method be professionally acceptable. Contreras, 195 F.3d at 490."Unscientific" and "not particularly helpful" tests clearly do not meet that standard. Thus, even under a clear error standard, the district court's determination that step three of the Craig test was satisfied requires reversal.

III. Standard of review

It is plain from a reading of the majority's opinion that it is heavily influenced by the fact that we are reviewing district court determinations that it believes to be of some complexity. Indeed, in oral argument, the state relied almost exclusively on the fact that the district court had considered the matter carefully and reached its conclusions following a long trial. This may have caused the majority to fail to examine sufficiently the legal inadequacies of the validation studies pointed out in the preceding sections of this dissent.

At each step in its analysis of job-relatedness, the majority emphasizes that the district court's determination was not clearly erroneous. However, as the majority recognizes, courts have not discussed the appropriate standard of review for a district court's ruling on test validation issues. In fact, the standard of review here should be the same as it was in Albemarle and Craig, where the Supreme Court and this court reversed a district court's determination that a test was valid based on errors in the application of the job-relatedness test similar to those committed by the district court here. At no point did either court suggest that the clearly erroneous test applied.

Although I believe that the district court's determination of job relatedness should be reversed regardless of the applicable standard of review, the proper standard is de novo. The majority fails to separate errors regarding the application of the various prongs of the Craig test, which involve primarily legal issues, from errors regarding the underlying factual evaluations of expert testimony and statistical significance. The errors here fall in the former category and are therefore subject to de novo review. See United States v. McConney, 728 F.2d 1195, 1201 (9th Cir. 1984) (en banc).2

IV. Conclusion

The defendants failed to meet their burden of demonstrating that the CBEST, with its discriminatory impact, is job related. The studies relied on by the CTC to validate the test did not differentiate among different teaching jobs, and the CTC did not demonstrate by professionally acceptable methods that the test items actually measured the skills identified in the job analysis. Title VII's requirement that tests be validated properly must be vigilantly enforced to ensure that tests that have a disparate impact on minorities are used only if they are job related for the position in question and if no alternative nondiscriminatory selection device exists. Plaintiffs' challenge to the CBEST implicates one of the most important objectives of Title VII -preventing employers from erecting arbitrary barriers which interfere with minority access to employment opportunities. Griggs v. Duke Power Co., 401 U.S. 424, 432 (1971). By erroneously affirming the district court's decision, we allow the State of California to perpetuate discrimination against qualified minority teachers, who are already seriously under represented in the California public school system, and, derivatively, against minority students as well. In the end, all Californians are the victims of the discriminatory tests the majority upholds. I respectfully dissent.

Notes:

1 See Bernard v. Gulf Oil, 841 F.2d 547, 567 (5th Cir. 1988) (remanding for specific findings because test for craftsmen positions was not validated for some positions, such as welding and carpentry, even though it was validated for others, such as pipefitting and boiler making); Walston v. County Sch. Bd., 492 F.2d 919, 926 (4th Cir. 1974) (holding, under a business necessity standard, that for the purposes of validating the use of the National Teacher Examinations a "job analysis for one teaching position (and the appropriate test for it) would not necessarily be suitable for another."); cf. Williams v. Ford Motor Co., 187 F.3d 533, 536-7, 542 (6th Cir. 1999);(holding content-validation study acceptable for unskilled workers at auto plant because workers were not hired for specific unskilled worker positions and were rotated between positions on a regular basis).

2 Clady v. County of Los Angeles , cited by the majority for the proposition that we have applied the clear error standard in cases involving test validation, is not to the contrary. In Clady , we applied the clear error standard because that case turned on subsidiary factual issues -primarily, credibility determinations of opposing witnesses -rather than the application of the Craig test to those factual findings. See Clady v. County of Los Angeles, 770 F.2d 1421, 1431 (9th Cir. 1985) (noting that the district court rejected the testimony of the appellants' expert, and stating that "[w]hen a trial judge's finding is based on his decision to credit the testimony of one or two or more witnesses, . . . that finding, if not internally inconsistent, can virtually never be clear error. " (citation omitted)).

The approach of the Fifth Circuit also does not support the majority's position. In Bernard v. Gulf Oil, 841 F.2d 547, 567 (5th Cir. 1988), the court remanded for specific findings because the district court upheld tests that were not validated for some positions, such as welding and carpentry, even though they were validated for pipefitting and boiler making. In doing so, the Fifth Circuit did not apply a clear error test. However, after the district court made findings based on expert testimony with respect to each of the particular jobs, the Fifth Circuit employed the clear error standard in reviewing the correctness of those factual findings. Bernard v. Gulf Oil, 890 F.2d 735, 743 (5th Cir. 1989). It is this second opinion that the majority cites.

FERNANDEZ, Circuit Judge, with whom RYMER, Circuit Judge, joins, Concurring in part and Dissenting1 in part:

I concur in the majority opinion with the exception of part IV, where the majority determines that a district court can refuse to award costs to the prevailing party simply because the court decides that the losing parties do not have much money, and are worthy because they have presented an important issue which is close and has some merit. As to that, I dissent.

To my mind, reasoning along that line bespeaks an improper bias in favor of plaintiffs. If the issue is important, both sides have a hand in seeing to it that the true public interest is vindicated, and, as it turns out, a defendant may well be the party correctly representing that public interest. Here we deal with the public's interest in having competent people running its schools and teaching its children, and Defendants showed that it is Plaintiffs who erred when they incorrectly thought that a competency test was overly burdensome and improper. That is to say, Plaintiffs' claim that they and those they represent should not have to meet the minimal competency requirements of the CBEST was wrong. The public interest has been vindicated, but not by them. Rather, Defendants properly represented the public's desires and aspirations for California's children. They were right, and should not be denied their costs.

Thus, I support the reasoning found in National Information Services, Inc. v. TRW, Inc., 51 F.3d 1470, 1471 (9th Cir. 1995), which recognized, as does the majority opinion, that "Rule 54(d)(1) creates a presumption in favor of awarding costs to the prevailing party." Because of that strong presumption, "[a] district court therefore generally must award costs unless the prevailing party is guilty of some fault, misconduct, or default worthy of punishment." Id. at 1472; see also Zenith Ins. Co. v. Breslaw, 108 F.3d 205, 207 (9th Cir. 1997). As we noted in National Information Services, 51 F.3d at 1471, it does not matter that the plaintiff sued in good faith and with no vexatious purpose, and even raised " `important and intricate' legal questions and included claims that were`not meritless.' " That is no reason to punish the defendant.

When a prevailing party is denied costs, that party is punished,2 but the punishment should not be exacted unless "the party has done something to deserve it." Id. at 1472. Of course, the district court should have broad equitable discretion to decide whether the prevailing party "deserves it," but that is where the discretion should generally3 stop. Therefore, all of the other reasons cited by the district court and the majority to justify denying costs to the prevailing parties in this case are improper. They, in fact, represent an abuse of discretion by the district court.

The majority opinion correctly notes that some of our prior decisions have upheld district courts' decisions to deny costs to the prevailing parties on the basis of the losing parties' limited financial conditions. See, e.g., Wrighten v. Metropolitan Hospitals, Inc., 726 F.2d 1346, 1358 (9th Cir. 1984); Moore v. Hughes Helicopters, Inc., 708 F.2d 475, 486 (9th Cir. 1983); National Organization for Women v. Bank of California, 680 F.2d 1291, 1294 (9th Cir. 1982). However, they did so without much explanation, and, unlike the majority, I would overrule those decisions instead of National Information Services.

The fact, if it is a fact,4 that Plaintiffs do not have access to much money is hardly a good reason to allow them to impose enormous costs upon Defendants, which have shown themselves to have been correct in the first place. I see nothing especially extraordinary about plaintiffs who are of modest means but still sufficiently well funded to carry out lengthy litigation. I see no reason to encourage them to cause great expense to their adversaries, safe in the knowledge that there is little downside to their efforts. Again, that is why this court has properly "recognized that denial of Rule 54(d) costs operates to punish the prevailing party for some impropriety during the course of litigation," and should otherwise be eschewed. Zenith, 108 F.3d at 207; see also Cherry v. Champion Intern. Corp., 186 F.3d 442, 447-48 (4th Cir. 1999) (holding that there should not be an exception to the presumption in favor of awarding costs to prevailing party because losing plaintiff is of "modest means," and disparity of resources is likewise an improper basis); Smith v. Southeastern Pennsylvania Transp. Authority, 47 F.3d 97, 100 (3d Cir. 1995) (holding that disparity in financial resources between the parties is not a valid basis to exempt losing party from paying costs).

Nor should it matter that Defendants are public entities. See Smith, 47 F.3d at 99 (denying costs would be unfair to taxpayers who subsidize prevailing defendant); Burroughs v. Hills, 741 F.2d 1525, 1538 (7th Cir. 1984) (Posner, J., concurring) (The plaintiffs' limited means claim "is entitled to little weight in a case such as this where the defendants are . . . public officials whose expenses of suit were borne by the United States, which is to say by the federal taxpayer. Most taxpayers are persons of limited means; most government revenue is raised from such persons, and not from the wealthy few. The denial of costs to the defendants in this case will not contribute, however slightly, to a more egalitarian distribution of the nation's wealth.").

I find it regrettable that the majority is using this case for the purpose of overruling National Information Services and enshrining a kind of plaintiffs-are-good-defendants-are-not test in its place. While the majority does not specifically say that it is taking that position, the denial of costs to Defendants surely has that effect. That is most unfortunate. See Mitchell v. City of Moore, 218 F.3d 1190, 1204 (10th Cir. 2000) (rejecting argument that "presumption in favor of costs should be relaxed when the prevailing party is the defendant in a civil rights case"); Cherry, 186 F.3d at 448 (holding that presumptive award of costs to prevailing party cannot be overcome because of "public interest" nature of the litigation); Contreras, 119 F.3d at 1295 (holding that civil rights plaintiffs do not warrant special treatment with respect to payment of costs to prevailing party); Jones, 789 F.2d at 1233 (same). In many cases, it is inequitable to require prevailing parties to bear their own attorney's fees. National Information Services' holding that unless the prevailing parties did something wrong, we should not impose the further inequity of depriving them of costs may not be all encompassingly perfect. However, it is much closer to perfect than the position we now adopt.

Thus, while I agree with most of the fine majority opinion, I must respectfully dissent from its decision regarding costs.

Notes:

1 Judges O'Scannlain and Kleinfeld concur in the portion of this opinion which dissents from part IV of the majority opinion.

2 Surely any lay person sees the winner's litigation expense as a punishing exaction. That, indeed, is a common complaint about our justice system; even if you win, you lose.

3 National Information Services used the word "generally," and while I see no real use for that loophole, perhaps there is some unforeseen additional consideration. For example, some courts have indicated that actual indigency might be a factor. See, e.g., Contreras v. City of Chicago, 119 F.3d 1286, 1295 (7th Cir. 1997); Jones v. Continental Corp., 789 F.2d 1225, 1233 (6th Cir. 1986). While I do not see why that should be so, perhaps there is no point in cost judgments against indigents, and granting costs would just waste the resources of everyone (including the courts). Thus, I, too, have used that destabilizing word.

4 There is no evidence in the record of the exact financial condition of any of the plaintiffs. The majority opinion's conclusion that Plaintiffs have limited resources seems to be inferred from evidence in the record which merely indicates what jobs are held by the individual plaintiffs and the amount of dues the organization plaintiffs receive from their members. The district court merely found a disparity of resources between the State and the Plaintiffs. Presumably that disparity would almost always exist, regardless of who litigated against the State.

KLEINFELD, Circuit Judge, with whom Circuit Judge O'Scannlain joins, concurring in part and dissenting in part.

I join in parts I(B), II, and III of Judge Graber's opinion, but not parts I(A) or IV; in part II of Judge Gould's opinion but not part I; that part of Judge Fernandez's opinion dissenting on costs but not his concurrence in part I(A) of Judge Graber's opinion. That is to say, I agree with Judge Graber that the test was adequately validated and did not violate the Civil Rights Act. And I agree that the judgment does not need to be vacated because of the district court's use of an in chambers expert (though Judge Tashima's concerns strike me as valid, and I might join in his view were I not confident that the result has to be the same regardless of the ex parte advice from the judge-selected expert). I agree with Judge Gould's dissent on whether Title VII applies. And I agree with Judge Fernandez that we should reverse the denial of costs.

I write separately for two related reasons. First, without at least a paragraph like the one above (and a scorecard to keep track of such paragraphs), readers of the opinion cannot figure out who stands where, and what the count is. So I have to write something. Second, our decision on whether Title VII applies, which generates most of our division (and the need for so incomprehensible a sentence as the one with which I begin this separate opinion), is entirely unnecessary and could better be avoided. We are not bound to avoid deciding, as we would be with a constitutional question,1 but it would be preferable for us to avoid deciding.2

The way to avoid deciding the issue of whether Title VII applies is the obvious and traditional formula:"Assuming without deciding whether Title VII applies, we conclude that it was not violated, because this test does not discriminate in violation of that law." I would have preferred that we had used that formula.

The reason why I advocate such avoidance is not the difficulty of the question. Judge Gould's analysis is compelling and clearly correct. Congress prohibited discrimination by employers with respect to employees. The defendants do not employ the plaintiffs or those whom the plaintiffs represent. So the Title VII prohibition does not speak to the situation.

Congress adopted the Civil Rights Act of 1964 after one of the great legislative battles of our time. It did not choose to impose the expense of justifying tests and other actions that affect employability on third parties other than employers.

The country suffered from massive direct and intentional race discrimination at that time. Considering the political challenge that the Civil Rights Act of 1964 posed for its advocates, and the skill and consumption of political capital it required of such great men as President Lyndon Johnson, Senator Hubert Humphrey, and Senator Everett Dirksen, we cannot assume that Congress would have gone any further than it did. Reading statutes as if they said what they do not say, in order to go further than the legislature did, vitiates careful legislative compromises. If we do not respect the compromises legislators make, how shall they be induced to make them?"If we were to `fully effectuate' what we take to be the underlying policy of the legislation, without careful attention to the qualifying words in the statute, then we would be overturning the nuanced compromise in the legislation, and substituting our own cruder, less responsive mandate for the law that was actually passed."3 Those who only got half a loaf from Congress frequently come to the federal courts for the other half, but their mail ought to be forwarded to Capitol Hill.

There are several reasons why it would be better for us to avoid deciding whether Title VII applies. One is that deciding the issue reduces the collegiality of our decision. By "collegiality" I do not mean civility or good feeling among colleagues, meanings sometimes inferred from the word. I mean the dictionary definition, "shared authority among colleagues,"4 so that we meld our individual voices into the voice of the court. An appellate court ought to speak collectively as nearly as possible, particularly when it speaks with the force of an en banc decision. In my experience, the way votes are accumulated for a collegial decision is mainly by deletion of matters not essential to the result, and on which there is disagreement.

In this case, the decision on whether Title VII applies is not essential to the result. Of the eleven judges on this en banc panel, nine agree that, whether Title VII applies or not, the test was adequately validated and would not violate Title VII. We could simply stop there. If the test is appropriate even if administered by an "employer," as the district court found and nine of us would affirm, then we need not decide whether a non-employer state board should be treated as if it were an employer for purposes of Title VII.

There are quite a few reasons why such collegiality is desirable. One is that the law is hard to figure out when a reader needs a scorecard to count votes on an issue by issue basis. It is much easier to know what the law is from Chief Justice Marshall's unanimous opinions, than from the old English Reports with individual judges' opinions reported seriatim. Careful jurists can fall into disagreement on how to keep score.5 When we do decide on an issue, despite the decision being inessential to the result, there is necessarily a question of whether our decision is mere dictum. The traditional definition of obiter dictum is a "remark by the way " or as "[s]tatements and comments in an opinion concerning some rule of law or legal proposition not necessarily involved nor essential to determination of the case in hand . . . and lack the force of an adjudication."6 Our resolution of inessential issues invites more litigation.

In this case, our reaching out to hold that Title VII applies to non-employers opens the door to more litigation, instead of shutting it. We have now created a circuit split on a national issue of great importance. As Judge Gould points out, all the other circuits to have ruled on whether Title VII (or the analogously construed ADEA provision) applies to non-employers such as state licensing boards have gone the other way. In a schools case in the Seventh Circuit,7 a veterinary board case out of the Third,8 and most closely analogous, a school teacher state certification examination case out of the Fifth,9 our sister circuits have gone the other way. (So have we, for that matter, in a dental board case.10 ) Though the majority opinion struggles to distinguish these cases, it is indeed a struggle, and the distinction is not solid enough for any state licensing board to rest on. A decision like today's, where we set up an inter-circuit conflict on a matter of great national importance, is colloquially known among those of us who do appellate law for a living as "cert bait." That is to say, we have furnished the Supreme Court with a reason to grant a petition for certiorari. As soon as we file our opinion, the highly capable lawyers involved will start calling around to arrange meetings relating to the certiorari petition and amicus briefs.

The majority opinion today says that we ought to decide the Title VII applicability issue because it is logical, but there is nothing illogical about assuming without deciding. In the study of logic, assuming without deciding is termed "conditional," "if p then q."11 Just as it is perfectly logical to say "if today is Thursday then tomorrow is Friday," without saying whether today is Thursday, it is perfectly logical to say that "if Title VII applies, it is satisfied," without saying whether Title VII applies.

The majority also says we should decide the Title VII issue even though we do not have to reach it, because the parties probably want to know the answer, particularly the State of California. Somehow I doubt that the State of California really wants to know that now it may have to spend hundreds of thousands, maybe millions, of dollars, and years of litigation, validating examinations and possibly all the other screening it does of all licensed professions. More likely, both sides just wanted to win this case and establish either that theCBEST examination was or was not permissible. These are school people. They probably care about schools, and want a final answer about the CBEST examination. We have undercut the finality of the answer by reaching out for "cert bait."There is another, and to my mind, especially important reason why we ought to avoid deciding this question on which we disagree. We may be wrong. On anything, we can be wrong. I am fairly confident that Judge Gould is right about Title VII, but I could be wrong. The scope and applicability of Title VII is an exceedingly important, and always controversial, question. Because we are not elected and always in every circumstance run a substantial risk of being wrong, the less we govern the better. That is not to say we should dodge the controversial. If the law is established and the facts show a violation, we are obligated to apply the law without regard to whether it is controversial. But we should restrain ourselves from deciding important public policy issues, when they are unnecessary to decision in the matter before us, we lack a consensus, all the other circuits to have decided them have gone the other way, and the law we establish is not clearly compelled. The people can govern themselves.

All that said, I am not going to make the score even harder to keep by withholding my judgment. We have reached the Title VII issue whether I like it or not. I cannot prevent us from reaching it. So I join Judge Gould in concluding that it does not apply because the lawsuit is not against the relevant employer.

Notes:

1 See, e.g., Ashwander v. Tennessee Valley Authority, 297 U.S. 288, 347 (1936) (Brandeis, J. concurring) (explaining the well-established principle of judicial restraint that courts should avoid declaring a statute unconstitutionally void when possible).

2 See Anderson v. U.S., 417 U.S. 211, 218 (1974) ("We think it inadvisable, however, to reach out . . . to pass on important questions of statutory construction when, simpler, and more settled, grounds are available for deciding the case at hand.").

3 Weyer v. Twentieth Century Fox Film Corp., 198 F.3d 1104, 1113 (9th Cir. 2000).

4 The American Heritage Dictionary 291 (2d college ed. 1985).

5 For example, in Ahumada-Aguilar, we were unable to agree on what the Supreme Court had established as law after the Court had decided a factually analogous case, but with four opinions and no majority opinion. U.S. v. Ahumada-Aguilar, 189 F.3d 1121 (9th Cir. 1999).

6 Black's Law Dictionary 455 (6th ed. 1990).

7 Equal Employment Opportunity Commission v. Illinois, 69 F.3d 167 (7th Cir. 1995).

8 George v. New Jersey Bd. of Veterinary Med. Exam'ers, 794 F.2d 113 (3d Cir. 1986).

9 Fields v. Hallsville Indep. Sch. Dist., 906 F.2d 1017 (5th Cir. 1990).

10 Haddock v. Board of Dental Examiners, 777 F.2d 462 (9th Cir. 1985).

11 Wesley C. Salmon, Logic 21 (1963).

GOULD, Circuit Judge, concurring in part and dissenting1 in part:

* I concur in Parts II, III, and IV of the majority's opinion, and in the judgment. I write separately, however, to dissent from Part I and to express my view to the contrary that Title VII does not apply to the CBEST. The majority extends the reach of Title VII far beyond what Congress intended and in so doing creates potential mischief for all of our states.2

II

Section 703(a)(1) of Title VII provides, in relevant part:

(a) It shall be an unlawful employment practice for an employer

(1) to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment . . . .

42 U.S.C. S 2000e-2(a)(1).

A natural reading of this section suggests that the "individual" it references is a potential, current, or past employee of the employer. That is the typical Title VII case. See, e.g., Oncale v. Sundowner Offshore Servs., Inc., 523 U.S. 75 (1998); Harris v. Forklift Sys., Inc., 510 U.S. 17 (1993); Ansonia Bd. of Educ. v. Philbrook, 479 U.S. 60 (1986); Alber marle Paper Co. v. Moody, 422 U.S. 405 (1975); Griggs v. Duke Power Co., 401 U.S. 424 (1971).

However, in a line of cases beginning with Sibley Memorial Hospital v. Wilson, 488 F.2d. 1338 (D.C. Cir. 1973), many courts -including this one -have extended Title VII liability to employers who discriminatorily interfere with an individual's employment relationship with a third party. See id. at 1341; see also Gomez v. Alexian Bros. Hosp., 698 F.2d 1019, 1021 (9th Cir. 1983). To extend Title VII's reach in this way requires not only an unnatural reading of the statute, but also a dubious inference about Congress' intent in enacting it. If Congress had intended to create liability for interference with third-party employment relationships, there is no discernable reason why it would have limited that liability to "employers," as defined in Title VII. See, e.g., Ehret v. Louisiana, 862 F. Supp. 1546, 1550 (E.D. La. 1992) (voluntary association of river pilots that interfered with third party employment relationship not subject to Sibley liability because it was not an "employer"). As the Seventh Circuit has noted:

It might be a good idea to impose liability on those who aid or abet violations of [anti-discrimination laws such as Title VII], but what sense would it make to confine that liability to persons or firms that happen to be employers? Since it would make very little sense that we can see, . . . we find it implausible to impute to Congress an intention to create, by lan guage not at all suggestive of any such intention, aider and abettor liability of one employer to the employees of another employer.

EEOC v. Illinois, 69 F.3d 167, 169 (7th Cir. 1995).3

This interpretation of the statute under Sibley and Gomez when applied to private employers at least advances the purpose of Title VII to eradicate discrimination in the workplace, and is consistent with the general directive that courts construe the statute broadly to effectuate that purpose. See Duffield v. Robertson Stephens & Co., 144 F.3d 1182, 1192 (9th Cir. 1998).

III

The same cannot be said, however, of the majority's further extension of Title VII to impose indirect liability on the State of California.4 The Supreme Court repeatedly has admonished that, absent a clear expression of congressional intent, courts must not interpret federal statutes as creating "in-roads by implication on state authority" to exercise what have historically been state police powers. Palmer v. Massachusetts, 308 U.S. 79, 84 (1939); see also Will v. Michigan Dep't of State Police, 491 U.S. 58, 65 (1989). Numerous courts have recognized that Sibley's liberal reading of the statute cannot extend Title VII liability to states that, in the course of exercising their police powers in good faith, may indirectly affect employment prospects. In Haddock v. Board of Dental Examiners, 777 F.2d 462 (9th Cir. 1985), for example, this court correctly observed that there is no evidence that when Congress amended Title VII to apply to state governments, it "intended to benefit anyone other than those actually employed by state governments or their subdivisions." Id. at 464 (emphasis added). We thus held that Title VII did not apply to the licensing activities of the California State Board of Dental Examiners. See id. Virtually all other courts to have considered Title VII's possible application to state licensing activities and to other exercises of traditional police powers have reached the same conclusion. See, e.g., Fields v. Hallsville Indep. Sch. Dist., 906 F.2d 1017, 1020 (5th Cir. 1990); George v. New Jersey Bd. of Veterinary Med. Exam'rs , 794 F.2d 113, 114 (3d Cir. 1986); United States v. New York State Dep't of Motor Vehicles, 82 F. Supp. 2d 42, 51 (E.D.N.Y. 2000); National Org. of Women v. Waterfront Comm'n of New York Harbor, 468 F. Supp. 317, 320 (S.D.N.Y. 1979); see also EEOC v. Illinois, 69 F.3d at 171 (because state regulation of teaching, even though pervasive, is exercise of police powers, Sibley liability does not apply).

The majority's holding in Part I.A strays from these cases and is inconsistent with the Supreme Court's controlling law and mandate on which they rest. It also creates a striking conflict among the circuits on an issue of importance to our states, is discordant with basic principles of federalism, and is unworkable in practice.

* The Supreme Court has consistently held that the establishment and regulation of a public school system lies at the core of a state's historic police powers; it is quintessentially a governmental, not a proprietary function. See Wisconsin v. Yoder, 406 U.S. 205, 213 (1972) ("Providing public schools ranks at the very apex of the function of a State."); Hadley v. Junior College Dist., 397 U.S. 50, 56 (1970) ("Education has traditionally been a vital governmental function."); Epperson v. Arkansas, 393 U.S. 97, 104 (1968) ("[P]ublic education in our Nation is committed to the control of state and local authorities.").

Because California regulates teaching pursuant to its police power, we should read Title VII as imposing liability on such regulation only if congressional intent to do so is clear. See United States v. Lopez, 514 U.S. 549, 580-81 (1995) ("[E]ducation is a traditional concern of the States . . . . In these circumstances, we have a particular duty to ensure that the federal-state balance is not destroyed. We start with the assumption that the historic police powers of the States are not displaced by a federal statute unless that was the clear and manifest purpose of Congress.") (Kennedy, J., concurring; citations and quotations omitted). As noted above, evidence of such an intent is far from clear. In fact, as this court recognized in Haddock, it is entirely absent. Courts have no business imputing to Congress an intent that Title VII's specific terms reach the sort of regulatory activities at issue in this case.

The majority turns away from this necessary conclusion by taking two routes to error. First, it argues that"[t]here is no overarching `licensing' exception to Title VII, " and "the state's high level of involvement in all areas of local public schools distinguishes this case" from the licencing cases cited above. This argument misperceives the issue and misapprehends applicable law. The rationale underlying the licensing cases has little to do with the extent of the state's involvement in the regulated activity, as the majority asserts, and everything to do with the principle that courts should not lightly infer a congressional intent to regulate in areas of peculiarly state concern. See, e.g., New York State Dep't of Motor Vehicles, 82 F. Supp. 2d. at 51.

Both the United States Supreme Court and the California Supreme Court recognize that the establishment and control of public schools is a peculiarly governmental function. See Lopez, 514 U.S. at 580-81; Butt v. State, 842 P.2d 1240, 1248, 1250 (Cal. 1992). That California's regulation of public schools is pervasive does not convert that activity from a governmental exercise of police power into a proprietary activity. Rather, California's pervasive regulation of public schools is the result of the state constitution's requirement that the state ensure equality of educational opportunity. See generally Butt, supra. The degree of regulation that California exerts over public schools shows that it is properly performing a governmental function in light of the state constitutional mandate that underlies that function.5

Second, on another erroneous path, the majority asserts that, because the CBEST applies only to public school employees, in imposing the test requirement California is acting in a proprietary capacity. This assertion fails to see the full picture of California's educational system. California's regulation of teacher qualifications is not confined to the public school system, but extends to all teachers. Private tutors and parents intent on home schooling must possess valid teaching credentials. Cal. Educ. Code S 48222. Similarly, private school teachers must be "capable of teaching, " and to permit the state to monitor that requirement private schools are required to file annually with the Superintendent of Public Instruction a statement listing each of their faculty members and their qualifications. Cal. Educ. Code SS 48222, 33190. Students who attend private schools that do not meet these standards are not excused from the state's compulsory education laws that require public school attendance. See id.; see also Cal. Educ. Code S 48200.

That the state has chosen to impose different qualification requirements on teachers in different settings does not mean that the CBEST is an exercise of proprietary, rather than governmental, powers. Instead, it reflects the differing circumstances and differing state interests that are implicated. For example, the California Constitution requires the state to "ensure basic educational equality" in its public schools. Butt, 842 P.2d at 1249. At the same time, under the United States Constitution, California may not require all children to attend those public schools, because "[t]he fundamental theory of liberty upon which all governments in this Union repose excludes any general power of the state to standardize its children by forcing them to accept instruction from public teachers only." Shinn v. People, 16 Cal. Rptr. 165, 172 (Ct. App. 1961) (quoting Pierce v. Society of Sisters, 268 U.S. 510, 535 (1925)). While the state may have a strong interest in ensuring uniformity of basic skills among public school teachers, it does not have the same interest in ensuring uniformity among private school teachers or private tutors. For this and perhaps other reasons, in exercising its police powers California might rationally impose different qualification requirements on teachers in these vastly different settings. See, e.g., People v. Turner, 263 P.2d 685, 688 (Cal. App. Dep't Super. Ct. 1953) (describing basis for distinction in qualification requirements for private tutors, home school teachers, and teachers in private schools). Further, even if a lack of basic skills were perceived to be a problem in all educational settings, it is well established in precedents assessing equal protection that a state may address such problems in a piecemeal fashion. See, e.g., City of New Orleans v. Dukes, 427 U.S. 297, 303-04 (1976).

B

Not only does the majority's holding miss the clear mark set by Supreme Court precedent, it also squarely conflicts with the decisions of the only two other circuits to have considered a similar issue. In Fields v. Hallsville Independent School District, 906 F.2d 1017 (5th Cir. 1990), the Fifth Circuit held that a Texas teacher skills test, much like the CBEST and applicable only to public schools,6 was merely a licensing exam and not subject to Title VII. Id. at 1020. The majority attempts to distinguish Fields by urging that in that case there was little evidence of the "control" over schools that is present here. The evidence that the Fields court was looking for, but did not find, was evidence of, for example, whether the state "played any role in the general hiring and firing of teachers" or was involved in their "daily supervision." Id. No such evidence existed there, and none exists here either. Hiring and firing of California's school teachers is performed by local school boards, as the majority concedes. And those boards are charged by statute with controlling the daily activities that teachers perform, and with evaluating their performance. See, e.g., Cal. Educ. Code SS 35020, 44660.

Similarly, in EEOC v. Illinois, the Seventh Circuit rejected the theory that Sibley liability applied to render the State of Illinois liable for discrimination in school districts' hiring and firing of teachers, despite the pervasive control that the state exerted over various aspects of the districts and their teachers, not unlike the control exerted by California upon which the majority's theory relies:

The State of Illinois exerts more control over public school teachers than over any private employees in the state and probably over any other persons for mally employed by local governments in the state. The state fixes not only a minimum salary for all teachers -which is after all not much different from fixing a minimum wage for private as well as public employees, and no one supposes that the fed eral government is the indirect employer of all the workers covered by the federal minimum-wage law -but also the number of days a teacher must work, what holidays he gets off, the amount of sick leave he is entitled to, his eligibility for and length of sab batical leave, the minimum lunch period, the terms of teachers' tenure, the rights of recalled teachers, and much else besides. 105 ILCS 5/24-1 et seq . But we do not think this makes the state the "real" employer of these teachers. So far as discrimination in hiring and firing on the basis of age or other for bidden characteristics is concerned, the key powers are, naturally, those of hiring and firing. Those pow ers are in the hands of the local school district, though constrained of course by the tenure provision of the state's school code. 105 ILCS 5/24-12. Fields v. Hallsville Independent School District, 906 F.2d 1017, 1020 (5th Cir. 1990) (per curiam), holds that a state's role in licensing public school teachers does not make them the state's employees. To similar effect, see Haddock v. Board of Dental Examiners , 777 F.2d 462, 464 (9th Cir. 1985). We think the present case is closer to Fields and Haddock than to the cases that classify the defendant as an indirect employer.

69 F.3d at 171.

C

The majority's theory that a state that extensively regulates an activity is subject to Sibley liability also is unworkable.

Where and how are we to draw a line, other than arbitrarily? States regulate a wide range of activities in ways that potentially interfere with individuals' employment opportunities with third parties. For example, California extensively regulates farm labor contractors, requiring them to have a state license, and regulating their relations both with farm laborers and the farms with whom they contract. See Cal. Labor Code S 1682 et seq. It heavily regulates the pharmacy industry, including licensing pharmacists, and regulating their authority, duties, and work hours. See Cal. Bus. & Prof. Code S 4051 et seq.; Cal. Labor Code S 850. It regulates school busing, including driver certification, whom buses may carry, and the features that buses must have. See Cal. Educ. Code S 39830 et seq. It regulates nursing and nursing education, requiring licenses for all nurses, and special licenses for public health nurses, including those employed by local, in addition to state, agencies. See Cal. Bus. & Prof. Code SS 675-79, 2700 et seq. It regulates harbors and navigation, including navigation, Cal. Harb. & Nav. Code S 240 et seq., vessels, id. S 399 et seq., masters and crews, id. S 790 et seq., and requires special licenses for pilots in the San Francisco Bay Area, id. S 1105 et seq.

Under the majority's theory, depending on the extent of the regulation in each of these areas, the State of California potentially could be liable under Title VII for the good faith exercise of its police powers to protect migrant farm workers, the public health, the safety of school children, and the environment and safety in San Francisco, San Pablo, and Suisin Bays. This result would impair federalism and is not a choice made by Congress.

The majority's opinion also calls into question this court's own decision in Haddock. The majority contends that Haddock is not inconsistent with its holding in Part I.A, arguing that in that case "licensing was the entire connection between the plaintiffs and the defendants." The State of California, however, is heavily involved in regulating the dental profession, including such diverse areas as setting licensing requirements, employment and compensation for acupuncturists in dental offices, the use of general anesthesia, and the facilities and equipment that dentists may use. See Cal. Bus. & Prof. Code S 1600 et seq. On the majority's theory, this regulation seems sufficient to render the State of California, if not the Board of Dental Examiners, liable under Title VII for any discriminatory impact in the dental licensing exam. Was Mr. Haddock's only problem that he sued the Board of Dental Examiners rather than the state?

These are only examples but illustrate that the majority's reliance on a "pervasive involvement" trigger for state Title VII liability either will extend Title VII into every aspect of a state's exercise of its police power, or will permit arbitrary application of Title VII depending on each court's interpretation and assessment of the scope of state regulatory control in varied fields. Neither result could be what Congress had in mind when it made it unlawful for "employers""to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment . . . ." 42 U.S.C. S 2000e-2(a)(1).

CONCLUSION

The majority's holding in Part I.A ignores the mandate of the Supreme Court that, absent clear evidence of a congressional intent, courts must refrain from construing federal statutes as regulating in areas where states traditionally have been sovereign in exercising their police powers. In addition, it conflicts with the only two other circuits to have considered the issue whether a state can be held liable for its school district's hiring and firing actions. The holding in Part I.A also impairs respected values of federalism because the majority applies Title VII to states' good faith exercise of their police powers. For these reasons, I respectfully dissent from that part of the majority opinion.

Notes:

1 Judges O'Scannlain and Kleinfeld join in part II, agreeing that Title VII does not apply to the CBEST.

2 If Title VI or Title VII were assumed to be applicable to the CBEST, I would also agree with the majority that the CBEST was properly validated.

3 This case arose under the Age Discrimination in Employment Act ("ADEA"), 29 U.S.C. S 623(a)(1). However, the Supreme Court has noted that because the anti-discrimination provisions of the ADEA and Title VII are identical, interpretation of one act applies with equal force to the other. See Trans World Airlines v. Thurston, 469 U.S. 111, 121 (1985); see also McKennon v. Nashville Banner Publ'g Co., 513 U.S. 352, 358 (1995) ("The ADEA and Title VII share common substantive features and also a common purpose.").

4 The State of California is not an employer of public school teachers. See Fry v. Board of Educ., 112 P.2d 229, 234 (Cal. 1941) (local school board and teacher have employer-employee relationship). Rather, individual school districts employ teachers. See Gonzalez v. California, 105 Cal. Rptr. 804, 807 (Ct. App. 1972). Thus, if Title VII applies to the state, it must apply to the state indirectly, under a Sibley-like theory.

5 That school districts are agents of the state for Eleventh Amendment purposes does not transform the state into an employer of all public school teachers, nor does it convert the state's regulation of education from a governmental to a proprietary function. First, the majority's assertion that school districts' Eleventh Amendment immunity arises from the fact that the state is "so entangled with the operation of California's local school districts," overstates the case. In Belanger v. Madera Unified School District, 963 F.2d 248 (9th Cir. 1992), we held that California's local school districts have Eleventh Amendment immunity not because of some putative overarching state entanglement, but because the districts satisfied the five-factor test summarized in Mitchell v. Los Angeles Community College District, 861 F.2d 198, 201 (9th Cir. 1988): (1) a money judgment againsta district would be satisfied out of state funds; (2) the school districts perform a central governmental function; (3) the school districts can sue and be sued in their own name; (4) the districts can take property in their own name; and (5) the districts have the corporate status of agents for school administration.

The fact remains, however, that the districts are distinct from the state in several respects that are relevant to the analysis here. For example, although the districts may be agents of the state for Eleventh Amendment purposes, the state is not vicariously liable for the torts of the districts or their teachers. See Butt, 842 P.2d at 1248. Similarly, the state is not vicariously liable for a district's breach of contract. See id. Finally, and most importantly, the state is not the employer of teachers, and districts have no authority to hire teachers on behalf of the state so as to make them state employees. See Gonzalez, 105 Cal. Rptr. at 807.

6 See State v. Project Principle, Inc., 724 S.W.2d 387, 391 (Tex. 1987); Tex. Educ. Code S 13.047 (1987).

TASHIMA, Circuit Judge, dissenting:

Among the issues raised by plaintiffs on appeal are the procedural errors ("procedural[ ] taint[ ]") which "result[ed in] the unreported influence of Dr. Klein." Plaintiffs explain their contention as follows:

We do not know the extent of Dr. Klein's ex parte communications with the court, nor the influence these communications had with the court, nor the extent to which [his] views and unfamiliarity with employment testing standards could have been tem pered by the heat of cross-examination. We know none of this because the district court did not require any report of the "advice" he received from Dr. Klein and, most importantly, denied Plaintiffs their right to cross-examine the court's expert.

Appellants' Opening Brief at 51. Because I cannot agree with the majority's cursory and mistaken treatment of the technical advisor issue in Part III, I respectfully dissent.

I.

Courts and jurists have long recognized that the "law should in some way effectively use expert knowledge wherever it will aid in settling disputes." Learned Hand, Historical and Practical Considerations Regarding Expert Testimony, 15 Harv. L. Rev. 40 (1901). "The only question is as to how it can do so best." Id. I agree with the majority that the district court retains the inherent authority to appoint a technical advisor in especially complex cases. Maj. op. at 13645. As Justice Brandeis, speaking for the Court, noted more than 80 years ago:

Courts have (at least in the absence of legislation to the contrary) inherent power to provide them selves with appropriate instruments required for the performance of their duties. This power includes authority to appoint persons unconnected with the court to aid judges in the performance of specific judicial duties, as they may arise in the progress of a cause.

Ex Parte Peterson, 253 U.S. 300 (1920) (approving appointment of an auditor and report to be made by him). In exercising that power, however, the district court must observe certain, minimal, procedural safeguards in order to insure that the advisor is unbiased and impartial, and that his participation is properly confined. Here, at the district court's invitation, a question of the advisor's bias and partiality was raised, but not resolved. The record is also completely devoid of any evidence of what role Dr. Klein played in the proceedings, the scope and basis of his influence, what materials or knowledge he may have called on in bringing that influence to bear, and what part his knowledge, expertise, and advice played in the district court's ultimate finding that CBEST is a valid test under Title VII. I would therefore vacate the judgment and remand to the district court for it to make findings on the bias and partiality issue, and to explain Dr. Klein's role, in accordance with the guidelines set forth below.

Even though we have never held explicitly that district courts may appoint technical advisors outside the confines of Federal Rule of Evidence 706,1 courts and commentators agree that a trial court's inherent authority to appoint a technical advisor has not been displaced by the federal rules. See Reilly v. United States, 863 F.2d 149, 158 (1st Cir. 1988); Note, Improving Judicial Gate keeping: Technical Advisors and Scientific Evidence, 110 Harv. L. Rev. 941, 949-50 (1997) (hereinafter "Technical Advisors"); cf. Hall v. Baxter Healthcare Corp., 947 F. Supp. 1387, 1392 & n.8 (D. Or. 1996) (appointing technical advisor under "inherent authority" of Fed. R. Evid. 104). Although the use of a technical advisor should be "the exception and not the rule," expert advice can be quite useful in a trial that involves unusually complex issues in fields "beyond the regular questions of fact and law with which judges must routinely grapple. " Reilly, 863 F.3d at 156-57. Moreover, because the district court is in a better position to decide if the appointment of a technical advisor is warranted, we should defer to its discretion with respect to whether or not to appoint one. See id. (reviewing appointment of technical advisor for an abuse of discretion).

The majority correctly concludes that this is one of those "rare cases" in which the appointment of a technical advisor under the inherent authority of the court is justified. Maj. op. at 13644; see Reilly, 863 F.2d at 156. As the original panel opinion pointed out, this case "involves the highly technical field of psychometrics, and presents problems of unusual complexity beyond the normal questions . . . with which judges routinely grapple." Association of Mexican American Educators v. California, 195 F.3d 465, 492-93 (9th Cir. 1999), reh'g en banc granted, 208 F.3d 786 (9th Cir. 2000).

II.

After three years of discovery, the district court found that it needed guidance on a host of complex, factual issues. As the district court noted in its order appointing technical advisor, "the Court comes face to face with many prickly problems requiring expertise in the esoteric fields of education and psychometrics including knowledge of theories about educational measurement and testing, cognitive psychology, statistics, and other fields pertaining to the CBEST and other cases." It then appointed Dr. Stephen P. Klein, a highly respected test validation expert, as its "Technical Advisor." Although I agree with the majority that the district court did not abuse its discretion in deciding to appoint a technical advisor, that is only the beginning of our inquiry, and not its end, as the majority seems to think.

The order appointing the technical advisor provides that he is to review all of the "direct testimony of the experts heretofore furnished to the Court,"2 briefs, selected pleadings, and depositions. Dr. Klein was also to be present in court during expert testimony, "take such notes as he deems proper and confer ex parte with the Court from time to time." It also noted that "[h]is role will be similar to that of a judicial clerk and, therefore, he will not be available for communication with or questioning by the parties." The order nowhere stated, however, what Dr. Klein was to do with all of this evidence and other information he was directed to absorb, except to "confer ex parte with the Court from time to time."

The use of a technical advisor is not without risks. First, whenever a court appoints a technical advisor, there is a danger that the court will rely too heavily on the expert's advice, thus compromising its role as an independent decisionmaker and the requirement that its findings be based only on evidence in the record. This risk is especially salient if the contents of the communications between the trial judge and the advisor is hidden from the parties (and appellate review), and where the parties have no opportunity to respond to the advisor's statements. See Technical Advisors, 110 Harv. L. Rev. at 957. Second, experts in the relevant field, particularly if it is a narrow and highly-specialized one, may be aligned with one of the parties; therefore, the district court must make every effort to ensure the technical advisor's neutrality, lest the advisor develop into, or give the appearance of being, an advocate for one side. Without some safeguards, the parties' confidence in the fairness of the trial will erode.

Thus, once the district court has appointed a technical advisor, its discretion concerning the advisor's role in the trial should not be unfettered. We have never had occasion to give definitive guidance to district courts on the use of technical advisors and this case clearly demonstrates why we should.3 Simply put, we are utterly unable, on this record, to review the propriety of Dr. Klein's advice to the district court because we have no idea what role he played in the district court's fact finding process. As indicated above, the district court laid out Dr. Klein's evidence-gathering duties in its appointing order, but that order does not state what Dr. Klein was to do with that store of knowledge. There is no evidence in the record that would provide any hint about the nature, scope, or substance of Dr. Klein's advice. We therefore do not know whether and, if so, to what extent, Dr. Klein may have impermissibly influenced the court's ultimate finding that CBEST is a valid test under Title VII: Did he draw upon his own expertise and knowledge of test validation to discount or contradict the opinion of a testifying expert on either side? Did he explain to the court why CBEST was valid, based either on his own opinion and/or on information (such as the scientific literature in the field, including his own writings) not in the record? Or was he asked only to review the court's proposed findings for technical accuracy? Moreover, plaintiffs have alleged that Dr. Klein was biased in favor of defendants and the district court did little to allay plaintiffs' concerns.

III.

Although we need not require strict adherence to any specific procedures, I would hold that a district court minimally must: (1) utilize a fair and open procedure for appointing a neutral technical advisor; (2) address any allegations of bias, partiality, or lack of qualification; (3) clearly define and limit the technical advisor's duties;4 (4) make clear to the technical advisor that any advice he gives to the court cannot be based on any extra-record information; and (5) make explicit, either through an expert's report or a record of ex parte communications, the nature and content of the technical advisor's advice. By adopting these minimal safeguards, the parties can be assured that the technical advisor appointment process is fair and that the technical advisor does not exercise undue influence on the district court. These procedures will also enable meaningful appellate review.

A. Neutrality of the Technical Advisor

The district court, of course, should have wide latitude when choosing a technical advisor, who, after all, is retained to aid the court's understanding of the difficult and complex technical issues presented in the case. The parties, however, should be given an opportunity to object to any proposed appointment based on bias, partiality, or lack of qualification. See Reilly, 863 F.2d at 159; Technical Advisors, 110 Harv. L. Rev. at 954-55; cf. Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 592 & n.10 (1993) (noting that, with respect to proffered scientific evidence, the trial judge must determine "the qualification of a person to be a witness") (quoting Fed. R. Evid. 104(a)). While the district court is not required to reject the appointment of a technical advisor merely because one side or the other objects, it must adequately address any concerns of bias, partiality, or lack of qualification that the parties raise.

Here, the district court properly allowed both sides an opportunity to review Dr. Klein's curriculum vitae and to submit written interrogatories to him concerning his experience and background.5 Plaintiffs objected to Dr. Klein's appointment on the grounds that: (1) he had worked closely with the California Commission on Teacher Credentialing, one of the defendants in this case; (2) his private practice focuses on supporting certification exams; (3) he has taken positions on issues raised in the case; and (4) he had worked closely with one of defendants' key experts, Dr. Mehrens.

We are unable to discern from the record whether any of these allegations are well-founded, but it is clear that the district court did not adequately address these objections. After plaintiffs objected to the appointment, the district court overruled the objection, stating that plaintiffs would have the "opportunity to examine him fully with respect to any views that he may have." This response to the objection might have been adequate, had the district court actually given plaintiffs the promised opportunity. As it turned out, however, the court's response to plaintiffs concerns was wholly inadequate because the district court later decided that it would not have Dr. Klein testify. There is nothing in the record to indicate that the district court seriously considered plaintiffs' objections,6 or that it took any other measure to insure Dr. Klein's neutrality, such as making its own independent inquiry. This omission is all the more troubling because the objections, if well founded, might well provide valid bases for disqualifying Dr. Klein from serving as a neutral technical advisor.

To avoid this situation in the future, I would require, absent mutual agreement by the parties on an acceptable technical advisor, that the district court meaningfully address, on the record, any colorable objection to the appointment of a specific technical advisor. Because the district court failed adequately to address plaintiffs' objections to Dr. Klein's appointment, I would remand to the district court for this determination to be made on the record.

B. The Technical Advisor's Role

The role of a technical advisor also should be carefully defined and limited to reduce the risk that the advisor will usurp the role of the court as fact finder, see Technical Advisors, 110 Harv. L. Rev. at 955, or of even giving that appearance. To this end, the district court should lay out the specific duties of the technical advisor, keeping in mind that the advisor's role should be that of "sounding board" and tutor who aids the court in understanding the "jargon and theory" relevant to the technical aspects of the evidence. Reilly, 863 F.2d at 158; see Technical Advisors, 110 Harv. L. Rev. at 955-56. This "job description" should make explicit that the technical advisor is not to contribute evidence (unless called as a witness) or to be an advocate on behalf of either party. Id., 110 Harv. L. Rev. at 955; see Reilly, 863 F.2d at 157. If the parties, the court, and the advisor all are clear on the advisor's role, there will be less likelihood that the advisor will overstep his bounds or that the parties may be led to believe that he has. Moreover, a "job description" that highlights those areas in which the district court needs aid or clarification may help to focus the parties' attention on issues that are important to the court and, therefore, aid the parties in addressing those concerns directly. See Technical Advisors, 110 Harv. L. Rev. at 955.

When the district court appointed Dr. Klein, as noted above, it ordered that he was "to review all the direct testimony of the experts, . . . the parties' briefs and proposed findings of fact and conclusions of law, and other such documents . . . as seems appropriate." It also explained that Dr. Klein should listen to the expert testimony, take notes, and confer with the court ex parte when necessary. The district court, however, failed to complete the list of Dr. Klein's critical duties. It did not describe the nature or scope of its consultations with Dr. Klein, e.g., whether his role was limited to reviewing the court's proposed findings for technical accuracy, whether he was to evaluate the opinions of the testifying experts, or whether he was to give the court his own, independent, expert opinion.

In the future, district courts should be more specific concerning the technical advisor's precise role. For example, the court could explain which technical concepts it finds troubling, and limit the expert's role to helping it comprehend those technical issues. See Hall v. Baxter Healthcare Corp., 947 F. Supp. at 1393-94 (listing specific questions submitted to the technical advisors). The district court may even find it helpful to invite the parties to participate in defining the expert's duties to reduce the risk of perceived unfairness and to help focus the issues. See Technical Advisors , 110 Harv. L. Rev. at 955.

C. Making a Record

Finally, the nature and scope of the advice provided to the district court by the technical advisor should be recorded in some manner. While I would not prescribe a mandatory procedure for making that record, such as a reporter's transcript of all ex parte conferences between the court and the technical advisor, obviously, some documentation is required -such as a report by the advisor, a summary of the advice given, or the court's statement on the record -of the court's interaction with the technical advisor.7

The district court, as well as other courts, likened the duties of a technical advisor to those of a law clerk and, thus, reasoned that the communications between the court and advisor may be kept confidential, i.e., concealed from the parties and the appellate court. See Reilly, 863 F.2d at 158. While there are some similarities between a law clerk and a technical advisor, that analogy is limited.8

In some important respects, a technical advisor is quite unlike a law clerk. A law clerk's function is to aid the judge in researching legal issues in cases pending before the court. Because the judge is an expert in the law and fully understands legal theory and analyses, it is unlikely, to say the least, that a law clerk will impermissibly usurp the judicial function. On the other hand, a technical advisor is brought in precisely because the judge is not familiar with the complex, technical issues presented in the case. See id. at 157 ("appointment of a technical advisor must arise out of some cognizable judicial need for specialized skills"). There is therefore an understandable concern that the technical advisor's opinion will carry undue weight with the judge. Cf. Daubert, 509 U.S. at 595 (noting that "[e]xpert evidence can be both powerful and quite misleading because of the difficulty in evaluating it") (internal quotations and citation omitted). In short, a judge can filter out "bad" legal advice or research from a law clerk; he or she is ill-equipped, however, to do the same with "bad" technical advice. Moreover, resolution of legal issues is committed to the judge qua judge and is subject to de novo review. On the other hand, factual issues, no matter how technical, are committed to the fact finder and, to be reviewed properly, must be based on the record made in the trial court.

Whatever method the court employs in making a record of the technical advisor's advice and counsel, there is no reason why a neutral technical advisor's advice should be shrouded in absolute secrecy. Concealing the nature of that expert advice can only erode confidence in the court's role as a neutral and independent decision maker.

IV.

These procedural safeguards are not especially burdensome and should not be difficult to apply. They are, however, essential in order to insure that the parties have confidence that the technical advisor is unbiased, impartial, and qualified, and that he has not unduly influenced the proceedings. Such minimal procedural safeguards are also required so that any challenge to the appointment of a technical advisor or to the degree of his participation in the case can meaningfully be reviewed on appeal. Here, other than a short order appointing Dr. Klein, the district court provided no explanation of his role. It also failed adequately to address the parties' written objections to Dr. Klein's neutrality.9

For the foregoing reasons, I would vacate the judgment and remand this case to the district court for its resolution of these issues and for such further proceedings as may thereafter be necessary.10

Notes:

1 I agree with the majority that Rule 706(a) has no application here because that provision only applies to testifying expert witnesses, and the district court did not call Dr. Klein to testify. The shortcoming in regard to Rule 706 was in the court's leading the parties to believe right up to the end of the trial that Dr. Klein would be called and then not calling him. The errors that the court did commit might well have been remedied had Dr. Klein been called as a witness and subjected to cross-examination.

2 As is not uncommon in bench trials, the district court required the direct testimony of expert witnesses to be submitted in written form and furnished to the court before the commencement of trial.

3 It is thus unfortunate that the majority gives the issue such cursory treatment.

4 To state conclusorily that the technical advisor's "role will be similar to that of a judicial law clerk" is insufficient. See discussion in Part III.C, infra.

5 Although it struck some of plaintiffs' interrogatories and refused to require Dr. Klein to respond to others designed to probe Dr. Klein's partiality.

6 Interestingly, defendants also expressed some concern that Dr. Klein's writings indicated a "strong predisposition for teacher assessment alternatives which are neither economically viable nor realistically feasible in California." There is also no indication that the district court addressed this concern.

7 In Reilly, the court suggested that a technical advisor should file an affidavit attesting to his compliance with the job description after his services have been rendered. See 863 F.2d at 159-60. This is one of a number of ways that a district court could ensure that the technical advisor does not exert any inappropriate influence on the outcome of the trial. See also Hall v. Baxter Healthcare Corp., 947 F. Supp. at 1393-94 (setting forth in detail the technical advisors' participation in the case, including the specific questions they were asked to answer); id. at 1394 & n.17, 1435-76 (setting forth as appendices the technical advisors' reports to the court).

8 I do agree that the unique role played by a technical advisor justifies ex parte communications between the judge and the advisor, subject to the limitations explained above.

9 The majority acknowledges that there is a "relative paucity of information in the record about Dr. Klein's interaction with the district court." Maj. op. at 13645. In spite of this deficiency in the record, it is willing to "assume that the district court did its job properly when we lack evidence to the contrary." Id. at 13645. That is not the law. The law is that we require a sufficient record and findings so that we can conduct a meaningful review. When the record is incomplete through no fault of the appellant, as is the case here, we should remand to permit a record and findings to be made so that we are able properly to review the issue. See, e.g., Morris v. Woodford, 229 (9th Cir. Oct. 5, 2000) ("We also emphasize that a more developed factual record with regard to those claims is necessary for meaningful appellate review.") (Graber, J.); Merrill v. Apfel, 224 F.3d 1083, 224 (9th Cir. 2000) (remanding to the Commissioner of Social Security where record lacked finding on whether claimant met disability requirement); United States v. Parilla, 114 F.3d 124, 125 (9th Cir. 1997) ("because the district court . . . made no specific factual findings regarding the evidence of sentencing entrapment, we vacate the defendant's sentence and remand for further proceedings"); United States v. Del Muro, 87 F.3d 1078, 1082 (9th Cir. 1996) (vacating and remanding to make findings under Fed. R. Crim. P. 32(c)(3)(D)); United States v. Robinson, 63 F.3d 889, 891-92 (9th Cir. 1995) (noting need for "more complete and specific findings"); United States v. Naranjo, 52 F.3d 245, 251 (9th Cir. 1995) (vacating and remanding because"[i]n the absence of specific findings on the record, we are uncertain as to what findings the district court relied on"); Carter v. Smith Food King, 765 F.2d 916, 924 (9th Cir. 1985) (remanding for the district court to address question "[t]he district court did not consider").

10 This remand does not, contrary to the majority's assumption, "undo this entire trial." Maj. op. at 13646. Whether that result eventually follows depends entirely on what develops on the limited remand that I propose. If it develops on remand that the technical advisor was partial and unduly influenced the court, why would that not amount to an abuse of discretion requiring a new trial? The majority avoids answering this question.

I would not, at this time, reach any of the other issues tendered on appeal on which I express no opinion. The en banc court should, however, retain initial jurisdiction over any further appeals after proceedings on remand have been completed in the district court to address any remaining issues.