Concrete Works of Colorado, Inc. v. City & County of Denver

321 F.3d 950, 2003 U.S. App. LEXIS 2396, 84 Empl. Prac. Dec. (CCH) 41,525, 2003 WL 294582

• Court: Court of Appeals for the Tenth Circuit
• Decided: February 10, 2003
• Docket: 00-1145
• Status: Published

Opinion

MURPHY, Circuit Judge.

I. INTRODUCTION

Plaintiff, Concrete Works of Colorado, Inc. (“CWC”) initiated this action in 1992, challenging the constitutionality of an affirmative action ordinance enacted by the City and County of Denver (hereinafter the “City” or “Denver”). The ordinance established participation goals for racial minorities and women on certain City construction and professional design projects. Denver has amended the ordinance twice since this lawsuit was initiated but it remains essentially unchanged for purposes of this case.

In 1993, the district court granted summary judgment in favor of Denver. See Concrete Works of Colo., Inc. v. City & County of Denver, 823 F.Supp. 821, 845 (D.Colo.1993) (“Concrete Works /”). After CWC appealed, this court reversed the grant of summary judgment and remanded the case for trial. See Concrete Works of Colo., Inc. v. City & County of Denver, 36 F.3d 1513, 1530-31 (10th Cir.1994) (“Concrete Works II”). On remand, a bench trial was held and the district court entered judgment in favor of CWC on its claims for injunctive and declaratory relief. See Concrete Works of Colo., Inc. v. City & County of Denver, 86 F.Supp.2d 1042, 1079 (D.Colo.2000) (“Concrete Works III”). The district court enjoined Denver from enforcing the ordinance. See id. CWC’s entitlement to damages was reserved and the district court directed entry of judgment under Rule 54(b) of the Federal Rules of Civil Procedure. See id. at 1044, 1079. Denver then brought this appeal.

Section II. of this opinion summarizes the challenged Denver ordinances and this litigation to date. Section III. generally presents the applicable law and the parties’ positions thereon. In Section IV., we examine the evidence upon which Denver relies to support the ordinances. At trial, Denver presented historical evidence, statistical evidence, and anecdotal evidence which are discussed respectively in subsections IV.A, IV.B., and IV.C. In Section V, we discuss the legal framework used by the district court to evaluate Denver’s evidence. Section VI. contains a discussion of CWC’s rebuttal evidence, including its challenges to Denver’s use of marketplace data and to the reliability of Denver’s disparity studies. In Section VIL, we address the question of narrow tailoring.

Exercising jurisdiction pursuant to 28 U.S.C. §§ 1291, 1292(a)(1), 1 we reverse and remand.

II. BACKGROUND

Comprehensive statements of the relevant facts and prior proceedings in this case are fully set out in our previous opinion and in the two district court orders. See Concrete Works II, 36 F.3d at 1515-17; Concrete Works III, 86 F.Supp.2d at 1043-46; Concrete Works I, 823 F.Supp. at 824-26. In particular, the most recent order of the district court contains a thorough description of the numerous studies commissioned by the City and the operation of the ordinances. See Concrete Works III, 86 F.Supp.2d at 1053-61, 1050-53. Consequently we have summarized only the most relevant facts.

The appellate record in this case is prodigious. The trial transcript, including opening and closing statements, exceeds three thousand pages and the entire appellate appendix exceeds ten thousand pages. Consequently, the parties’ ability to cogently frame the issues and provide record support for them respective positions was particularly critical. Though we are under no obligation to do so, this court made every reasonable effort to locate pertinent portions of the record when a citation to such was omitted by the parties.

A. The Ordinances

In 1990, the City promulgated an affirmative action program codified as Ordinance No. 513 (the “1990 Ordinance”). The 1990 Ordinance established the May- or’s Office of Contract Compliance (“MOCC”) and delegated to that office “authority to promulgate such rules and regulations and/or informal guidelines as may be necessary to effectuate the purposes” of the 1990 Ordinance. Subject to exemptions adopted by the Manager of Public Works, the 1990 Ordinance applied to all contracts for which bidding was required before the City could make an award.

The 1990 Ordinance contained annual goals for the utilization of minority business enterprises (“MBEs”) and women business enterprises (“WBEs”). Of the total dollars spent annually for construction contracts with the City, the goal was 16% to MBEs and 12% to WBEs. The annual goals for professional design and construction services were 10% of annual expenditures to MBEs and 10% to WBEs. MBEs were defined in the 1990 Ordinance as businesses: (1) at least 51% owned by one or more eligible minorities and (2) with daily business operations controlled by one or more eligible minorities. Minorities were defined as persons of “Black, Hispanic, Asian-American, or American Indian descent.” WBEs were defined as businesses: (1) at least 51% owned by one or more women and (2) with daily business operations controlled by one or more women. To participate in the program, both MBEs and WBEs were required to obtain certification from the City. Notwithstanding the requirement to meet annual goals, the MOCC could set individualized M/WBE participation goals for specific City construction and professional design projects. On some projects, goals were set at zero.

Prime contractors and subcontractors who bid on City contracts were required to commit to the goals and requirements set forth in the 1990 Ordinance. Bidders could comply with the 1990 Ordinance by meeting the project participation goals or by demonstrating sufficient good faith efforts to meet those goals. Bidders could meet the good-faith requirements by demonstrating that they sought to subcontract with M/WBEs but were unsuccessful or that they rejected a certified MBE or WBE because it did not submit the lowest bid or was not qualified. If the bidder failed to meet either the good-faith or project-participation requirements, the bid was considered “not responsive.”

In 1996, the City replaced the 1990 Ordinance with Ordinance No. 304 (the “1996 Ordinance”). The district court stated that the 1996 Ordinance differed from the 1990 Ordinance as follows:

The 1996 Ordinance ... amended the 1990 Ordinance by expanding the definition of “covered contracts” to include limited categories of privately financed projects on City-owned land; added updated information and findings to the statement of factual support for continuing the program; refined the requirements for W/MBE certification and graduation; mandated the use of MBEs and WBEs on change orders and expanded sanctions for improper behavior by MBEs, WBEs or majority owned contractors in failing to perform the affirmative action commitments made on City projects. It changed the definition of American-Indian descent by eliminating the provision for 25% blood quantum alternative to tribal membership.

Concrete Works III, 86 F.Supp.2d at 1049. The 1996 Ordinance was amended in 1998 by Ordinance No. 948 (the “1998 Ordinance”). Annual participation goals for both MBEs and WBEs were reduced to 10% of total dollars spent on construction projects and 10% of total dollars spent on professional design contracts. A substantive change prohibited an MBE or a WBE, acting as a bidder, from counting self-performed work toward project goals.

B. The Lawsuit

In January 1992, CWC filed a complaint in federal district court alleging that the 1990 Ordinance violated the Equal Protection Clause of the Fourteenth Amendment. CWC is a Colorado construction firm owned and operated by a non-minority male. CWC alleged that it lost three contracts with the City because it failed to comply with the M/WBE participation goals or meet the good-faith requirements set out in the 1990 Ordinance. CWC sought both damages and injunctive relief. After a motion for summary judgment filed by the City was granted by the district court, CWC appealed. This court reversed, concluding that genuine issues of material fact existed, making the grant of summary judgment inappropriate. See Concrete Works II, 36 F.3d at 1530-31. The case was remanded for trial. 2 Id. at 1531.

The district court conducted a bench trial in February and June 1999 on the constitutionality of the three ordinances. The court ruled in favor of CWC and concluded that the ordinances violated the Fourteenth Amendment. See Concrete Works III, 86 F.Supp.2d at 1079. The court enjoined the City from enforcing the 1990 Ordinance, the 1996 Ordinance, and the 1998 Ordinance. See id. The court reserved ruling on CWC’s claim for damages. See id. The City then brought this appeal.

III. STANDARD OF REVIEW AND BURDENS OF PROOF

A. Race-Based Remedial Measures

CWC argues that the ordinances violate the Equal Protection Clause of the Fourteenth Amendment which provides that “[n]o State shall ... deny to any person within its jurisdiction the equal protection of the laws.” U.S. Const, amend. XIV, § 1. Because the use of racial preferences is a “highly suspect tool,” the race-based measures contained in the ordinances are subject to strict judicial scrutiny. City of Richmond v. J. A. Croson Co., 488 U.S. 469, 493, 109 S.Ct. 706, 102 L.Ed.2d 854 (1989) (plurality opinion) (“[T]he purpose of strict scrutiny is to ‘smoke out’ illegitimate uses of race ... [and] ensure[ ] that the means chosen ‘fit’ [the] compelling goal so closely that there is little or no possibility that the motive for the classification was illegitimate racial prejudice or stereotype.”). To withstand CWC’s challenge, the race-based measures in the ordinances must serve a compelling governmental interest and must be narrowly tailored to further that interest. See Adarand Constructors, Inc. v. Pena, 515 U.S. 200, 227, 115 S.Ct. 2097, 132 L.Ed.2d 158 (1995) (“Adarand III ”); Wygant v. Jackson Bd. of Educ., 476 U.S. 267, 274, 106 S.Ct. 1842, 90 L.Ed.2d 260 (1986) (plurality opinion).

Denver asserts that it has a compelling interest in remedying racial discrimination within its jurisdiction. A clear majority of the Supreme Court has expressly held that “[a] State’s interest in remedying the effects of past or present racial discrimination may in the proper case justify a government’s use of racial distinctions.” Shaw v. Hunt, 517 U.S. 899, 909, 116 S.Ct. 1894, 135 L.Ed.2d 207 (1996). A plurality of the Court has also stated that a governmental entity “can use its spending powers to remedy private discrimination, if it identifies that discrimination with the particularity required by the Fourteenth Amendment.” Croson, 488 U.S. at 492, 109 S.Ct. 706 (plurality opinion). Because “an effort to alleviate the effects of societal discrimination is not a compelling interest,” Denver can demonstrate that its interest is compelling only if it satisfies two conditions. Shaw, 517 U.S. at 909-10, 116 S.Ct. 1894 (emphasis added). First, it must identify the past or present discrimination “with some specificity.” Id. at 909, 116 S.Ct. 1894 (quotation omitted). Second, it must also demonstrate that a “strong basis in evidence” supports its conclusion that remedial action is necessary. Id. at 910, 116 S.Ct. 1894 (quotation omitted).

Denver can meet its burden without conclusively proving the existence of past or present racial discrimination. See Concrete Works II, 36 F.3d at 1522 (“[T]he Fourteenth Amendment does not require a court to make an ultimate judicial finding of discrimination before a municipality may take affirmative steps to eradicate discrimination.”). Denver may rely on “empirical evidence that demonstrates ‘a significant statistical disparity between the number of qualified minority contractors ... and the number of such contractors actually engaged by the locality or the locality’s prime contractors.’ ” Id. (quoting Croson, 488 U.S. at 509, 109 S.Ct. 706 (plurality opinion)). Furthermore, Denver may rely on statistical evidence gathered from the six-county Denver Metropolitan Statistical Area (MSA). See id. at 1520. Denver may supplement the statistical evidence with anecdotal evidence of public and private discrimination. See id. at 1520-21; see also id. at 1520 (“Personal accounts of actual discrimination or the effects of discriminatory practices may, however, vividly complement empirical evidence.”).

“Neither Croson nor its progeny clearly state whether private discrimination that is in no way funded with public tax dollars can, by itself, provide the requisite strong basis in evidence necessary to justify a municipality’s affirmative action program.” Id. at 1529. Denver, however, clearly may take measures to remedy its own discrimination or even to prevent itself from acting as a “passive participant in a system of racial exclusion practiced by elements of the local construction industry.” Croson, 488 U.S. at 492, 109 S.Ct. 706 (plurality opinion) (quotation omitted). Thus, Denver may establish its compelling interest by presenting evidence of its own direct participation in racial discrimination or its passive participation in private discrimination. See Concrete Works II, 36 F.3d at 1519 & n. 7.

The question of whether Denver has demonstrated a strong basis in evidence is a question of law. See id. at 1522. As such, we review that question and any attendant legal questions de novo. Underlying factual findings are reviewed for clear error. See id.

Once Denver meets its burden, CWC must introduce “credible, particularized evidence to rebut [Denver’s] initial showing of the existence of a compelling interest.” Adarand Constructors, Inc. v. Slater, 228 F.3d 1147, 1175 (10th Cir.2000) (“Adarand VII”). “[R]ebuttal evidence may consist of a neutral explanation for the statistical disparities.” Coral Constr. Co. v. King County, 941 F.2d 910, 921 (9th Cir.1991) (cited with approval in Concrete Works II, 36 F.3d at 1531). CWC can also rebut Denver’s statistical evidence “by (1) showing that the statistics are flawed; (2) demonstrating that the disparities shown by the statistics are not significant or actionable; or (3) presenting contrasting statistical data.” Id.; see also Eng’g Contractors Ass’n of S. Fla., Inc. v. Metro. Dade County, 122 F.3d 895, 916 (11th Cir.1997) (same); Contractors Ass’n of E. Pa., Inc. v. City of Phila., 6 F.3d 990, 1007 (3d Cir.1993) (same). This court has repeatedly emphasized that the burden of proof at all times remains with CWC to demonstrate the unconstitutionality of the ordinances. 3 See Adarand VII, 228 F.3d at 1176 (‘We reiterate that the ultimate burden of proof remains with the challenging party to demonstrate the unconstitutionality of an affirmative-action program.” (quotation omitted)); Concrete Works II, 36 F.3d at 1522-23; see also Wygant, 476 U.S. at 277-78, 106 S.Ct. 1842 (plurality opinion) (“The ultimate burden remains with the [plaintiff] to demonstrate the unconstitutionality of an affirmative-action program.”).

B. Gender-Based Remedial Measures

This court applies intermediate scrutiny to the gender-based measures contained in the ordinances. See Concrete Works II, 36 F.3d at 1519. To withstand CWC’s challenge, Denver must establish an “exceedingly persuasive justification” for those measures. United States v. Virginia, 518 U.S. 515, 524, 116 S.Ct. 2264, 135 L.Ed.2d 735 (1996) (quotation omitted). Denver can meet its burden by demonstrating that the gender-based preferences “serve[ ] important governmental objectives” and are “substantially related to achievement of those objectives.” Id. (quotation omitted). Neither this court nor the Supreme Court has developed a framework for analyzing equal protection challenges to gender-based remedial measures. See Eng’g Contractors Ass’n, 122 F.3d at 909 (“The Supreme Court has not addressed the question explicitly, and there is a similar dearth of guidance in the reported decisions of other federal appellate courts.”). Further, the parties have not provided this court with any comprehensive arguments on this issue. CWC implicitly advocates that the gender-based classifications must also survive strict judicial scrutiny. Denver argues that the statistical and anecdotal evidence it presented is sufficient to survive strict scrutiny so, a fortiori, the gender-based measures necessarily survive intermediate scrutiny.

To meet its burden of demonstrating an important governmental interest, Denver must show that the gender-based measures in the ordinances were based on “reasoned analysis rather than through the mechanical application of traditional, often inaccurate, assumptions.” Miss. Univ. for Women v. Hogan, 458 U.S. 718, 726, 102 S.Ct. 3331, 73 L.Ed.2d 1090 (1982). Thus, the evidentiary basis necessary to demonstrate Denver’s important governmental interest may be something less than the “strong basis in evidence” required to justify race-based remedial measures. See Eng’g Contractors Ass’n, 122 F.3d at 909; Contractors Ass’n of E. Pa., Inc., 6 F.3d at 1010 (“Logically, a city must be able to rely on less evidence in enacting a gender preference than a racial preference because applying Croson’s evi-dentiary standard to a gender preference would eviscerate the difference between strict and intermediate scrutiny”).

Denver argues that it can satisfy its burden of production without introducing evidence showing its active or passive participation in gender discrimination. This approach has been embraced by both the Ninth and Eleventh Circuit Courts of Appeal. See Ensley Branch, NAACP v. Seibels, 31 F.3d 1548, 1580 (11th Cir.1994) (“Under the intermediate scrutiny test, a local government must demonstrate some past discrimination against women, but not necessarily discrimination by the government itself. One of the distinguishing features of intermediate scrutiny is that, unlike strict scrutiny, the government interest prong of the inquiry can be satisfied by a showing of societal discrimination in the relevant economic sector.”); Coral Constr. Co., 941 F.2d at 932 (“Unlike the strict standard of review applied to race-conscious programs, intermediate scrutiny does not require any showing of governmental involvement, active or passive, in the discrimination it seeks to remedy.”). We need not resolve this issue, however, because Denver has introduced evidence that links the City to gender discrimination in the local construction industry. See §§ IV.C. & VI.A.1., infra.

IV. DENVER’S EVIDENCE SUPPORTING ITS COMPELLING INTEREST AND ITS IMPORTANT GOVERNMENTAL INTEREST

A. Historical Evidence

At trial, Denver introduced evidence detailing its construction contracting prae-tices before the 1990 Ordinance. In 1973, the City Council enacted an ordinance creating an Affirmative Action Office (“AAO”) within its Department of Public Works (“DPW”). 4 According to testimony from the City’s first Affirmative Action Officer, Wesley Martin, the AAO sought to ensure that minority contractors were hired to participate in City construction projects. Martin further testified that minority contractors were available but the City’s rules, guidelines, and biases operated to effectively bar them from participating in City contracting. In 1977, the City Council passed a resolution establishing a voluntary program aimed at increasing minority participation in City contracting. Martin testified that the resolution had very little impact on how City projects were bid, mainly because the program had no enforcement mechanism.

In 1977, the Department of Housing and Urban Development (“HUD”) commenced an investigation into a grievance filed by the Minority Association of Contractors. The grievance alleged that minority contractors were not being utilized on Denver-based, federally funded projects in violation of applicable federal statutes. HUD provided the City with a preliminary investigative report dated September 30, 1977 in which HUD concluded,

The [C]ity failed to take those reasonable actions to overcome the effects of conditions which resulted in limited participation in the benefits of the [Community Development Block Grant] Program, and failed to make reasonable efforts to meet the special needs of the minority contractors which in effect resulted in minority contractors not taking full advantage of the [Community Development Block Grant] Program.

Martin testified that this HUD report was significant because it was the first time the City “was actually told that they were in apparent non-compliance with affirmative action requirements.”

Martin also testified that the HUD report led Congresswoman Patricia Schroeder to request an investigation by the General Accounting Office (“GAO”). The GAO evaluated, inter alia, the DPW’s compliance with federal affirmative action requirements for minority construction contractors. In a report released on September 25, 1978 (the “GAO Report”), the GAO concluded that certain DPW contracting practices “appeared to have a significant negative effect on minority and other categorical groups of contractors covered by [federal] affirmative action requirements.” These practices included requiring contractor prequalification, advertising for most bids only on a limited basis, and providing inadequate time to submit a bid • proposal. See Concrete Works II, 36 F.3d at 1524 n. 11. Appended to the GAO Report was a chart detailing the DPW’s utilization of minority contractors on federally funded City projects. That chart indicates that a total of $55,477,000 was awarded for such contracts between July 1, 1975 and December 31, 1977. Of that total, $2,476,000, or 4.46% was awarded to minority firms.

In 1979, the United States Department of Transportation (“DOT”) threatened to withdraw federal financial assistance for contracting projects at Denver’s Stapleton International Airport unless the City took measures to facilitate minority participation on Stapleton projects. In a letter to the City’s mayor, the DOT asserted that the DPW’s prequalification requirement, while neutral on its face, was unjustified and operated to bar minority contractors from obtaining DPW contracts. The letter directed the City to eliminate or modify the prequalification requirement and to develop and authorize an affirmative action plan and procedure. In 1980, the City Council adopted an affirmative action program which applied to all contracts funded by the DOT. The plan contained percentage participation goals for both MBEs and WBEs.

A report on the utilization by the DPW of minority and women contracting firms prepared by the AAO and dated April 2, 1983, indicates that 9.4% of all DPW contract dollars were awarded to MBEs in 1977, 5.6% in 1978, 4.3% in 1979, 17.2% in 1980, 14.7% in 1981, and 24.1% in 1982. With respect to WBEs, 1.1% of all DPW contract dollars were awarded to WBEs in 1980, 5 0.34% in 1981, and 1.9% in 1982. Martin testified that the information on MBE and WBE project participation was provided by the contractors and that the AAO had no procedure by which it could monitor actual participation. He characterized the utilization numbers as “overstated.” Although the AAO report estimated the total number of MBEs and WBEs in the Denver MSA, it did not estimate their availability as a percentage of all construction firms.

In 1983, the City began to consider expanding its affirmative action program to include all DPW construction projects, not just those receiving federal funding. The City held a public hearing at which the City Council heard testimony from minority contractors and other individuals regarding utilization of MBEs and WBEs on local construction projects. The testimony included specific examples of discrimination encountered in the Denver construction industry. Many minority contractors testified that they worked on projects that had federal requirements for minority par ticipation but were almost completely excluded from City projects without federal affirmative action requirements. Additionally, the Director of Governmental Affairs for the Associated General Contractors of Colorado stipulated at the hearing that there was discrimination in the industry against minorities and women. The City Council subsequently enacted Ordinance 246, Series of 1983, which set goals for MBE and WBE participation in all City construction projects managed by the DPW. The annual goal for MBE participation was 20% of dollars spent. For WBE participation, the goal was 5% of dollars spent.

In 1984, the year after Ordinance 246 was passed, the AAO reported that MBE participation on all DPW construction contracts was 28%; WBE participation was 6%. In 1985, MBE participation was 21% and WBE participation was 7%. In 1986, the AAO reported MBE participation on all DPW projects at 20% and WBE participation at 7%. In 1987, MBE participation was 26% and WBE participation was 9%.

Ordinance 246 was set to expire in 1988. In that year, the DPW surveyed local contractors by soliciting written responses to questionnaires. The City Council also conducted public hearings on the utilization of MBEs and WBEs on DPW projects. The stated objective of the hearings was “[t]o determine the current MBE and WBE utilization levels on [DPW] projects and to assess their overall capabilities; also, to investigate the extent and impact of any past discriminatory practices or barriers to MBE and WBE participation on [DPW] projects; and to identify any special problems affecting MBEs and WBEs in specific areas of the construction industry.” Based on the responses to the questionnaires and the extensive testimony presented at the hearings, the City Council determined that Ordinance 246 should be extended with modification. It, therefore, enacted Ordinance No. 424, Series of 1988, which, inter alia, set higher annual participation goals for MBEs and WBEs.

B. Statistical Evidence — The Disparity Studies

In 1989, Denver hired, inter alia, Browne, Bortz & Coddington, Inc. and Harding & Ogborn (collectively “BBC”) to assess the propriety of the DPW goals program in light of the Supreme Court’s decision in Croson. BBC issued a final report on June 2, 1990 (the “1990 Study”). It determined that data showing MBE and WBE participation on most DPW construction projects was “tainted” by federal and City affirmative action programs that had been in place for more than a decade. Consequently, the 1990 Study analyzed the availability and utilization of MBE and WBE construction and design firms on City bond projects from the 1970’s and 1980’s. These projects were not subject to the goals program. Denver argues that this data provides a true measure of MBE and WBE utilization on public contracts.

The conclusions reached by BBC were expressed, in part, in the form of disparity indices. A “disparity index is calculated by dividing the percentage of MBE and WBE participation in City contracts by the percentage of MBEs and WBEs in the relevant population of local construction firms. A disparity index of 1 demonstrates full MBE and WBE participation, whereas the closer the index is to zero, the greater the MBE and WBE underutilization.” Concrete Works II, 36 F.3d at 1524 n. 10. Data from eight City bond projects undertaken between 1972 and 1976 showed disparity indices of less than 0.63 for MBEs and less than 0.29 for WBEs. On a bond project for a renovation of the Museum of Natural History, the disparity indi-ces were 0.48 for MBEs and 0.40 for WBEs. Finally, for 1985 housing bond projects, the disparity indices were 0.43 for MBEs and 0.09 for WBEs.

The 1990 Study also examined MBE and WBE utilization in the overall Denver MSA construction market, both public and private. Because DPW construction contracts represented only 2% of all construction in the Denver MSA, BBC believed that the data would not be skewed by the DPW goals program and would reflect the utilization of MBEs and WBEs in the market in which Denver obtained its construction and professional design services. Disparity indices for 1977, calculated by using Census Bureau data, were 0.44 for MBEs and 0.46 for WBEs. For 1982, the disparity indices were 0.46 for MBEs and 0.30 for WBEs. Additional disparity indices for 1989 were calculated using data obtained from telephone surveys conducted by a private telemarketing firm retained by BBC. The disparity indices calculated from this data were 0.43 for MBEs and 0.42 for WBEs.

Finally, BBC interviewed representatives of MBEs, WBEs, majority-owned construction firms, and government officials. They also reviewed testimony given at hearings held in 1988 by the DPW and the Denver City Council. Based on this information, the 1990 Study concluded that, despite Denver’s efforts to increase MBE and WBE participation in DPW projects, some Denver employees and private contractors engaged in conduct designed to circumvent the goals program. In an effort to ensure that projects were awarded to certain contractors, Denver employees avoided the goals program by using change orders to existing contracts rather than putting new work out to bid. Employees also characterized some major construction projects as “remodeling” because remodeling projects fell under the auspices of the Department of General Services (“DGS”) which had no goals program. Other responses indicated that prime contractors continued to call WBEs they knew were no longer in business and counted those calls as good-faith efforts to meet the goals program. Others bid shopped in an effort to prevent MBEs and WBEs from submitting the lowest bid, or characterized subcontractors as suppliers and then contended the goals program did not apply because not enough work was subcontracted. After reviewing the statistical and anecdotal evidence contained in the 1990 Study, the City Council enacted the 1990 Ordinance.

In 1991, BBC prepared an additional study analyzing the utilization of MBEs and WBEs in the goods, services, and remodeling industries (the “DGS Study”). BBC analyzed, inter alia, data for construction and remodeling contracts issued by the DGS. During the relevant period, these contracts were not subject to a goals program but were most likely subject to bonding and prevailing wage requirements. The disparity indices calculated for 1989 DGS remodeling projects were 0.14 for MBEs and 0.47 for WBEs. For 1990 remodeling projects, the disparity in-dices were 0.19 for MBEs and 1.36 for WBEs.

After this court decided Concrete Works II, Denver commissioned another study by BBC (the “1995 Study”). Using 1987 Census Bureau data, the 1995 Study again examined utilization of MBEs and WBEs in the construction and professional design industries within the Denver MSA. The census data included information on employment and revenues for proprietor-ships, partnerships, and Subchapter S corporations with ten or fewer stockholders. Information on C corporations was unavailable. According to the 1995 Study, firms were classified as minority or women-owned “if the sole owner or at least half of the partners or shareholders were minorities or women.” The 1995 Study acknowledged that the Census Bureau data on Hispanic, Asian, and Native American firms were generated from sampling and underrepresented total marketplace utilization and total availability of those firms. The 1995 Study, however, also noted that “Census information indicates that the same relationships between utilization and availability presented in [the] report would also be found if the entire population of Hispanic, Asian and Native American-owned firms were represented.”

The 1995 Study also concluded that MBEs and WBEs were more likely to be one-person or family-run businesses. The study concluded that Hispanic-owned firms were less likely to have paid employees than White-owned firms but that Asian/Native American-owned firms were more likely to have paid employees than White or other minority-owed firms. To determine whether these factors explained overall market disparities, the 1995 Study used the census data to calculate disparity indices for all firms in the Denver MSA construction industry 6 and separately calculated disparity indices for firms with paid employees and firms with no paid employees.

[[Image here]]

The Census Bureau information was also used to examine average revenues per employee for Denver MSA construction firms with paid employees. Hispanic, Asian, Native American, and women-owned firms with paid employees all reported lower revenues per employee than majority-owned firms. A comparison of revenues per employee for Black-owned firms was not included in the 1995 Study because of the small number of Black-owned constructions firms with paid employees.

The 1995 Study also used 1990 census data to calculate rates of self-employment within the Denver MSA construction industry. The data indicated that 5.8% of Blacks and 6.2% of Hispanics working in the construction industry were self-employed compared with 14.8% of Whites working in the construction industry. For women, 4.8% were self-employed compared to 12.6% of men. Data from the professional design industry demonstrated that 3.3% of women in that industry were self-employed compared to 13.0% of men. The study concluded that the disparities in the rates of self-employment for Blacks, Hispanics, and women persisted even after controlling for education and length of work experience. The 1995 Study controlled for these variables but reported that Blacks and Hispanics working in the Denver MSA construction industry were less than half as likely to own their own businesses as were Whites of comparable education and experience. White women in the construction industry owned businesses at 38% of the rate expected given their education and experience. Additionally, women working in the Denver professional design industry owned businesses at 30% of the expected rate. BBC was unable to perform an analysis of business-ownership rates for Blacks, Hispanics, Asians, and Native Americans working in the professional design industry because the necessary data was unavailable.

In late 1994 and early 1995, BBC also conducted a telephone survey of 2920 construction firms doing business in the Denver MSA. The survey collected information on firm revenue; length of time in business; ethnic, racial, and gender status of business ownership; work history with the City; qualifications; interest in future work with the City; and other characteristics. Information was obtained for all construction and professional design firms, including C corporations. Based on information obtained from the survey, BBC calculated percentage utilization and percentage availability of MBEs and WBEs. Percentage utilization was calculated from revenue information provided by the responding firms. Percentage availability was calculated based on the number of MBEs and WBEs that responded to the survey question regarding revenues. Using these utilization and availability percentages, the 1995 Study showed disparity indices of 0.64 for MBEs and 0.70 for WBEs in the construction industry. In the professional design industry, disparity indices were 0.67 for MBEs and 0.69 for WBEs. The 1995 Study concluded that the disparity indices obtained from the telephone survey data were more accurate than those obtained from the 1987 census data because the data obtained from the telephone survey was more recent, had a narrower focus, and included data on C corporations. 7 Additionally, it was possible to calculate disparity indices for professional design firms from the survey data.

The 1995 Study also contained a summary of a disparity study conducted in 1993 for the Denver Housing Authority (the “DHA Study”) and a 1992 disparity study conducted for the Regional Transportation District (the “RTD Study”). Because the RTD had an affirmative action program in place, the RTD Study examined the utilization of minority- and women-owned firms by the private sector in the Denver area marketplace. The RTD Study, inter alia, examined both prime contracting and subcontracting in the Denver MSA construction industry. The disparity indices shown in the RTD Study are summarized below. 8

The DHA Study examined the utilization of M/WBEs by the Denver Housing Authority for the years 1986, 1987, 1988, and 1992. Both census data and information from the DHA’s vendor file were used to measure availability. The DHA Study found disparities between the utilization and availability of M/WBEs in some areas in some years, including years that DHA had an affirmative action program in place.

After it was presented with the 1995 Study, the City enacted the 1996 Ordinance. The 1996 Ordinance made several changes to the 1990 Ordinance. See Concrete Works III, 86 F.Supp.2d at 1049; supra § II.A.

In 1997, the City retained National Economic Research Associates, Inc. (“NERA”) to conduct a study to estimate the availability of MBEs and WBEs and to examine, inter alia, whether race and gender discrimination limited the participation of MBEs and WBEs in construction projects of the type typically undertaken by the City (the “NERA Study”). The NERA Study used a more sophisticated method to calculate availability than the earlier studies conducted by BBC. The study first identified the construction specialities and geographic areas in which the City spent the bulk of its construction funds. The MOCC provided NERA with information for the years 1991-1996 relating to the City’s construction contracts. That information included the name of the project, the name and address of each contractor and subcontractor, the task performed, the contract type, the amount paid to each contractor, and the total value of each project. NERA excluded projects funded by the federal government and projects relating to the construction of Denver International Airport because they were not representative of the typical City construction project. Public library projects funded by the 1990 Library Bond Program were prorated because their large size was atypical. NERA used the firms’ postal zip codes to determine their location.

Each construction contract or task performed was then assigned a four-digit Standard Industrial Classification (“SIC”) code. SIC codes catagorize businesses by specialization and are used by the Census Bureau to report economic statistics. NERA then used this data to summarize the City’s construction expenditures by project type and geographic area. The study showed that more than 84% of the City’s construction dollars were paid to firms located in Adams, Arapahoe, Boulder, Denver, Douglas, and Jefferson counties. Further, forty SIC codes accounted for 95% of all City construction expenditures.

NERA used the geographic and specialization information to calculate M/WBE availability. Availability was defined as “the ratio of M/WBE firms to the total number of firms in the four-digit SIC codes and geographic market area relevant to the City’s contracts.” The total number of firms was obtained from Dun & Bradstreet’s Marketplace database, The number of M/WBE firms was obtained from a directory compiled by BBC from twenty-nine different sources. NERA then successfully located 1002 of the 1283 M/WBEs in the directory. The information it obtained from these firms was used to adjust for the possibility of overcounting M/WBEs. NERA also sampled firms in the Dun & Bradstreet Marketplace database to make an undercount adjustment. The final result was a weighted average of availability for each racial group and for women. Availability was calculated for both prime contractors and subcontractors in each group.

The NERA Study then compared M/WBE availability and utilization in the Colorado construction industry. The statewide market was used because neces sary information was unavailable for the Denver MSA. NERA believed the statewide data was relevant because the Denver MSA accounted for nearly 60% of the total value of construction work done in Colorado. Additionally, data collected in 1987 by the Census Bureau was used because more current data was unavailable. The NERA Study calculated disparity in-dices 9 for the statewide construction market in Colorado as follows: 0.41 for African American firms, 0.40 for Hispanic firms, 0.14 for Asian and other minorities, and 0.74 for women-owned firms. NERA considered a disparity to be substantially significant if it was less than 0.80 because the Equal Employment Opportunity Commission considers a selection rate for any group that is 80% or less of the selection rate for the group with the highest rate to be evidence of adverse impact. Cf 29 C.F.R. § 1607.4(D).

The NERA Study also contained an analysis of whether African Americans, Hispanics, or Asian Americans working in the construction industry are less likely to be self-employed than similarly situated Whites. Using data from the Public Use Microdata Samples (“PUMS”) of the 1990 Census of Population and Housing, NERA obtained a sample of 3075 individuals working in the construction industry, of whom 1874 resided in the Denver MSA. NERA was able to obtain the following information for each individual: race; whether the individual was self-employed or worked for someone else; sex; marital status; age; education; access to capital including dividend and interest income, spouse’s income, and home ownership; number of children living at home; and “personal handicaps.” NERA considered these factors possible determinants of self-employment. Using this information, NERA conducted two probit regressions: one for Colorado and one for the Denver MSA. The study concluded that in both Colorado and the Denver MSA, African Americans, Hispanics, and Native Americans working in the construction industry have lower self-employment rates than Whites. Asian Americans had higher self-employment rates than Whites. NERA concluded that the high rate of self-employment for Asian Americans was consistent with studies showing that immigrants are more likely to be self-employed than individuals born in the United States. Close to 60% of the Asian Americans in the PUMS data used by NERA were immigrants.

Using the availability figures calculated earlier in the study, the NERA Study then compared the actual availability of M/WBEs in the Denver MSA with the potential availability of M/WBEs if they formed businesses at the same rate as Whites with the same characteristics.

[[Image here]]

Finally, the NERA Study examined whether self-employed minorities and women in the construction industry have lower earnings than white males with similar characteristics. Using linear regression analysis, NERA was able to compare business owners with similar years of education, of similar age, doing business in the same geographic area, and having other similar demographic characteristics. Even after controlling for several factors, the results showed that self-employed African Americans, Hispanics, Native Americans, and women had lower earnings than white males. In the Denver MSA, the earnings of self-employed women in the construction industry were 59% of the earnings of similarly-situated white men. The earnings disparity for African Americans was 51%, for Hispanics it was 75%, and for Native Americans it was 26%. No disparity was shown for Asian Americans.

NERA also conducted a mail survey of both M/WBEs and non-M/WBEs to obtain anecdotal evidence on their experiences in the construction industry. Approximately 17% of the M/WBEs who received the questionnaire responded. Of the M/WBEs who responded, 35% indicated that they had experienced at least one incident of disparate treatment within the last five years while engaged in business activities. These business activities included, inter alia: (1) applying for a commercial loan, (2) applying for a bond, (3) obtaining quotes from suppliers, (4) bidding or working on private and public sector prime contracts, (5) bidding or working on private and public sector subcontracts, (6) receiving payments for subcontracting work, (7) being required to do inappropriate or extra work on a project, and (8) having to meet unnecessarily strict quality or performance standards. The survey also posed the following question: “How often do prime contractors who use your firm as a subcontractor on public sector projects with [M/WBE] goals or requirements ... also use your firm on public sector or private sector projects without [M/WBE] goals or requirements?” Fifty-eight percent of minorities and 41% of white women who responded to this question indicated they were “seldom or never” used on non-goals projects.

M/WBEs were also asked whether the following aspects of procurement made it more difficult or impossible to obtain construction contracts: (1) bonding requirements, (2) insurance requirements, (3) large project size, (4) cost of completing proposals, (5) obtaining working capital, (6) length of notification for bid deadlines, (7) prequalification requirements, and (8) previous dealings with an agency. This question was also asked of non-M/WBEs in a separate survey. With one exception, 10 M/WBEs considered each aspect of procurement more problematic than non-M/WBEs. To determine whether a firm’s size or experience explained the different responses, NERA conducted a regression analysis that controlled for age of the firm, number of employees, and level of revenues. The results again showed that with the same, single exception, M/WBEs had more difficulties than non-M/WBEs with the same characteristics. Additionally, the results of a follow-up telephone survey of M/WBE and non-M/WBE firms not responding to the mail survey led NERA to conclude that it is likely the disparities are greater than indicated. The telephone survey indicated that non-M/WBE firms were more likely to have responded to the mail survey than M/WBEs if they had actually experienced the difficulties delineated in the survey question.

After the NERA Study was completed, the City enacted the 1998 Ordinance. The 1998 Ordinance reduced the annual goals to 10% for both MBEs and WBEs and eliminated a provision which previously allowed M/WBEs to count their own work toward project goals. See Concrete Works III, 86 F.Supp.2d at 1050.

C. Anecdotal Evidence

The district court’s memorandum and order contains a comprehensive synopsis of the anecdotal evidence presented by Denver at trial. See id. at 1071-74. That evidence was extensive and included the testimony of the senior vice-president of a large, majority-owned construction firm who stated that when he worked in Denver, he received credible complaints from minority and women-owned construction firms that they were subject to different work rules than majority-owned firms. He also testified that he frequently observed graffiti containing racial or gender epithets written on job sites in the Denver metropolitan area. Further, he stated that he believed, based on his personal experiences, that many majority-owned firms refused to hire minority or women-owned subcontractors because they believed those firms were not competent.

Several M/WBE witnesses testified that they experienced difficulty prequalifying for private sector projects and projects with the City and other governmental entities in Colorado. See id. at 1071-72. One individual testified that her company was required to prequalify for a private sector project while no similar requirement was imposed on majority-owned firms. Several others testified that they attempted to prequalify for projects but their applications were denied even though they met the prequalification requirements.

Other M/WBEs testified that their bids were rejected even when they were the lowest bidder; that they believed they were paid more slowly than majority-owned firms on both City projects and private sector projects; that they were charged more for supplies and materials; that they were required to do additional work not part of the subcontracting arrangement; and that they found it difficult to join unions and trade associations. See id. at 1072-73. There was extensive testimony detailing the difficulties M/WBEs experienced in obtaining lines of credit. One WBE testified that she was given a false explanation of why her loan was declined; another testified that the lending institution required the co-signature of her husband even though her husband, who also owned a construction firm, was not required to obtain her co-signature; a third testified that the bank required her father to be involved in the lending negotiations.

The most poignant anecdotal testimony involved recitations of racially and gender-motived harassment experienced by M/WBEs at work sites. Women were called “bitches” and Blacks were called “nigger” or “dumb nigger.” One seventy-three year old truck driver was called a “dumb, f-ing Mexican.” Even more disturbing was the testimony that minority and female employees working on construction projects were physically assaulted and fondled, spat upon with chewing tobacco, and pelted with two-inch bolts thrown by males from a height of eighty feet.

V. THE LEGAL FRAMEWORK APPLIED BY THE DISTRICT COURT

The district court began its examination of the City’s evidence with the disparity studies conducted by BBC and NERA. The court criticized the studies because they did not answer the following six questions posed by the court for the first time in its memorandum and order:

(1) Is there pervasive race, ethnic and gender discrimination throughout all aspects of the construction and professional design industry in the six county Denver MSA? (2) Does such discrimination equally affect all of the racial and ethnic groups designated for preference by Denver and all women? (3) Does such discrimination result from policies and practices intentionally used by business firms for the purpose of disadvantaging those firms because of race, ethnicity and gender? (4) Would Denver’s use of those discriminating firms without requiring them to give work to certified MBEs and WBEs in the required percentages on each project make Denver guilty of prohibited discrimination? (5) Is the compelled use of certified MBEs and WBEs in the prescribed percentages on particular projects likely to change the discriminatory policies and programs that taint the industry? (6) Is the burden of compliance with Denver’s preferential program a reasonable one fairly placed on those who are justly accountable for the proven discrimination?

Concrete Works III, 86 F.Supp.2d at 1066— 67.

Believing that its six questions set the proper legal framework for analyzing the City’s evidence, the district court criticized Denver’s disparity studies and refused to give weight to much of Denver’s evidence because it did “almost nothing to answer” those questions and because the methodology used in the studies “was not designed to answer the relevant questions.” Id. at 1067, 1071. The court’s questions, however, misstate controlling precedent and Denver’s burden at trial.

Read in context with the other five questions and the record, it is clear from the first question that the district court believed Denver was required to prove the existence of discrimination. Instead of asking whether Denver had demonstrated strong evidence from which an inference of past or present discrimination could be drawn, the question asks whether Denver’s evidence shows that there is pervasive discrimination. See id. at 1066. The second and third questions then refer to “such discrimination” and the sixth question asks whether the burden of compliance is placed on those firms accountable for the “proven discrimination.” Id. (emphasis added). It appears the district court may have been persuaded by CWC’s erroneous and unsupported statement in its written closing argument that Denver had the “burden of establishing by a preponderance that not only were there inferences of discrimination, but in fact that the inferences were correct.” 11 Denver, however, bore no such burden.

In Concrete Works II, this court clearly-stated that “the Fourteenth Amendment does not require a court to make an ultimate finding of discrimination before a municipality may take affirmative steps to eradicate discrimination.” 36 F.3d at 1522. Denver’s initial burden was to demonstrate that strong evidence of discrimination supported its conclusion that remedial measures were necessary. Strong evidence is that “approaching a prima facie case of a constitutional or statutory violation,” not irrefutable or definitive proof of discrimination. Croson, 488 U.S. at 500, 109 S.Ct. 706. The burden of proof at all times remained with CWC to prove by a preponderance of the evidence that Denver’s “evidence did not support an inference of prior discrimination and thus a remedial purpose.” Adarand VII, 228 F.3d at 1176 (quotation omitted).

Consistent with its second question, the district court concluded that “the aggregation of [all racial and ethnic groups] as equally victimized by discrimination and equally entitled to the preferential remedies is particularly problematic for Fourteenth Amendment equality analysis.” Concrete Works III, 86 F.Supp.2d at 1069 (emphasis added). In Croson, a majority of the Court noted that Richmond’s affirmative action program included racial groups “that, as a practical matter, may never have suffered from discrimination in the construction industry in Richmond.” 488 U.S. at 506, 109 S.Ct. 706. Consequently, the Court questioned whether Richmond could justify the program as an attempt to remedy past discrimination. See id. Denver, however, did introduce evidence of discrimination against each group included in the ordinances. Thus, Denver’s evidence does not suffer from the problem discussed by the Court in Croson. The district court, however, apparently believed Denver could not satisfy its burden of introducing strong evidence unless it was able to show that each group suffered equally from discrimination. Croson imposes no such requirement.

The district court’s third question also misstates the applicable law. 12 Underlying that question is the district court’s conclusion that Denver must demonstrate that the private firms directly engaged in any discrimination in which Denver passively participates do so intentionally, with the purpose of disadvantaging minorities and women. The district court provided no support for its conclusion and this court could find none. The Croson majority concluded that a “city would have a compelling interest in preventing its tax dollars from assisting [local trade] organizations in maintaining a racially segregated construction market.” 488 U.S. at 503, 109 S.Ct. 706. Thus, Denver’s only burden was to introduce evidence which raised the inference of discriminatory exclusion in the local construction industry and linked its spending to that discrimination.

The Supreme Court has clearly stated that the inference of discriminatory exclusion can arise from statistical disparities. See id. Accordingly, we conclude that Denver can meet its burden through the introduction of statistical and anecdotal evidence alone. To the extent the district court required Denver to introduce additional evidence to show discriminatory motive or intent on the part of private construction firms, the court erred. Denver was under no burden to identify any specific practice or policy that resulted in discrimination. Neither was Denver required to demonstrate that the purpose of any such practice or policy was to disadvantage women or minorities. To impose such a burden on a municipality would be tantamount to requiring direct proof of discrimination and would eviscerate any reliance the municipality could place on statistical studies and anecdotal evidence. Accord Associated Gen. Contractors of Cal., Inc. v. Coalition for Econ. Equal., 950 F.2d 1401, 1416 n. 11 (9th Cir.1991).

Guided by the erroneous third question, the court concluded that Denver’s disparity studies were flawed because they “do not generate a fair inference that there are discriminatory barriers to participation in the construction industry that are different from societal discrimination.” Concrete Works III, 86 F.Supp.2d at 1071 (emphasis added). The district court provided no authority for its conclusion that the City must demonstrate that any identified discrimination in the construction industry is not a reflection of societal discrimination. Denver, in fact, is not required to show that the discriminatory practices in the construction industry were unique to that industry or differed from societal discrimination. Under the district court’s approach, a municipality is without the power to remedy even proven discrimination in the construction industry as long as that discrimination merely reflects general societal discrimination. Denver offers the persuasive hypothesis that the district court’s position arose from a misinterpretation of Croson’s admonition against reliance on general societal discrimination to support affirmative action programs. See Croson, 488 U.S. at 497, 109 S.Ct. 706 (plurality opinion). We are not suggesting that societal discrimination without more would suffice. Indeed, the converse is true. The Croson plurality’s statements, however, have no application when, as in this case, the municipality presents specific evidence of discrimination in the local construction industry. If such evidence is presented, it is immaterial for constitutional purposes whether the industry discrimination springs from widespread discriminatory attitudes shared by society or is the product of policies, practices, and attitudes unique to the industry. Denver’s statistical and anecdotal evidence is relevant because it identifies discrimination in the local construction industry, not simply discrimination in society. The genesis of the identified discrimination is irrelevant and the district court erred when it discounted Denver’s evidence on that basis.

The district court’s erroneous view of the law affected its analysis of Denver’s evidence. After considering the anecdotal evidence, the court found that the evidence supported the conclusion “that women and minority groups are disadvantaged in trying to compete in the construction industry because of the prevalence of negative views about them.” Id. at 1074; see also id. (“In summary, the anecdotal evidence shows that race, ethnicity and gender affect the construction industry and those who work in it.”). Notwithstanding that finding, the court deemed the anecdotal evidence unpersuasive because it did not show that the negative views prevalent in the construction industry were different than “any societal views.” Id.; see also id. at 1073 (“It cannot be determined whether these [incidents involving harassment at work sites] are manifestations of societal prejudices or fairly attributable to employ ers as a business policy.”). The district court’s erroneous belief that Denver was required to show the existence of specific discriminatory policies and that those policies were more than a reflection of societal discrimination resulted in the court improperly discounting much of Denver’s statistical and anecdotal evidence.

The district court’s fourth question appears to be the basis upon which the court rejected the evidence Denver presented on marketplace discrimination. See § VI.A.1., infra. The question expresses the erroneous legal conclusion that a municipality may only remedy its own discrimination. This conclusion is contrary to both our holding in Concrete Works II and the plurality opinion in Croson. It appears to be a reflection of the erroneous statement made by CWC in its closing argument. See supra n. 12. This court recognized in this very case that “a municipality has a compelling interest in taking affirmative steps to remedy both public and private discrimination specifically identified in its area.” Concrete Works II, 36 F.3d at 1529 (emphasis added). In Concrete Works II, we remanded this case for trial and stated, “The record before us does not explain the Denver government’s role in contributing to the underutilization of MBEs and WBEs in the private construction market in the Denver MSA, and this may well be a fruitful issue to explore at trial.” Id. at 1529-30. We also stated that “we do not read Croson as requiring the municipality to identify an exact linkage between its award of public contracts and private discrimination.” Id. at 1529. Our comments indicate that Denver can meet it burden of demonstrating its compelling interest with evidence of private discrimination in the local construction industry coupled with evidence that it has become a passive participant in that discrimination. See id. Thus, contrary to the wording of question four, Denver was clearly not required to demonstrate that it is “guilty of prohibited discrimination” to meet its initial burden.

Likewise, the district court’s fifth question manifests the erroneous conclusion that Denver was required to demonstrate the ordinances will change discriminatory practices and policies in the local construction industry. Denver’s program can survive the equal protection challenge brought by CWC if Denver can show that the ordinances are narrowly tailored to remedy Denver’s participation in the identified discrimination; there is no requirement that the ordinances must also eliminate the discrimination. In fact, any such requirement would be illogical. If firms persisted in their discrimination, they could effectively defeat all affirmative action legislation.

Finally, contrary to the district court’s sixth question, the ordinances do not have to be tailored to place the burden of compliance only on those firms accountable for the discrimination. A plurality of the Supreme Court has stated that “[a]s part of this Nation’s dedication to eradicating racial discrimination, innocent persons may be called upon to bear some of the burden of the remedy.... [S]ueh a sharing of the burden by innocent parties is not impermissible.” Wygant, 476 U.S. at 280-81, 106 S.Ct. 1842 (quotations omitted). The proper focus is on whether the burden on third parties is “too intrusive” or “unacceptable.” See id. at 283, 106 S.Ct. 1842; United States v. Paradise, 480 U.S. 149, 182, 107 S.Ct. 1053, 94 L.Ed.2d 203 (1987) (plurality opinion).

The sixth question also appears to be implicated in the district court’s statement rejecting Denver’s disparity studies: “While statistical studies may suggest that some showings of disparity may create an inference of discrimination, they say noth ing about who is responsible for such discrimination.” Concrete Works III, 86 F.Supp.2d at 1070. The district court then intimated that it would be illogical to assume that “male Caucasian contractors will only do business with other male Caucasians even where that is contrary to their economic interests.” Id. Such an assumption, while perhaps economically illogical, nonetheless is consistent with the district court’s own observation that, “[discriminating behavior ... is irrational and not subject to the often used legal standard of conduct — a reasonable person under the same or similar circumstances.” Id. at 1064. Additionally, this court has previously concluded that Denver’s statistical studies which compare utilization of M/WBEs to availability, support the inference that “local prime contractors” are engaged in racial and gender discrimination. Concrete Works II, 36 F.3d at 1529. Thus, Denver’s disparity studies should not have been discounted because they fail to specifically identify those individuals or firms responsible for the discrimination.

The six questions posed by the district court as its aggregate litmus test contain misstatements or misapplications of the legal principles that govern equal protection cases like the one before the court. Not only did the district court analyze the ordinances and Denver’s evidence supporting them through an incorrect legal framework, it also discounted Denver’s studies as biased because they failed to address the six questions. See Concrete Works III, 86 F.Supp.2d at 1071. According to the court, this bias was evident when the articulated purposes of the studies were contrasted with the court’s questions. See id. Because the court’s six questions incorrectly state Denver’s burden, however, the studies cannot be criticized as biased on that basis. We conclude the district court’s framework imposed a greater burden on Denver than that required by applicable law. Accordingly, the court failed to give sufficient weight to Denver’s evidence. See id. (criticizing Denver’s disparity studies because “the methodology was not designed to answer the relevant questions”); id. at 1074 (discounting Denver’s anecdotal evidence because it “does not answer the six questions considered in the court’s evaluation of the statistical evidence.”).

VI. CWC’s ARGUMENTS AND REBUTTAL EVIDENCE

A. The Disparity Studies

1. Use of Marketplace Data

The district court, inter alia, concluded that the 1990 Study, the 1995 Study, the NERA Study, and the other disparity studies upon which Denver relied were significantly flawed because they measured discrimination in the overall Denver MSA construction industry, not discrimination by the City itself. See id. at 1070 (“The question, [Denver says], is what happens in the market without [the ordinances]. That may be consistent with scientific methodology but it does not square with the applicable law.”). The court may have believed that marketplace data is irrelevant because such data would not answer question four: “Would Denver’s use of those discriminating firms without requiring them to give work to certified MBEs and WBEs in the required percentages on each project make Denver guilty of prohibited discrimination?” Id. at 1066. The district court’s conclusion, however, is directly contrary to our holding in Adarand VII that evidence of both public and private discrimination in the construction industry is relevant. See 228 F.3d at 1166— 67.

Consistent with the district court’s conclusion, CWC continues to argue, as it did in Concrete Works II, that marketplace data is irrelevant because the ordinances can be justified only by evidence of discrimination by the City itself or by prime contractors while working on City projects. See supra n. 12. CWC’s argument is based on language used in Croson but taken out of context by CWC and by memorandum orders issued by two federal district courts. See Croson, 488 U.S. at 502, 109 S.Ct. 706; Associated Util. Contractors of Md., Inc. v. Mayor & City Council, 83 F.Supp.2d 613, 619 (D.Md. 2000); Webster v. Fulton County, 51 F.Supp.2d 1354, 1368-69 (N.D.Ga.1999), summarily affd, 218 F.3d 1267 (11th Cir. 2000). In Croson, the Court criticized Richmond’s data, in part, because it did not show “what percentage of total city construction dollars minority firms now receive as subcontractors on prime contracts let by the city.” Croson, 488 U.S. at 502, 109 S.Ct. 706. Relying on this language, CWC argues that the City cannot meet its burden of demonstrating strong evidence of discrimination unless it demonstrates that it directly participates in discrimination or indirectly participates by utilizing contractors who discriminate on City projects. It further argues that evidence of marketplace discrimination is irrelevant to this determination and can never assist the City in meeting its burden.

CWC’s argument is not supported by the language it cites from Croson when that language is read in context. In the Croson majority opinion, the Court noted, by example, some of the flaws in the evidence presented by the city of Richmond. In addition to criticizing the city because it did not present data showing the percentage of city dollars received by minority subcontractors on city construction projects, the Court also faulted Richmond’s evidence that MBE membership in local contractors’ associations was extremely low. See id. at 503, 109 S.Ct. 706. The Court’s criticism, however, was not that the evidence of low participation in such associations was irrelevant. Instead, the Court criticized the information because it did not consider nondiscriminatory explanations for the low level of membership. See id. Importantly, the Court specifically concluded that if Richmond had linked the evidence showing low MBE participation in trade associations to the number of local MBEs eligible for membership, any resulting statistical disparity, if large enough, could support an inference of discriminatory exclusion. See id. If such an inference could be drawn, “the city would have a compelling interest in preventing its tax dollars from assisting these organizations in maintaining a racially segregated construction market.” Id. Thus, contrary to CWC’s miseharacterization of the Court’s statements in Croson, the Court actually concluded that evidence of marketplace discrimination could be used by a municipality to meet its burden of producing strong evidence but that Richmond failed to meet its burden because, inter alia, it failed to offer any such evidence.

The conclusion reached by the majority in Croson that marketplace data is relevant in equal protection challenges to affirmative action programs is consistent with the approach later taken by the Court in Shaw v. Hunt. In Shaw, a majority of the Court relied on the majority opinion in Croson for the broad proposition that a governmental entity’s “interest in remedying the effects of past or present racial discrimination may in the proper case justify a government’s use of racial distinctions.” 517 U.S. at 909, 116 S.Ct. 1894. The Shaw Court did not adopt any requirement that only discrimination by the governmental entity, either directly or by utilizing firms engaged in discrimination on projects funded by the entity, was remediable. The Court, however, did set out two conditions which must be met for the governmental entity to show a compelling interest. “First, the discrimination must be identified discrimination.” Id. at 910, 116 S.Ct. 1894 (quotation omitted). The City can satisfy this condition by identifying the discrimination, “ ‘public or private, with some specificity.’ ” Id. (quoting Croson, 488 U.S. at 504, 109 S.Ct. 706) (emphasis added). The governmental entity must also have a “strong basis in evidence to conclude that remedial action was necessary.” Id. (quotation omitted). Thus, Shaw specifically stated that evidence of either public or private discrimination could be used to satisfy the municipality’s burden of producing strong evidence.

CWC’s argument that the marketplace data is irrelevant is also inconsistent with binding precedent in this circuit. In Ada-rand VII, we specifically concluded that evidence of marketplace discrimination can be used to support a compelling interest in remedying past or present discrimination through the use of affirmative action legislation. See 228 F.3d at 1166-67 (“[W]e may consider public and private discrimination not only in the specific area of government procurement contracts but also in the construction industry generally; thus any findings Congress has made as to the entire construction industry are relevant.” (emphasis added)). Further, in this very case we rejected the argument CWC reasserts here that marketplace data is irrelevant and remanded the case to the district court to determine whether Denver could link its public spending to “the Denver MSA evidence of industry-wide discrimination.” Concrete Works II, 36 F.3d at 1529. We clearly stated that evidence explaining “the Denver government’s role in contributing to the underuti-lization of MBEs and WBEs in the private construction market in the Denver MSA ” was relevant to Denver’s burden of producing strong evidence. Id. at 1530 (emphasis added).

Consistent with our mandate in Concrete Works II, the City attempted to show at trial that it “indirectly contributed to private discrimination by awarding public contracts to firms that in turn discriminated against MBE and/or WBE subcontractors in other private portions of their business.” Id. The City can demonstrate that it is a “ ‘passive participant’ in a system of racial exclusion practiced by elements of .the local construction industry” by compiling evidence of marketplace discrimination and then linking its spending practices to the private discrimination. Croson, 488 U.S. at 492, 109 S.Ct. 706 (O’Connor, J., joined by Rehnquist, C.J. and White, J.). Therefore, evidence of marketplace discrimination is not only relevant but, in this ease, it is essential to the City’s claim that it is an indirect participant in private discrimination. Consequently, we again reject CWC’s argument and conclude that the district court’s determination that the marketplace data was irrelevant was a legal error that significantly affected the court’s analysis of Denver’s evidence.

At trial, the City presented testimony from M/WBEs doing business in the Denver MSA to support its assertion that it is a “ ‘passive participant’ in a system of racial exclusion practiced by elements of the local construction industry” because it contracts with firms which discriminate against women and minorities. Id. At least eight M/WBEs testified that general contractors who use them on City construction projects refuse to use them on private projects. The witnesses specifically named the contractors who engaged in this practice. An employee of the MOCC identified eighteen of those contractors as firms that have performed City contracts. Based on this testimony, the district court found that,

The City does not want to pay tax dollars to support firms that discriminate against other firms because of their race, ethnicity and gender. Yet, the anecdotal evidence shows that Denver has repeatedly and knowingly done just that. During the taking of testimony about the experiences of minority and woman-owned firms in dealing with other contractors on projects not involving Denver, the City’s lawyers carefully demonstrated that those same contractors often do business with the City.

Concrete Works III, 86 F.Supp.2d at 1075-76 (discussing narrow tailoring). CWC does not challenge this finding. The anecdotal evidence supporting the district court’s finding would not be sufficient on its own to support Denver’s burden of demonstrating a strong basis in evidence for its conclusion that remedial action was necessary. However, this evidence links Denver’s spending to private discrimination. See Concrete Works II, 36 F.3d at 1529. Consequently, we conclude that the anecdotal evidence and the district court’s factual finding based on that evidence amply support Denver’s position that it indirectly contributes to private discrimination in the Denver MSA construction industry. See Croson, 488 U.S. at 492, 109 S.Ct. 706 (O’Connor, J., joined by Rehnquist, C.J. and White, J.) (“It is beyond dispute that any public entity, state or federal, has a compelling interest in assuring that public dollars, drawn from the tax contributions of all citizens, do not serve to finance the evil of private prejudice.”).

CWC’s argument that the lending discrimination studies and business formation studies presented by Denver are irrelevant is also foreclosed by circuit precedent. In Adarand VII, we concluded that evidence of discriminatory barriers to the formation of businesses by minorities and women and fair competition between M/WBEs and majority-owned construction firms shows a “strong link” between a government’s “disbursements of public funds for construction contracts and the channeling of those funds due to private discrimination.” 228 F.3d at 1167-68. Evidence that private discrimination results in barriers to business formation is relevant because it demonstrates that M/WBEs are precluded at the outset from competing for public construction contracts. See id. at 1168. Evidence of barriers to fair competition is also relevant because it again demonstrates that existing M/WBEs are precluded from competing for public contracts. See id. Thus, like the studies measuring disparities in the utilization of M/WBEs in the Denver MSA construction industry, studies showing that discriminatory barriers to business formation exist in the Denver construction industry are relevant to the City’s showing that it indirectly participates in industry discrimination. 13

The City presented evidence of lending discrimination to support its position that M/WBEs in the Denver MSA construction industry face discriminatory barriers to business formation. Denver introduced a disparity study prepared in 1996 and sponsored by the Denver Community Reinvestment Alliance, Colorado Capital Initiatives, and the City. The study’s stated purpose was to “determine what barriers, if any, small businesses face when seeking credit from financial institutions, and whether there is any difference in treatment among small business owners seeking credit based on their ethnicity.” The study ultimately concluded that “despite the fact that loan applicants of three different racial/ethnic backgrounds in this sample were not appreciably different as businesspeople, they were ultimately treated differently by the lenders on the crucial issue of loan approval or denial.” In Adaraná VII, this court concluded that this very study, among other evidence, “strongly supported] an initial showing of discrimination in lending.” 228 F.3d at 1170; see also id. at 1170 n. 13 (“Lending discrimination alone of course does not justify action in the construction market. However, the persistence of such discrimination ... supports the assertion that the formation, as well as utilization, of minority-owned construction enterprises has been impeded.” (citation omitted)). The City also introduced anecdotal evidence of lending discrimination in the Denver construction industry. See Concrete Works III, 86 F.Supp.2d at 1072-73.

CWC did not present any evidence that undermines the reliability of the lending discrimination evidence but simply repeats the argument, foreclosed by circuit precedent, that it is irrelevant. The district court criticized the evidence because it failed to determine whether the discrimination resulted from discriminatory attitudes or from the neutral application of banking regulations. See Concrete Works III, 86 F.Supp.2d at 1072. We have already concluded, however, that discriminatory motive can be inferred from the results shown in disparity studies. Because the district court’s criticism is not directed at the methodology used or conclusions drawn in the lending discrimination study, it does not undermine the study’s reliability as an indicator that the City is passively participating in marketplace discrimination. Indeed, in Adarand VII we took “judicial notice of the obvious causal connection between access to capital and ability to implement public works construction projects.” 228 F.3d at 1170. The district court’s other criticisms are erroneous in light of this court’s holding that the study “support[s] an initial showing of discrimination in lending.” Id.

Denver also introduced evidence of discriminatory barriers to competition faced by M/WBEs in the form of business formation studies that were included in the BBC and NERA Studies. The 1990 Study and the 1995 Study both showed that all minority groups in the Denver MSA formed their own construction firms at rates lower than the total population but that women formed construction firms at higher rates. The 1997 NERA Study employed a more sophisticated analysis to examine self-employment rates and controlled for gender, marital status, education, availability of capital, and personal/family variables. As discussed, supra, the NERA Study concluded that African Americans, Hispanics, and Native Americans working in the construction industry have lower rates of self-employment than similarly situated Whites. Asian Americans had higher rates. The NERA Study did not calculate business-formation rates for women working in the construction industry. The NERA Study also concluded that minority and female business owners in the construction industry, with the exception of Asian American owners, have lower earnings than white male owners. This conclusion was reached after controlling for education, age, marital status, and disabilities.

Although CWC asserts that the district court correctly concluded that the business formation studies cannot be used to justify the ordinances, the court’s conclusion con flicts with our holding in Adarand VII. See 228 F.3d at 1167-68. “[T]he existence of evidence indicating that the number of [MBEs] would be significantly (but un-quantifiably) higher but for such barriers is nevertheless relevant to the assessment of whether a disparity is sufficiently significant to give rise to an inference of discriminatory exclusion.” Id. at 1174. Further, the district court’s conclusion that no inference of discrimination can be drawn from the studies because such a conclusion would require the “problematic assumption of a direct relationship between the rate of formation of new businesses and the number of persons working as employees in the industry” erroneously ignores the fact that the NERA Study controlled for variables like education and level of experience. Concrete Works III, 86 F.Supp.2d at 1066.

In its written closing argument, CWC asserted that the business formation study included in the 1995 Study is flawed because it did not control for variables such as the type of educational degree received and types of work experience. 14 Conceding that the NERA Study was “better,” CWC nevertheless argues that it too is flawed because it did not control for, inter alia, “quality of education” or “culture.” The district court faulted the 1995 Study because it did not control for “marital status, veteran status, availability of other sources of income and hours worked during the previous year.” Concrete Works III, 86 F.Supp.2d at 1057-58. The court also noted that the NERA Study failed to control for “prior business experience, religion, cultural history and whether parents were self-employed.” Id. at 1060.

Even assuming that it is possible to adequately measure variables like quality of education and culture, neither the district court nor CWC provided any explanation of how the failure to control for any of these variables undermines the reliability of the business formation studies. While the report prepared by CWC’s expert, George LaNoue, indicated that cultural differences or immigration status may affect business formation rates, LaNoue did not explain what he meant by the term “cultural differences,” did not conduct a study that controlled for these variables and, more importantly, did not testify that controlling for the variables would eliminate the disparities. “[Gjeneral criticism of disparity studies, as opposed to particular evidence undermining the reliability of the particular disparity studies ... is of little persuasive value.” Adarand VII, 228 F.3d at 1173 n. 14.

In sum, the district court erred when it refused to consider or give sufficient weight to the lending discrimination study, the business formation studies, and the studies measuring marketplace discrimination. That evidence was legally relevant to the City’s burden of demonstrating a strong basis in evidence to support its conclusion that remedial legislation was necessary. See id. at 1167-68. CWC’s generalized denouncements of the studies fall far short of the “credible, particularized evidence” necessary to rebut the City’s evidence. Id. at 1175.

2. Variables

Beyond its conclusion that Denver’s disparity studies were largely or wholly irrelevant because they measured marketplace discrimination and not discrimination by the City itself or by contractors working on City projects, the district court identified numerous perceived flaws in the studies. Acknowledging that disparity studies may, when properly conducted, constitute persuasive evidence in equal protection cases, the district court relied on language in Crosoris plurality opinion to set the standards those studies must meet. According to the district court,

the probative force of statistical disparity studies depends upon whether the data used provide meaningful measurements of the number of minority firms “qualified” and “willing and able to perform a particular service” as well as the number actually used in public contracting, directly or indirectly.

Concrete Works III, 86 F.Supp.2d at 1065 (quoting Croson, 488 U.S. at 501, 109 S.Ct. 706). As we have already concluded, disparity studies that measure the use of M/WBEs in the relevant marketplace, in addition to those that measure the direct or indirect use of M/WBEs in public contracting, are also probative. The court, however, also discredited the conclusions reached in Denver’s disparity studies either because those studies failed to address the six questions developed by the court or because they failed to control for certain variables.

a. Size and Experience

In Concrete Works II, this court reversed the grant of summary judgment, in part, because it was unclear from the record whether the 1990 Study overstated the availability of M/WBEs in the Denver construction market. We noted that “a disparity index calculated on the basis of the absolute number of MBEs in the local market may show greater underutilization than does data that takes into consideration the size of MBEs and WBEs.” Concrete Works II, 36 F.3d at 1528. At trial, both parties had the opportunity to present evidence on the effect firm size and capacity had on the disparities shown in Denver’s studies.

CWC challenges Denver’s disparity studies as unreliable because the disparities shown in the studies may be attributable to firm size and experience rather than discrimination. CWC’s argument is based on a report submitted by its expert, George LaNoue, which contains an extensive discussion of firm size and experience. LaNoue first asserts that M/WBEs are generally smaller and less experienced than majority firms. He then contends that smaller and less experienced firms are less qualified and less able to undertake City construction projects. He concludes that because Denver’s disparity studies do not control for firm size and experience, the M/WBE availability figures used in those studies are inflated. Thus, CWC argues the disparity indices calculated in the studies are unreliable because they are not based on the availability of only those firms qualified, willing, and able to work on City projects.

The district court noted that Denver’s disparity studies contain no analysis of “the actual qualifications and capacities of the MBEs and WBEs in the Denver MSA,” but instead assume that, at any given time, all M/WBEs are available to perform each contract. Concrete Works III, 86 F.Supp.2d at 1065-66. The court criticized Denver’s assumption as “implausible” and concluded that “[aggregating all of the MBEs and WBEs in estimating availability without regard for the size of the businesses ... is a serious flaw in the methodology and impairs the value of the results.” Id. at 1066, 1068.

Denver acknowledges that M/WBEs are generally smaller than majority-owned firms in terms of revenues and number of employees and are slightly less experienced. Additionally, in his written report, another CWC expert, John Lunn, asserts that the only conclusion that can be drawn from Denver’s studies is “that minority- and women-owned firms are smaller and less experienced than the average of all construction firms in the Denver construction market.” Although CWC did not conduct its own marketplace disparity study that controlled for firm size and experience, its argument that the disparity studies are unreliable because they fail to control for size and experience is deserving of consideration in light of the uncontrovert-ed evidence that M/WBEs are generally smaller and less experienced than majority firms.

Denver counters, however, that a firm’s size has little effect on its qualifications or its ability to provide construction services and that M/WBEs, like all construction firms, can perform most services either by hiring additional employees or by employing subcontractors. At trial, Denver introduced evidence that the median number of employees of all construction firms in the Denver MSA is three and presented testimony that even firms with few permanent employees can perform large, public contracts by hiring additional employees or subcontractors and renting equipment. Additionally, the district court found that “most firms have few full-time permanent employees and must grow or shrink their performance capacity according to the volume of business they are doing.” Id. at 1064.

CWC responds that elasticity itself is relative to size and experience; M/WBEs are less capable of expanding because they are smaller and less experienced. The district court found that a firm’s ability to expand “depends upon [its] access to increased resources, including workers with the needed skills, equipment, material and operating capital.” Id. The court did not find that a firm’s size affected its ability to obtain these resources but concluded that “the firm’s access to information, its reputation in the community and the skills of its managers” all had an effect. Id. Thus there is no finding by the district court that smaller, less experienced firms are less able to expand.

Even if we assume that M/WBEs are less able to expand because of their smaller size and more limited experience, CWC does not respond to Denver’s argument and the evidence it presented showing that experience and size are not race- and gender-neutral variables and that M/WBE construction firms are generally smaller and less experienced because of industry discrimination. The lending discrimination and business formation studies both strongly support Denver’s argument that M/WBEs are smaller and less experienced because of marketplace and industry discrimination. In addition, Denver’s expert David Evans testified that discrimination by banks or bonding companies would reduce a firm’s revenue and the number of employees it could hire. Robin Hackett, the owner of a WBE, testified that she has difficulty finding and retaining white male employees because they are ridiculed by others in the industry for working at a WBE. Hackett also testified that she is excluded from participating in a labor pool of non-union employees shared by other local firms. Those firms sharing this labor pool are able to “bid a bigger volume of work and not worry about manning the projects.” Even Lunn, CWC’s expert, admitted during cross-examination that the size and revenue differences between M/WBEs and majority-owned firms could be due in part to the presence of discrimination.

Denver also argues that CWC’s argument lacks merit because the 1990 Study, the DGS Study, the 1995 Study, and the NERA Study all controlled for size and the 1995 Study controlled for experience. It asserts that the 1990 Study measured revenues per employee for construction M/WBEs and concluded that the resulting disparities, “suggest! ] that even among firms of the same employment size, industry utilization of MBEs and WBEs was lower than that of non-minority male owned firms.” Similarly, the 1995 Study controlled for size, calculating, inter alia, disparity indices for firms with no paid employees which presumably are the same size. CWC responds by simply making the bald allegation that the 1995 Study’s control for size is a “pretense” and that the control is “meaningless.” This glib response is wholly inadequate to rebut Denver’s argument and the conclusions drawn in the disparity studies.

Based on the uncontroverted evidence presented at trial, we conclude that the district court did not give sufficient weight to Denver’s disparity studies because of its erroneous conclusion that the studies failed to adequately control for size and experience. Denver is permitted to make assumptions about capacity and qualification of M/WBEs to perform construction services if it can support those assumptions. The assumptions made in this case are consistent with the evidence presented at trial and support the City’s position that a firm’s size does not affect its qualifications, willingness, or ability to perform construction services and that the smaller size and lesser experience of M/WBEs are, themselves, the result of industry discrimination. Further, CWC did not conduct its own disparity study using marketplace data and thus did not demonstrate that the disparities shown in Denver’s studies will decrease or disappear if the studies controlled for size and experience to CWC’s satisfaction. Consequently, CWC’s rebuttal evidence is insufficient to meet its burden of discrediting Denver’s disparity studies on the issue of size and experience.

b. Specialization

The district court also faulted Denver’s disparity studies because they do not control for firm specialization. See Concrete Works III, 86 F.Supp.2d at 1068 (“Aggregating all of the MBEs and WBEs in estimating availability without regard for ... the particular services or type of work in which they specialize is a serious flaw in the methodology and impairs the value of the results.”). The court’s criticism would be appropriate only if there was evidence that M/WBEs are more likely to specialize in certain construction fields. CWC has failed to marshal any such evidence. Its appellate brief contains only the concluso-ry statement that M/WBEs “tend to be ... congregated in certain construction specialities.” This assertion is unsupported by any citation to the record and CWC does not direct this court to any corroborative study or other evidence. Further, there is no identified evidence showing that certain construction specializations require skills less likely to be possessed by M/WBEs. In fact, CWC’s statement appears to contradict the testimony of the City’s expert, Dr. Evans, that the data he reviewed showed that MBEs were repre sented “widely across the different [construction] specializations.” 15

In his expert report, Lunn criticizes Denver’s disparity studies because they aggregate construction firms without regard to specialization or size. Although Lunn’s report contains an example that illustrates his criticism, he conceded during cross-examination that the example was purely hypothetical and was not based on any data from the Denver MSA construction industry even though he had access to that data. Lunn also testified that he had not done any investigation into whether aggregation bias caused the disparities shown in Denver’s studies.

Not only does CWC offer no support for its position that M/WBEs are clustered in certain construction specialities, but it has failed to demonstrate that the disparities shown in Denver’s studies are eliminated when there is control for firm specialization. In contrast, the NERA Study, which controlled for SIC-code subspecialty yet still showed disparities, provides support for Denver’s argument that firm specialization does not explain the disparities.

c. Bidding

In his report, CWC’s expert, LaNoue, contends that the M/WBE availability data used in Denver’s disparity studies is unreliable because it is not a measure of only those firms actually bidding on City construction projects. LaNoue asserts that, “All other measures than bidding are merely proxies for availability.... [U]n~ less a firm makes a bid, it is not actually available for public contracting.” La-Noue’s position appears to constitute an implicit conclusion that only studies which equate M/WBE availability with actual bids on a contract-by-contract basis can be used to support Denver’s burden. To calculate availability at the level of certainty urged by CWC, Denver would be compelled to survey each MBE, WBE, and majority firm each time bids are sought for a construction project to determine which firms actually bid on the project, either as a prime contractor or as a subcontractor. While this approach may provide an accurate count of available firms, it says nothing about whether those firms are qualified. If availability is calculated on an individual basis using LaNoue’s approach, it is possible that unqualified firms would be included in the availability figure simply because they bid on a particular project. Conversely, qualified firms would not be included if, for any reason, they chose not to bid on the project. Thus, LaNoue’s approach itself illustrates why disparity studies must make assumptions about availability as long as the same assumptions can be made for all firms. 16

CWC does not identify any evidence showing that M/WBEs bid on City projects at a different rate than non-M/WBE firms. Thus, it has not demonstrated that differences in bidding practices may explain the disparities found in Denver’s studies. Additionally, we do not read Cro-son to require disparity studies that measure whether construction firms are able to perform a particular contract. The studies must only determine whether the firms are capable of “undertak[ing] prime or subcontracting work in public construction projects.” Croson, 488 U.S. at 502, 109 S.Ct. 706.

3. Stock Data

John Lunn, CWC’s expert, criticized Denver’s studies because they used “stock data” and not “flow data.” Studies utilizing stock data provide information on an industry at a discrete point in time. Flow data show changes that occur in the industry over time. Lunn testified that substantial changes in a market situation and substantial changes in flows do not show up in studies utilizing the stock concept. He then speculated that a study utilizing flow data might show no disparities even though an analysis of stock data may indicate that discrimination is present. Lunn, however, illustrated his point by using a hypothetical involving law school admissions: he did not test his hypothesis on actual flow data from the Denver construction market and he did not testify that any substantial change had occurred in the Denver construction industry which would make a study using flow data more reliable than the disparity studies conducted by BBC and NERA.

Denver’s individual studies examined the Denver construction industry from the 1970s through 1997. Each study showed disparities between M/WBEs and non-M/WBEs. Thus, considered in the aggregate the disparity studies themselves undermine CWC’s unsupported assertion that the use of flow data would explain the disparities.

B. Utilization of M/WBEs on City Projects

CWC argues that Denver cannot demonstrate a compelling interest because it ov-erutilizes M/WBEs on City construction projects. This argument is an extension of CWC’s argument that Denver can justify the ordinances only by presenting evidence of discrimination by the City itself or by contractors while working on City projects. Because we have concluded that Denver can satisfy its burden by showing that it is an indirect participant in industry discrimination, CWC’s argument relating to the utilization of M/WBEs on City projects goes only to the weight of Denver’s evidence.

In Concrete Works II, this court concluded that CWC had raised a genuine issue of material fact relating to Denver’s reliance on data from City-funded construction projects not subject to the ordinances or the goals program which predated the ordinances. We noted that one set of data “reveal[ed] extremely low MBE and WBE utilization.” Concrete Works II, 36 F.3d at 1526 (referring to the DGS Study). Similarly, data analyzing the utilization of M/WBEs on bond projects for which goals were never set also showed underutilization of M/WBEs. See id. at 1527 (referring to the 1990 Study). In contrast to this data, however, the record also contained studies showing very strong M/WBE utilization on City projects subject to the goals program. See id. at 1525. Consistent with this court’s mandate in Concrete Works II, at trial Denver sought to demonstrate that the utilization data from projects subject to the goals program was tainted by the program and “reflect[ed] the intended remedial effect on MBE and WBE utilization.” Id. at 1526.

In the pretrial order, the parties stipulated that the City’s utilization of MBEs 17 exceeded their availability in 18.75 of the 20.75 years measured since 1978. CWC asserts that this indicates there is no unde- rutilization of MBEs by the City and, thus, data on the utilization of MBEs on City projects provides no support for the race-based remedial measures. 18 Denver argues that the non-goals data is the better indicator of past discrimination in public contracting than the data on all City construction projects.

At trial, Denver presented evidence that the high utilization of MBEs on City projects subject to the goals program was the result of the program and not the absence of discrimination. Particularly persuasive was Denver’s evidence that MBE utilization rates on City construction projects declined significantly after the goals program was amended in 1989. One amendment eliminated the minimum annual goals for MBE participation and replaced them with a system whereby goals were set for each individual project. Another amendment required MBEs seeking certification to demonstrate that they had suffered from past discrimination. The drop in utilization rates can thus be linked to a relaxation of the City’s affirmative-action efforts. CWC presented no evidence that otherwise explains the decrease in utilization rates after the amendments became effective.

Denver also presented uncontroverted testimony that the goals and non-goals projects were similar. One of Denver’s experts testified that the same pool of construction firms worked on both the goals and non-goals projects. Additional evidence indicated that projects that should have been performed by the DPW were diverted to the DGS to avoid the goals program. This demonstrates that there was at least some overlap in the character of the goals projects administered by DPW and the non-goals projects administered by the DGS. Again, CWC presented no evidence that the character of the projects was significantly different or that they involved different pools of prime contractors. Finally, CWC’s argument that overutilization of MBEs on City projects rendered MBEs unavailable to work on private projects has little persuasive value in light of the extensive evidence on industry elasticity and Denver’s evidence that in 1989, DPW projects accounted for less than 4% of all MBE revenues. We conclude that Denver presented ample evidence to support the conclusion that the evidence showing M/WBE utilization on City projects not subject to the ordinances or the goals programs is the better indicator of discrimination in City contracting.

CWC also attacks the non-goals data presented by the City and the conclusions drawn from that data. Because CWC takes the position that the City can only remedy its own discrimination or the discrimination of contractors while working on City projects, it asserts that the non-goals data is the only relevant data presented by Denver. It further argues that the data provides insufficient support for Denver’s burden. We have rejected CWC’s argument that the marketplace data is irrelevant but agree that the non-goals data is also relevant to Denver’s burden. We note, however, that Denver did not rely heavily on the non-goals data at trial but focused primarily on the marketplace studies to support its burden.

CWC attempts to rebut the non-goals evidence by noting flaws in some of the disparity indices calculated on non-goals projects in the 1990 Study. CWC first criticizes the portion of the 1990 Study which examined the eight local bond projects undertaken between 1972 and 1976. CWC asserts that the disparity indices calculated for these bond projects are meaningless because they were calculated using 1977 availability figures. This valid criticism undermines the persuasive value of these disparity indices. CWC also challenges the conclusions drawn in the portion of the 1990 Study that analyzed five private housing projects constructed in 1985 with revenue bonds but not subject to the goals program. The flaws CWC notes in this study are legitimate and support its position that the minority utilization figures are inaccurate and, thus, the subject disparity indices in the 1990 Study may not be reliable.

CWC also asserts that Denver cannot rely on “the Peat Marwick study” to support the ordinances because that study found no evidence of discrimination in bonding by the City. Unfortunately, we cannot evaluate the merits of CWC’s argument because it has provided no record citation directing us to this study and we were unable to locate it in the appellate record. We also note that while it was discussed in Concrete Works I, it is not referred to in Concrete Works III. After a time-consuming search of the trial transcript, unguided by any record citation, we were able to locate and review only one piece of relevant evidence: the testimony of a witness, Wesley Martin. On cross-examination the following exchange occurred between Martin and CWC’s counsel,

Q: —you would agree that they did find there that there was no evidence of bonding discrimination in the bonding industry?

A: I believe they termed it a little bit differently than that, if I recall from reading that. They said that there was no immediate evidence of discrimination evident there, but I don’t think they came to a conclusion that it did not exist.

This testimony provides no support for CWC’s one-sentence argument that the Peat Marwick study found “no evidence of discrimination, either by Denver or by the bonding industry.” Because we cannot review the study itself, we cannot determined whether it rebuts Denver’s other evidence and we conclude that CWC’s argument is waived.

The City also relied on the DGS Study, the GAO Report, and the DOT’s threat to withdraw federal funding as evidence of City discrimination on non-goals projects. CWC does not rebut or even discuss the DGS Study which showed disparity indices of 0.14 for MBEs and 0.47 for WBEs on 1989 DGS remodeling projects. On 1990 DGS remodeling projects the disparity in-dices were 0.19 for MBEs and 1.36 for WBEs. With respect to the GAO Report, CWC asserts that the only finding reached in that report was that the DPW’s prequa-lification requirement had a disparate impact on minorities. CWC then summarily argues that this finding does not support the conclusion that the City was discriminating because the DPW’s prequalification requirements were “race neutral.” The GAO Report, however, concluded that the prequalification system was applied in “an inconsistent and subjective manner” and that it “appeared to have a greater adverse effect on small and minority contractors.” The Report also concluded that the DPW’s advertising procedures for bid proposals and its lack of a procedure to assure that contractors had adequate time to prepare bid proposals had a greater negative effect on smaller and/or less experienced contractors and minority contractors. Denver has already conceded that MBEs are generally smaller and less experienced than majority-owned firms and has introduced evidence indicating that both characteristics are the result of industry discrimination. See § VI.A.2.a, supra.

CWC also asserts that Denver’s reliance on the GAO Report is misplaced because the Report contained an incorrect minority availability figure of 11%. In the 1990 Study, BBC used census data to estimate MBE availability in 1977 19 at 2%. The GAO Report, however, did not calculate disparity indices or attempt to determine whether minority contractors were being underutilized. As CWC itself concedes, the Report concluded only that Denver’s contracting practices and policies adversely affected small contractors and minority contractors more than majority contractors. CWC’s criticism does nothing to rebut this finding. As the Report itself indicates, the GAO did not rely on the 11% availability figure to reach its conclusions:

Since Denver had not established specific goals for minority contractor participation in federally assisted construction contracts, no measure was available to determine whether the current 5-per-cent level of participation was reasonable. However, Denver identified about 11 percent of the construction contracting firms in the Denver metropolitan area as being owned by minorities.

Although the GAO Report is not particularly compelling because it contains very little analysis, the conclusions drawn in the Report do provide some support for Denver’s position that it engaged in discriminatory conduct. CWC’s arguments are insufficient to rebut those conclusions.

CWC’s attempt to discredit Denver’s reliance on the DOT’s threats to withdraw federal funding is also unpersuasive. Like the GAO Report, the DOT found that Denver’s prequalification requirements operated as a barrier to minority firms. In 1979, the DOT threatened to initiate administrative proceedings to terminate federal funding for Stapleton International Airport unless Denver modified its prequalification procedures and developed an affirmative action plan applicable to Stapleton. We again reject CWC’s argument that the DOT’s findings do not show discriminatory conduct by Denver because the prequalifi-cation requirements were race-neutral. CWC also asserts that the DOT’s findings are flawed because the DOT erroneously determined that minority contractor utilization was 6% in 1978 and 5% in 1979. CWC again points to the 1990 Study prepared by BBC which calculated MBE availability in 1977 at 2% and argues that, based on the DOT’s utilization figures, MBEs were actually overutilized on DOT-funded projects. CWC does not provide any record citation to support its assertion that the DOT determined minority utilization to be 6% in 1978 and 5% in 1979. We have reviewed all the materials in the appellate record relating to the City’s dispute with the DOT and could not locate the source of CWC’s information or confirm its accuracy. Thus, we are unable to consider CWC’s argument. In addition, we question whether an accurate disparity index can be calculated by using availability figures from one study and utilization figures from a second study. There is simply no way to determine whether similar data-collection methods were used in the two studies.

In sum, Denver has demonstrated that the utilization of M/WBEs on City projects has been affected by the affirmative action programs that have been in place in one form or another since 1977. Thus, the non-goals data is the better indicator of discrimination in public contracting. CWC’s criticisms of the non-goals disparity indices reported in the 1990 Study per suasively rebuts that data. CWC, however, is unable to persuasively rebut the other non-goals evidence, including the DGS Study, the GAO Report, and the DOT’s threatened administrative action. We conclude that, on balance, the non-goals data provides some support for Denver’s position that its belief that racial and gender discrimination existed in public contracting before the enactment of the ordinances was supported by strong evidence.

C. Revenue Comparisons

The 1995 Study contained tables showing that revenues per employee were lower for Hispanic, Asian, Native American, and women-owned construction firms in the Denver MSA than for majority-owned firms. CWC argues that this data on revenues is flawed. In his expert report, CWC expert, LaNoue, asserted that prime contractors report gross revenues differently than subcontractors: a subcontractor counts only those revenues earned by its own employees while a prime contractor also includes amounts it receives but pays out to subcontractors. At trial, LaNoue testified that “all the evidence suggests” that “white, male-owned firms are more likely to be prime contractors.” Thus, CWC argues that the revenue comparisons contained in the 1995 Study overstate the revenues of majority-owned firms.

CWC’s argument rises or falls on the correctness of the statement in LaNoue’s expert report that “MWBEs, as newer, smaller firms, tend more often to be subcontractors than prime contractors.” La-Noue did not identify any study or other evidence upon which he based his statements. In fact, availability data reported in Table 2 of the NERA Study appear to contradict LaNoue’s statement. That data shows that both MBEs and WBEs are almost equally likely to be prime contractors as they are to be subcontractors. “When an expert opinion is not supported by sufficient facts to validate it in the eyes of the law, or when indisputable record facts contradict or otherwise render the opinion unreasonable, it cannot support a jury’s verdict.” Brooke Group Ltd. v. Brown & Williamson Tobacco Corp., 509 U.S. 209, 242, 113 S.Ct. 2578, 125 L.Ed.2d 168 (1993).

LaNoue’s statements, at best, provide tenuous support for CWC’s argument. However, Denver does not challenge the district court’s finding, apparently based on LaNoue’s unsupported testimony, “that small minority firms most often work as subcontractors.” Concrete Works III, 86 F.Supp.2d at 1070. Thus, we must credit CWC’s argument but conclude it is only marginally persuasive and further note that Denver’s reliance on the revenue-per-employee data was minimal.

To support its related argument that majority-owned firms have greater revenues per employee because they are larger and more experienced, CWC offered the testimony of Craig Long. Long, who has worked in the Denver construction industry for twenty-eight years, testified that larger and/or more experienced firms are more competitive and more profitable per employee because they, inter alia: (1) have more access to cash so they can take advantage of early-payment discounts; (2) can obtain lower prices from suppliers if they have established long-term relationships with them; and (3) can obtain credit at lower rates. We have already concluded, however, that M/WBEs are less able to benefit from the advantages Long describes because of discrimination.

CWC also argues that the revenue comparisons are flawed because they do not identify the source of the revenues. CWC takes the position that Denver cannot use evidence of revenue differences to show that it is a passive participant in marketplace discrimination unless it can demonstrate that all revenues were derived from work done in the Denver MSA. It appears uncontested that Denver’s studies did not exclude revenues that Denver-based firms earned on projects outside the Denver MSA. See id. (noting that the data used in the NERA Study “do not limit revenues to what was earned by work done in Colorado”). LaNoue testified that ascertaining the source of firm revenues is important because it identifies the entities or individuals responsible for the discrimination. Given our conclusion, supra, that Denver’s other evidence supports the conclusion that it is a passive participant in marketplace discrimination, CWC’s criticisms regarding the source of revenues are irrelevant.

D. Anecdotal Evidence

Faced with the prodigious anecdotal evidence presented by Denver at trial, CWC argues that many of the examples of discrimination are not “severe” or “extreme.” The anecdotal evidence, however, included several incidents involving profoundly disturbing behavior on the part of lenders, majority-owned firms, and individual employees. The anecdotal testimony revealed behavior that was not merely sophomoric or insensitive, but which resulted in real economic or physical harm. While CWC also argues that all new or small contractors have difficulty obtaining credit and that treatment the witnesses characterized as discriminatory is experienced by all contractors, Denver’s witnesses specifically testified that they believed the incidents they experienced were motivated by race or gender discrimination. They supported those beliefs with testimony that majority-owned firms were not subject to the same requirements imposed on them.

There is no merit to CWC’s argument that the witnesses’ accounts must be verified to provide support for Denver’s burden. Anecdotal evidence is nothing more than a witness’ narrative of an incident told from the witness’ perspective and including the witness’ perceptions. In this case, the anecdotal evidence was not subject to rigorous cross-examination. See id. at 1071 (“This court limited the number of witnesses to be called for this purpose to avoid duplication under F.R.E. 403, received lay opinions under F.R.E. 701, expansively accepted hearsay and applied a liberal standard of relevance. Cross examination of these witnesses was restricted to avoid unduly prolonging the trial.”). Denver was not required to present corroborating evidence and CWC was free to present its own witnesses to either refute the incidents described by Denver’s witnesses or to relate their own perceptions on discrimination in the Denver construction industry.

After considering Denver’s anecdotal evidence, the district court found that the evidence “shows that race, ethnicity and gender affect the construction industry and those who work in it” and that the egregious mistreatment of minority and women employees “had direct financial consequences” on construction firms. Id. at 1074, 1073. The court, however, failed to credit any of Denver’s anecdotal evidence because it did not answer the six questions posed by the court. See id. at 1074 (“[TJaken as a whole, the anecdotal evidence does not answer the six questions considered in the court’s evaluation of the statistical evidence.”). As we conclude supra, however, the district court’s six questions represent a misstatement of the governing law and imposed an erroneously high burden on Denver. Based on the district court’s findings regarding Denver’s anecdotal evidence and our review of the record, we conclude that the anecdotal evi dence provides persuasive, unrebutted support for Denver’s initial burden. Cf. Int’l Bhd. of Teamsters v. United States, 431 U.S. 324, 339, 97 S.Ct. 1843, 52 L.Ed.2d 396 (1977) (concluding that anecdotal evidence presented in a pattern or practice discrimination case was persuasive because it “brought the cold [statistics] convincingly to life”).

E. Summary

The record contains extensive evidence supporting Denver’s position that it had a strong basis in evidence for concluding that the 1990 Ordinance and the 1998 Ordinance were necessary to remediate discrimination against both MBEs and WBEs. The information available to Denver and upon which the ordinances were predicated, indicated that discrimination was persistent in the local construction industry and that Denver was, at least, an indirect participant in that discrimination.

Before enacting the 1990 Ordinance, Denver was informed by the GAO that the DPW was likely not complying with federal affirmative-action requirements. A year after the GAO Report, the DOT threatened to withdraw federal funding for Sta-pleton International Airport because the DOT took the position that minority contractors were not being utilized at appropriate levels. The City then held public hearings at which extensive testimony was given detailing discrimination against MBEs and WBEs on local construction projects. In 1988, the City solicited written responses to questionnaires and conducted additional hearings on the issue of discrimination in the local construction industry.

The City then retained BBC to evaluate the utilization of M/WBEs on local construction projects. BBC ultimately produced the 1990 Study which showed large disparities in the utilization of MBEs and WBEs on construction projects undertaken in the Denver MSA. Further, the 1990 Study indicated that there were disparities in the utilization of M/WBEs on City projects. The anecdotal evidence collected by BBC and summarized in the 1990 Study strongly suggested that there was ongoing discrimination against MBEs and WBEs in the Denver construction industry.

Before enacting the 1998 Ordinance, Denver had the opportunity to evaluate the considerable amount of information contained in the 1995 Study. That study used both census data and information obtained from a telephone survey to examine the utilization of MBEs and WBEs in the Denver construction and professional design industries. The census data showed disparities in the utilization of MBEs and the survey data showed disparities in the utilization of MBEs and WBEs. The 1995 Study also showed that Blacks and Hispanics working in the Denver construction industry had lower rates of self-employment than Whites and that women had lower rates of self-employment than men. The disparities in self-employment rates persisted even though the 1995 Study controlled for education and length of work experience.

The 1995 Study also contained a summary of the RTD Study and the DHA Study. The RTD Study showed disparities in the utilization of M/WBE prime contractors and subcontractors in the Denver MSA construction industry. The DHA Study found some disparities in the utilization of M/WBEs on DHA projects. In addition, the separate DGS Study showed disparities in the utilization of MBEs and WBEs on DGS construction and remodeling projects.

The City then retained NERA to examine whether discrimination limited M/WBE participation in contracting. The NERA Study showed disparities in the utilization of M/WBEs in the Colorado construction industry. It also concluded that African Americans, Hispanics, and Native Americans working in the Denver MSA construction industry have lower self-employment rates than Whites. The NERA Study then used the information on self-employment rates to calculate the potential availability of MBEs if they formed construction businesses at the same rate as Whites. It then compared potential availability to the actual availability of MBEs in the Denver MSA. In the case of African American, Hispanic, and Native American prime contractors and subcontractors, potential availability exceeded actual availability.

The NERA Study also examined the earnings of self-employed minorities and women in the construction industry. The study showed disparities in the earnings of self-employed women, African Americans, Hispanics, and Native Americans. Finally, the results of a mail survey conducted by NERA showed that M/WBEs engaged in business activities experience disparate treatment at a higher rate than non-M/WBEs.

To meet its initial burden, Denver was not required to unequivocally establish the existence of discrimination nor was it required to “negate all evidence of non-discrimination.” Concrete Works II, 36 F.3d at 1530. After properly considering all relevant evidence presented by Denver at trial and the district court’s undisputed factual findings, we conclude that Denver met its initial burden of producing strong evidence of racial discrimination in the Denver construction industry. Denver has also shown that the gender-based measures were based on reasoned analysis. Moreover, although CWC does not raise the issue, we conclude that Denver had a strong basis in evidence to conclude that action was necessary to remediate discrimination against M/WBEs before it adopted both the 1990 Ordinance and the 1998 Ordinance. See Shaw, 517 U.S. at 910, 116 S.Ct. 1894 (“[T]he institution that makes the racial distinction must have had a strong basis in evidence to conclude that remedial action was necessary, before it embarks on an affirmative-action program.” (quotation omitted) (emphasis in original)).

To rebut Denver’s evidence, CWC was required to “establish that Denver’s evidence did not constitute strong evidence of such discrimination.” Concrete Works II, 36 F.3d at 1523. CWC cannot meet its burden of proof through conjecture and unsupported criticisms of Denver’s evidence. ' Rather, it must present “credible, particularized evidence.” Adarand VII, 228 F.3d at 1175. CWC has not met its burden. Its legal arguments urging us to disregard Denver’s non-goals data, marketplace data, lending discrimination study, and business formation study lack merit. Further, its criticisms of the methodologies used in those studies, albeit legitimate in some very limited circumstances, merely nip at the edges of Denver’s evidence and are insufficient to undermine the reliability of the studies. CWC hypothesized that the disparities shown in the studies on which Denver relies could be explained by any number of factors other than racial discrimination. However, it did not conduct its own marketplace disparity study controlling for the disputed variables and has presented no other evidence from which this court could conclude that such variables explain the disparities. Although CWC advanced a seemingly meritorious argument that the size and experience of M/WBEs may explain the disparities, Denver’s additional evidence persuasively dispelled that argument.

“[Wjhether a strong basis in evidence of past or present discrimination exists, thereby establishing a compelling interest for the municipality to enact a race-conscious ordinance, is a question of law.” Concrete Works II, 36 F.3d at 1522. We conclude that Denver has demonstrated that a strong basis in evidence supported its conclusion that remedial action was necessary to remedy racial discrimination in the Denver construction industry. Further, Denver has shown that the gender-based measures in the 1990 and 1998 Ordinances were based on reasoned analysis. Consequently, we conclude that Denver has shown that it has a compelling interest in remedying racial discrimination in the Denver construction industry and that it has an important governmental interest in remedying gender discrimination. CWC has failed to rebut Denver’s showing.

VII. NARROW TAILORING

Having concluded that Denver has demonstrated a compelling interest in the race-based measures and an important governmental interest in the gender-based measures, this court must now examine whether the ordinances are narrowly tailored to serve the compelling interest and are substantially related to the achievement of the important governmental interest.

Shortly after CWC filed this lawsuit in 1992, Denver moved for summary judgment. The United States District Court for the District of Colorado granted Denver’s motion. Judge Sherman G. Finesilver prepared an order which was published in the Federal Supplement. See Concrete Works I, 823 F.Supp. at 821. The court concluded that Denver had established a compelling interest in its affirmative-action plan. See id. at 841. The court also concluded that Denver’s program was narrowly tailored. 20 See id. at 844-45. CWC appealed the grant of summary judgment and that appeal culminated in our decision in Concrete Works II. This court reversed the grant of summary judgment on the compelling-interest issue and concluded that CWC had waived any challenge to the narrow tailoring conclusion reached by the district court.

Because Concrete Works does not challenge the district court’s conclusion with respect to the second prong of Croson’s strict scrutiny standard — i.e. that the Ordinance is narrowly tailored to remedy past and present discrimination — we need not address this issue. The summary allegation in amici’s brief that the ordinance is not narrowly tailored does not suffice to preserve this issue on appeal.

Concrete Works II, 36 F.3d at 1531 n. 24.

“[A] legal decision made at one stage of litigation, unchallenged in a subsequent appeal when the opportunity to do so existed, becomes the law of the case for future stages of the same litigation, and the parties are deemed to have waived the right to challenge that decision at a later time.” Capps v. Sullivan, 13 F.3d 350, 353 (10th Cir.1993) (quotation omitted); see also Demarest v. Price, 130 F.3d 922, 942 n. 9 (10th Cir.1997); Martinez v. Roscoe, 100 F.3d 121, 123 (10th Cir.1996). Other circuit courts of appeal have recognized that this rule is “necessary to the orderly conduct of litigation” and ensures that a party which fails to challenge a ruling in a first appeal does not “stand better as re gards the law of the case than one who had argued and lost.” Laffey v. Northwest Airlines, Inc., 740 F.2d 1071, 1089 (D.C.Cir.1984); Fogel v. Chestnutt, 668 F.2d 100, 109 (2d Cir.1981).

Denver argues that the district court erred when it addressed the narrow tailoring issue on remand. The law of the case doctrine, however, is “only a rule of practice in the courts and not a limit on their power.” United States v. Monsisvais, 946 F.2d 114, 116 (10th Cir.1991). Although the law of the case doctrine does not implicate a court’s jurisdiction, a district court’s authority to deviate from law of the case is circumscribed by three “exceptionally narrow” exceptions:

(1) when the evidence in a subsequent trial is substantially different; (2) when controlling authority has subsequently made a contrary decision of the law applicable to such issues; or (3) when the decision was clearly erroneous and would work a manifest injustice.

Huffman v. Saul Holdings Ltd. P’ship, 262 F.3d 1128, 1133 (10th Cir.2001) (quotation omitted). The district court did not rely on any of these exceptions when it revisited the narrow tailoring issues in Concrete Works III and CWC does not argue that any exception applies. In fact, CWC’s only response to Denver’s law-of-the-case argument is contained in a short footnote in its appellate brief. CWC then proceeds to advance what appear to be many of the same merits arguments the district court rejected when it granted summary judgment for Denver on the narrow tailoring issue in 1993. See Concrete Works I, 823 F.Supp. at 841-45.

In its footnote, CWC does assert that the two prongs of strict scrutiny are inseparable. We presume that CWC is arguing that the compelling interest analysis cannot be conducted without also conducting the narrow tailoring analysis. CWC’s argument is clearly foreclosed by our treatment of this case in Concrete Works II; we reversed the grant of summary judgment on the compelling-interest prong without addressing the narrow-tailoring prong, concluding that it had been waived. See 36 F.3d at 1531 & n. 24.

CWC also argues that the 1998 Ordinance was not before the district court when it first ruled on the narrow tailoring issues. CWC suggests that the law of the case doctrine did not bar the district court from addressing the question of whether that ordinance is narrowly tailored. This argument would appear to implicate the second exception to the law of the case doctrine, i.e., the evidence presented at trial was substantially different than the evidence before the district court when it ruled on the narrow tailoring issue the first time. CWC’s argument, however, must fail because CWC does not identify any significant or material differences between the 1990 Ordinance and the 1998 Ordinance. Further, the portion of its brief dedicated to the narrow tailoring issue contains no argument that the amendments made to the original ordinance since Judge Finesilver ruled on the issue disadvantage them to a greater degree or otherwise affect the narrow tailoring issue. Any such argument would be inconsistent with the district court’s unchallenged conclusion that Denver’s program has “been in effect since 1990 without substantial change.” Concrete Works III, 86 F.Supp.2d at 1079.

We conclude that the district court lacked authority to address the narrow tailoring issue on remand because none of the exceptions to the law of the case doctrine are applicable. The district court’s earlier determination that Denver’s affirmative-action measures are narrowly tailored is law of the case and binding on the parties.

VIII. CONCLUSION

The City has demonstrated a compelling interest in remediating racial discrimination in the Denver construction industry and an important governmental interest in remediating gender discrimination in that industry. Further the City’s affirmative-action program is narrowly tailored. Consequently, the judgment of the district court enjoining Denver from enforcing the 1998 Ordinance and declaring the 1990 Ordinance and the 1998 Ordinance unconstitutional is reversed. The case is remanded with instruction to enter judgment for Denver.

2 . CWC filed an amended complaint on April 9, 1997, adding allegations related to the 1996 Ordinance.

3 . We are not persuaded by CWC's unsupported assertion that this is "an impossible and unconstitutional burden.”

4 . The DPW is the City agency responsible for most City construction contracting.

5 . Statistics were not collected for WBEs until 1980.

6 . The 1995 Study also used census data to examine marketplace utilization of minority and women-owned firms in all industries in the Denver MSA. We agree with the district court that the relevance of these statistics is unclear. See Concrete Works III, 86 F.Supp.2d 1042, 1056 (D.Colo.2000).

7 . The district court incorrectly stated that the survey data did not include information on C corporations. See Concrete Works III, 86 F.Supp.2d at 1056. Instead, it was the Census Bureau data that did not include C corporations.

8 . The RTD Study expressed the disparity indi-ces in percentages. For purposes of consistency, we have converted the percentages to decimal form,

9 . See note 8, supra.

10 . The only aspect of procurement that did not cause greater difficulties for M/WBEs than for non-M/WBEs was previous dealings with an agency.

11 . Although CWC's closing argument contains no support for this statement, it embodies the position taken by its expert, George LaNoue, a professor of political science at the University of Maryland. LaNoue summarized his criticisms of Denver’s disparity studies by stating, ''[Tjhe ultimate test is of course whether a set of data that lacks the relevant variables can ever be used to prove discrimination, and the proof of discrimination is what I think is required here.”

12 . Like the district court’s first question, the third question appears to be derived from the erroneous statement in CWC’s written closing argument that Denver must prove “the existence of a pattern of deliberate exclusion of women and minority contractors in Denver” and that "this pattern of exclusion must be unique to Denver, caused by Denver and its contractors, and not be the result of general societal discrimination.” Although the closing argument contains no support for this statement, in its appellate brief, CWC relies on the following statement taken from a footnote in a memorandum opinion issued by the Northern District of Illinois: "Prejudiced attitudes are sometimes the personal attitudes of the supervisor rather than being reflective of the employer’s policy.” Builder’s Ass’n of Greater Chicago v. County of Cook, 123 F.Supp.2d 1087, 1114 n. 13 (N.D.Ill.2000).

13 . CWC states, without elaboration, that Ada-rand VII is “inapposite” because it involved a federal program and because a petition for a writ of certiorari was filed with the Supreme Court. CWC’s unsupported attempt to distinguish Adarand VII is wholly without merit. As an aside, we note that the writ of certiorari granted in Adarand VII was dismissed as improvidently granted. See Adarand Constructors, Inc., v. Mineta, 534 U.S. 103, 122 S.Ct. 511, 515, 151 L.Ed.2d 489 (2001).

14 . The argument in CWC’s appellate brief on this issue consists of only the following single sentence: "The Trial Court found that Denver had failed to consider the proper and major variables.” This sentence ends with a footnote which reads, "For a detailed explanation of the factors considered for variables and the controls used and why they are not meaningful or reliable, see Plaintiff’s Closing Statement.” This court is under no obligation to consider arguments not fully set forth in a party’s appellate brief, including arguments incorporated by reference to prior pleadings or other materials. See Gaines-Tabb v. ICI Explosives, USA, Inc., 160 F.3d 613, 623-24 (10th Cir.1998). We strongly disapprove of this practice not only because it hinders this court’s ability to review the merits of the argument, but it unfairly allows the party to avoid the page and word limitations imposed on appellate filings. In this case, the deficiencies in CWC's appellate brief are particularly problematic because the closing statement, to which this court is directed, itself incorporates arguments made in another separate document. This practice is improper under the rules of appellate procedure.

15 . Dr. Evans’ testimony was given in response to the following question posed during his cross-examination by CWC: "[A]re MWBEs generally in the same construction specialities as non-MWBEs in the Denver area?”

16 . A municipality's study can be challenged by demonstrating that it did not control for a variable that has a negative effect on the qualifications, willingness, or abilities of M/WBEs. The municipality would then be required to show that the variable does not effect the M/WBEs' availability or that the variable itself is not race- or gender-neutral. This is precisely the situation presented in this case.

17 . There was no analogous stipulation with respect to the utilization of WBEs.

18 . CWC’s argument relating to the City’s alleged overutilization of MBEs is limited to CWC’s challenge to Denver's production of strong evidence. It does not argue that the stipulation is relevant to the narrow tailoring analysis.

19 . The GAO Report contained a table showing the utilization of minority contractors on federally funded contracts awarded by the DPW between July 1, 1975 and December 31, 1977.

20 . Although the district court acknowledged that the gender-based remedial measures in Denver’s plan should be subject to intermediate scrutiny, it analyzed those measures under the strict-scrutiny standards. See Concrete works I, 823 F.Supp. at 829 (”[A]n ordinance surpasses or fails constitutional standards in light of the most stringent applicable level of scrutiny.”).