2019 UT App 39
THE UTAH COURT OF APPEALS
CALIFORNIA COLLEGE INC., STEVENS-HENAGER COLLEGE INC., COLLEGEAMERICA SERVICES INC., COLLEGEAMERICA DENVER INC., AND COLLEGEAMERICA ARIZONA INC., Appellees, v. UCN INC. AND INCONTACT INC., Appellants.
Opinion No. 20160120-CA Filed March 21, 2019
Third District Court, Salt Lake Department The Honorable Heather Brereton No. 090907053
David M. Bennion, Zack L. Winzeler, and Alan S. Mouritsen, Attorneys for Appellants Robert E. Mansfield and Steven J. Joffee, Attorneys for Appellees
JUDGE MICHELE M. CHRISTIANSEN FORSTER authored this Opinion, in which JUDGES KATE APPLEBY and DAVID N. MORTENSEN concurred.
CHRISTIANSEN FORSTER, Judge:
¶1 This is an interlocutory appeal asking us to determine whether the district court abused its discretion when it denied inContact’s pretrial motion to exclude certain expert witness testimony. InContact contends that the data set underlying the experts’ opinions at issue failed to meet the standards of reliability set forth by rule 702 of the Utah Rules of Evidence. Because both expert opinions admitted by the court relied on data the parties agree was, to an extent, unreliable, we determine that a threshold showing of reliability was not shown. California College v. UCN
Accordingly, we vacate the district court’s ruling denying inContact’s motion to exclude and remand for further proceedings.
BACKGROUND
¶2 The appellant in this case is inContact Inc., formerly known as UCN Inc. We use its current name for clarity. The appellees are a consortium of educational entities including California College Inc., Stevens-Henager College Inc., CollegeAmerica Services Inc., CollegeAmerica Denver Inc., and CollegeAmerica Arizona Inc. 1 For convenience, we refer to them collectively as “the College,” mindful that the appellees are separate entities and without ascribing any legal status to the term.
¶3 InContact provided enhanced telephone services to assist the College in recruiting students. From February 2006 to April 2007 (Damage Period), the College’s offices “were frequently unable to receive inbound calls,” and “inbound calls were often incorrectly routed, delayed, or simply lost.” These problems allegedly left the College without full telephone capabilities for a total of approximately 108 hours.
¶4 In December 2009, the College filed a complaint against inContact, alleging multiple causes of action. The College sought damages equal to its lost profits, asserting that the telephone problems led to lower recruitment, which in turn led to reduced student enrollment and therefore lost profits. The College retained two experts with experience in calculating damages:
1. California College Inc. does business as California College for Health Sciences and Independence University.
20160120-CA 2 2019 UT App 39 California College v. UCN
Ted Tatos, a statistician, and Richard Hoffman, a certified public accountant.
¶5 Tatos estimated the number of student enrollments that the College lost during the Damage Period. Tatos used a method known as regression analysis to develop a statistical model for estimating the number of enrollments the College should have had but for the malfunctioning telephone services. 2 To conduct the regression analysis and develop the model, Tatos relied upon data collected by the College. The data reflected, among other things, how many potential students had contacted the College, how many interviews were conducted, and how many students had enrolled. Tatos compared the figures from January 2003 to March 2006 (before the Damage Period) and June 2007 to December 2010 (after the Damage Period) with the figures from the Damage Period. These figures ostensibly were drawn from the College’s internal operating reports (OPS Reports). Using his regression model, Tatos estimated that the College suffered 1,254 lost student enrollments as a result of the faulty telephone services.
¶6 Hoffman used Tatos’s estimate of lost enrollments to calculate the College’s lost profits. He first determined the average net tuition dollars the College received for each student who enrolled during 2006 (roughly $19,000) and subtracted the costs the College would have incurred to generate that tuition (roughly $3,300). Hoffman then multiplied this result by the
2. Regression analysis is a statistical tool concerned with estimating the relationship between different variables and the influence of one variable on other variables. See Reed Constr. Data Inc. v. McGraw-Hill Cos., 49 F. Supp. 3d 385, 396–401 (S.D.N.Y. 2014) (providing a detailed explanation of regression analysis). For example, regression analysis may be used to test the predictive strength of snowfall on ski tourism.
20160120-CA 3 2019 UT App 39 California College v. UCN
number of lost enrollments calculated by Tatos and arrived at approximately $19.7 million in lost profits. 3 Hoffman’s original report proffered these analyses and his damages estimate.
¶7 InContact moved to exclude Tatos and Hoffman as expert witnesses. It submitted two rebuttal expert reports—one from Patrick Kilbourne, a forensic accountant, and one from Steven Waters, an economist. Kilbourne’s report cast doubt on the accuracy of the data set Tatos used, while Waters’s report challenged the reliability of Tatos’s statistical model based upon the data.
¶8 Kilbourne first noted that the values assigned to the variables Tatos used were incorrect. Instead of relying on the actual OPS Reports, Tatos used a “summary” prepared by Carl Barney, one of the owners of one of the colleges. Barney drew some values from the OPS Reports but also supplemented those with his own estimates. It was Barney’s “summary,” prepared specifically for the purpose of the lawsuit, that had been provided to Tatos and upon which Tatos based his regression analysis and the resulting statistical model. Kilbourne also identified some of the flaws in the OPS Reports. Because the OPS Reports each included twelve to eighteen months of rolling data, a particular month appeared in multiple reports and the numbers reflected for that month could vary for a variety of reasons. For example, the OPS Reports that included January 2007 recorded the number of interviews conducted during that month variously as 582, 594, 622, 631, and 654. Tatos used the lowest value in his analysis, without explanation, and Kilbourne
3. InContact notes that this amount of lost profits exceeds the amount of the College’s actual annual profits before and during the Damage Period. In 2004, before the Damage Period, the College’s profits were $4,263,322. In 2005, the profits were $1,223,191 and in 2006, the profits were $2,453,204.
20160120-CA 4 2019 UT App 39 California College v. UCN
criticized this approach because it resulted in a model that maximized the estimated shortfall. Moreover, Kilbourne noted that many of the numbers provided by Barney and used by Tatos did not match any of the OPS Reports. 4
¶9 Waters’s report first rebutted the concept of lost enrollments, suggesting that the changing unemployment rate produced lower enrollments—a variable not accounted for in Tatos’s model. Had Tatos incorporated the unemployment rate as a variable, Waters suggested, Tatos would have found that the data predicted no lost enrollment. Waters then opined that Tatos’s model was overly sensitive to changes in the input values. To demonstrate this shortcoming, Waters plugged different sets of values into Tatos’s statistical model. Waters first used values from the most recent OPS Reports, noting that the College’s lawyers had described those figures as “the ‘most accurate and up-to-date’ data.” Waters concluded that, “[b]y simply using [these values] instead of the [values] used by Mr. Tatos, I find that Mr. Tatos’s model estimates [lost enrollments] of 588—a decrease of over 53% from 1,254—and this is without accounting for the unemployment rate.” Waters also applied two other sets of numbers to Tatos’s model: Tatos’s data through 2007, which resulted in a calculation of 358 lost starts, and the OPS report data through 2007, which resulted in a calculation of 275 lost starts. In other words, Waters opined that (1) Tatos’s model was flawed because it did not account for any change in the unemployment rate; and (2) even if not flawed in this
4. On appeal, the College concedes that “the initial data provided to [Tatos and Hoffman] was mistakenly not derived entirely from OPS Reports and included certain inaccurate information.” For example, the numbers recorded in Barney’s summary only matched the OPS Reports for two of the forty-six months between February 2005 and December 2008.
20160120-CA 5 2019 UT App 39 California College v. UCN
manner, Tatos’s model was too sensitive because it generated drastically different results when the input values were changed. 5
¶10 In response, the College submitted an updated expert report from Hoffman. Hoffman’s updated report acknowledged that the values used in his original report were inaccurate. In his updated report, Hoffman noted that the College had “decided that the data used by Dr. Waters provided the most reasonable estimate of” the inquiry, interview, and enrollment figures. Hoffman therefore performed his calculations anew, starting with 588 lost enrollments and multiplying that by the same net per-student profit margin he had calculated before. In essence, Hoffman utilized the same numbers Waters had used to show that Tatos’s model was oversensitive, but neither Waters nor Hoffman had independently verified the data set by checking against the raw numbers. On this basis, Hoffman re-estimated the College’s lost profits at roughly $9.2 million. 6
5. InContact asserts, and we agree, that Waters was not opining that the newest OPS Reports were the most accurate or that Tatos’s model, based on a regression analysis of inaccurate numbers, could generate accurate results so long as the right inputs were used.
6. Several months after Hoffman’s updated report, in response to inContact’s Statement of Discovery Issues filed with the court seeking the production of the OPS Reports, the College asserted, “After reviewing its OPS reports, as well as the student data identified by Dr. Waters as coming from the most recent OPS reports, [the College] determined that the data pulled from the OPS reports by Dr. Waters, which reflects monthly student data from the most recent OPS reports, reflects the most accurate and (continued…)
20160120-CA 6 2019 UT App 39 California College v. UCN
¶11 After Hoffman submitted his updated report, both Kilbourne and Waters updated their expert reports as well. InContact’s experts continued to disagree with the analysis and conclusions reached by Hoffman. Waters asserted, “[T]he analysis I put forward in [my first] report was done for the sole purpose of demonstrating the sensitivity of [the College’s] findings. I did not, and still do not, consider [the College’s expert’s] model to be statistically reliable.” Similarly, in his updated report, Kilbourne asserted,
[B]oth Dr. Waters and I identified data contained in [the OPS Reports] generated by [the College] in the normal course of their business that was substantially different from the data supplied to and relied upon by Mr. Hoffman. I did not opine, and I do not understand that Dr. Waters opined, that either data set was accurate or inaccurate, but rather that [the College] had produced multiple documents with conflicting data.
Kilbourne concluded that the integrity and accuracy of the data set relied on by both Tatos and Hoffman was questionable and that they had relied on irrelevant data in their analyses.
¶12 InContact moved to exclude Hoffman and Tatos as expert witnesses, along with their respective opinions. The district court held a hearing on the motion and noted inconsistencies in the data provided to and relied on by Tatos, stating, “[T}he regression analysis conducted by Mr. Tatos was not conducted reliably in this case because it was based on flawed data and insufficient data under Rule 702(b).” See Utah R. Evid. 702(b).
(…continued) reasonable data upon which to analyze [the College’s] operating metrics.”
20160120-CA 7 2019 UT App 39 California College v. UCN
The district court therefore “exclude[d] any testimony based upon the original data, including testimony from Mr. Tatos and testimony from Mr. Hoffman to the extent he relied upon Mr. Tatos’[s] work with the unreliable data.” 7
¶13 Following the district court’s order excluding the expert testimony based upon Tatos’s original data, the case was assigned to a different judge who revisited that decision. Although no transcript of this hearing was included in the record on appeal, the parties agree the court sua sponte determined that it would reconsider the motion to exclude Tatos and Hoffman from trial. Two days later, inContact filed an objection to the court’s decision to reconsider. On December 11, 2015, the court again held a hearing regarding the motion to exclude the College’s expert witnesses.
¶14 At that hearing, inContact argued that “the data on which [the College] relied both in the first [report], but even in the second [report], is unreliable data.” InContact highlighted Tatos’s reliance on Barney’s summary of the OPS Reports rather than the actual reports, but it also asserted that the underlying OPS Reports were “inherently unreliable.” As an example, inContact noted that the data for a given month varied from report to report: “Then it happens again and again and again; and the numbers vary wildly. So those numbers become really problematic for making any kind of a reliable assessment. Which number do you take? Which operations report do you take [the number from]” to represent a particular month?
7. The court did not order Hoffman’s exclusion as an expert witness. It determined that Hoffman’s updated report, which relied in part on the values Waters used to discredit the initial report, was sufficiently reliable to be admitted into evidence. We address the apparent contradiction later in this opinion. Infra ¶¶ 27–29.
20160120-CA 8 2019 UT App 39 California College v. UCN
¶15 InContact also explained that its expert—Waters—lacked access to all of the College’s data, and therefore he relied on the value reported in the most recent OPS Report for a given month. InContact stressed that Waters had not vouched for the reliability of the data and characterized his report as unreliable: “‘Even if you do [the analysis] with the numbers that [the College]” says are “the most reliable numbers, it’s either 588, 358, or 275 [lost starts.]’” And inContact pointed out that Waters’s final opinion was that there were zero lost starts. Thus, according to inContact, both Tatos’s and Hoffman’s reports were based on inherently unreliable raw data.
¶16 The court conceded that “[i]n a case like this with business records kept the way they’re kept, you’re never going to have a perfect number, it seems, because of the rolling nature of the operation reports” but noted that “we don’t have to have perfect numbers.” The court stated that it could not rule on “whose regression analysis and methodology was better” because that was “an issue for the fact finder.” Instead, the court framed the issue as whether the expert opinions of Tatos and Hoffman were reliable and, if not, the unreliability was “because [the experts] didn’t use good data.”
¶17 The court eventually ruled that inContact’s arguments “go to the weight and credibility to be given to the opinions of [the College’s] expert witnesses, not to the admissibility of those opinions.” The court found that “the facts and data upon which [the College’s] expert witnesses have relied are sufficiently reliable to satisfy the minimal threshold standard imposed by Rule 702 [of the Utah Rules of Evidence]. Although the facts and data may not be perfect, perfection is not required to satisfy Rule 702’s threshold standard.” The court noted that it was the jury’s role, not the court’s, to determine the weight and credibility of the experts’ opinions flowing from the data. The court therefore vacated its earlier order excluding Tatos’s testimony, denied the motion to
20160120-CA 9 2019 UT App 39 California College v. UCN
exclude Tatos’s and Hoffman’s testimony, and ruled that Tatos and Hoffman would be permitted to offer expert testimony during the trial. InContact sought interlocutory review, which this court granted.
ISSUE AND STANDARD OF REVIEW
¶18 InContact contends that, pursuant to rule 702 of the Utah Rules of Evidence, the district court erred or abused its discretion in not excluding Tatos’s and Hoffman’s opinions because those opinions were not based upon sufficient facts and data. We review a district court’s decision as to the admissibility of expert witness testimony for an abuse of discretion and will not reverse that decision unless it exceeds the limits of reasonability. ConocoPhillips Co. v. Utah Dep’t of Transp., 2017 UT App 68, ¶ 12, 397 P.3d 772.
ANALYSIS
¶19 The district court initially excluded the expert opinion testimony of Tatos and Hoffman as “flawed” and “insufficient” to the extent those opinions rested on the results of Tatos’s model based on his original data. See Utah R. Evid. 702(b). Following assignment of the case to a different judge, the district court reconsidered this decision and reversed course in a second ruling, determining instead that these expert opinions—though imperfect—would be admitted.
¶20 As an initial matter, we note that the district court’s reconsideration of the expert witness issue was not inappropriate. “While a case remains pending before the district court prior to any appeal, the parties are bound by the court’s prior decision, but the court remains free to reconsider that decision.” IHC Health Services, Inc. v. D & K Mgmt., Inc., 2008 UT
20160120-CA 10 2019 UT App 39 California College v. UCN
73, ¶ 27, 196 P.3d 588. And, two district court judges presiding over the same case, “while different persons, constitute a single judicial office.” In re R.B.F.S., 2012 UT App 132, ¶ 12, 278 P.3d 143 (quotation simplified). Thus, a district court judge may “revisit[] a previously decided issue during the course of a case, regardless of whether the judge has changed or remained the same throughout the proceedings” so long as the issue has not been raised before and ruled on by an appellate court. Mid- America Pipeline Co. v. Four-Four, Inc., 2009 UT 43, ¶ 11, 216 P.3d 352.
¶21 We begin with the district court’s second ruling: “The Court finds that the facts and data upon which [the College’s] expert witnesses have relied are sufficiently reliable to satisfy the minimal threshold standard imposed by Rule 702.” The expert witnesses referred to were Tatos and Hoffman. Because the only numbers used by Tatos were drawn from Barney’s summary, we understand the court’s statement to mean that non-expert, party- representative Barney’s summary of the data, or a subset of that summary, prepared for this lawsuit, was a sufficient basis for the experts’ testimony.
¶22 Rule 702(b) describes when certain expert opinion is admissible:
Scientific, technical, or other specialized knowledge may serve as the basis for expert testimony only if there is a threshold showing that the principles or methods that are underlying in the testimony
(1) are reliable,
(2) are based on sufficient facts or data, and
(3) have been reliably applied to the facts.
20160120-CA 11 2019 UT App 39 California College v. UCN
Utah R. Evid. 702(b). “Importantly, [rule 702] require[s] the plaintiff to make only a threshold showing of reliability.” Eskelson ex rel. Eskelson v. Davis Hosp. & Med. Center, 2010 UT 59, ¶ 12, 242 P.3d 762 (quotation simplified). This threshold “is not so rigorous as to be satisfied only by [methodology or data] that are free of controversy.” Id. (quotation simplified); cf. State v. Woodard, 2014 UT App 162, ¶ 26, 330 P.3d 1283 (holding that one method of fingerprint analysis was sufficiently reliable to be admitted into evidence despite the existence of several competing methodologies and the lack of a nationwide standard method). While an expert may rely on his or her own interpretation of data that have a foundation in the evidence, even if the data is in dispute, the expert “cannot give opinion testimony that flies in the face of uncontroverted” facts or data. Eskelson, 2010 UT 59, ¶ 16 (quotation simplified). Thus, while a range of values for a given variable may be admissible under rule 702, there must be some evidence underpinning the values used by the expert.
¶23 Applying rule 702, district courts “act as gatekeepers to screen out unreliable expert testimony.” State v. Lopez, 2018 UT 5, ¶ 20, 417 P.3d 116 (quotation simplified). The court, “view[ing] proposed expert testimony with rational skepticism,” id. (quotation simplified), must determine whether the proponent has met the threshold showing that the method used is “‘generally accepted by the relevant expert community’” or that the “principles underlying [the expert’s] testimony are ‘reliable, based upon sufficient facts or data, and have been reliably applied to the facts,’” id. ¶ 22 (ellipses omitted) (quoting Utah R. Evid. 702).
¶24 As applied here, the data used by Tatos and Hoffman to generate their opinions must have some foundation indicating its reliability before the court may admit the data, the models developed from that data, or the interpreted results of the models. However, the College concedes that “the initial data
20160120-CA 12 2019 UT App 39 California College v. UCN
provided to [Tatos and Hoffman] was mistakenly not derived entirely from OPS Reports and included certain inaccurate information.” And on appeal, the College does not direct us to any evidence that the original data set used by Tatos to create his regression model was somehow reliable despite including inaccurate information.
¶25 The district court’s ruling states that the data set given to Tatos was reliable. Now on appeal, however, even the proponent of Tatos’s testimony admits that the data set “included certain inaccurate information.” Because the College admits that the data given to Tatos contained some inaccurate values and did not identify any evidence rehabilitating the balance of those data, we cannot endorse the district court’s conclusion that the data met the threshold of reliability required by rule 702. And Tatos does not claim that he in any way independently verified, even minimally, the accuracy of the data set Barney originally supplied. We therefore reject the district court’s conclusion that simply assuming the reliability of Barney’s summary endows the data with any degree of actual reliability. See, e.g., Maddox v. Claytor, 764 F.2d 1539, 1552 (11th Cir. 1985) (“If the tested disparity is based on erroneous assumptions or suffers from flaws in the underlying data, then standard deviation analysis is foredoomed to yield an equally faulty result.”).
¶26 Further, Tatos used unverified and summary data provided by Barney to create his regression model. The results of that analysis were also admitted by the district court. But the nature of regression analysis is such that a regression model based on flawed data will itself be flawed, because a “best fit” model that uses erroneous data may not reasonably approximate the actual values. See generally Franklin M. Fisher, Multiple Regression in Legal Proceedings, 80 Colum. L. Rev. 702, 704, 707 (1980) (discussing the process of regression analysis); Kevin Gilmartin & Elizabeth Hartka, Using Regression Analysis to Compute Back Pay, 31 Jurimetrics J. 289, 295–
20160120-CA 13 2019 UT App 39 California College v. UCN
96, 298 (1991) (noting, as a pitfall of regression analysis, the design risk presented by “inclusion of tainted variables” and the fact that “an ill-designed regression model could result in an even more inequitable distribution of relief”); Michael J. Saks et al., 179 Annotated Reference Manual on Scientific Evidence (2d ed.), Reference Guide on Multiple Regression, at *15–16 (noting that multiple regression analysis assumes that the variables have been measured accurately and that inaccurate data will decrease the reliability of the results). The College never explained why Tatos’s model, developed from erroneous and unreliable data, should nevertheless be considered reliable, even by the low bar set by rule 702. Rule 702 requires an expert’s opinion to rest on “sufficient facts and data.” Utah R. Evid. 702(b). Accordingly, we conclude that it was an abuse of discretion for the district court to admit Tatos’s regression model and its results, because the model was developed from “certain inaccurate information.”
¶27 We next consider whether Hoffman’s opinions are admissible. 8 Hoffman’s initial opinion had two operative parts. First, Hoffman calculated the value of each lost start. Second, he calculated the total amount lost by multiplying the number of lost starts—estimated by Tatos to be 1,254—by the estimated amount lost per-lost-start. Hoffman later modified his damages opinion by substituting the number of lost starts from Waters’s rebuttal opinion—588—for Tatos’s original number. Because
8. The district court did not directly rule upon whether Hoffman’s updated expert report was admissible. Instead, it determined that the data on which it relies—Waters’s calculation—was sufficiently reliable. Because we conclude otherwise, we address the admissibility of Hoffman’s updated report in order to provide direction to the district court and parties on remand.
20160120-CA 14 2019 UT App 39 California College v. UCN
Hoffman relied on Waters’s results, we must also consider Waters’s conclusion.
¶28 Waters ran moderately different, hypothetical data through Tatos’s statistical model and arrived at a number of lost starts that is less than half of Tatos’s result. Due to the widely divergent results (1,254 and 588), Waters concluded that Tatos’s statistical model itself was not reliable as it projected more dramatic results with only moderate changes to the input data. This also supported inContact’s broader assertion that the original data did not support the proposition that the phone system caused any lost starts. Neither Waters nor inContact presented foundational evidence supporting the reliability of Tatos’s statistical model. Indeed, the purpose for Waters’s inputting new numbers through the model was to establish the model’s complete unreliability. In modifying his expert opinion, Hoffman simply adopted the number of lost starts from Waters’s calculation without any explanation other than acknowledging Waters had used them. No foundational evidence appears to have been presented regarding the reliability of Hoffman’s opinion—specifically the statistical model’s result that informs his conclusion—other than Hoffman’s own statement that Waters’s report provided “the most reasonable estimate” of the relevant data.
¶29 Because nothing in the record establishes that Hoffman has any expertise in regression analysis, there is no basis to conclude that Hoffman could determine whether Waters’s report was a reasonable estimate, let alone “the most reasonable estimate.” Indeed, Hoffman fails to acknowledge that Waters in no way endorsed the data he utilized as reliable. Instead, for only rhetorical purposes, Waters used data the College’s lawyers—not its experts—had suggested was better than the previously inaccurate data. Accordingly, given the paucity of foundational support, we must conclude that the district court erred in determining that the College met the threshold showing
20160120-CA 15 2019 UT App 39 California College v. UCN
of reliability required under rule 702(b) to admit Hoffman’s expert opinion.
¶30 When the district court revisited its ruling, it determined that the second set of data—the set of numbers used by Waters in the rebuttal opinion showing the oversensitivity of Tatos’s model—was sufficiently reliable to meet the threshold requirement under rule 702. Accordingly, the court admitted, without limitation, the expert opinion testimony of Tatos and Hoffman. This was error. The court’s ruling implied that any regression analyses, models, or results based upon Waters’s data critical of Tatos’s model, satisfied rule 702. But, as we have explained, the court’s determination that the data set was admissible pursuant to rule 702 was an abuse of discretion. It follows that it was improper to admit an expert opinion whose factual basis relied on the flawed data. Although the district court did not directly rule upon the admissibility of Hoffman’s updated expert opinion, Hoffman’s updated opinion is based upon flawed data and a correspondingly flawed statistical model. As a result, that opinion should have been excluded.
CONCLUSION
¶31 The district court abused its discretion by admitting Tatos’s expert opinion and Hoffman’s initial expert opinion, because those opinions were developed from a data set both parties acknowledge is unreliable. The district court also erred when it determined that the second set of numbers—Waters’s assumed data—was sufficiently reliable to meet the threshold requirement of rule 702. We therefore vacate the district court’s order and remand for further proceedings consistent with this opinion.
20160120-CA 16 2019 UT App 39