Concord Music Group, Inc. v. Anthropic PBC

• Court: District Court, N.D. California
• Decided: May 23, 2025
• Docket: 5:24-cv-03811
• Status: Unknown

Opinion

1 2 3 4 UNITED STATES DISTRICT COURT 5 NORTHERN DISTRICT OF CALIFORNIA 6 7 CONCORD MUSIC GROUP, INC., et al., Case No. 24-cv-03811-EKL (SVK)

8 Plaintiffs, ORDER ON JOINT DISCOVERY 9 v. SUBMISSIONS

10 ANTHROPIC PBC, Re: Dkt. Nos. 328, 341, 345 11 Defendant.

12 Before the Court are the Parties’ Joint Discovery Submissions regarding three discovery 13 disputes. First, Defendant Anthropic PBC (“Anthropic”) seeks to compel Plaintiffs (“Publishers”) 14 to disclose and produce currently undisclosed prompts to and outputs from Claude (and related 15 settings) that Publishers and their agents submitted and obtained in the course of their legal 16 investigation. Dkt. 328 at 2 (relating to Interrogatory no. 1 and Requests for Production (“RFP”) 17 nos. 37-43). Second, the Parties disagree as to the appropriate scope of the sampling protocol in 18 response to this Court’s prior order. Dkt. 341; see also Dkt. 318. Third, Publishers challenge 19 certain confidentiality designations made by Anthropic as overbroad and improper. Dkt. 345. 20 These matters came on for hearing on May 13, 2025. Having considered the Parties’ arguments, 21 the relevant law and the record in this action, the Court DENIES without prejudice Anthropic’s 22 request to compel production of Publishers’ prompts, outputs, and settings; ORDERS 23 production of a sample of 5 million prompt-output pairs according to the protocol set forth 24 below; and SUSTAINS-IN-PART and OVERRULES-IN-PART Publishers’ challenge to 25 Anthropic’s confidentiality designations. 26 //// 27 //// 1 I. DKT. NO. 328: ANTHROPIC’S MOTION TO COMPEL PRODUCTION OF PUBLISHERS’ UNDISCLOSED PROMPTS AND OUTPUTS AND THE 2 SETTINGS USED THEREFORE 3 A. Legal Standard 4 Federal Rule of Civil Procedure 26 protects disclosure of documents and tangible things 5 “prepared in anticipation of litigation or for trial.” Fed. R. Civ. P. 26(b)(3)(A); see In re Grand 6 Jury Subpoena (Mark Torf/Torf Env't Mgmt.), 357 F.3d 900, 906 (9th Cir. 2004). This includes 7 attorney work product, which may be either fact work product or opinion work product. 8 “[O]pinion work product” includes “an attorney's mental impressions, conclusions, opinions, or 9 legal theories developed in anticipation of litigation.” Republic of Ecuador v. Mackay, 742 F.3d 10 860, 869 n.3 (9th Cir. 2014); see Fed. R. Civ. P. 26(b)(3)(B). It “is virtually undiscoverable.” Id. 11 However, “[t]he privilege derived from the work-product doctrine is not absolute. Like 12 other qualified privileges, it may be waived.” United States v. Sanmina Corp., 968 F.3d 1107, 13 1119 (9th Cir. 2020). “Similar to the waiver of the attorney-client privilege, a litigant can waive 14 work-product protection to the extent that he reveals or places the work product at issue during the 15 course of litigation.” Id. In other words, under the “fairness principle,” a party cannot “us[e] the 16 privilege as both a shield and a sword.” Id. at 1117 (explaining the fairness principle in context of 17 attorney-client privilege). 18 B. Discussion 19 In this case, Publishers conducted a pre-suit investigation into potential infringement by 20 Anthropic and relied upon certain prompts and outputs from their investigation in support of their 21 allegations of infringement. See Dkt. 1; Dkt. 337-2 (Ex. B); Dkt. 369 at 1-2. Publishers 22 represent that they have produced all prompts and outputs on which they have relied, totaling 23 nearly 5,000 prompt-output pairs. Dkt. 369 at 2-3. Anthropic has now served broad discovery 24 requests directed to all prompts and outputs including those not relied upon by Publishers – in 25 other words, those that presumably did not support claims of infringement. See Dkts. 328-1–2 26 (Interrogatory no. 1 and RFP nos. 37-43). In the joint statement, Anthropic argues that either (a) 27 the unrelied-upon prompts/outputs are not privileged or (b), if privileged, that the privilege has 1 Anthropic’s initial argument, that the information it seeks (undisclosed prompts and 2 outputs, and the settings therefore) is not privileged is unpersuasive. Publishers cite cases where 3 courts, including in this District, have found precisely this information to constitute attorney work 4 product. E.g., Tremblay v. OpenAI, Inc., No. 23-cv-03223-AMO, 2024 WL 3748003, at *2-*3 5 (N.D. Cal. Aug. 8, 2024) (explaining that the underlying magistrate judge’s order had found “that 6 the account settings and negative test results [were] fact work product [but] that Plaintiffs waived 7 the ability to assert work product” and holding that this was a misapplication of law because “the 8 ChatGPT prompts were queries crafted by counsel and contain counsel’s mental impressions and 9 opinions about how to interrogate ChatGPT” and were thus opinion, not fact, work product). 10 Anthropic distinguishes only Tremblay’s denial of waiver, based on the plaintiffs’ use of outputs 11 there as being unlike this case, but does not distinguish the basic finding that the failed prompts 12 and related settings are attorney work product. See Dkt. 328 at 5. This Court agrees with 13 Tremblay in that regard. 14 The closer issue is the extent to which Publishers have waived the attorney work product 15 protection. Publishers admit that they have relied on certain prompts and outputs in their First 16 Amended Complaint (Dkt. 337) and in various other filings, including their original and renewed 17 motions for preliminary injunction and supporting declarations but also represent that they have 18 produced all prompts and outputs relied upon. See Dkt. 369 (“Publishers have produced all 19 Claude prompts and outputs from their investigation on which they have relied upon to date. In 20 total, Publishers have produced approximately 4,659 Claude prompt/output records….”); see also 21 Dkt. 372 (“Hrg. Tr.”) at 50:16-51:18. Under the “fairness principle” or sword-and-shield doctrine, 22 there has been at least a limited waiver here; indeed, Publishers have produced nearly 5,000 23 prompt-output pairs upon which they rely. The issue is how far the waiver reaches. 24 The Ninth Circuit has made clear that “the scope of [a work product] waiver must be 25 ‘closely tailored ... to the needs of the opposing party’ and limited to what is necessary to rectify 26 any unfair advantage gained….” Sanmina, 968 F.3d at 1124. Anthropic claims waiver by 27 pointing to Publishers’ “ease of use” or “massive use” allegations in the complaint – for example, 1 scale in order to enable those models to generate responses to user prompts that infringe 2 Publishers’ copyrighted lyrics.” Dkt. 1, ¶ 122; see Dkt. 328 at 1, 4 (citing Dkt. 1, ¶¶ 8, 108, 122). 3 Anthropic has argued that it needs access to unrelied-upon prompts and outputs in order to rebut 4 such allegations. Dkt. 328 at 4 (“This argument fails if those successes required hundreds of 5 unsuccessful attempts to coax purportedly infringing outputs from Claude. Anthropic thus has a 6 compelling need for the full picture.”). But Publishers’ prompts and outputs are cited only as 7 examples of infringement; at this early stage, it remains unclear what evidence Publishers will 8 rely on to prove the “ease of use” or “massive use” allegations. See Dkt. 337, ¶¶ 8, 156, 170 9 (paragraphs corresponding to ¶¶ 8, 108 and 122 of the original complaint). Thus, at this stage of 10 the litigation, Anthropic’s discovery requests and the attendant waiver argument are overbroad. 11 RFP no. 38, for example, requests “Documents sufficient to show each of the prompts 12 [Publishers] entered into Claude … regardless of whether the intent was to generate allegedly 13 infringing, including … the user or organization account used, any system prompts and/or 14 temperature setting sued, and the date and time on which the prompt was entered.” This calls for 15 a sweeping subject-matter waiver for all unrelied-upon prompts and outputs and goes too far 16 regardless of whether the prompts, settings and corresponding output are fact work product or 17 opinion work product. Thus, given the broad nature of Anthropic’s requests, requiring the 18 production of all undisclosed prompts, settings and outputs is neither “closely tailored” to 19 Anthropic’s needs nor limited to what is necessary under the fairness principle. See Sanmina, 968 20 F.3d at 1124. 21 This Court therefore DENIES Anthropic’s requests without prejudice to Anthropic 22 seeking to discover, as necessary, specific facts supporting specific contentions or opinions 23 disclosed in the course of the litigation. 24 //// 25 //// 26 //// 27 //// 1 II. DKT. NO. 341: SAMPLING PROTOCOL FOR RFP NOS. 50-51 2 This dispute arises from Publishers’ requests for prompts and outputs from Claude 3 products that relate to song lyrics, for which this Court previously ordered that a “statistically 4 significant sample would be proportional to the needs of the case.” See Dkts. 306, 318. The 5 Parties have submitted competing proposals along with expert declarations, and the Court has had 6 an opportunity to question the experts. See Dkts. 341, 341-2, 352, 372. 7 At the outset, the Court notes that during the hearing, Publishers asked this Court to 8 examine Anthropic’s expert, Ms. Chen and strike her declaration because at least one of the 9 citations therein appeared to have been an “AI hallucination”: a citation to an article that did not 10 exist and whose purported authors had never worked together. See Hrg. Tr. at 7:11-11:25. The 11 Court gave Anthropic time to investigate the circumstances surrounding the challenged citation. 12 Having considered the declaration of Anthropic’s counsel and Publishers’ response (Dkts. 371, 13 373), the Court finds this issue is a serious one—if not quite so grave as it at first appeared. 14 Anthropic’s counsel protests that this was “an honest citation mistake” but admits that 15 Claude.ai was used to “properly format” at least three citations and, in doing so, generated a 16 fictitious article name with inaccurate authors (who have never worked together) for the citation at 17 issue. Dkt. 371, ¶¶ 3, 6. That is a plain and simple AI hallucination. Yet the underlying article 18 exists, was properly linked to and was located by a human being using Google search; so, this is 19 not a case where “attorneys and experts [have] abdicate[d] their independent judgment and critical 20 thinking skills in favor of ready-made, AI-generated answers….” Contra Dkt. 373 at 2 (quoting 21 Kohls v. Ellison, No. 24-cv-3754 (LMP/DLM), 2025 WL 66514 at *4 (D. Minn. Jan. 10, 2025). 22 A remaining serious concern, however, is Anthropic’s attestation that a “manual citation check” 23 was performed but “did not catch th[e] error.” Dkt. 372, ¶ 6. It is not clear how such an error— 24 including a complete change in article title—could have escaped correction during manual cite- 25 check by a human being. Furthermore, although the undersigned’s standing order does not 26 expressly address the use of AI by parties or counsel, Section VIII.G of Judge Lee’s Civil 27 Standing Order requires a certification “that lead trial counsel has personally verified the content’s 1 verification has occurred here. In sum, the Court STRIKES-IN-PART Ms. Chen’s declaration, 2 striking paragraph 9, and notes for the record that this issue undermines the overall credibility of 3 Ms. Chen’s written declaration, a factor in the Court’s conclusion. 4 Turning to the substance of the dispute, based upon the Parties’ agreed upon parameters, 5 the Court adopts the values of a 95% confidence level corresponding to a “Z-score” of 1.96 and an 6 expected prevalence of 0.00006. Dkt. 352, ¶ 12; Dkt. 341-2, ¶¶ 5, 12. The disagreement between 7 the Parties is with respect to the margin of error: Anthropic offers a 25% margin of error, while 8 Publishers would require a 5% margin. Dkt. 3, 7. During the hearing, the Parties’ experts testified 9 that there was room within the 5 to 25% (Publisher’s expert Mr. Buchan) or 10 to 50% (Ms. Chen) 10 range—i.e., agreeing that anywhere from 10 to 25% would be statistically valid. See Hrg. Tr. at 11 26:19-28:1, 32:21-33:2. Based upon the foregoing, the Court determines that a margin of error of 12 approximately 11.3% is within the range that will yield a representative sample and will 13 appropriately balance the associated burden against the needs of the case. Calculating a sample 14 from these values according to the experts’ formulas, the Court ORDERS Anthropic to produce a 15 sample in accordance with the following protocol: 16 • Anthropic shall produce a total of 5 million prompt-output pairs (i.e., each 17 containing a given prompt and the corresponding Claude output); 18 • These prompt-output pairs shall be drawn equally from pre-suit and post-suit data, 19 i.e., with 2.5 million records from between September 22 and October 18, 2023, 20 and 2.5 million records from between October 19, 2023 and March 22, 2024; and 21 • These prompt-output pairs shall be randomly selected from the respective periods. 22 Anthropic will produce this sample as soon as practicable and no later than the deadline for 23 substantial completion of remaining document productions, currently July 14, 2025. 24 //// 25 //// 26 //// 27 //// 1 III. DKT. NO. 345: ANTHROPIC’S CONFIDENTIALITY DESIGNATIONS 2 With regard to the final dispute, Publishers challenge three categories of “Highly 3 Confidential—Attorneys’ Eyes Only” (“HC-AEO”) designated materials: (1) Anthropic’s 9,418 4 Claude prompt-output records produced to-date; (2) various Claude use statistics and figures; and 5 (3) two training datasets used by Anthropic. Dkt. 345 at 1. Publishers’ concerns regarding over- 6 designation by Anthropic are not unfounded. The protective order (“PO”) in this case requires the 7 Parties to “take care to limit any [confidentiality] designations to specific material that qualifies 8 under the appropriate standard [and, to] the extent practicable … designate for protection only 9 those parts of material documents, [etc.] that qualify.” Dkt. 293, § V.1. While some level of 10 confidentiality is appropriate for the materials, a blanket HC-AEO designation is not. 11 First, with regard to Claude prompts and outputs generally, the Court is cognizant not only 12 of the roughly 9,000 prompts produced to-date but also of the forthcoming production of the 5 13 million record sample. This Court has previously recognized that Anthropic’s “users have a 14 privacy interest at stake” in their use of Claude. Dkt. 318. 4. Anthropic does not object to down- 15 designating prompts and outputs to “Confidential” rather than “HC-AEO,” and Publishers are 16 unable to identify prejudice apart from the assertion that it makes quoting from prompts and 17 outputs in public filings more difficult. Accordingly, recognizing that it would be impractical if 18 not impossible to require Anthropic to review and redact each prompt-output pair to maintain 19 users’ privacy, Anthropic may designate prompts and outputs categorically as “Confidential” at 20 the outset under the PO. However, Anthropic is cautioned that the standard for designation under 21 the PO and this Court’s recognition of a general privacy interest are separate from any inquiry into 22 whether documents in the public record should be sealed.1 23 For the Second category, Anthropic’s use of the “HC-AEO” designation for Claude use 24 1 Under the PO, confidential information is any information that is “nonpublic, sensitive, or 25 proprietary information, the disclosure of which could harm the privacy or business interests of any person.” Dkt. 293, § II.6. But a Party seeking to maintain documents under seal must meet 26 either the “good cause” or “compelling reasons” standard. Ctr. For Auto Safety v. Chrysler Grp., 809 F.3d 1092, 1099 (9th Cir. 2016). Accordingly, if Anthropic submits specific prompts or 27 outputs in connection with a filing, the burden falls on Anthropic to evaluate whether it can meet ] statistics and figures has been overbroad. Anthropic did not meaningfully defend its designation 2 || of information related to the “sampling numbers” at the hearing, agreeing that such statistics could 3 || be down-designated to “Confidential” and subcategorized as needed, but instead pointed to more 4 || detailed information in its response to Interrogatory no. 4 that it asserts was properly designated. 5 || Urg. Tr. at 68:23-71:3. Having reviewed Anthropic’s submission under seal, (Dkt. 368), the Court 6 || agrees that Anthropic’s query counts, broken down to such detail, are the type of sensitive metrics 7 || that may properly be maintained as “HC-AEO” material. Anthropic shall down-designate its other 8 statistics, including less granular and less comprehensive use-counts, to “Confidential.” 9 Third and finally, the Court agrees with Anthropic that, as a general matter, the training 10 || datasets used to train Claude are competitively sensitive and may properly be maintained as “HC- 11 AEO” material. Of course, if Anthropic has publicly disclosed the training datasets at issue, then 12 || confidentiality is not appropriate. But Anthropic maintains that it “has never publicly disclosed 13 || using either dataset to train Claude.” Dkt. 345 at 10. Neither Publishers’ citations to what 14 || Anthropic’s competitors have disclosed about their products, nor their citations to a 2021 article 3 15 || that predates Claude and does not mention Claude, refute this representation. Accordingly, a 16 || Anthropic may maintain this information as “HC-AEO” material. 2 17 SO ORDERED. 18 19 || Dated: May 23, 2025 20 Season vaYl SUSAN VAN KEULEN 2] United States Magistrate Judge 22 23 24 25 26 27 28