Victor Stanley, Inc. v. Creative Pipe, Inc.

MEMORANDUM AND ORDER

PAUL W. GRIMM, United States Magistrate Judge.

The plaintiff, Victor Stanley, Inc. (“VSI” or “Plaintiff’) filed a motion seeking a ruling that five categories of electronically stored documents produced by defendants Creative Pipe, Inc. (“CPI”) and Mark and Stephanie Pappas (“M. Pappas”, “S. Pappas” or “The Pappasses”) (collectively, “Defendants”) in October, 2007, are not exempt from discovery because they are within the protection of the attorney-client privilege and work-produet doctrine, as claimed by the Defendants. VSI argues that the electronic records at issue, which total 165 documents, are not privileged because their production by Defendants occurred under circumstances that waived any privilege or protected status. Alternatively, as for a subset of nine email communications from M. Pappas to a computer forensics expert Defendants retained to assist them with producing electronically stored information (“ESI”), VSI contends that the attorney-client privilege is inapplicable, and with regard to another two email communications (one draft, the other actually sent) from M. Pappas to one of his attorneys, VSI contends that they are neither privileged nor protected. Finally, as for two email communications from M. Pappas to two of his attorneys, VSI argues that they are beyond the scope of the attorney-client privilege because they fall within the erime/fraud/tort exception. Defendants acknowledge that they produced all 165 electronic documents at issue to VSI during Rule 34 discovery, but argue that the production was inadvertent, and therefore that privilege/protection has not been waived. As to the various email communications, Defendants argue that they are within the scope of the attorney-client privilege and work-product protection, and that the erime/fraud/tort exception is not applicable. The motion has been fully briefed, Paper Nos. 212, 221, 225, and 230, and I find that a hearing is not necessary. Local Rules of the United States District Court for the District of Maryland, Rule 105.6. For the reasons that follow, I find that all 165 electronic documents are beyond the scope of the attorney-client privilege and work-product protection because assuming, arguendo, that they qualified as privileged/protected in the first instance,¹ and assuming further that Defendants properly complied with their obligation to particularize any claims of privilege/protection imposed by Fed.R.Civ.P. 26(b)(5), Local Rules of the United States District Court for the District of Maryland, Appendix B, Discovery Guideline 9.c (“Discovery Guideline”), and the orders of this court detailing how such assertions must be demonstrated once they were challenged by VSI,² the privilege/protection was waived by the voluntary production of the documents to VSI by Defendants.

Background Facts

The following facts are not subject to dispute. The Defendants’ first Rule 34 response was a “paper production,” not ESI, made in May 2007. Pl.’s Supp’l Mem. 3, Paper No. 221. Plaintiff objected to its sufficiency, and following a hearing, the court ordered the parties’ computer forensic experts to meet and confer in an effort to identify a joint protocol to search and retrieve relevant ESI responsive to Plaintiffs Rule 34 requests. Id. This was done and the joint protocol prepared. Pl.’s Supp’l Mem. Ex. 9, Paper No. 221. The protocol contained detailed search and information retrieval instructions, including nearly five pages of keyword/phrase search terms. It is noteworthy that these search terms were aimed at locating responsive ESI, rather than identifying privileged or work-product protected documents within the population of responsive ESI. After the protocol was used to retrieve responsive ESI, Defendants reviewed it to locate documents that were beyond the scope of discovery because of privilege or work-product protection. Counsel for Defendants had previously notified the court on March 29, 2007, that individualized privilege review of the responsive documents “would delay production unnecessarily and cause undue expense.” Pl.’s Letter of Mar. 29, 2007, Paper No. 79. To address this concern, Defendants gave their computer forensics expert a list of keywords to be used to search and retrieve privileged and protected documents from the population of documents that were to be produced to Plaintiff. Id. However, Defendants’ counsel also acknowledged the possibility of inadvertent disclosure of privileged/protected documents, given the volume of documents that were to be produced, and requested that the court approve a “clawback agreement” fashioned to address the concerns noted by this court in Hopson v. Mayor of Baltimore, 232 F.R.D. 228 (D.Md.2005).³ In response, the court held a telephone conference to discuss the proposed clawback agreement, and thereafter issued a letter order requesting additional briefing by the parties “regarding the burdens associated with conducting a privileged [sic] review of the information to be produced in the time frame required by [the] discovery [schedule] in this case.” Letter Order, Apr. 24, 2007, Paper No. 92. However, on April 27, 2007, Defendants’ counsel notified the court that because Judge Garbis recently had extended the discovery deadline by four months, Defendants would be able to conduct a document-by-doeument privilege review, thereby making a clawback agreement unnecessary. Defs.’ Letter of Apr. 27, 2007, Paper No. 93. Accordingly, Defendants abandoned their efforts to obtain a clawback agreement and committed to undertaking an individualized document review.

Following their privilege review, Defendants made their ESI production to Plaintiff in September 2007. Pl.’s Supp’l Mem. 5, Paper No. 221. It is noteworthy that by the time of this production, Defendants had discharged their local attorneys, Messrs. Mohr and Ludwig from Meyer, Klipper & Mohr, and brought in new counsel.⁴

After receiving Defendants’ ESI production in September, 2007, Plaintiffs counsel began their review of the materials. They soon discovered documents that potentially were privileged or work-product protected and immediately segregated this information and notified counsel for Defendants of its production, following this same procedure each time they identified potentially privileged/protected information. Pl.’s Supp’l Mem. Exs. 11-15, Paper No. 221. Defendants’ Counsel, Mr. Schmid, responded by asserting that the production of any privileged or protected information had been inadvertent. Pl.’s Supp’l Mem. Ex. 17, Paper No. 221. Defendants also belatedly provided Plaintiff with a series of privilege logs, purportedly identifying the documents that had been withheld from production pursuant to Fed.R.Civ.P. 26(b)(5). Defs.’ Opp’n Mem. Exs. 4, 6, and 9, Paper No. 225.

The parties disagree substantially in their characterization of how Defendants conducted their review for privileged and protected documents before the ESI productions were made to Plaintiff. Defendants contend that after the joint ESI search protocol was implemented and the responsive ESI identified, their computer forensics expert, Ms. Gene-vive Turner, “conducted a privilege search using approximately seventy different keyword search terms ... [that] had been decided upon previously by Mr. Pappas, his former attorney, Christopher Mohr, and another attorney, F. Stephen Schmid---- All documents which were returned during the keyword search were segregated and provided to one of Mr. Pappas’ attorneys, John G. Monkman, Jr. for the first phase of the pre-production privilege review.” Defs.’ Opp’n Mem. 4, Ex. 1 (Pappas Aff.) and Ex. 3 (Monkman Aff.), Paper No. 225. This characterization, however, is somewhat misleading. In actuality, after the joint retrieval protocol had been executed, Ms. Turner determined that there were some ESI files (4.9 gigabytes) that were in text-searchable format and others (33.7 gigabytes) that were not. Defs.’ Opp’n Mem. Ex. 2 (Turner Aff. 11. 7), Paper No. 225. Turner conducted a search for privileged material on the text-searchable files using the seventy keywords developed by M. Pappas, Mohr and Schmid. As to the nontext-searchable files, she produced them to Monkman for manual privilege review. Turner Aff. H116-7. Monkman reviewed each of the files identified as privileged/protected by Turner based on her keyword searches. Monkman Aff. H 7. Additionally, Monkman and M. Pappas teamed up to begin doing a “page-by-page” manual privilege review of the nontext-searchable ESI files. Id. at H 8. According to Monkman: “[t]he second phase of review consisted of page-by-page review of ... [the non text-searchable ESI files], which was undertaken by Mr. Pappas and me. However, due to the compressed schedule and time constraints in reviewing these tens of thousands of documents within the time permitted, this review was undertaken by reviewing the page titles of the documents. Documents whose page titles indicated that the privilege might be applicable were reviewed in then-entirety by Mr. Pappas or me. This was the only way for us to complete the unwieldy review of these documents within the time permitted”. Id.

The foregoing affidavits create the impression that the keyword search Turner conducted on the text-searchable ESI files, using the seventy keywords developed by M. Pappas and his attorneys, successfully culled out the privileged/protected documents; and this status was confirmed by Monkman’s review, and they were withheld from production. Further, the Defendants’ characterization of the privilege review suggests that as to the non-text searchable files, Pappas and Monkman did all that could be reasonably expected of them in the time allowed to make the ESI production, which was to review only the title page of the documents and not their entire content. From the affidavits Defendants provided, the court is left to infer that the text-searchable documents that were not flagged by the keyword search Turner conducted were produced to the Plaintiff, as well as the nontext-searchable files that Monk-man and M. Pappas determined were not privileged or protected based on their limited title-page review. This is because the Defendants fail to delineate exactly which documents were and were not provided to the Plaintiff, or where the 165 documents at issue were located within the ESI productions made to the Plaintiff.

The implied conclusion that the court is invited to draw, from the limited information provided by the Defendants, is that the 165 documents that are the subject of the present motion were contained within the population of nontext-searchable ESI files that were produced by the Defendants to the Plaintiff, making their production inadvertent. However, this inference is not so easily drawn.

First, the Defendants are regrettably vague in their description of the seventy keywords used for the text-searchable ESI privilege review, how they were developed, how the search was conducted, and what quality controls were employed to assess their reliability and accuracy. While it is known that M. Pappas (a party) and Mohr and Schmid (attorneys) selected the keywords, nothing is known from the affidavits provided to the court regarding their qualifications for designing a search and information retrieval strategy that could be expected to produce an effective and reliable privilege review. As will be discussed, while it is universally acknowledged that keyword searches are useful tools for search and retrieval of ESI, all keyword searches are not created equal; and there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search or relying exclusively on such searches for privilege review. Additionally, the Defendants do not assert that any sampling was done of the text searchable ESI files that were determined not to contain privileged information on the basis of the keyword search to see if the search results were reliable. Common sense suggests that even a properly designed and executed keyword search may prove to be over-inclusive or under-inclusive, resulting in the identification of documents as privileged which are not, and non-privileged which, in fact, are. The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive. There is no evidence on the record that the Defendants did so in this case. Rather, it appears from the information that they provided to the court that they simply turned over to the Plaintiff all the text-searchable ESI files that were identified by the keyword search Turner performed as non-privileged, as well as the non-text searchable files that Monkman and M. Pappas’ limited title page search determined not to be privileged.

The Plaintiff paints an entirely different picture of the Defendants’ privilege review. VSI vigorously disputes Defendants’ assertion that the text-searchable ESI received by Defendants’ computer forensic expert, Turner, following the execution of the joint search and retrieval protocol was in a format that was difficult to search for privileged or protected materials. Plaintiff contends that it was able to do a keyword search of the text-searchable ESI produced by Defendants in about one hour using a “readily-available desktop search tool.” PL’s Reply Mem. 3 and Ex. 1 (Slaughenhoupt Aff. 11 6), Paper No. 230. VSI further contends that the non-text-searchable files that Monkman and M. Pappas reviewed by looking at the title pages consisted primarily of image files, such as photographs, catalogs, and drawings, which are not likely to contain privileged or protected information. Id. at 119. Most importantly, however, the Plaintiff argues that the Defendants’ complaint-that they could not effectively conduct a privilege review of the nontext-searchable files because there were so many of them-is a red herring because “the privileged materials [that are the subject of this motion] were all in text and thus were all searchable using standard text search tools. Contrary to Mr. Pappas’ assertion, a majority of the .PDF files in the ESI were searchable using readily available search tools. The ESI contained 9008 .PDF files, the majority of which were searchable and the remaining could have been made searchable using readily available OCR software and/or the native OCR Text Recognition tool within Adobe Acrobat.” Id. at 118.

Thus, according to the Plaintiff, the Defendants have waived any claim to attorney client privilege or work-product protection for the 165 documents at issue because they failed to take reasonable precautions by performing a faulty privilege review of the text-searchable files and by fading to detect the presence of the 165 documents, which were then given to the Plaintiff as part of Defendants’ ESI production. As will be seen, under either the Plaintiffs or Defendants’ version of the events, the Defendants have waived any privilege or protected status for the 165 documents in question.

Applicable Law

As this court discussed in some detail in Hopson, 232 F.R.D. at 235-38, courts have taken three different approaches when deciding whether the inadvertent production to an adversary of attorney client privileged or work-product protected materials constitutes a waiver. Under the most lenient approach there is no waiver because there has not been a knowing and intentional relinquishment of the privilege/protection; under the most strict approach, there is a waiver because once disclosed, there can no longer be any expectation of confidentiality; and under the intermediate one, the court balances a number of factors to determine whether the producing party exercised reasonable care under the circumstances to prevent against disclosure of privileged and protected information, and if so, there is no waiver. Id. As also noted in Hopson, the Fourth Circuit Court of Appeals has yet to decide which approach it will follow, although individual district courts within the circuit have adopted the intermediate balancing approach. Id. at 236 n. 18; see also Cont’l Cas. Co. v. Under Armour, Inc., 537 F.Supp.2d 761, 768 n. 3 (D.Md.2008). As Hopson pointed out, however, a careful reading of the Fourth Circuit’s decisions regarding waiver of the attorney-client privilege, albeit in contexts not closely related to the facts of this case,⁵ suggest that it is more inclined to adopt the strict approach than the intermediate or lenient one. Hopson, 232 F.R.D. at 236-37. Under the strict approach, there is no legitimate doubt that Defendants’ production of the 165 asserted privileged/protected documents waived the attorney-client privilege and work-product protection.⁶ Even under the intermediate test, however, the result would be the same.⁷

The intermediate test requires the court to balance the following factors to determine whether inadvertent production of attorney-client privileged materials waives the privilege: (1) the reasonableness of the precautions taken to prevent inadvertent disclosure; (2) the number of inadvertent disclosures; (3) the extent of the disclosures; (4) any delay in measures taken to rectify the disclosure; and (5) overriding interests in justice. McCafferby’s, Inc., v. Bank of Glen Burnie, 179 F.R.D. 163, 167 (D.Md.1998) (citing cases). The first of these factors militates most strongly in favor of a finding that Defendants waived the privilege in this case.

Assuming that the Plaintiffs version of how Defendants conducted their privilege review is accurate,⁸ the Defendants obtained the results of the agreed-upon ESI search protocol and ran a keyword search on the text-searchable files using approximately seventy keywords selected by M. Pappas and two of his attorneys. Defendants, who bear the burden of proving that their conduct was reasonable for purposes of assessing whether they waived attorney-client privilege by producing the 165 documents to the Plaintiff, have failed to provide the court with information regarding: the keywords used; the rationale for their selection; the qualifications of M. Pappas and his attorneys to design an effective and reliable search and information retrieval method; whether the search was a simple keyword search, or a more sophisticated one, such as one employing Boolean proximity operators;⁹ or whether they analyzed the results of the search to assess its reliability, appropriateness for the task, and the quality of its implementation. While keyword searches have long been recognized as appropriate and helpful for ESI search and retrieval, there are well-known limitations and risks associated with them, and proper selection and implementation obviously involves technical, if not scientific knowledge. See, e.g., United States v. O’Keefe, 537 F.Supp.2d 14, 24 (D.D.C.2008) (“Whether search terms or ‘keywords’ will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguisties____ Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.”); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331, 333 (D.D.C.2008), (“[Djetermining whether a particular search methodology, such as keywords, will or will not be effective certainly requires knowledge beyond the ken of a lay person (and a lay lawyer)____”);¹⁰ In re Seroquel Prods. Liab. Litig., 244 F.R.D. 650, 660 n. 6, 662 (M.D.Fla. 2007) (criticizing defendant’s use of keyword search in selecting ESI for production, noting the failure of the defendant to provide information “as to how it organized its search for relevant material, [or] what steps it took to assure reasonable completeness and quality control” and observing that “while key word searching is a recognized method to winnow relevant documents from large repositories ... [e]ommon sense dictates that sampling and other quality assurance techniques must be employed to meet requirements of completeness.”); The Sedona Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. 189, 194-95, 201-02 (“[Ajlthough basic keyword searching techniques have been widely accepted both by courts and parties as sufficient to define the scope of their obligation to perform a search for responsive documents, the experience of many litigators is that simple keyword searching alone is inadequate in at least some discovery contexts. This is because simple keyword searches end up being both over- and under-inclusive in light of the inherent malleability and ambiguity of spoken and written English (as well as all other languages).”). Id. at 194-95. To address this known deficiency, the Sedona Conference suggests as best practice points, inter alia:

Practice Point 3. The choice of a specific search and retrieval method will be highly dependent on the specific legal context in which it is to be employed.

Practice Point 4. Parties should perform due diligence in choosing a particular information retrieval product or service from a vendor.

Practice Point 5. The use of search and information retrieval tools does not guarantee that all responsive documents will be identified in large data collections, due to characteristics of human language. Moreover, differing search methods may produce differing results, subject to a measure of statistical variation inherent in the science of information retrieval.

Practice Point 6. Parties should make a good faith attempt to collaborate on the use of particular search and information retrieval methods, tools and protocols (including as to keywords, concepts, and other types of search parameters).

Practice Point 7. Parties should expect that their choice of search methodology will need to be explained, either formally or informally, in subsequent legal contexts (including in depositions, evidentiary proceedings, and trials).

Id.; and Information Inflation: Can the Legal System Adapt, 13 Rich. J.L. & Tech. 10, at *38, 40 (as cited at www.westlaw.com) (“[I]t is not surprising that lawyers and those to whom they delegate search tasks may not be particularly good at ferreting out responsive information through use of simple keyword search terms.... Accordingly, the assumption on the part of lawyers that any form of present-day search methodology will fully find ‘all’ or ‘nearly all’ available documents in a large, heterogeneous collection of data is wrong in the extreme.”).

Use of search and information retrieval methodology, for the purpose of identifying and withholding privileged or work-product protected information from production, requires the utmost care in selecting methodology that is appropriate for the task because the consequence of failing to do so, as in this case, may be the disclosure of privileged/protected information to an adverse party, resulting in a determination by the court that the privilege/protection has. been waived. Selection of the appropriate search and information retrieval technique requires careful advance planning by persons qualified to design effective search methodology. The implementation of the methodology selected should be tested for quality assurance; and the party selecting the methodology must be prepared to explain the rationale for the method chosen to the court, demonstrate that it is appropriate for the task, and show that it was properly implemented. In this regard, compliance with the Sedona Conference Best Practices for use of search and information retrieval will go a long way towards convincing the court that the method chosen was reasonable and reliable, which, in jurisdictions that have adopted the intermediate test for assessing privilege waiver based on inadvertent production, may very well prevent a finding that the privilege or work-product protection was waived.

In this case, the Defendants have failed to demonstrate that the keyword search they performed on the text-searchable ESI was reasonable. Defendants neither identified the keywords selected nor the qualifications of the persons who selected them to design a proper search; they failed to demonstrate that there was quality-assurance testing; and when their production was challenged by the Plaintiff, they failed to carry their burden of explaining what they had done and why it was sufficient.

Further, the Defendants’ attempt to justify what was done, by complaining that the volume of ESI needing review and time constraints presented them with no other choice is simply unpersuasive. Defendants were aware of the danger of inadvertent production of privileged/protected information and initially sought the protections of a non-waiver agreement such as that discussed in Hop-son, supra. Had they not voluntarily abandoned their request for a court-approved non-waiver agreement, they would have been protected from waiver. Instead, they advised the court that they did not need this protection and elected to do a document-by-document privilege review. According to Defendants version of the facts, when they undertook an individualized review of the nontext-searchable ESI and determined that they could only review the title pages, they neither sought an extension of time from the court to complete an individualized review nor reinstated their request for a court-approved non-waiver agreement, despite their awareness of how it would have provided protection against waiver. In these circumstances, Defendants’ protests that they did their best and that their conduct was reasonable rings particularly hollow.

The remaining factors to be assessed under the intermediate test may be quickly disposed of. The Defendants produced 165 asserted privileged/protected documents to the Plaintiff, so this case does not present an instance of a single document slipping through the cracks. Further, the court’s in camera review of the documents reflects that many of them are email and other communications between the Defendants and their various attorneys, as well as draft discovery responses, documents relating to settlements in unrelated litigation, comments from M. Pappas to counsel regarding discovery responses, and email correspondence between M. Pappas and Ms. Turner, the ESI forensic expert retained by Defendants. Thus, the disclosures were substantive — including numerous communications between defendants and their counsel. As noted by other district courts within the Fourth Circuit, any order issued now by the court to attempt to redress these disclosures would be the equivalent of closing the barn door after the animals have already run away. FDIC v. Marine Midland Realty Credit Corporation, 138 F.R.D. 479, 483 (E.D.Va. 1991) (“Any order issued now by the court would have only limited effect; it could not force NBNE to forget what has already been learned”); Parkway Gallery Furniture, Inc. v. Kittinger/Pennsylvania House Group, 116 F.R.D. 46, 52 (M.D.N.C.1987) (“[W]hen disclosure is complete, a court order cannot restore confidentiality and, at best, can only attempt to restrain further erosion.”). And, while the precise dates of the disclosures of the documents at issue are not clear from the record — since the Defendants made a series of ESI productions over a several week period — it is noteworthy that the Defendants did not discover the disclosure, but rather the Plaintiff made the discovery and notified the Defendants that potentially privileged/protected ESI had been produced. Therefore, this is not an instance in which a party inadvertently produced privileged information to an adversary, discovered the disclosure promptly, and then took immediate steps to inform the adversary that they had received the information inadvertently, thus demanding that it be returned.

While Defendants’ counsel did assert privilege and inadvertent production promptly after being notified by the Plaintiff of the production of possible privileged/protected information, the more important period of delay in this case is the one-week period between production by the Defendants and the time of the discovery by the Plaintiff of the diselosures-a period during which the Defendants failed to discover the disclosure. Finally, the Defendants have pointed to no overriding interests in justice that would excuse them from the consequences of producing privileged/protected materials. The Plaintiff is blameless, but the Defendants are not, having failed to take reasonable precautions to prevent the disclosure of privileged information, including the voluntary abandonment of the non-waiver agreement that the Plaintiff was willing to sign. Every waiver of the attorney-client privilege produces unfortunate consequences for the party that disclosed the information. If that alone were sufficient to constitute an injustice, there would never be a waiver. The only “injustice” in this matter is that done by Defendants to themselves. Marine Midland Realty Credit Corp., 138 F.R.D. at 483 (“It is seldom ‘fundamentally unfair’ to allow the truth to be made public, and under the circumstances ... the Court finds that it would ■not be fair to reward Rowe’s carelessness [in disclosing privileged materials] with a protective order.”). Accordingly, even under the intermediate test, the Defendants are not insulated from waiver.

Sufficiency of Defendants’ Assertion of Privilege/Protection

In addition to arguing that the Defendants waived any privileged or protected status that the disclosed documents had, the Plaintiff also appears to contend that the Defendants failed to properly establish the existence of the privilege and protection asserted in the first instance. Specifically, the Plaintiff argues that despite the requirements of Fed.R.Civ.P. 26(b)(5) and Discovery Guideline 5.c of this court that claims of privilege must be particularized, the Defendants failed to meet this obligation, and further, failed to comply with an order of this court¹¹ regarding the proper way to assert privilege in responding to discovery requests. See Pl.’s Supp’l Mem. 1, 6, 10, Paper No. 221; Pl.’s Reply Mem. 1, Paper No. 230. While Defendants do not deny that they failed to comply with the court’s order, they contend that when the parties met to confer regarding the sufficiency of the Defendants’ privilege/protection claims, the parties “essentially” came to an agreement that for all but eleven of the 165 documents at issue, the asserted privilege/proteetion was legitimate.¹² Accordingly, Defendants argue that if the waiver issue is decided in their favor, there are only eleven documents for which the assertion of privilege/protection is challenged by Plaintiff. Defs.’ Opp’n Mem. 6-7, Paper No. 225. Having decided the waiver issue against the Defendants, there is no need to reach the question of whether privilege/protection was properly asserted in the first instance. However, given the recurring problems associated with resolving disputed privilege/protection claims during discovery, it would be helpful to state the procedures that need to be followed in this process for the benefit of future cases.

While the scope of discovery in civil cases broadly encompasses facts relevant to the claims and defenses raised in the pleadings, and, on a showing of good cause, may even be extended to facts relevant to the subject matter of the litigation, Fed.R.Civ.P. 26(b)(1), it does not include privileged information. Id. Similarly, work-product protected information is beyond the reach of discovery unless the requesting party makes a showing of substantial need for the information and inability to obtain its substantial equivalent without undue hardship. Fed.R.Civ.P. 26(b)(3). Because the responding party is entitled to refuse to produce requested discovery if it is privileged or work product protected, the rules require that when doing so, the responding party must “describe the nature of the documents, communications, or tangible things not produced or disclosed— and do so in a manner that, without revealing information itself privileged or protected, will enable other parties to assess the claim.” Fed.R.Civ.P. 26(b)(5)(A)(ii). This requirement was added in the 1993 amendments to the rules of civil procedure, and in the words of the advisory committee:

The party [asserting privilege/protection] must also provide sufficient information to enable other parties to evaluate the applicability of the claimed privilege or protection. Although the person from whom the discovery is sought decides whether to claim a privilege or protection, the court ultimately decides whether, if this claim is challenged, the privilege or protection applies. Providing information pertinent to the applicability of the privilege or protection should reduce the need for in camera examination of the documents.

Fed.R.Civ.P. 26 advisory committee’s note. Neither the rule nor the advisory committee comment specifies exactly how the party asserting privilege/protection must particularize its claim. The most common way is by using a privilege log, which identifies each document withheld, information regarding the nature of the privilege/protection claimed, the name of the person making/receiving the communication, the date and place of the communication, and the document’s general subject matter. See, e.g., Discovery Guideline 9.c.; Paul W. Grimm, Charles S. Fax, & Paul Mark Sandler, Discovery Problems and Their Solutions, 62-64 (2005)(“To properly demonstrate that a privilege exists, the privilege log should contain a brief description or summary of the contents of the document, the date the document was prepared, the person or persons who prepared the document, the person to whom the document was directed, or for whom the document was prepared, the purpose in preparing the document, the privilege or privileges asserted with respect to the document, and how each element of the privilege is met for that document.”). Id. at 62-63.

In actuality, lawyers infrequently provide all the basic information called for in a privilege log, and if they do, it is usually so cryptic that the log falls far short of its intended goal of providing sufficient information to the reviewing court to enable a determination to be made regarding the appropriateness of the privilege/protection asserted without resorting to extrinsic evidence or in camera review of the documents themselves. New judges find that the privilege log is ever sufficient to make the discrete fact-findings needed to determine whether a privilege/protection was properly asserted and not waived. Further, because privilege review and preparation of privilege logs is increasingly handled by junior lawyers, or even paralegals, who may be inexperienced and overcautious, there is an almost irresistible tendency to be over-inclusive in asserting privilege/protection. While some of this tendency is understandable given the consequences of mistakenly producing privileged/protected information, the experience of many judges is that when the documents themselves are reviewed, it often turns out that a much smaller percentage of documents actually meet the requirements of the asserted privilege/protection than was claimed by the asserting party. Counsel should be wary of filing a response to a Rule 34 document production request that asserts privilege/protection as a basis for refusing to make requested production without having a factual basis to support each element of each privilege/proteetion claimed for each document withheld, because doing so is a sanctionable violation of Fed.R.Civ.P. 26(g).

Requesting parties also know of the limited utility of privilege logs (for they likely have served similar privilege logs in response to their adversary’s discovery requests), and thus, when they receive the typical privilege log, they are wont to challenge its sufficiency, demanding more factual information to justify the privilege/protection claimed. This, in turn, is often met with a refusal from the producing party, and it does not take long before a motion is pending, and the court is called upon to rule on the appropriateness of the assertion of privilege/protection, often with the producing party’s “magnanimous” offer to produce the documents withheld for in camera review. In camera review, however, can be an enormous burden to the court, about which the parties and their attorneys often seem to be blissfully unconcerned.

For example, in order for the court to determine whether the attorney-client privilege was properly asserted regarding a particular document, the court must make the following fact determinations:

(1) the asserted holder of the privilege is or sought to become a client; (2) the person to whom the communication was made (a) is a member of the bar of a court, or his subordinate and (b) in connection with this communication is acting as a lawyer; (3) the communication relates to a fact of which the attorney was informed (a) by his client (b) without the presence of strangers (c) for the purpose of securing primarily either (i) an opinion on law or (ii) legal services or (iii) assistance in some legal proceeding, and not (d) for the purpose of committing a crime or tort; and (4) the privilege has been (a) claimed and (b) not waived by the client.

In re Allen, 106 F.3d 582, 600 (4th Cir.1997) (holding also that “a district court’s holding that the attorney-client privilege does not protect communications rest[s] essentially on determinations of fact”). Id. at 601. Sometimes the document itself makes this clear, such as when the lawyer writes to the client to provide an opinion and the correspondence reflects that it is a confidential communication. Often times, however, it is impossible to determine if the privilege applies without extrinsic evidence, which must be provided by affidavit, deposition transcript, or other source. The time it takes the court to review this extrinsic evidence on a document-by-document basis can be extensive, particularly given the tendency of lawyers to be over-inclusive in the assertion of privilege/protection in the first place. It should go without saying that the court should never be required to undertake in camera review unless the parties have first properly asserted privilege/protection, then provided sufficient factual information to justify the privilege/protection claimed for each document, and, finally, met and conferred in a good faith effort to resolve any disputes without court intervention. United States v. Zolin, 491 U.S. 554, 571-72, 109 S.Ct. 2619, 105 L.Ed.2d 469 (1989) (“[W]e cannot ignore the burdens in camera review places upon the district courts, which may well be required to evaluate large evidentiary records without open adversarial guidance by the parties.... Before engaging in in camera review ... ‘the judge should require a showing of a factual basis adequate to support a good faith belief by a reasonable person’ ... that in camera review of the materials may reveal evidence to establish the claim [of privilege/protection]”.) (internal citations omitted); United States v. Family Practice Assocs. of San Diego, 162 F.R.D. 624, 627 (S.D.Ca.1995) (“Prior to an in camera review there must first be a sufficient evidentiary showing of a legitimate issue as to application of a privilege or other protection. In camera review should not replace effective adversarial testing of the claimed privileges and protection.”); Diamond State Ins. Co. v. Rebel Oil Co., 157 F.R.D. 691, 700 (D.Nev. 1994)(“In camera review is generally disfavored. It is not to be used as a substitute for a party’s obligation to justify its withholding of documents. In camera review should not replace the effective adversarial testing of the claimed privileges and protections. Resort to in camera review is appropriate only after the burdened party has submitted detailed affidavits and other evidence to the extent possible.”) (internal citations omitted); and Caruso v. Coleman Co., 1995 WL 384602, at *1 (“[R]esort to in camera review is appropriate only after the burdened party has submitted detailed affidavits and other evidence to the extent possible. Unfortunately, this court is put in the untenable position of having to speculate in order to determine which privileges apply to the individual documents. A court has the right to refuse to engage in such speculation, since the burden of proving the attorney-client or work-product privileges rests on the party claiming the privilege.”) (internal citations omitted) (emphasis in original); Weber v. Pa-duano, No. 02 Civ. 3392, 2003 WL 161340, at *14 (S.D.N.Y. Jan. 22, 2003) (holding in camera review should not be undertaken routinely, but only after the party asserting privilege has submitted an adequate record to support the claim); Bourne of New York City v. AmBase Corp., 150 F.R.D. 465, 475 (S.D.N.Y.1993) (“AmBase’s suggestion of in camera review in lieu of an evidentiary presentation is misplaced. Such review ... is not, however, to be routinely undertaken, particularly in a case involving a substantial volume of documents, as a substitute for a party’s submission of an adequate record in support of its privilege claims.”); and King v. Conde, 121 F.R.D. 180, 190 (E.D.N.Y.1988) (“After giving defendants an opportunity to respond (with a brief and possible supplemental affidavits or declarations), the court can determine whether the defendants have made the requisite threshold showing to invoke the privilege. If the court finds that the defendant has not satisfied its threshold burdens, direct disclosure is in order. If the threshold burdens are met, the court may then review the materials at issue in camera and decide which, if any to withhold from disclosure.”).

All of this has led to some fairly strongly worded statements from courts about what a party must do to substantiate its claim of privilege or protection. See Parkway Gallery Furniture, Inc. v. Kittinger/Pennsylva-nia House Group, 116 F.R.D. 46, 48 (M.D.N.C.1987) (“Disputes over whether the attorney-client privilege has been waived through inadvertent production of the documents or on the basis of the fraud or crime exception to the privilege often involve contested facts necessitating an evidentiary showing. Generally, the proponent or party claiming rights or benefit of an assertion bears the burden of establishing his contention.”); Caruso, 1995 WL 384602, at *1 (“A general allegation of privilege is insufficient. Instead, a clear showing must be made which sets forth the items or categories objected to and the reason for that objection. Accordingly, the proponent must provide the court with enough information to enable the court to determine the privilege, and the proponent must show by affidavit that precise facts exist to support the claim of privilege.”); United States ex rel. Burns v. Family Practice Associates of San Diego, 162 F.R.D. 624, 627-28 (S.D.Cal.1995) (finding Defendant’s failure to make out a factual showing by “detailed affidavits or other evidence” waived privilege); Nutmeg Ins. Co. v. Atwell, Vogel & Sterling, 120 F.R.D. 504, 510 (W.D.La. 1988) (“In considering whether a proponent of the privilege is entitled to protection, the courts must place the burden of proof squarely upon the party asserting privilege. Accordingly, the proponent must provide the court with enough information to enable the court to determine privilege, and the proponent must show by affidavit that precise fact exist to support the claim of privilege.”); Church of Scientology Intern, v. U.S. Dept. of Justice, 30 F.3d 224, 231 (1st Cir.1994) (“These declarations are written too generally to supplement the index in any meaningful way.... Thus, none of the functions of the index ... are served: the declarations do not demonstrate careful analysis of each document by the government; the court has not been assisted in its duty of ruling on the applicability of an exemption; and the adversary system has not been visibly strengthened.”); United States v. First State Bank, 691 F.2d 332, 335 (7th Cir.1982) (“A taxpayer need not reveal so many facts that the privilege becomes worthless but he must at least identify the general nature of that document, the specific privilege he is claiming for that document, and facts which establish all the elements of the privilege he is claiming. These allegations must be supported by affidavits.”); In re French, 162 B.R. 541, 548 (Bankr.D.S.D.1994) (“Debtor did, indeed, file an affidavit claiming the privilege, but the timing was off, and, even then, it was in the form of a ‘blanket’ assertion rather than articulated specific facts giving rise to a privilege.”).

Thus, insuring that a privilege or protection claim is properly asserted in the first instance and maintained thereafter involves a several step process. First, pursuant to Fed.R.Civ.P. 26(b)(5), the party asserting privilege/proteetion must do so with particularity for each document, or category of documents, for which privilege/proteetion is claimed. At this first stage, it is sufficient to meet the initial burden by a properly prepared privilege log. If, after this has been done, the requesting party challenges the sufficiency of the assertion of privilege/protection, the asserting party may no longer rest on the privilege log, but bears the burden of establishing an evidentiary basis — by affidavit, deposition transcript, or other evidence — for each element of each privilege/protection claimed for each document or category of document. A failure to do so warrants a ruling that the documents must be produced because of the failure of the asserting party to meet its burden. If it makes this showing, and the requesting party still contests the assertion of privilege/protection, then the dispute is ready to submit to the court, which, after looking at the evi-dentiary support offered by the asserting party, can either rule on the merits of the claim or order that the disputed documents be produced for in camera inspection.

In this case, the court made it clear to the Defendants that they were obligated to follow this procedure. See Letter Order, Dec. 28, 2007, Paper No. 194, and Pl.’s Letter of Feb 20, 2007 at 3, Paper No. 212, (citing to the Transcript of Dee. 21, 2007 Telephone Hearing, pp 16-17, when the court outlined the procedures to be used when submitting a privilege log). The Plaintiff argues that Defendants failed to comply with the court’s order, and the Defendants have not demonstrated that they did. Had I not ruled that any privilege/proteetion already was waived, then the effect of a failure by the Defendants to comply with the court’s order regarding the proper manner in which to assert privilege/protection would have warranted an order to produce the materials for failure to carry the burden of demonstrating the existence of the privilege/proteetion claimed.

Conclusion

For the reasons stated, the court finds that the Defendants waived any privilege or work-product protection for the 165 documents at issue by disclosing them to the Plaintiff. Accordingly, the Plaintiff may use these documents as evidence in this case, provided they are otherwise admissible. In this regard, the Plaintiff has only sought use of the documents themselves, and the court has not been asked to rule, and accordingly does not, that there has been any waiver beyond the documents themselves.

1 . The 165 documents were produced to me for review in camera. Having done so, it is apparent that many do not qualify as attorney-client privileged or work-product protected. For example, the following documents were asserted to be privileged or protected, yet the court's in camera review discloses that these assertions are without merit: Doc. No. 18 (discovery request from Plaintiff to Defendant); Doc. Nos. 28, 32 (email between employee of Creative Pipe to M. Pappas, not discussing any materials that legitimately could be characterized as confidential); Doc. Nos. 24, 60 (email from Plaintiff's attorney to Defendants' attorney); Doc. Nos. 56, 61-65 (email between M. Pappas and G. Turner, Defendants’ ESI expert, regarding payment); Doc. Nos. 105, 111, 130-133, 148-149, 151-158 (pictures of products, such as benches, trash can); Doc. No. 143 (page from invoice M. Pappas from attorney, no confidential information contained). It should he noted that the Defendants' failure to comply with the court's order of December 28, 2007, Paper No. 194, regarding how to handle assertion of privilege/protection claims resulted in an absence from the record of the factual basis to support their claims.

2 . This court informed Defendants that they had the burden of providing an evidentiary basis to establish each element of the attorney-client privilege and work-product protection for each document at issue. Letter Order, Dec. 28, 2007, Paper No. 194. Notwithstanding, Defendants failed to do so, relying instead on the privilege logs that they provided to VSI, which did little more than briefly identify and describe each document and identify the basis for the refusal to produce it. As will be explained in this memorandum and order, when a party refuses to produce documents during discovery on the basis that they are privileged or protected, it has a duty to particularize that claim. Fed.R.Civ.P. 26(b)(5), Discovery Guideline 9.c; Caruso v. Coleman Co., CIV. A. No. 93-CV-6733, 1995 WL 384602, at * 1, (E.D.Pa. June 22, 1995); Bowne of New York City v. AmBase Corp., 150 F.R.D. 465, 474 (S.D.N.Y.1993); In re Pfohl Bros. Landfill Litig., 175 F.R.D. 13, 20 (W.D.N.Y. 1997); United States v. Kovel, 296 F.2d 918, 923 (2d Cir.1961). While a privilege log that complies with Discovery Guideline 9.c is an acceptable way to do so initially, once the claims of privilege/protection have been challenged by the requesting party, the producing party must then establish an eviden-tiary basis to support the privilege/protection claim. Failure to do so results in a forfeiture of the privilege/protection claimed. Bowne, 150 F.R.D. at 474 (holding that if the party claiming privilege fails to provide sufficient detail to demonstrate all legal requirements to make out the privilege, the claim must be rejected); Fox v. California Sierra Fin. Servs., 120 F.R.D. 520, 524 (N.D.Cal.1988) (finding that a party claiming privilege as basis for withholding discovery must properly identify each document and the basis for the privilege claimed); In re Pfohl Bros., 175 F.R.D. at 20 (holding "[mjere conclusory or ipse dixit assertions of privilege” fail to satisfy the burden of demonstrating the applicability of a privilege).

3 . In Hopson, this court discussed the dangers inherent in using non-waiver agreements, such as "clawback” or "quick-peek" agreements, and noted that reliance on them could nonetheless result in a determination that privilege and work-product protection had been waived, notwithstanding the agreement, given the current state of the substantive law regarding privilege waiver. Hopson, 232 F.R.D. at 236-38. The court further identified a process that could be employed within the boundaries of existing privilege waiver law that would significantly improve the likelihood of avoiding privilege waiver. The court noted:

[I]t is essential to the success of this approach in avoiding waiver that the production of inadvertently produced privileged electronic data must be at the compulsion of the court, rather than solely by the voluntary act of the producing party, and that the procedures agreed to by the parties and ordered by the court demonstrate that reasonable measures were taken to protect against waiver of privilege and work product protection.

Id. at 240. Defendants' counsel were aware of the requirements of Hopson. Pl.’s Letter of Mar. 29, 2007, Paper No. 79. The court’s request for additional briefing regarding the burdens associated with conducting privilege review within the time allotted for Defendants to produce the ESI to Plaintiff was aimed at developing a factual record that would permit a Hopson compliant non-waiver agreement to be approved by the court.

4 . It also is worth noting that Defendants’ current counsel, James Rothschild, of Anderson Coe and King LLP, and Joshua Kaufman, of Venable LLP entered their appearance after all the events that are relevant to resolving the pending dispute had taken place and are not responsible for any of the actions or inactions that contributed to the court's ruling.

5 . None of the Fourth Circuit cases reviewed in Hopson examined privilege waiver in the context of a voluminous document production during discovery in a civil case, and none of them considered the extra challenges of preventing privilege waiver posed by handling voluminous production of ESI, which is a relatively new phenomenon. The advisory committee notes to recently amended Fed.R.Civ.P. 26(b)(5) acknowledge these challenges:

The Committee [on the Rules of Practice and Procedure] has repeatedly been advised that the risk of privilege waiver and the work necessary to avoid it, add to the costs and delay of discovery. When the review is of electronically stored information, the risk of waiver, and the time and effort required to avoid it, can increase substantially because of the volume of electronically stored information and the difficulty in ensuring that all information to be produced has in fact been reviewed.

Fed.R.Civ.P. 26 advisory committee's note. Notwithstanding this recognition, however, the recently adopted rules of civil procedure relating to ESI do not effect any change in the substantive law of privilege waiver, as was discussed in some detail in Hopson, supra, because the Rules Enabling Act precludes creation or abrogation of any privilege by ordinary rule making. This is reserved for Congress. 28 U.S.C. § 2074(b) (1988). Following the Hopson decision, however, the Advisory Committee on the Rules of Evidence conducted hearings on this issue and, following public comment, proposed a new rule of evidence: Rule 502. The Committee approved the proposed rule and the Judicial Conference then forwarded it to Congress where it was passed by the Senate as S. 2450. It is still pending in the House of Representatives. If enacted by Congress, Proposed Federal Evidence Rule 502 would solve the problems Hopson discussed and protect against privilege waiver under circumstances similar to those presented in this case if the parties entered into a non-waiver agreement that meets the requirements of the proposed rule, and the court, in turn, approved it. Until this happens, however, the procedures identified in Hopson are the only ones that provide a possible means of avoiding waiver in those jurisdictions that have not recognized the intermediate approach to waiver by inadvertent production (and, as noted, the Defendants initially sought to enter a non-waiver agreement such as discussed in Hopson, but then abandoned this effort). Should the issue of privilege waiver by inadvertent production of voluminous ESI be considered by the Fourth Circuit at some time in the future, it may be hoped that the court will be cognizant of the unique problems presented with regard to avoiding privilege waiver presented by ESI discovery, as well as the fact that the approval of Proposed Evidence Rule 502 by the Committee on the Rules of Evidence, as well as the Judicial Conference, recognizes a need to provide relief in this difficult area. The substantive law of privilege is not rigid and inflexible, Hopson, 232 F.R.D. at 240(citing Jaffee v. Redmond, 518 U.S. 1, 8, 116 S.Ct. 1923, 135 L.Ed.2d 337 (1996)), but is governed by principles of the common law as interpreted “by the courts of the United States in the light of reason and experience.” Fed.R.Evid. 501. Experience has now shown that ESI discovery presents unique, heretofore unrecognized, risks of waiver of privilege or work-product protection even when the party asserting the privilege or protection has exercised care not to waive it. The approval of Proposed Evidence Rule 502 by the Judicial Conference is a reasoned response to this new experience, but still pending in Congress. For those courts that have yet to decide which approach to follow regarding the inadvertent disclosure of privileged material during ESI discovery, the commentary to the proposed rule is worthy of consideration.

6 . As noted in Continental Casualty Co. v. Under Armour, Inc., 537 F.Supp.2d 761 (D.Md.2008), if documents qualify as both attorney-client privileged and work-product protected, separate analysis is required to determine whether inadvertent production constitutes waiver. However, the majority view is that disclosure of work-product material in a manner that creates a substantial risk that an adversary will receive it waives the protection. Id. at 772-73 (citing Restatement (Third) of the Law Governing Lawyers § 91 (2000)). In this case. Defendants’ voluntary, though inadvertent, production of the 165 documents directly to counsel for the Plaintiff waived any work-product protection they may have had. Id.

7 . Citing dicta in Hopson, 232 F.R.D. at 237 n. 27, Defendants argue that state privilege waiver law controls the determination of whether the inadvertent production of privileged ESI waived the privilege, at least as to the supplemental state law claims that have been pleaded by Plaintiff. Defs.’ Opp’n Mem. 10 n. 4, Paper No. 225. And, as they correctly note, the Maryland Court of Special Appeals has adopted the intermediate test in Elkton Care Center Associates, Ltd. Partnership v. Quality Care Management, 145 Md.App. 532, 805 A.2d 1177 (2002). However, as this court more recently pointed out in Continental Casualty Co., 537 F.Supp.2d at 768 n. 3, (citing cases), the majority of federal courts that have addressed the issue of what privilege law to apply in federal cases where both federal and state claims are pending, and where the law of privilege is different under federal law than it is under state law, have concluded that federal privilege law trumps state privilege law. If for no other reason than an appreciation of the shortness of life, a court ought not to be required to parse out competing outcomes under differing state and federal privilege law to apply to the same core facts presented in litigation that spawned both federal and state claims is a time consuming and challenging task. I agree that following the majority view is a better approach, and so adopt it in this decision. Consequently, federal privilege waiver law will apply to both the federal and state claims.

8 . Which, on the record before me, the Defendants do not rebut.

9 . Keyword searching may be accomplished in many ways. The simplest way is to use a series of individual keywords. Using more advanced search techniques, such as Boolean proximity operators, can enhance the effectiveness of keyword searches. Boolean proximity operators are derived from logical principles, named for mathematician George Boole, and focus on the relationships of a "set” of objects or ideas. Thus, combining a keyword with Boolean operators such as "OR," "AND,” "NOT,” and using parentheses, proximity limitation instructions, phrase searching instructions, or truncation and stemming instructions to require a logical order to the execution of the search can enhance the accuracy and reliability of the search. The Sedo-na Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. (2007) at 200, 202, 217-18 ("Sedona Conference Best Practices”); Information Inflation: Can the Legal System Adapt?, 13 Rich. J.L. & Tech. 10 (2007) at *37-41 (as cited at www.westlaw.com). In addition to keyword searches, other search and information retrieval methodologies include: probabilistic search models, including "Bayesian classifiers” (which searches by creating a formula based on values assigned to particular words based on their interrelationships, proximity, and frequency to establish a relevancy ranking that is applied to each document searched); “Fuzzy Search Models” (which attempt to refine a search beyond specific words, recognizing that words can have multiple forms. By identifying the "core” for a word the fuzzy search can retrieve documents containing all forms of the target word); “Clustering” searches (searches of documents by grouping them by similarity of content, for example, the presence of a series of same or similar words that are found in multiple documents); and "Concept and Categorization Tools” (search systems that rely on a thesaurus to capture documents which use alternative ways to express the same thought). See Sedona Conference Best Practices, supra, at 217-23.

10 . The O'Keefe and Equity Analytics opinions have raised the eyebrows of some commentators who have expressed the concern that they "en-graft [Fed.R.Evid.] 702 (and [Fed.R.Evid. 104(a) into discovery ... [which, it is feared] would multiply the costs of discovery”, and, it is argued, this is a "path [that] is rife with unintended consequences”. See, e.g., Rule 702 and Discovery of Electronically Stored Information, 8 Digital Discovery & E-Evidence (BNA) No. 5, at p. 3 (May 1, 2008). A careful reading of O'Keefe and Equity Analytics, however, should allay these concerns. In neither case did the court expressly hold that Fed.R.Evid. 702 and 104(a) were "engrafted” into the rules of discovery in civil proceedings (indeed, neither opinion even mentions Rule 104(a)). Instead, Judge Facciola made the entirely self-evident observation that challenges to the sufficiency of keyword search methodology unavoidably involve scientific, technical and scientific subjects, and ipse dixit pronouncements from lawyers unsupported by an affidavit or other showing that the search methodology was effective for its intended purpose are of little value to a trial judge who must decide a discovery motion aimed at either compelling a more comprehensive search or preventing one. Certainly those concerned about the O'Keefe and Equity Analytics opinions would not argue that trial judges are not required to make fact determinations during discovery practice. Indeed, such fact determinations inundate them. For example, deciding whether ESI discovery is not reasonably accessible because of undue burden or cost (Fed.R.Civ.P. 26(b)(2)(B)) involves factual determinations, as does determining whether discovery sought is too expensive or burdensome under Fed.R.Civ.P. 26(b)(2)(C); determining whether sanctions should be imposed for failing to preserve ESI or if the loss was a result of the routine, good faith operation of an electronic information system under Fed.R.Civ.P. 37(e); or determining whether documents withheld from disclosure are privileged or protected. Certainly the court is entitled to reliable factual information on which to make such rulings. It cannot credibly be denied that resolving contested issues of whether a particular search and information retrieval method was appropriate— in the context of a motion to compel or motion for protective order — involves scientific, technical or specialized information. If so, then the trial judge must decide a method’s appropriateness with the benefit of information from some reliable source — whether an affidavit from a qualified expert, a learned treatise, or, if appropriate, from information judicially noticed. To suggest otherwise is to condemn the trial court to making difficult decisions on inadequate information, which cannot be an outcome that anyone would advocate. For example, in the analogous technical area of sampling ESI, courts have recognized the need to have expert assistance to develop a valid random sampling protocol. See, e.g., In re Vioxx Products Liability Litigation, No. 06-30378, 06-30379 2006 WL 1726675, at *2 n. 5 (5th Cir. May 26, 2006) ("By random sampling, we mean adhering to a statistically sound protocol for sampling documents.... The parties must provide expert assistance to the district court in constructing any protocol.”); Manual for Complex Litigation (Fourth) § 11.446 (2004) ("The complexity and rapidly changing character of technology for the management of computerized materials may make it appropriate for the judge to ... call on the parties to provide the judge with expert assistance, in the form of briefings on the relevant technological issues.”). Indeed, it is risky for a trial judge to attempt to resolve issues involving technical areas without the aid of expert assistance. In American National Bank 6 Trust Co. v. Equitable Life Assurance Society, 406 F.3d 867, 879 (7th Cir.2005), the court reversed a magistrate judge's sanctions ruling that was predicated on sampling methodology the judge developed, and which the appellate court characterized as "arbitrary” and lacking "logical foundation.”

Moreover, if the court is to be given scientific or technical information to resolve a contested discovery matter, what standards should govern its evaluation? Should the court ignore a purported ESI expert's lack of qualifications if that shortcoming is demonstrated by the party opposing his opinion? Should the court accept opinions shown to be unsupported by sufficient facts or based on demonstrably unreliable methodology? The answer is obviously "No.” Viewed in its proper context, all that O’Keefe and Equity Ana-lytics required was that the parties be prepared to back up their positions with respect to a dispute involving the appropriateness of ESI search and information retrieval methodology— obviously an area of science or technology — with reliable information from someone with the qualifications to provide helpful opinions, not conclu-sory argument by counsel. The goal of Federal Rule of Evidence 702 is to set standards to determine whether information is "helpful” to those who must make factual determinations involving disputed areas of science, technology or other specialized information. The rule is one of common sense, and reason-opinions regarding specialized, scientific or technical matters are not “helpful” unless someone with proper qualifications and adequate supporting facts provided such an opinion after following reliable methodology. That these common sense criteria are found in the rules of evidence does not render them off-limits for consideration during discovery. It is not unusual for pretrial factual determinations in civil cases to look to the Federal Rules of Evidence for assistance in resolving fact disputes. Indeed, in summary judgment practice, Fed.R.Civ.P. 56(e) requires that the parties support their motions with “such facts as would be admissible in evidence.” The message to be taken from O'Keefe, Equity Analytics, and this opinion is that when parties decide to use a particular ESI search and retrieval methodology, they need to be aware of literature describing the strengths and weaknesses of various methodologies, such as The Sedona Conference Best Practices, supra, n. 9, and select the one that they believe is most appropriate for its intended task. Should their selection be challenged by their adversary, and the court be called upon to make a ruling, then they should expect to support their position with affidavits or other equivalent information from persons with the requisite qualifications and experience, based on sufficient facts or data and using reliable principles or methodology-

For those understandably concerned about keeping discovery costs within reasonable bounds, it is worth repeating that the cost-benefit balancing factors of Fed.R.Civ.P. 26(b)(2)(C) apply to all aspects of discovery, and parties worried about the cost of employing properly designed search and information retrieval methods have an incentive to keep the costs of this phase of discovery as low as possible, including attempting to confer with their opposing party in an effort to identify a mutually agreeable search and retrieval method. This minimizes cost because if the method is approved, there will be no dispute resolving its sufficiency, and doing it right the first time is always cheaper than doing it over if ordered to do so by the court. Additionally, cost can be minimized by entering into a court-approved agreement that would comply with Hopson, or if enacted, Proposed Evidence Rule 502. In addition, there is room for optimism that as search and information retrieval methodologies are studied and tested, this will result in identifying those that are most effective and least expensive to employ for a variety of ESI discovery tasks. Such a study has been underway since 2006, when the National Institute of Standards and Technology (NIST), an agency within the U.S. Department of Commerce, embarked on a cooperative endeavor with the Department of Defense to evaluate the effectiveness of a variety of search methodologies. This project, known as the Text Retrieval Conference (TREC), evolved into the Tree LegalTrack, a research effort aimed at studying the e-discovery review process to evaluate the effectiveness of a wide array of search methodologies. This evaluative process is open to participation by academics, law firms, corporate counsel and companies providing ESI discovery services. See: http://trec-legdl.umiacs.umd.edu. The next test will occur in the summer of 2008. The goal of the project is to create industry best practices for use in electronic discovery. This project can be expected to identify both cost effective and reliable search and information retrieval methodologies and best practice recommendations, which, if adhered to, certainly would support an argument that the party employing them performed a reasonable ESI search, whether for privilege review or other purposes.

11 . See, e.g., Letter Order, Dec. 28, 2007, Paper No. 194 ("[W]ithin 30 days, Defendant[s] shall provide to Plaintiff an affidavit or other similar evidentiary support to establish each element of each privilege or work product protection asserted for each document for which privilege or work product is claimed.”).

12 . As noted supra, footnote 1, the court’s in camera review of the documents confirmed that there were numerous documents for which no legitimate claim of privilege or protection could be sustained.