Covad Communications Company v. Revonet Inc ( 2009 )

UNITED STATES DISTRICT COURT
FOR THE DISTRICT OF COLUMBIA
____________________________________
)
COVAD COMMUNICATIONS )
COMPANY, )
)
Plaintiff, )
)
v. ) Civil Action No. 06-1892 (CKK/JMF)
)
REVONET, INC., )
)
Defendant. )
____________________________________)
MEMORANDUM OPINION
This case has been referred to me for the resolution of discovery disputes. The issues
now before the Court are (1) whether plaintiff Covad Communications Company (“Covad”)
should be permitted to conduct a forensic search of defendant Revonet, Inc.’s (“Revonet”)
computers and servers; (2) how those searches should be conducted; and (3) who should pay for
them.
I. Background.
The history of this case appears in previous opinions. See generally Covad
Communications Co. v. Revonet, 254 F.R.D. 147 (D.D.C. 2008); Covad Communications Co. v.
Revonet, 250 F.R.D. 14 (D.D.C. 2008). Briefly, Covad hired Revonet to run a marketing
campaign for Covad’s voice over Internet protocol (“VoIP”) business. As part of the campaign,
Revonet made outbound phone calls to find persons who might be interested in Covad’s services
and also fielded calls from persons who were seeking information about Covad. These practices
were known as generating outbound and inbound leads, respectively. Covad now alleges that
Revonet expropriated information that belonged to Covad in violation of the contract between
the parties.
As further discussed in the findings of fact, Revonet used a database called the Federated
Database to house all of its sales lead information regardless of the source. Covad has alleged
that Revonet took sales lead data – stored in the Federated Database – that belonged to Covad
and gave it to other customers. Understandably, there is no dispute that the content of the
Federated Database and the evidence within that database itself and elsewhere are the very heart
of this lawsuit. Thus, I have expressed my view, and the parties have agreed, that the database
itself, and not just the data that it holds, should be considered an exhibit in this case and the
parties must find a way to examine it so that they can make the appropriate arguments and try
this case. Cf. Smith v. Café Asia, 246 F.R.D. 19, 21 (D.D.C. 2007).
The parties first brought the issue of the database to the Court’s attention in December
2008, when their attempts to reach an agreement as to how to search it failed. At that time, I
ordered the parties to file proposed protocols for searching the Federated Database. See Minute
Order (12/3/09). The parties filed their protocols on December 12, 2009, but unfortunately each
proposal was based on a number of untested factual assumptions about Revonet’s computer
systems. Accordingly, I set the matter for an evidentiary hearing for the purpose of resolving
those factual disputes. See Memorandum Order (1/13/09) [#67]. The hearing took place on
three days: March 4, 2009, March 24, 2009, and April 15, 2009.
II. The Hearing.
At the hearing, Revonet presented Fred Purdue, who was qualified as an expert in
Information Technology and SQL databases. Purdue is a consultant who, though he did not
2
inspect the servers themselves, spoke to a number of Information Technology (“IT”) personnel at
Revonet. Purdue testified about a number of server failures at Revonet in 2007 after this
litigation was filed. He also opined that it would be risky to make a forensic copy of the servers
because they are very old. Purdue then gave a cost estimate of $47,000 for copying and
analyzing Revonet’s drives, a figure that did not include legal expenses associated with the
actual forensic examination. He did not discuss any individual computers, whether they exist,
whether they should be imaged, or how much that might cost.
Next, Brice Anderson, a former salesperson who is now Director of Channel Sales
Strategies at Revonet, testified. Anderson testified that Revonet tracked phone numbers that
people used to call into Revonet’s call centers to determine where callers found out about Covad.
When calls came in to the call center, Revonet would input information about the caller into a
program called RPM. There was no specific field to indicate that a call was an inbound lead, but
Anderson testified that one could tell that from looking at the entry in RPM, though he did not
explain how.
At some point around the time that Covad and Revonet stopped doing business, Revonet
began the Gold campaign, in which it selected certain leads out of the Federated Database and
copied them over into the Gold campaign. Anderson conducted a search and concluded that
about 2,000 inbound Covad leads were assigned to the Gold campaign, and of that, about 25 of
them were sold to other companies. Anderson also testified that in the summer of 2006, Revonet
caused all leads from Covad that were designated as inbound leads to be hidden, so they do not
appear to most Revonet employees in RPM.
Finally, Covad called Scott Ellis, a forensic computer analyst, to testify as an expert
3
witness. Ellis contradicted Purdue’s contention that it would be extremely time consuming and
expensive to make forensic copies of the drives. Ellis also testified that the fact that Revonet
used a web-based interface for their salespeople to access the Federated Database was important
because computers that access the Internet make temporary Internet cache files, so he would want
to look at those PCs. He would also want to look at the PCs of IT Specialists who were moving
data around. Ellis ultimately gave a rough estimate of $25,000 for the forensic investigation.
During Ellis’ testimony, Covad questioned him about some e-mails that indicate that
leads may have been distributed by e-mail in addition to through the Gold Campaign. Covad’s
counsel indicated that only four external e-mails were produced by Revonet.
III. Findings of Fact.
From that testimony, I make the following findings of fact:
1. Revonet has one database, the Federated Database, that contains all of its campaigns.
Transcript of Proceedings (3/24/09) (“3/24/09 Tr.”) 64:19-24.
2. Within the Federated Database, entries are assigned to different campaigns and identified
as such. 3/24/09 Tr. 64:22-24. Entries are not specifically designated as “inbound” or
“outbound” in the database, however. 3/24/09 Tr. 83:7-11.
3. The Covad campaign dealt with both “inbound” and “outbound” leads. 3/24/09 Tr.
52:24-53:3. Revonet purchased lists of contacts from outside sources and called those
contacts to promote Covad’s services. Those sources were designated as “outbound.”
3/24/09 Tr. 53:5-54:9. Revonet also fielded inquiries about Covad from contacts who
called in to Revonet. Those sources were designated as “inbound.” 3/24/09 Tr. 55:1-16.
4
4. In 2006, Revonet created a new campaign called the “Gold Campaign.” 3/24/09 Tr.
58:17-20. To populate the Gold Campaign, Revonet selected leads from other campaigns
that it considered to be good leads. 3/24/09 Tr. 59:18-60:21. Criteria for inclusion into
the Gold Campaign included past success for another client, such as Covad. 3/24/09 Tr.
60:9-13.
5. Approximately 14,000 leads were copied into the Gold Campaign. 3/24/09 Tr. 76:5-6.
To create the campaign, Revonet IT personnel made complete copies of leads that already
existed in other campaigns and placed the copies into the Gold Campaign. 3/24/09 Tr.
62:9-10. The only difference between the copies was that the field indicating which
campaign the lead belonged to was changed to Gold. 3/24/09 Tr. 83:15-25. The entries
were not moved. Instead, they were duplicated, so the original leads would still appear in
the appropriate campaigns. Id.
6. For inbound leads, Revonet used phone numbers to determine what advertisement or
mailer, or other source the caller was responding to. 3/24/09 Tr. 55:17-22.
7. The Gold Campaign leads were marketed in a program called Leads on Demand. 3/24/09
Tr. 63:23-64:9. Through Leads on Demand, Revonet offered five free leads taken from
the items in the Gold Campaign to prospective customers. 3/24/09 Tr. 66:11-16. It also
sold leads to customers in bundles. 3/24/09 Tr. 67:4-12.
8. Revonet set up a web portal for its customers to access the five free leads and the leads
that were purchased in bundles. 3/24/09 Tr. 74:20-75:6. The web portal was called
OneView, and it was a SharePoint database. Id. The SharePoint database was housed on
the same server as the Federated Database.
5
9. Sally Lefferts was responsible for putting leads on the OneView portal. 3/24/09 Tr. 75:7-
10. Before she made them available to new customers, she removed prior customer data.
10. When Covad was a client of Revonet’s, the Federated Database was housed on a server
called OVNC SQL 02, which was located in New Canaan, CT. Transcript of Proceedings
3/4/09 AM (“3/4/09 AM Tr.”) 33:21-34:1. In July 2007, Revonet moved the SQL 02
server to its office in Sioux Falls, SD. 3/4/09 AM Tr. 34:8-22. During the move, the
server crashed, and the data on the server was lost. Id. Revonet was able to restore the
server using a back-up tape. 3/4/09 AM Tr. 34:23-35:1.
11. When the SQL 02 server data was restored, it was restored onto a server called OVNC
SQL 01. The Federated Database still resides on SQL 01. 3/4/09 AM Tr. 34:23-35:1.
12. In July 2007, the e-mail exchange server that Revonet used in Sioux Falls crashed. 3/4/09
AM Tr. 33:3-6. Revonet sought to restore the server from a back-up tape, but determined
that the server had not been backed up appropriately. Id. Accordingly, Revonet searched
users’ PCs and retrieved e-mail files and loaded those onto an exchange server, Revmail
two, that was located in New Canaan. 3/4/09 AM Tr. 33:7-11.
13. In the fall of 2007, Revmail two began experiencing problems. 3/4/09 AM Tr. 35:3-7.
Revonet IT professionals then purchased some new drives for the retired SQL 02 server,
and repurposed some drives from another retired server to build a new server called
Revmail. 3/4/09 AM Tr. 35:8-11. The data from Revmail two was then moved to
Revmail. Id.
14. Sales associates at Revonet use a program called RPM to access the Federated Database.
RPM is a web-based user interface that is capable of running reports, but generally does
6
not contain any data. 3/4/09 AM Tr. 21:6-15. RPM is located on a server called OVNC
WEB 01. 3/4/09 AM Tr. 26:8-9.
15. The OneView web portal is currently not operational. 3/4/09 AM Tr. 23:2-16.
II. Analysis.
A. Forensic Imaging.
As an initial matter, Covad seeks to make forensic images of Revonet’s drives and
computers to preserve information as it currently exists. The expense associated with taking a
forensic image is small compared with the cost of the forensic search itself and provides both
parties and the Court with the best possible present depository of the crucial information in the
database. The process is obviously not as time consuming as the forensic search itself and
lessens some of the cost of forensic searching since the search can be done off-site rather than on
the premises housing the servers or computers. Any interference with business operations will be
insignificant because I will require the searching to be done over a weekend. While I appreciate
that Revonet’s servers are nearing or past their life span, I credit Ellis’s testimony that age alone
does not render the items so fragile that the process of taking the images will destroy the servers.
Creating a forensic image is no more burdensome than using the server for everyday business
activities and may ultimately benefit Revonet by creating a forensic record of its data before it
uploads the data onto new servers, whether it rents the space (the so-called “cloud
configuration”) or buys new ones.1 For those reasons, Revonet shall permit Scott Ellis to make
forensic images of the following items:
1
There was testimony from Perdue that Revonet is considering transferring its data to
servers that are owned by companies that provide space on their servers for a fee. Access to the
data is then procured by use of the Internet; hence, the appellation that the data is “in the clouds.”
7
Server Characteristics in Intervening events Where it is now
2006
Revmail one Sioux Falls e-mail Crashed in 2007, no unknown, but it has
exchange server back-up was not been repurposed
available – best guess, Sioux
Falls in storage
Revmail two New Canaan e-mail Revmail one e-mails unknown, but it has
exchange server that could be not been repurposed
recovered from user – best guess, Sioux
PCs were loaded onto Falls in storage
Revmail two.
Revmail two was
moved to Sioux Falls
Revmail two began
experiencing
problems and was
retired.
Revmail Did not exist See OVNC SQL 02 Sioux Falls, Revmail
(below) is the e-mail
exchange server for
Revonet.
OVNC SQL 02 Home of the During transport to Now called Revmail,
Federated Database Sioux Falls, an SQL 02 is currently
and SharePoint external drive on the in Sioux Falls and is
databases, located in server crashed or the e-mail exchange
New Canaan sustained damage and server for Revonet.
caused the whole It’s not clear what
server to crash. happened to the
drives that were in
Testimony suggests the original SQL 02.
that the drives from To the extent the
SQL 02 were original drives still
removed, and exist, they should be
replaced with two copied.
drives from a retired
8
ISA server and a few
newly purchased
drives. The
information from
Revmail two was
then loaded onto
SQL 02.
OVNC SQL 01 Don’t know When SQL 02 In Sioux Falls,
crashed, Revonet currently houses the
used a back-up to Federated Database.
restore it onto SQL
01.
OVNC WEB 01 Home of RPM N/A Home of RPM
Personal Computers N/A N/A N/A
that interfaced with
any of the above
mentioned drives
between the date
when Covad and
Revonet entered into
a contract and
December 31, 2006.
If there is any dispute about whether a given computer was used during the relevant time
period, Ellis shall make a forensic copy of the computer in any event and preserve it pending
further order of this Court. The resulting forensic images will be safeguarded by Ellis who will
preserve them inviolate. He shall only use these forensic copies of the devices or electronic
depositories at issue to conduct forensic searches.
B. Forensic Searching.
1. Federated Database, SharePoint Database, and RPM (OVNC SQL 01, OVNC SQL 02,
and OVNC WEB 01).
Revonet has explained that its databases were housed on a back-end database server. In
9
the Federated Database subsets of data are segregated by “campaign.” Thus there is a Covad
campaign, an XO campaign, a Verizon campaign, a Gold campaign, etc. within the Federated
Database. Revonet claims that Covad inbound leads would only appear in the Covad campaigns
and the Gold campaign. Accordingly, Revonet argued in its proposed protocol that it should only
have to produce those two campaigns because nothing else would be relevant.
Covad maintains that it needs access to full forensic copies of the drives because it needs
access to historical data about the database, not just the database itself. Much of that historical
data will not be visible by looking at the database alone.
Rule 34 permits a party to “copy . . . electronically stored information” but is limited by
Rule 26, which allows only for discovery of “any non-privileged matter that is relevant.” Fed. R.
Civ. P. 26(b)(1).2 Revonet does not argue that the information at issue is privileged, because it
falls within a recognized common law privilege but only that it is confidential because it may
expose information that Revonet is obliged by contract to keep confidential. Confidentiality is
not a basis for withholding information in the ordinary course if it can be protected by a
protective order (which I will gladly sign) restricting access to confidential information to certain
persons and prohibiting its use for any purpose other than the limited purpose of the forensic
copying and searching that I am ordering. Furthermore, the testimony of Purdue and Ellis
suggests that there will likely be pieces of information located on the server but outside of the
Federated Database that would tend to show what Revonet did with the data because of the way
that SQL databases usually work and how the electronically stored information was used by
2
Note that “relevance” is, in turn, defined to include information “reasonably calculated
to lead to the discovery of admissible evidence.” Fed. R. Civ. P. 26(b)(1).
10
Revonet. As I have explained, that information is certainly relevant. Indeed, it is the very heart
of this lawsuit.
The advisory committee note to Rule 34 cautions against making forensic examination
the default, however, and encourages courts to “guard against undue intrusiveness.” Fed. R. Civ.
P. 34, Advisory Comm. Note (2006). Because these examinations raise issues of confidentiality
and can produce thousands of documents that have to be reviewed for relevance and privilege,
“compelled forensic imaging is not appropriate in all cases, and courts must consider the
significant interests implicated by forensic imaging before ordering such procedures.” John B. v.
Goetz, 531 F.3d 448, 460 (6th Cir. 2008); see also White v. Graceland Coll. Ctr. for Prof. Dev. &
Lifelong Learning, Inc., No. 07-CV-2319, 2009 WL 722056, at *7-8 (D. Kan. Mar. 19, 2009)
(quoting Advisory Committee Notes to 2006 amendments for the proposition that forensic
imaging is not appropriate in all cases and noting that courts permit imaging when there is trade
secret or electronic evidence at issue, or where discovery responses suggest some inconsistency
or impropriety).
The precise scope of the limitations suggested by the 2006 Amendment to Rule 34, as it
applies to forensic inspection of a relevant object as opposed to a forensic search for documents,
is unclear because the sections of that Rule denominated “Procedure” do not specify what, if any
requirements, must be met before a court permits the inspection. The request must, however, be
a request “within the scope of Rule 26(b).” Fed. R. Civ. P. 34(a).
Rule 26(b)(2)(B) provides that a party “need not provide discovery of electronically
stored information from sources the party identifies as not reasonably accessible because of
undue burden or cost.” Despite that showing, the court may nevertheless order discovery from
11
such sources if the demanding party shows good cause. Id. In making this determination, the
court must “consider[] the limitations of Rule 26(b)(2)(C).” Id. Those limitations are, of course,
those that pertain to all discovery, whatever form it takes, and require a court to limit the extent
of discovery if it determines that (1) the discovery sought is unreasonably cumulative or
duplicative or can be obtained from other sources that are more convenient, less burdensome or
expensive; (2) the requesting party has had ample opportunity to obtain the information by other
discovery; (3) “the burden or expense of the proposed discovery outweighs its likely benefit,
considering the needs of the case, the amount in controversy, the parties’ resources, the
importance of the issues at stake in the action, and the importance of the discovery in resolving
the issues.” Fed. R. Civ. P. 26(b)(2)(C).
Since the latter rule pertains to all discovery, whether forensic inspection of an object
under Rule 34(a)(1)(A) should also be viewed through the prism of the data on it not being
reasonably accessible under Rule 26(b)(2)(B) (that requires good cause to search for data that is
not reasonably accessible) is of little moment. As a form of discovery, forensic inspection is
unquestionably subject to the balancing required by Rule 26(b)(2)(C) whenever any discovery is
challenged as an undue and unnecessary burden. I am therefore required to use that calculus to
ascertain whether the forensic imaging should be permitted.
As I have explained, Revonet argued that forensically searching the Federated Database
server is unduly burdensome because the servers to be searched are old and may crash, its
business will be disrupted while the search is conducted, and the search may yield data that is
subject to confidentiality agreements with other clients. None of these arguments are
compelling. The searches will be done on copies of the forensic images, so the servers
12
themselves will not be affected after the initial copies are made. The search will be conducted on
the weekend and concerns about confidentiality can easily be alleviated through a protective
order.
On the other hand, the “needs of the case . . . the amount in controversy . . . the
importance of the issues at stake in the action, and the importance of the discovery in resolving
the issues,” Rule 26(b)(2)(C)(iii), surely establish that the potential benefit of the forensic search
outweighs its burden.
First, there is simply no other way in which to seek this information, and it should
establish once and for all what Revonet did with the Covad data that was in the Federated
Database. Indeed, the investigation may have a determinative impact on whether this case will
survive. If the forensic imaging shows little use of the data from Covad by Revonet, Covad’s
financial motivation to continue this lawsuit may disappear or at least lead to an early settlement.
If, on the other hand, the data contradicts Revonet’s assertions, made throughout the lawsuit, that
Revonet made little use of the Covad data, then the shoe will be on the other foot. In either
event, the data simply must be found and analyzed before the parties and the Court can get their
hands around what this case is truly worth. It is simply one of those perhaps unusual cases where
paradoxically “the amount in controversy,” “the likely benefit,” and the “needs of the case” can
only be ascertained by permitting the discovery sought.
Second, while the issues at stake in this commercial controversy may be not be of interest
to any one but these litigants, they still have a right to its speedy and just conclusion, which, in
my view, will be advanced by the forensic imaging. I know of no other way to examine the data
and how it was used by Revonet.
13
Third, while Revonet’s complaints at the hearing that it is in difficult financial
circumstances may have merit, the forensic searching may produce the information it needs to
decide how much more of its resources (including money for legal fees) to devote to this case.
Thus, on balance, I conclude that Revonet’s objections to the forensic searching, while
meritorious, are overwhelmed by the necessity of the forensic searching that I am ordering. I will
also meet Revonet’s concerns about confidentiality with a protective order. I will therefore order
that Ellis shall complete a forensic search of the OVNC SQL 01, OVNC SQL 02, and OVNC
WEB 01 to capture data of whatever type or kind that would tend to show (1) Revonet’s receipt
of information from Covad or (2) Revonet’s modification of the information from Covad, or (3)
Revonet’s use of the information from Covad in the period (a) from the date of the contract
between Covad and Revonet until December 31, 2006, or (b) from the date of the contract
between Covad and Revonet to the present. Ellis shall then provide the parties with a written
report of his findings. He shall also retrieve the information and make it available, in a
reasonably usable format, to the parties and their counsel.
2. E-mail servers (Revmail one, Revmail two, and Revmail).
In addition to conducting a forensic examination of the database servers, Covad wants to
forensically examine Revonet’s e-mail exchange servers. Covad offers three justifications in
support of its request: (1) Revonet’s e-mail production is facially incomplete; (2) Revonet’s e-
mail production suggests that it was using e-mail to give leads to clients; and (3) Revonet’s
servers failed after litigation was initiated but before discovery commenced in earnest.
There is certainly authority for the proposition that if a party’s e-mail production suggests
that she is intentionally hiding things, or failing to take appropriate steps to respond to discovery,
14
a forensic examination may be appropriate. Preferred Care Partners Holding Corp. v. Humana,
Inc., No. 08-CV-20424, 2009 WL 982460, at *14-15 (S.D. Fla. Apr. 9, 2009) (allowing forensic
exam at producing party’s expense because of discovery failings, inconsistencies and defendant’s
“print and purge” campaign); Peskoff v. Faber, 244 F.R.D. 54, 59 (D.D.C. 2007) (allowing
forensic examination because documents were conspicuously absent in certain time periods and
circumstances suggested that absence was the result of party’s misconduct). However, it does
not automatically follow that every time a litigant alleges that the other party’s electronic
production is deficient in some regard the appropriate next step is to conduct a forensic
examination. See Exec. Air Taxi Corp. v. City of Bismark, N.D., 518 F.3d 562, 569 (8th Cir.
2008) (finding no deficiency with e-mail production to warrant forensic examination; party’s
mere suspicion not enough); Sup. Prod. P’ship v. Gordon Auto Body Parts Co., Ltd., No. 06-CV-
916, 2009 WL 690603, at *2-3 (S.D. Ohio Mar. 12, 2009) (refusing to allow forensic search as a
sanction for discovery misconduct where party seeking exam had not provided any evidence that
relevant e-mails had been lost or destroyed by inadequate litigation hold).
It is the rare case that a litigant does not allege some deficiency in the production of
electronically stored information, particularly e-mail. All too many entities lack records
management policies that are aggressively enforced, and records keeping may be a function not
of an enterprise wide policy but determined by the idiosyncratic habits of the various users. In
such a situation, the possibility that one user saved everything while another saved nothing may
lead to curious gaps in the e-mails that are produced and an inability to explain why any are
missing. While such productions cast little glory on the companies that produce them, I cannot
find any authority in the cases to date that permit a court to conclude that allegations of
15
deficiencies in themselves automatically require a forensic search whenever a party claims that
there are, for example, fewer e-mails from a person or about a subject or transmitted in a given
time than the party expected to find. This would result in forensic examinations in virtually
every case, which would increase the cost of litigation involving electronically stored information
markedly not only because of the cost of the examination itself, but also because it would yield
information that would have to be sorted for relevance and privilege. Experience shows that the
latter, involving expensive reviewing of “e-mail strings” by lawyers, paralegals or by vendors to
whom the work has been outsourced, may dwarf the cost of the search. Fear of the consequences
of consistently ordering forensic searches of e-mail servers should, at a minimum, require a
showing that permits the finding, albeit based on circumstantial evidence, that the producing
party has not complied with Rule 26(g) because the production is not “complete and correct as of
the time it was made.” Fed. R. Civ. P. 26(g)(A).
As evidence of Revonet’s alleged shortcomings, Covad identified e-mails that had been
produced recently, noted that only four external e-mails had been produced, and identified
various alleged deficiencies with search terms. Notably, however, Revonet asked Covad’s
counsel repeatedly to suggest search terms, and Covad’s counsel did not respond to any requests.
Given that history it is unfair to allow Covad to fail to participate in the process and then argue
that the search terms were inadequate. This is not the kind of collaboration and cooperation that
underlies the hope that the courts can, with the sincere assistance of the parties, manage e-
discovery efficiently and with the least expense possible. See The Sedona Conference
Cooperation Proclamation (2008), available at http://www.thesedonaconference.org/content/tsc_
cooperationproclamation.
16
While there is some evidence to suggest that Revonet’s production may have been
incomplete, this is a much closer case than Peskoff or Preferred Care Partners (discussed above).
Fortunately, I may be in a better position to assess the completeness of Revonet’s production
because of the forensic search of the databases that I am permitting. That search may yield
important information about the use of e-mail by Revonet that may permit more certain
conclusions about whether a search of the e-mail servers can be justified under Rule 26(2)(C).
Accordingly, I will reserve decision on whether a forensic examination of the e-mail servers to
look for items that should have been produced in response to Covad’s requests is appropriate
until Ellis has submitted his expert reports and the parties have had the opportunity to confer and,
if necessary, brief the issue of whether, in light of what he has found, a forensic search of the e-
mail servers is appropriate.
Covad does provide compelling justification for one forensic search of e-mail servers that
I will permit.
The original Revmail one server that once resided in Sioux Falls crashed after this case
had been filed, and there were no back-ups. Revonet had a responsibility to take reasonable steps
to preserve relevant information once it had notice of litigation. While the destruction of e-mails
here was clearly inadvertent, Revonet has a responsibility to take appropriate steps to try to
recover those e-mails, and it is not clear that it has ever done so. Orrell v. Motorcarparts of Am.,
Inc., No. 3:06-CV-418, 2007 WL 4287750, at *7 (W.D.N.C. Dec. 5, 2007) (“The fact that
plaintiff’s home computer allegedly ‘crashed’ – as opposed to having been ‘wiped’ as the work
laptop was – in no way eliminates the Plaintiff’s burden to do all she could under those
circumstances to preserve evidence.”). There was no evidence presented regarding what efforts
17
Revonet took to extract data from the two e-mail servers that have since been retired, particularly
the first one, which crashed completely. Because we need to determine what, if any, data was
lost before evaluating what effect the server crash had on the adequacy of production, Ellis
should first examine the forensic images of Revmail one and Revmail two to determine whether
it is possible to do a forensic search of either of them.
The only e-mail server that is currently operational at Revonet is Revmail. Thus,
Revonet’s document request responses would have come from searches run on Revmail.
Revmail one and Revmail two were both in operation during the time when Covad and Revonet
were working together. Thus, it is very possible that there were e-mails created during the time
that Revmail one and Revmail two were operational that would be relevant to this case. What is
not clear, however, is the degree to which data may exist on Revmail one or Revmail two that
was not transferred to Revmail. Thus, a comparison should be made between Revmail one and
Revmail and between Revmail two and Revmail to determine what, if any, data exists on the
non-operational servers that does not exist on Revmail. These comparisons should yield the
universe of e-mails that have not yet been considered for production (if Revmail one and
Revmail two can be searched in their present states).
The ultimate goal of any such search is to ascertain whether there is any recoverable data
on Revmail one and Revmail two that would tend to show (1) Revonet’s receipt of information
from Covad, or (2) Revonet’s modification of the information from Covad, or (3) Revonet’s use
of the information from Covad in the period (a) from the date of the contract between Covad and
Revonet until December 31, 2006, or (b) from the date of the contract between Covad and
Revonet to the present. Thus, Ellis shall search the universe of data on all three servers to
18
ascertain whether there is data on Revmail one and Revmail two that is not on Revmail and that
meets the criteria I have just identified. He shall submit a written report to the parties explaining
his conclusion that there is, or is not data that fits the descriptions I have laid out in this section.
He shall carefully identify the data he has found that meets the criteria (1) and (2) that is on either
Revmail one or Revmail two but is not on Revmail. Once he does that we shall address the next
step to be taken. I note here only that I appreciate that if a search for additional e-mail becomes
appropriate the necessity of reviewing the data for possibly privileged material may arise that
may not be eliminated by the protective order I sign because, as I have already noted, there is a
difference between privileged and confidential material. I note here my intention to do
everything possible and in particular to use the provisions of Rule 502 of the Federal Rules of
Evidence to forge an agreement that will reduce the necessity and cost of such a review to the
minimum.
3. Individual PCs.
Like the Federated Database, the individual PCs are relevant because they may contain
information that demonstrates how the relevant data was used at Revonet. Accordingly, Ellis
shall create a forensic image of each PC and conduct a search of it to ascertain if it contains data
that would tend to show (1) Revonet’s receipt of information from Covad, or (2) Revonet’s
modification of the information from Covad, or (3) Revonet’s use of the information from Covad
in the period (a) from the date of the contract between Covad and Revonet until December 31,
2006, or (b) from the date of the contract between Covad and Revonet to the present. If he finds
indications that such data exists, he shall report to the parties what indications he has found and
whether he believes additional searches will yield additional data.
19
4. Additional Searching.
Once Ellis has submitted his reports pursuant to this Opinion, the parties shall meet and
confer within one week to discuss whether any additional searching would be worthwhile and
justified considering the anticipated costs. The parties are certainly free to agree to any further
searches, but that agreement must be in writing and provided to Ellis as instructions for the
search. If the parties cannot agree, the party who seeks to conduct additional searches must file a
motion for additional searching explaining why the results of the search conducted, the history of
the controversy between the parties and of this litigation, the controlling law and any other factor,
justify the additional search sought. That party shall indicate in the filing whether it will pay the
cost of such additional searching and, if not, why the cost should be shifted to its opponent.
C. Costs.
It has long been the presumption in this country that the producing party bears the cost of
production and “[a]ny principled approach to electronic evidence must respect that presumption.”
Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 317 (S.D.N.Y. 2003). “Production” of
electronic information can quickly become quite expensive and disproportionate to what is truly
in stake in the lawsuit. For one, retrieving the information, even when not done through forensic
methods, may still require the assistance of a technological expert or consultant to provide
guidance as to how the search should be conducted to gather the requisite information at the
lowest possible cost. Additionally, the universe of items to be considered for production is ever
expanding with the ubiquity of e-mail and other forms of electronic communication, such as
instant messaging and the recording of voice messages. Electronic data is difficult to destroy and
storage capacity is increasing exponentially, leading to an unfortunate tendency to keep
20
electronically stored information even when any need for it has long since disappeared. This
phenomenon–the antithesis of a sound records management policy-leads to ever increasing
expenses in finding the data and reviewing it for relevance or privilege.
As this case demonstrates, these new technologies have the capacity to be outcome
determinative but often at significant expense. Thus the courts are required to strike a balance
between allowing the requesting party to take full advantage of the technologies available to it
and protecting the producing party from having to pay to leave no stone unturned. Resting all of
the costs of electronic discovery on the producing party may create a perverse incentive on the
part of the requesting party to dispense with reason and restraint and unleash every new
technology under the sun to try and find information that supports the requesting party’s claims.
Fortunately for me, the financial burden of “producing” electronic information in this
case is relatively low, and I will require Revonet to bear that cost. When I say “producing,”
however, I am referring to the costs of satisfying Revonet’s obligations under the Federal Rules
of Civil Procedure to either (1) make items available for examination, or (2) produce documents.
See Fed. R. Civ. P. 34. For the devices or depositories that I have determined that Revonet must
make available for examination: OVNC SQL 01, OVNC SQL 02, OVNC WEB 01, and user
PCs, Revonet satisfies its obligation to produce those items by paying for a forensic copy of those
items to be made. Once Revonet has made those items available to Covad’s expert, I see no
reason why Revonet should be required to pay for its opponent’s expert witness to analyze the
evidence. Accordingly, I will require Covad to bear the costs of conducting the analyses I
described in §§ III.B.1 and III.B.3 of this Opinion once I have signed any Protective Order that
Revonet and Covad prepare for me.
21
For the items that I am ordering searches of for the purpose of producing documents:
Revmail one, Revmail two and Revmail, I will order that Revonet bear the costs of imaging these
servers and the costs of the limited searches that I have ordered in § III.B.2 of this Opinion. I am
requiring Revonet to bear this cost based on my conclusion that Revonet should have taken all
reasonable steps to recover the allegedly lost data at the time it was lost.
I will consider the question of who should pay for any additional searches if and when
that issue is before the Court.
III. Conclusion.
For the reasons discussed herein, Revonet shall make the items I have identified available
for imaging and searching. An Order accompanies this Memorandum Opinion.
Date: May 27, 2009 /S/
JOHN M. FACCIOLA
U.S. MAGISTRATE JUDGE
22