City Hall

No Malicious Intent in Dallas Data Loss, Investigators Say

The city manager, assistant city manager, chief financial officer, chief information officer, former council member Jennifer Gates and the police department, all knew about the data loss months before telling anyone else.
The city manager, assistant city manager, chief financial officer, chief information officer, former council member Jennifer Gates and the police department, all knew about the data loss months before telling anyone else. Michael Förtsch on Unsplash
With a few clicks of a mouse and a handful of key strokes, more than 23 terabytes of city data disappeared early last year. One former staffer with Dallas’ Information and Technology Services Department is largely to blame, according to independent investigators hired by the city. They say it was all a preventable accident.

Back in August, the Dallas County District Attorney’s Office sent out a press release saying they’d recently learned several terabytes of police data was deleted. Other city officials knew about the data loss as early as April but neglected to tell anyone. When he found out in August, Dallas Mayor Eric Johnson said he felt "blindsided" by the news.

Four months ago, Dallas hired the law firm Kirkland and Ellis to investigate how the data loss happened and how another like it could be prevented in the future. This week, Kirkland and Ellis released the results of its investigation, which was led by former U.S. Attorney Erin Nealy Cox, who's now a partner at the firm.

The investigation involved a review of city and police records, as well as interviews with 28 city and third-party witnesses and multiple follow-up interviews.

The probe found that the data loss was the result of a series of actions by a city IT tech who was responsible for system backups and archiving in late March 2021. He was the “backup technician.”

In fall 2020 and spring 2021, the city’s IT services migrated Dallas servers from the cloud to servers at City Hall. They did this because data usage on the cloud was more expensive than the city had budgeted for. The city had a secondary cloud storage facility in Arizona at the time that was contributing to the escalated costs.

The backup tech started accidentally deleting data when the migration began in January, not in March as previously stated by the city.

During the migration process, the report says, the backup tech made some mistakes. He didn’t copy DPD data from the existing cloud-based storage systems to the new City Hall servers. Instead of copying the actual data, he copied the placeholder files that indicated the data had been archived. Thinking he’d properly backed up the data to the City Hall servers, the backup tech deleted the data from the cloud.

A terabyte of data is equal to about "6.5 million document pages, commonly stored as Office files, PDFs, and presentations," according to cloud storage company Dropbox.

While migrating the servers, the backup tech also deleted settings in a city software responsible for archiving DPD family violence and other police data. This resulted in the loss of even more data.

All of this seems to have happened because of the backup tech’s flawed understanding of the city systems he worked on, and it’s the result of a lack of training. The backup tech had only received two training sessions since 2018.

“During the course of our investigation, we did not uncover any evidence that the backup technician had malicious intent or criminal purpose in deleting the data,” the investigative report said. “Ultimately, any conclusion in this regard will be up to the appropriate law enforcement agency tasked with investigating this matter.”

Approximately 23.94 terabytes were lost during the migration. Only 3.26 terabytes were recovered from other sources. The investigators said all users should consider the rest of the data permanently lost and unrecoverable in its original format.

"All of us are cogs in a machine. We all have to do our part to make it work. The problem is, the parts are people and people make mistakes." – Rebekah Perlstein, attorney

tweet this
Other than the cost, the investigation determined the effects of the data loss to be relatively limited. With a couple exceptions, DPD reported that every file identified as potentially lost was found in another location. The DA’s office also reported to investigators that as of January the data loss hasn’t had a significant impact on prosecutions. But, this could change.

“It remains possible that there will be additional impacts in the future, although ITS, DPD, and the DA’s office are taking steps to minimize any future impact,” the report said.

Rebekah Perlstein, a criminal defense attorney with the law offices of Scott H. Palmer, P.C., said that to her knowledge none of her cases have been affected. But that doesn't mean they haven't been. “You may not be able to see the effects right now, but that doesn’t mean they’re not there," she said. “If history tells us anything, it’s that effects are not immediate."

She also said it's difficult to determine whether her cases were affected because they don't always get access to all the available data. In a child pornography case she was handling, evidence provided by police didn’t include some photos and a video interview of her client. It looked like “essentially, the entirety of the case” was missing, she said.

Months went by until the photos and taped interview were produced by prosecutors. She said there’s no way for her to know if that was the result of a data loss or someone just not uploading all the available evidence.

“I don’t know what else DPD has that they didn’t upload,” she said. “All of us are cogs in a machine. We all have to do our part to make it work. The problem is, the parts are people and people make mistakes.”

A few things could have been done to prevent this from happening the way it did. IT services could have implemented safeguards, like enabling the “soft delete” feature in the city’s cloud-based storage facility, which would have made the data recoverable. The IT department could have also kept a second copy of the migrated data until it ensured the migration was successful.

The backup tech didn’t have adequate training on the city systems. While investigators said he should have asked the department for more training on the systems, his superiors should have made sure he was adequately trained in the first place.

"It remains possible that there will be additional impacts in the future, although ITS, DPD, and the DA’s office are taking steps to minimize any future impact." – Kirkland and Ellis

tweet this
City officials also shouldn’t have waited months to tell anyone about the data loss. It should have been identified as a critical incident, and the city’s incident response plan should have been activated at the time the data loss occurred.

Recovery efforts are still underway at the city’s IT department. They’re looking for copies of potentially lost documents in other city systems using keyword searches to find affected cases. So far, the IT department has gone through about 36% of the nearly 17,500 cases affected by the data loss. “It is not an effort to recover the original data, which ITS understands to be permanently lost,” the investigators said.

The investigators still aren’t sure if the city’s recovery efforts will be worth it when they’re done, which will be some time later this year. This is because the data isn’t recoverable in its original format and many affected files may never be needed in the future. Moving forward, the investigators recommended the city be selective in what data it works to recover first. All of this could cost as much as $750,000 in time and city resources.

To prevent such a data loss in the future, investigators said, the IT department needs better procedures for data migration; the city should make sure every department’s IT needs are covered; and DPD should consider hiring a chief information officer who fully understands the department’s IT systems and needs, as well as an advocate for them when budget season comes around.
KEEP THE DALLAS OBSERVER FREE... Since we started the Dallas Observer, it has been defined as the free, independent voice of Dallas, and we'd like to keep it that way. With local media under siege, it's more important than ever for us to rally support behind funding our local journalism. You can help by participating in our "I Support" program, allowing us to keep offering readers access to our incisive coverage of local news, food and culture with no paywalls.
Jacob Vaughn, a former Brookhaven College journalism student, has written for the Observer since 2018, first as clubs editor. More recently, he's been in the news section as a staff writer covering City Hall, the Dallas Police Department and whatever else editors throw his way.
Contact: Jacob Vaughn