The Autopsy of Journalspace.com
Or: Mirroring is not Backup.
Or: Lock his accounts, fire him, and escort him out.
This is a sad story about how a six year old company was done in by bad backup strategy, poor HR practices, and bad network administration. Some of the facts aren’t exactly clear, but here’s what I’ve been able to piece together so far.
Journalspace.com was a free blog web site that hosted its servers with an outside hosting company. Sometime in the past (it’s not clear when) an employee of the hosting company was caught stealing and was fired.
On his way out, the bad guy sabotaged some servers. There is speculation that Journalspace’s server was among them. It could have been done by someone else, or by an application failure, or an accident; but whatever the cause, Journalspace’s main database was over-written, erasing all the data in it.
Journalspace’s backup plan was simple – Just RAID the drives! What could be better than mirroring plus automatic failover? It’s WAY faster than traditional backups, it’s more automatic, and it’s built right into the operating system. It’s faster, better and cheaper than backups!
RAID works constantly and automatically. Sometimes you get an automated email that says, basically, “Hey, it’s all OK now, but sector 1012 had an issue last night and I copied from the Mirror, marked the sector, and there was not a microsecond of downtime. – your little RAID buddy, lookin’ out for you.”
Unfortunately for Journalspace, their backup plan did exactly what it was supposed to do – mirror changes from one drive to the other, in real time. So, when the database containing all the company’s (and their customers’) data was erased, RAID mirrored the erasure to the other drive, in real time.
“Hey, everything’s OK. I heard you erased your main database, so I went ahead and also erased it for you on the Mirror – your little RAID buddy, lookin’ out for you.”
Journalspace sent their drives to DriveSavers, who specializes in recovering data from hard drives, but were unsuccessful. With all data and credibility gone, the company publically admitted its mistakes, apologized, put their domain name on EBay, and closed up. As of December 8, the bidding is at $2600.
So, what killed Journalspace? Erasure of the main database. They should have had a backup of the main database offsite where the bad guy couldn’t erase it. They shouldn’t have confused “Mirroring” with “Backup.” A proper backup could have easily saved the company.
But there were contributing factors. The hosting company, upon finding out about the bad guy stealing from the company, should have (in rapid succession) disabled his access to the network, fired him, and escorted him out of the building immediately, without allowing him to touch a keyboard. That sounds cruel and embarrassing, but it is the proper way to terminate a formerly trusted employee, especially in high-tech.
Many people with their egos on the line and not much time to think can, in anger, temporarily justify almost any behavior. Jails are full of them. This bad guy probably really thought it was a good idea to sabotage computers belonging to his company. He probably thinks differently now.
This is probably best summed up by another blogger on this subject who paraphrased Strother Martin (Cool Hand Luke, 1967), “What we’ve got here is a failure to administrate.”
Failure to properly administrate the network, the company’s security practices, its vendor, and the vendor’s HR practices all contributed to Journalspace’s downfall.
Having a proper backup instead of simple Mirroring could have saved the company.
