Let me tell you a scary story.
When we plan our backups, the idea is to have a duplicate copy of our data on something separate from where it is normally stored and used. In theory, it is unlikely that both sources will fail simultaneously.
Even very small businesses would be devastated by a loss of data or an extended network outage. On Monday I had a simultaneous failure of a server and the primary backup device. We were dealing with the potential loss of a lot of data and a long outage indeed.
The firm will live on because there was a secondary backup device.
The moral of the story is: if your data is business critical, pay attention to your backups and use more than one device to hold them.
Fortunately the call this Monday morning does not happen frequently. The server running Small Business Server 2003 in a small law office was not responding.
- The firm’s Internet connection runs through a second network card in the server, so no one had Internet access. (That used to be the preferred way to set up Small Business Server. The proliferation of inexpensive firewall devices has changed that and SBS 2008 will not support that setup; instead it will assume that small businesses have a SonicWall or Snapgear firewall/router or something like it.)
- Firm documents were completely inaccessible. Individual My Documents folders could be opened (they’re stored on the server but a local copy is stored on the individual workstations using “Offline Files”) but they’re rarely used – everything important is in the shared “Company” folder.
- Outlook is running in cached mode so all copies could be opened, but of course no mail could be sent or received.
- Specialized programs run from the server were unavailable – Abacus, Timeslips.
A very bad thing.
The server is aging and a little underpowered, running on a single IDE hard drive. The symptoms made me think that the hard drive had failed. I got a replacement and stood ready to restore the server from the backup image.
I could not open the file folder where the backups were stored.
The primary backup device was a Buffalo Terastation Pro II, less than a year old. I’ve set up several of them, I use one myself – 1Gb network attached storage, with four 250Gb hard drives running in a RAID5 array, meaning if any one of the hard drives fails, the box continues working with no interruption other than a beep to remind you to change the bad drive.
I hooked up my notebook, set the IP address within the range being used by the Terastation – and I couldn’t open the file folder on the Terastation.
I opened Internet Explorer and got a login screen to the Terastation’s web interface but it would not finish loading the main screen so I could use any of the Terastation’s built-in tools.
The next two hours were spent trying to talk to the Terastation – confirming IP addresses, checking firewall settings, hooking the Terastation up to my office network in case it wanted a working DHCP or WINS server, trying to do soft resets, pulling various combinations of the hard drives, and a lot of other things. At the end of that time it threw up a “Kernel error” message on its little LCD screen and nothing I did gave me any hope that it would recover. Sure, I’ll call Buffalo for warranty support tomorrow but that wasn’t going to help a law office that was completely down.
We had also been doing backups to external hard drives on Thursday night, and swapping between two external USB hard drives every Friday.
I was able to use the external hard drive to restore the server to life as of Thursday night at midnight. Documents created or edited on Friday are lost but Outlook is completely up to date – all changes on Friday were synced from the offline copies, and Exchange Defender delivered all mail that arrived while the server was down.
Do you appreciate why I was sweating? If we hadn’t had that second backup device, we’d have been left staring at each other with nothing to do but sharpen knives – seppuku in my case, murder in my client’s case.
I’ve got some Small Business Server clients that are not currently using two different backup devices. I’ll be contacting them to urge them to buy more external hard drives or an NAS or whatever will provide extra redundancy.
If you’ve got a single external hard drive for your computer, get another one and rotate them.
If you’re backing up onto CDs, buy an external hard drive and start using it for backups – and occasionally keep backing up onto the CDs!
Look into online backups, but also use a local device.
Backup backup backup! I hate losing data!
My guess is that the hard drives in the Terastation are just fine and the failure is deeper in the hardware. (The Terastation was working at least until Friday night. There was no indication of a power surge or something else that took out both the server and the Terastation. This is a very weird coincidence.) A Google search turns up lots of complaints about the Terastation. I take that with a grain of salt because every device has generated a score of complaints that could be turned up in a Google search.
There is a part of this story that is full of magic and light and goodness. The software used by this client is StorageCraft ShadowProtect, and it is just swell. I’ll tell you about it someday.
