Monday, October 18, 2010

Back up and backup!

This is my first post on BroadBand Nation as a guest, thanks to Michael/FreedomFire for the kind invitation.

Isn't it just so tedious to hear someone bleat on about how important backups are? Well, yes it is, until disaster strikes & you find yourself with CATASTROPHIC data loss and NO BACKUP... suddenly backups become very, very useful. I found myself in just that situation a couple of weeks ago, so here's my personal tale of woe and some advice to readers to prevent this happening to you...

I thought I'd be smart and install a webmin module to administer my Asterisk PBX. I've been casually looking for such a module for the last few years, ever since I first installed asterisk at my home office. My Asterisk installation is quite small, providing VOIP services over my ADSL connection. I have a handful of SIP extensions, a couple of SIP trunks, a wired SIP deskphone, a SIPURA FXS adaptor for a standard DECT cordless and a SIP client on my Android phone. Anyway, I came across the Thirdlane asterisk PBX manager module for webmin, so thought I'd give it a go. I happily downloaded it and installed it through the Webmin interface, and was quite un-impressed how it didn't seem to pick up any of the details of my existing Asterisk instalation. I was just about to start poking around in the setup instructions/README files when I noticed my SIP deskphone's normal display had been replaced with the lost-comms-with-the-server-message "Waiting for proxy server..." - oops!

A slow realisation of what might have happened gradually dawned on me.... I ssh'd into the Asterisk's Linux box and manually checked the /etc/asterisk directory.... Disaster! All my config files had been overwritten with default files from the Thirdlane webmin module! I began to sweat. Where was my backup??? I knew that I'd manually backed up the asterisk config a long while (couple of years!) ago, but was not confident I knew where the backup was. I knew I'd made several changes to the PBX in the mean time, several of them only recently, so I knew I was in trouble!

This quickly taught me 2 things: 1. the importance of backups and 2. Even without a regular backup, how I wished I'd backed-up manually before installing the webmin module! A little less haste would have saved hours of stress.

After 30 minutes of searching old backup disks, I eventually found the most recent backup - which was dated Feb 2008! Oh dear. This was going to be fun. I over-wrote the /etc/asterisk directory with this old version & set to work. Lucky really that I had made some changes in the last couple of weeks, so I had a pretty good picture in my head of what the config should be, so I was quite quickly able to get the PBX back into shape.

To prevent being left without recent backups in the future, I knocked up a quick shell-script to be run by cron that zips up the /etc/asterisk directory and scp the resulting zip file to another machine that is also running dropbox, then from there dropbox does its magic and copies to a couple of other machines (one at another location) and also backs up to the cloud.

Disaster averted but it chewed up a couple of hours of my time that could have been better spent!

So I guess the moral of the story is threefold:

  1. Make sure you have a backup regime in place that backs up both locally and to another physical location (i.e. "The Cloud" or another off-site location). A golden rule of backups is that it's not really a backup unless it exists in two physically seperate locations!

  2. Make sure you're notified if the backup process fails for whatever reason.

  3. Bite the bullet and try and restore a backup. It's no good saving all that data unless you know it's good! This isn't as onerous as it sounds, virtualisation packages (like VMWare, virtualbox etc.) are invaluable for creating test instances of systems that you can try and restore data onto...

After all that, and to add some more to my pain, in my haste to install I hadn't noticed that the Thirdlane PBX manager module for webmin that so kindly overwrote my asterisk config was only a trial-licenced version, and would only work with less that "4 lines". So it wouldn't interact with my config at all. Thanks. A. Lot.

So, once more, back up & BACKUP!

Labels: , ,


Post a Comment

<< Home