Monday 23 August 2010

Backing Up My Life With Amazon Web Services

Amazon's Simple Storage System or S3 for short, is what I have decided to use from now on (for now until I discover yet something else) to backup my life.

S3 seems to be the solution I have been looking for for ages. A simple, flexible, no-contract, reliable and easily accessible storage facility that resides entirely in the cloud. Its cost is based on exactly what you use; rather than renting a fixed amount of space on some server, you just pay AWS for what you use of their services. So if you upload N amount of data you will be charged for storing N amount of data per month. If you access that data for COPYing GETting or PUTting commands then you will be charge per command. So you can use it as little or as much as you like.

There is a perfect little application called s3sync which is a small ruby program which syncs data you specify to S3. It works very much like rsync. This is ideal as it is a command line program so I can run it on my web server which contains a lot of data I wouldn't like to loose, such as family photos/videos.

I altered my backup script to now use this new method, based on a great tutorial I found on the web, which is dated as 2006, so this has been around for a long time.
#!/bin/bash #Backup script for server

#set variable of date for labelling date=`date +%F`

cd /home/user/backupdata/

#remove oldest mysql backup rm `ls -t *mysql* | tail -n 1`

#Dump mysql databases mysqldump --all-databases > /home/user/backupdata/${date}_mysql_backup

cd /home/user/bin/s3sync/ export AWS_ACCESS_KEY_ID=My Amazon Access Key ID
export AWS_SECRET_ACCESS_KEY=My Amazon Secret Access Key
export SSL_CERT_DIR=/home/user/bin/s3sync/certs
export AWS_CALLING_FORMAT=SUBDOMAIN
ruby s3sync.rb -r -v --ssl --exclude="/home/user/some/dir" --delete /home/user mybucket:
ruby s3sync.rb -r -v --ssl --delete /var/www mybucket:


I have this setup as a cron job once a week.

Friday 20 August 2010

Upgrade Drupal Core and Restore from Mysql Backup

For too long the admin pages in my http://drupal.org site have been bugging me to upgrade from 6.12 core to the latest, which recently for to 6.19. I have been putting it off after reading the upgrade.txt which describes how to do it as it sounding a little lengthy and I just never had the time to do it.

I tried once but it didn't work. I went 'more or less' through the process described in upgrade.txt, which accompanies the download, but after doing it the admin pages still said it was out of date, as if I hadn't done anything, which was odd as I had deleted almost everything and replaced it with the new version.

Anyway, I came back to it recently and decided to sort it out. I downloaded the latest core and followed the instructions to the letter. But once again, my admin pages still showed me at 6.12, as if I had just done nothing!

This was obviously a bit annoying as I had very carefully followed the documentation. So I put my problem out in the #drupal-support IRC room on irc.freenode.net and started getting some helpful hints to start tracking down the problem.

The very best bit of information I got was a suggestion to use drush. drush is a command line program that you run in a terminal on the web server, within the site root. It is a simple little program that can perform such functions as checking the current status of the site, check for updates of modules, install those modules, clear cache, run update.php etc. The beauty is the feedback it gives you, highlighting any problems with error messages to help track down problems.

The first problem I came across when trying to update all modules was a duplication error, where one 'piece of code' was conflicting with another of the same name. This was preventing modules from updating. So after tracking down and removing the duplicate files, all modules (except core) updated seamlessly.

Now every time I ran the command drush pm-update or drush up drupal to try and get the core module, drupal, to update from 6.12 to 6.19, it would run through the apparent process with successful messages, but then if I ran them again, it would keep saying it was at 6.12 and needed updating. So the update was failing.

Trouble was that drush wasn't reporting any error messages and even drush core-status would confusingly say it was at 6.19, but everything else said 6.12.

I tried to track down any duplications in my site's directory but there was nothing. So I finally decided to start afresh and wipe everything out and install 6.19 as new.

To ensure I would not loose all my site content, including nodes, images, modules, theme, functionality I needed to ensure I backed up everything I could possibly need.

I made a complete copy of my site's root directory, in this case /var/www/. I also made an up to date dump of the mysql database, very important as this is where everything is contained.

To make a backup of the mysql database:
mysqldump drupal > drupal_dump.sql

Obviously ensure you back up the right database. In this case my database was called 'drupal', but it could have been called anything. If you need to add certain user and password options, add -u username -p password after mysqldump. This will create a text file containing the whole database. I wasn't sure how I would use it yet, but I just knew backing it up was the thing to do if I stood any chance of putting all my data back into the new site.

Next was to make the bold move and empty var/www. I then dropped the database in mysql.

To drop a database, enter the mysql environment with 'mysql' with admin privileges, then simply enter the command DROP DATABASE drupal;. I then re-created the database so it was new and empty, but using the same name: CREATE DATABASE drupal;

I then, perhaps niavely, believe that I have to use a unique user for each mysql database I have so I issue the following command to grant privileges to my chosen user on the drupal database:
GRANT ALL PRIVILEGES ON drupal.* TO myuser@localhost IDENTIFIED BY 'password';

Next I downloaded and unpacked the latest drupal core:

cd /var/www/
wget http://ftp.drupal.org/files/projects/drupal-6.19.tar.gz
untar xvf drupal-6.19.tar.gz


I then went to my root page in a web browser and ran the standard install script to get a new, empty up to date drupal installation.

Next was to get all my existing data back into my site for the mysql backup to try and make it appear as if everything is just as it was before.

The key to getting all content and settings back is to write the mysql dump back into mysql into the newly created, albeit same named, drupal database. But before this will work, you have to put all the modules that were present in the installation when the mysqldump was performed back in place so that the database doesn't get confused thinking it has modules installed that are not present.

To find out which modules were installed I created a temporary database in mysql and wrote the backup into that just so I could examine it. To write a mysqdump into a new database:

mysql drupal_tempdb < drupal_dump.sql

I then wanted to view what modules were installed in this database backup. To do this I viewed what tables were in the database in the mysql environment:

USE drupal;
SELECT * FROM system WHERE TYPE = 'module';


This showed me all the entries in the system table containing the modules. I could then examine this list to see what modules were installed and also, importantly, which versions of each module. This is though where the backup of /var/www/ comes in handy, as all the modules that were installed are contained in here. So using the list in mysql, which also tells me the location of each module. So I just copied each module back from the backup into the right location in the new installation.

Once I was happy that I had replaced all the existing modules, I wrote my drupal database dump in to the new database and ran update.php............... and it was mostly back!

All my themes were back (after copying the theme dir from backup), all my nodes and views were working. There were just some files missing, such as uploaded images, but again copying these back in from the back up solved this.

Saturday 7 August 2010

Seesmic Desktop on Ubuntu


I have been using Seesmic (#seesmic @seesmic) on my Android phone (HTC Desire) for a while now and must say I like it a lot. Just works nice and looks good. Plus support for Facebook is coming soon and I like the 'all-in-one' type of approach.

I had a look at the great Seesmic web app. and it looks great, works anywhere you have a web browser. Also noted they have a desktop client too, which runs using Adobe Air. Adobe Air is avaible to install from the repos in ubuntu so I gave it a go.

First:

sudo apt-get install adobeair


Once installed, this added Adobe Air Application Installer to the Applications > Accessories menu.

Download the air file from the seesmic website

Open that from the menu and it prompts you to browse for an air file to install, simply follow on-screen instructions to install the app. You will be prompted for your admin password.

Once installation is complete follow the simple set up wizard to add Twitter and Facebook accounts.

Great app., check it out.