A BACKUP STRATEGY FOR LINUX VIA CD-R

For a Linux server which has no high-capacity tape drive fitted but which does have access to a CD burner (either on-board or via another networked machine), and plenty of spare disc for some large temporary files, a useful and convenient alternative is to use multiple CDs instead.

I use a simple little script called backup that uses find, cpio and fsplit (#) to create a series of cpio archive fragments totalling about 2100Mb that contain all the files of interest on my system. The fragment size is currently set to 630 Mb (by the script) so they can fit onto normal CDs. The chunks are automatically named by backup and fsplit by appending 000, 001 etc onto a date-derived file name base. (# See below to obtain fsplit.)

My CD burner is actually on an adjacent Windoze 98 PC, but I've installed Samba on the Linux server to give Windoze full access to the Linux file system. So once the "chunks" have been created on the Linux server, I start the Adaptec CD-burner program on Windoze, point it's file window over at the /usr/local/backups directory on the Linux machine, and start burning the CDs. (If you don't have Samba running on your Linux system, you could alternatively fetch the chunks using FTP.) Of course, if you have a CD burner on the Linux server itself, you can just create them locally - using (eg) CdRoast.

Either way, once the (Joliet format) CDs have been created, they can then be read at any time on the Linux server's CD drive.


(#) NOTE - When you download a copy of fsplit program (a Linux PC binary), you'll notice that I've appended a .bin onto its name. This is just to stop Netscape from trying to display it as text. So after you download it, you should rename it using "mv fsplit.bin fsplit". You can then type fsplit -h for usage (and/or look at the backup script). I keep my copy in /usr/local/bin, incidentally.

If you prefer (and/or have the need), you can download the source (fsplit.c) and compile it up yourself. Just stick it in an empty directory and type "make fsplit" and Bob's your auntie.


The script currently accepts only one parameter (a number) to specify the maximum age of files (in days) which are to be included. If you don't specify a number (the usual mode of operation), file age is just ignored.


Full backup example

Running backup "as is" will create (in /usr/local/backups) a set of 630Mb archive chunks, a couple of errors files, and a list file, looking something like this:

-rw-r--r--    1 root     root     630000000 Jun  9 22:24 200106092207.cpio.000
-rw-r--r--    1 root     root     630000000 Jun  9 22:32 200106092207.cpio.001
-rw-r--r--    1 root     root     630000000 Jun  9 22:44 200106092207.cpio.002
-rw-r--r--    1 root     root     199682048 Jun  9 22:58 200106092207.cpio.003
-rw-r--r--    1 root     root           88 Jun  9 22:16 find_errors
-rw-r--r--    1 root     root           15 Jun  9 22:17 errors
-rw-r--r--    1 root     root      2867480 Jun  9 22:16 list

It's always worth browsing through the find_errors and errors files. On Linux, find_errors will typically contain a couple of harmless complaints about access to the /proc directory. This occurs even though /proc is in the scripts exceptions list because find traverses every directory - it's the following sed command in the pipe that does all the filtering.

The cpio errors file, errors, should only contain a final block-count figure.

The first time you use the script (or after you make changes to it), you should also have a quick browse through list to make sure there are no surprises. That is - make sure that the most obvious directories (such as /etc, /usr/lib ...) are all present and accounted for. And do the reverse check - that none of the directories which should have been excluded are present.

If all looks well, you can then copy each of your chunks onto CDs (1 per CD).

I typically also copy list onto the first CD for later convenience. This contains the list of the files that were actually backed up.

If you forget to include list on the CD, it can be regenerated later (*) using:

%<root> cat *.cpio.00* | cpio -it > list

(*) You have to retrieve all the archive pieces off your CDs and back onto your hard disc before you can do this, of course.

Incidentally - if you want a long listing using this procedure (ie: as per the ls -l directory listing style), just add a -v, ie:

%<root> cat *.cpio.00* | cpio -itv > list

or even just

%<root> cat *.cpio.00* | cpio -itv | less


Making intermediate (delta) backups

Once a full backup has been made, many subsequent backups may then be done as deltas. With backup, a delta is assumed if you supply a number of days as a parameter. So for example - if you last did a full backup 4 weeks (28 days) ago, you could now cover yourself with a delta via:

%<root> backup 29

This will pick up only those files that have been touched in the last 29 days (I usually add +1 just to play safe).

You can do a number of deltas over a period of weeks or months, and provided you always specify a number of 'days' for the delta that at least covers the period of time since your last full backup, you can then stage a recovery via the original full backup followed a 2nd, minor 'refresh' recovery using your latest delta.

The reason for doing deltas is of course that - whereas your full backup may have consumed 4 or 5 (or more) CDs - subsequent deltas will only require a single CD until such time as you add (and/or change) a total of 630Mb worth of files. And that will often be several months.

Anyway, once you reach that point (ie: where a delta needs more than 1 CD to hold it), it may be a good time to do another full backup!


General usage comments

The implicit assumption with this script is that the entire system will be backed up except for any directory trees included in the sed RE (regular-expression). You would almost certainly want to change this RE in backup for your own system, but keep in mind that each line needs to be terminated with a reverse slash ("\"), and that because the forward slash ("/") is used for a delimiter in the REs, they must always be escaped with a reverse slash if you want them taken literally. So for example, the path /usr/local/backups/ in the sed part of the pipe must be presented to sed as /\/usr\/local\/backups\//.

Because I now find myself with about 9Gb lying around in my /usr/local tree, I generally find that I need to do some tidying up before I run this script! Probably not a bad thing thing in itself (although having a $70,000 high-speed tape jukebox like we have at the work QTH would certainly make life much easier :-)

But if I just run backup from a cold start, I usually find that it wants to create 9 or 10 CDs, and that just seems plain silly. I've considered using gzip with the script to reduce the number of CDs required, but this is difficult to arrange when you need to create output file chunks of a pre-specified and reasonably exact size to fill a CD.

So there's only one method available for getting the number of CDs down - and that's to ignore selected directories. This is where the "tidy up" comes in.

What I do before running backup is to go into each of /usr, /usr/local, and /usr/local/src  (to name the three main offenders) and run the du command as follows:

%<root> du -sk * > DUSAGE

I can then whizz around looking into each of these DUSAGE files to see where most of the megabyte mass lies, and shuffle directories around appropriately. For example, in my /usr/local/src  tree, I may have 500Mb worth of directories of externally sourced stuff that can quite happily be moved down into /usr/local/src/nobackup. That way, the script will skip them but they're still available if I do need them again. (And of course I can always move them straight back up immediately after the backup anyway :-)

Alternatively, I might find some new, big fat directories in the Linux system area that deserve to be added to the backup exclusion list. Three that I did recently add (after installing Mandrake 7.2) were /usr/share, /usr/src, and /usr/doc. This eliminated about 700Mb of stuff, and will have no particular effect on my recovery process when and if I need it.

Of course, you may spend an hour doing this sort of thing and then run backup only to find that you're still dissatisfied with the result. The only option is to iterate - go around again and either (a) move still more stuff into one of the backup-exclusion 'exclusion' directories, or (b) [where the directories themselves can't be moved because they're in the system area] add new exclusion directories to the sed RE pipe in backup.

Doubtless this all sounds rather messy, but unless you're prepared to fork out the necessary dollars for a high-speed tape drive that will back up 20Gb or thereabouts for you, you really have no choice. The good thing about this system (if any) is that the real effort (as described above) is at backup time. Recovery (when you finally do need it) is relatively straightforward.


Verifying the result

Okay - so you've made a backup, and you can now see a few files called 200?????????.cpio.00? (etc) sitting there. You're about to start burning CDs, and yet a little voice in the back of your head is whispering "How do I really know that these fragments can be reliably pieced back together for a recovery if and/or when the evil day comes ...?"

Well, it's easy enough to check your cpio archive pieces - just type:

%<root> cat *.cpio.* | cpio -it > xx

and if all is well, the check file xx will end up with the same list of files that are in list. You can quickly verify the number of files in each using something like: wc list xx

If you'd like to do a more exact verification (eg: using diff), you'll need create the check-list of file names in xx in same format as those in list (ie: in relative form), so instead use something like:

%<root> cat *.cpio.* | cpio -it | sed s'/^/\.\//' > xx

to add the dot-slash at the start of each name, then use the diff command, viz: diff list xx

Using the wc and/or diff commands as just shown is a bit academic, of course. Just having the cpio -it line run to completion on your fragments and seeing your file list regenerated in xx with no error messages should convince you that your archive pieces are indeed capable of being strung together at any time in the future and used for a recovery, because cpio must still read all your fragments from beginning to end to generate such a list.


Types of recovery procedures

To retrieve one or more files, grab your latest CD set (4 CDs in my case), copy the chunks from each CD into an empty directory, and then re-assemble them back to a single cpio archive file using a simple shell "for-loop" such as (*) :

for f in *cpio.[0-9]*
do
cat $f >> archive; && rm $f
done

This simple 'for-loop' solution assumes that no file called archive exists to begin with.

(*) Linux (etc) caveat

When doing a recovery using the above on a Linux system (Mandrake 7.2) with a 5 CD set - approx 2700Mb worth, I made the rude discovery that Linux in general still suffers from the 2Gb size limit (for any one individual file). The system started building archive via the above for-loop, reached 2Gb in size, and crashed out with a 'File Too Large' error message. The same file size restriction exists in many other older Unix systems (ie: earlier versions of Solaris, FreeBSD, etc), and in these cases you'll need to use a more direct recovery method such as:

cd junk
cat *.cpio.00* | cpio -idmv 2>errors

This will dump the archived tree into the 'junk' sub-directory. As given above, the command assumes that the archive chunks are also in the 'junk' sub-directory.

Because this method doesn't rebuild the original archive, it's also quicker and require less disc space, so it's probably a better method anyway.


Doing a full restore

Overall, the archive may be used for recovery via a number of slightly different methods, depending on what you need on the day.

Method 1 - if a complete recovery is needed (after a disc trash or similar disaster), you would typically:

(a) Do a fresh install of Linux (preferably the same version), then
(b) Copy the archive pieces off your CDs into the root directory, then
(c) type (as root) the following command:

%<root> cd /
%<root> cat *.cpio.00* | cpio -idmv 2>errors

This will install any missing files, and it will also update any existing files which need it. It will NOT overwrite any files which have an identical or a more recent datestamp than the corresponding file in the cpio backup archive, so be careful not to inadvertantly touch or modify too many system files following such a fresh install before running this recovery, or any such inadvertantly touched files will miss out on being updated.

Incidentally, any files from the archive which do fail to be updated back onto disc for this (or any) reason will score a line in the errors file giving the full file name and a description of the problem, so it's always a good idea to check the errors file after any such cpio recovery (ideally, it should be empty, of course).

You can also run cpio with the additional "u" flag (ie: cpio -idmvu < archive ) - this unconditionally writes every file from the archive to disc. I prefer to avoid this where possible (it makes me feel a trifle nervous), but where you've just re-installed the OS onto a set of blank system discs and you need a complete recovery in a hurry, you may feel inclined to do this anyway.


To restore selected files only

Method 2 - cd into an empty temp dir somewhere, copy all your CD chunks into it, and start by extracting the entire archive using the same command as above. For example:

%<root> cd /usr/local/tmp
%<root> cat *.cpio.00* | cpio -idmv 2>errors

All directories and files will then be pulled out and installed under this temporary directory with a dir structure which mirrors the original system tree under / - but transplanted. You can then fetch individual files out of the usr/local/tmp tree and cp or mv them at your leisure.

If you need to refresh entire directory trees, then (assuming you'd pulled out a whole archive into /usr/local/tmp as just described), let's suppose you'd now like to automatically refresh (as necessary) all files in the real (live) /etc/sysconfig tree.

Just do this:

%<root> cd /usr/local/tmp/etc/sysconfig
%<root> find . | cpio -pdmv /etc/sysconfig 2>errors

This uses cpio's pass through mode (-p) to update all files in the /etc/sysconfig tree which NEED updating from the recovered ones in /usr/local/tmp/etc/sysconfig. I've done this sort of thing quite often. You can use the same method to recover any directory tree.

Remember - because we unpacked the archive into /usr/local/tmp, everything in that directory will mirror what used to be in the root directory on the day the backup backup was made. So (eg) the old /etc directory will now appear as /usr/local/tmp/etc, the old /usr as /usr/local/tmp/usr, and so on.

Incidentally - if you do update just about anything in or below /etc as per the above example, you may need to reboot before going too much further. Linux keeps all its startup stuff in and below /etc, and a warm reboot is the quickest way of picking up any configuration data that may have been modified.

Method 3 (tricky) - retrieve only selected files from the cpio archive, using cpio's own "pattern matching" syntax.

Believe me - this is difficult to get working, and can waste a half a day. I've tried it in the past ... the trick is getting path slashes to match, and putting asterixes in the right places. The command required is something like:

cat *.cpio.00* | cpio -idmv 'pattern' 2>errors

where pattern matches the file(s) that you want (but good luck :)

I have made this method work, by the way. A worthwhile start is to check the contents of the archive (using cpio -it <archive) to find a file name which is close to the start. Then experiment with the pattern-matching syntax until you can pull that file out. It boggles me though as to why (when I can damn well see a file in the archive called ./bin/fred) why I can't just put that exact name-string into the cpio pattern without having to muck around so much ...

Which is exactly why I usually just get impatient and use method 2 :-) Much easier when you've got all the files just sitting there in a handy side directory ...


CAVEATS and COMMENTS

The usual comments re my "legal irresponsibility" apply if you use any of this.
If the size of the filesystem portion being archived is X megabytes in size, you must have around X megabytes of spare disc space to use backup, and about the same amount to stage a recovery.
The best arrangement is to have a suitable CD burner on the same machine which has created the archive pieces. Personally, I'm not that fortunate yet - my burner's on a Windoze box. If you do have to move the archive pieces over to another machine to burn the CDs and you use FTP, just make quite sure you use binary mode to do it! A quick and easy way of verifying this is to look at the chunk sizes on the destination system and ensure that they're identical to the originals. (Better still - if you have the bandwidth - avoid FTP and burn the CDs across a network mount!)
Remember particularly that files are archived in the form of relative pathnames, so they will always be restored relative to your current working directory. So cd to an empty directory somewhere before you start, or (if you intend a full, direct emergency recovery) into "/" before you "unload" the archive. Pulling out, for example, the entire /usr tree into the /etc tree just because you happened to be in /etc when you started a recovery can be very messy to clean up (yes - I did do this on a really bad day many years ago.
Why use cpio instead of tar for all this? Well, I was "brought up" on cpio (System V Unix from Nat Semi), and I think it's more flexible in its recovery than tar. And on Linux, at least (thanks to its GNU heritage), the man entry for tar is yet another one of these utterly infuriating and arrogant "we don't keep this man entry up to date. Use the tex doco if you want the real story ..." thingees. Whereas at least cpio has a normal, proper unix man entry. (But if you feel more comfortable with a tarball, then just modify the script to use tar.)
It constantly amazes me as to how obscurely and badly written the doco for both (GNU) tar and cpio is. Their functionality is hardly what anyone would call world shattering stuff. The Sun (Solaris) tar and cpio man entries are actually fairly reasonable. It's a pity the great GNU folks couldn't just take some lessons in good documentation from that (BSD and SysV) camp instead of going off and inventing some obscure new system that nobody knows how to drive. Grrrr ...
I've also provided a page re backing up Windoze PCs using a similar method.

Q: What is worse than not making a backup?
A: Making a backup that doesn't work!

After making a backup using any new system, always invest the time to do a trial recovery, even if it's just a single file or directory being pulled into a spare directory somewhere. Never trust any backup program (including this one) until you've see it actually work full circle.
I once saw a department make backups for over a year (every Friday night) before discovering on the critical day that they were all useless. They were most disconcerted by this discovery.
(Granted - the guy that set it all up for them was a dickhead that I later had to sack anyway - but it can still happen to any of us. Don't let it happen to you - you must perform some trial recoveries before you trust any new backup system.)


Miscellaneous thoughts ...

The above may all sound somewhat complicated to the Unix "newbie", but in reality, it is no more than a series of very simple steps - plus 3 recovery alternatives, depending on your needs. It provides a backup strategy which (with the exception of my simple little fsplit program and your CD-burner software) is all plumbed up from standard bits and pieces of good old Unix.

On the other hand, comprehensive high-speed tape-based backup systems are not cheap. A reasonably good one will set you back around $AU30,000, and you can spend 2 to 5 times that amount if you want the real up-market equipment. And these systems are not just costly to purchase - they have a steep learning curve as well. So there is still no easy road.

I've used one or two commercial tape-based systems over the years, of course, and to be honest, they make me nervous. You're completely at the mercy of the company who designed and manufactured the tape unit, and the company who produced the backup/recovery software. Open systems, on the other hand, give me a warm, cozy feeling because (a) I can see what they're doing, and (b) I can go look up the 'man' entries for the different bits and get a full understanding. (Hell - with Linux, I can even get out the damn source and change it if I like ... :-)

And no - this system probably won't impress your average Windoze-soaked MS Certified IT manager. (But who cares ... in my experience, they usually can't recover files for their users anyway :-)


Other CD backup options

 CDBKUP is a suite of perl programs by John-Paul Gignac and is also worth a look. Features full or incremental backups, standard GNU tarballs, support for multi-session CDs, and can also split large backups between multiple CDs.


Changes and updates

December 5, 2001 (Vers 2.6): Remove another bug in backup script introduced in version 2.5 (arrgghhhh ...) - a typo on FIND_ARGS was causing delta's to run as full backups. Again it's thanks to Justin Noack, JN Computer Care for quickly pointing this bug out. (And let's hope that this is the last change)

November 21, 2001 (Vers 2.5): Remove a bug in backup script which affected a restore into an empty directory. The find command for archive creation was being used with an arg of -type f and this was excluding directory information (UID, GID and permissions) from the archive. Result of this was that a restore into an empty directory would create all sub-directories with UID.GID = root.root and perms of 700. (File perms were okay.) Thanks to Justin Noack, JN Computer Care and Coloma Community Schools.

October 8, 2001: Add a new section to this web page called "Verifying the result". Plus other minor tidy-ups to improve readability.

June 18, 2001: Add a new section to this web page called "Making intermediate (delta) backups".

June 10, 2001: (backup vers 2.4) Add '-H crc' flag to cpio command in script to checksum everything and to enable > 64K i-nodes. Update the recovery strategy sections of this page to overcome the 2Gb file size limit of some systems (eg: Linux).

June 9, 2001: Add some more 'exclusion' directories to script after installing Mandrake 7.2

March 11, 2001: Updated fsplit to take stdin, so the backup script now pipes the output of cpio through fsplit without the need to generate an intermediate (full cpio archive) first. (And just try to do that with NT or Win2000 or whatever they call it this year :-)


  Some general observations re backups, etc, as related to Unix and MS Windows. Written just after yet another NT disaster :-(

  Back to (Bluehaze) software archive page

 Bluehaze home page

Admin (Bluehaze Solutions). Last revised: Tuesday 26th March, 2002 (correction to one sentence)