Date: Fri, 28 Apr 2006 13:33:22 +0700 (ICT)
From: Olivier Nicole <on@cs.ait.ac.th>
To: dipak@banyan.cs.ait.ac.th, adul@banyan.cs.ait.ac.th,
        on@banyan.cs.ait.ac.th
Subject: Restore disk /usr crash on Amanda server

Hi,

On Monday the /usr disk crashed on Amanda server.

The good things are that the system can boot in single mode without the
/usr partition and that partition was saved on tape.

The bad things are that Amanda software was deleted because it resides
in /usr and that many utilities (gzip, tar) are not available in
single user mode.

I did the following to solve the problem:

1) I booted the machine in single user mode.

2) I rebuilt the filesystem for /usr, using the utilities fdisk,
   dicklabel and newfs that are located in /sbin.

3) I mounted the newly created /usr partition with the command 

   mount -a

   This will report that vinum0 and vinum1 cannot be mounted, that is
   expected. If any vinum partition is needed (to mount /holding or
   /holding2) use:

   vinum stripe -n vinum0 /dev/da0s1g /dev/da2s1g

   vinum stripe -n vinum1 /dev/da1s1e /dev/da3s1e

   For /holding and /holding2 respectively.

4) I located the last backup of the /usr partition for the machine
   amanda, for example looking back in the amanda verify messages; I
   mounted that tape in the tape drive.

5) I issued the command:

   /amanda/amrestore -p /dev/nsa0 amanda usr | /amanda/tar --numeric-owner -xpGvf -

   This takes advantage of copies of amrestore, tar and gzip, that
   have been statically compiled and that reside in Amanda home
   directory (on the root partition). Note that amrestore must be
   build with the exact path to Gzip program: /amanda/gzip.

6) I rebooted in multi user mode.

7) I restored /usr/local/etc/amanda from the copy made on
   database.cs.ait.ac.th machine in /database/amanda/amanda.

   This directory contains the configuration for Amanda as well as the
   tape list. This directory is copied to database.cs.ait.ac.th
   machine after each run with amdatabase command.

8) I restored /usr/local/var/amanda/amanda/CSIM-set-1 from the copy
   made on database.cs.ait.ac.th machine in
   /database/amanda/CSIM-set-1.

   This directory contains the index databases for Amanda. This
   directory is copied to database.cs.ait.ac.th machine after each run
   with amdatabase command.

9) I restored /usr/local/var/amanda/amanda/passwd from the copy
   made on database.cs.ait.ac.th machine in
   /database/amanda/passwd.

   This file contains the password used for rsynch backup done by
   amdatabase command.

Done.

--------------

The daily email with subject AMANDA DATABASE INFO has changed; a first
part was added. This email is generated by the command amdatabase.

This email has three parts:

- first part is a copy of the tapelist /usr/local/etc/amanda/tapelist

  It is important to save the tapelist after every run, so in case of
  restoring, the sequence of tapes is preserved.

- second part is the report of rsync of the database files from
  Amanda. Database files hold information like what was saved on what
  tape every day. It is important to keep a back-up of the database.

  The data are on amanda:/var/amanda/ and amanda:/usr/local/etc/amanda
  and they are rsync'ed to database:/database/amanda after the back-up
  has completed. Rsync runs on the gigabit network only.

- thrid part is the result of "amadmin <config> export", you can use
  the equivalent import command to rebuild the database, and the file
  is also human readable.

--------------

This email is inspired from http://wiki.zmanda.com/index.php/Restoring_files

Olivier