Thu Sep 17 11:08:28 +07 2020

Managing multiple disks with virtual tapes for Amanda: chg-vdisk

The system chg-vdisk

Amanda introduced virtual tapes (vtape) to store backups on a disk instead of magnetc tapes. At the time, Amanda could pnly use a single disk to store virtual tapes; if you had several disks, it was suggested to use some disk concatenation like RAID 0.

While a solution, I think that it is not the best solution: if a single disk is to fail, the RAID is not available anymore and one lose all the backups. Using a RAID of higher level is not ideal, I would prefer to trade the reliability of RAID for more backup storage and more incremental backups.

To that effect, I have modified the tape changer for vtapes chg-disk to work on a bunch of disks.

Each disk is mounted at a mounting point in Unix. At this mounting point there must be a directory amanda that belongs to the user and group running Amanda server.

In the directory amanda there must exist a file called disklabel that contains a unique name for that disk that will be used further in the configuration. The directory amanda also contains the slots as defined by the vtapes. The slots are numbered in sequence across all the vdisks, 1 to a for the first disk, a+1 to b for the second disk, b+1 to c for the third, etc.

A configuration chg-vdisk.conf file must be created in Amanda configuration directory (where the file amanda.conf is saved). It will be described later.

Two files, chg-vdisk.cache.db and chg-vdisk.current, are created by chg-vdisk in that same Amanda configuration directory and should not be changed by the user.

The configuration file chg-vdisk.conf

The configuration file of chg-vdisk resides in the Amanda configuration directory and follow the syntax of Config::IniFiles of Perl, with sections between square brackets ([...]) and a list of parameter=value.

  • [syslog] defines options used by syslog:
    • facility= the facility used to log syslog messages.
  • [options] defines general options:
    • path= the path of Amanda configuration directory;
    • pid_file= the path of the file to store chg-vdisk pid;
    • lock_file= the path of the file that chg-vdisk can use for locking operation;
    • timeout= (obsolete, see below) the timeout in seconds to wait for autmount;
    • signal= (obsolete) the signal thatthe operator can send to chg-vdisk once the requested disk has been mounted.
  • [mountpoints] defines the list of mount points used by the disks, each parameter if of the form:
    • /path/to/the/mountpoint=1 where /path/to/the/mountpoint is the equivalent of the disk mountpoint in /etc/fstab. Note that the =1 is dictated by the file format that must associate a value to each entry.
  • [disklabel] for each disk in the system, there is a section named by the disklabel of the disk:
    • first-slot= is the first slot that exist on that disk;
    • last-slot= is the last slot that exist on that disk.
  • [email] (obsolete) defines the email address of the operator:
    • mailto= is the email address used for the recipient, uses Amanda configuration value by default;
    • mailfrom= is the email address used for the sender;
    • mailrepeat= is the frequency, in seconds, to send repeated email.
  • [week] and [holiday] (obsolete) define the date and time when an operator is present and can swap disks (see below). The syntax is equivalent to the one used for CSIM access control.

Adding a disk to the system

This presuppose that the disk is running and attached to the Unix server.

If you want to add a new disk to the system, there are two steps for initialisation of the new vdisk and one step for mounting the disk; if you need to mount a disk that has already been initialised and used, for example an old disk that has been retired but still holds backups, jump to the step 3:

  1. initialize the new vdisk and create the slots:

    You must create directory amanda and the file disklabel at the root of the vdisk; these must belong to the user and group running Amanda.

    Then the slots are created with the script ~on/create_slot:

    sudo ~on/create_slot disklabel #slot

    where disklabel is the label used in the file disklabel and #slot is the number of slots you want to create on that disk.

    For CSIM, each vtape is 110GB, so a 6TB disk would allow about 53 slots.

    Note that the slots are created with number starting at zero;

  2. initialize new new vtapes in the slots of the new vdisk:

    The script ~on/label_slot creates new vtapes in each slots of the disk disklabel; it uses amlabel internally and must be run by the user amanda.

    You can run the command:

    sudo -u amanda ~on/label_slot disklabel

    where disklabel is the same as above. Vtapes will have a sequential number starting from the maximum number found in the file tapelist;

  3. mount the vdisk:

    This step renumbers all the slots of the vdisks accessible on the server to make sure that the slot number are in sequence, this is because Amanda tape changer expects slots that are numbered in sequence from 1 to the maximum while adding a disk may introduce gaps or duplicated.

    This script does not change the vtapes or the contents of the backups that are present on the disks, only the slot numbers are modified, it is safe to run this scipt on disks that alread contain backups.

    The script ~on/renum_slot do the renumbering and issue a piece of configuration that must be updated manually inside chg-vdisk.conf. This script nust be run by root.

    You can simply issue the command:

    sudo ~on/renum_slot

    as a result you will get some information of the form:

    [disk CSIM-set-10]
    # mounted on /host/localhost/ada2
    first-slot=1
    last-slot=27
    
    [disk CSIM-set-11]
    # mounted on /host/localhost/ada1
    first-slot=28
    last-slot=54
    
    [disk CSIM-set-13]
    # mounted on /host/localhost/ada7
    first-slot=55
    last-slot=72
    

    This is to replace the corresponding section in chg-vdisk configuration file.

Removing a disk from the system

It may be necessary to remove a disk from the system if the disk has failed and became unusable or if a newer and larger disk would be installed.

Once the disk jas been physically removed from the server, two steps must be done:

  1. renumber the slots for the remaining disks:

    The script ~on/renum_slot is used and the output must be imported into chg-vdisk configuration, as described above;

  2. make Amanda aware that the vtapes are not mounted anymore and should be marked no-reuse:

    The script ~on/unlabel_slot can be used to make a list of vtapes listed as reuse in the file tapelist but that are not loaded in any slot. It must be used by TODO.

    The script outputs a list of vtapes that can be used to manually issue a command amadmin no-reuse. If the disk has been destroyed, the command TODO may be prefered to completely remove the vtapes from tapelist.

Disk automouting on FreeBSD

The other files created by chg-vdisk

  • chg-vdisk.current: the notion of a tape currently loaded in the tape changer is moot for vtapes, but it is part of the API for the tape changers in Amanda. This file keeps track of the tape "currently" in the changer.
  • chg-vdisk.cache.db: the system needs to keep a link between the slots and the vtape present in each slot. This information does not change frequently so it would a waste of resources to compute it again each time the the vdisk changer is called. This database keeps track of the vtapes mounted in the slots.

    When a certain vtape is needed, the slot number in the chache is checked first, and if the vtape on disk at this slot is the expected one, chg-vdisk continues its its operation. Else, if the vtape in the slot is different from the one cached, all the disks are read, one slot at a time and a new cache is saved.

Some other features

i originally envisioned chg-vdisk to work with USB attached disks, where a limited number of USB slots were available. The system would have been able to find out which of the disks were currently pluged in and request an operator to switch disk appropriately.

USB disk were slow and SATA interface rendered all that part very obsolete. But that is the reason of the part regarding email and holidays in the configuration file.

Finally, chg-vdisk should be modified to run as a multi-changer as it can write on several disks or files at the same time, but that has never been the bottleneck with vtapes, so it runs one vtape at a time.


Posted by Olivier | Permanent link | File under: administration, backup

Thu Jul 16 11:44:23 +07 2020

Add MAC address to the filter for SMC smart classroom

Smart classroom are set to use the specific SSID SMC.In order to prevent anyone using that SSID, a filtering has been setup on the MAC addresses of the devices that are allowed to connect to that wireless network.

The operating system DD-WRT used on the WiFi access points in CSIM does not allow MAC filtering per SSID, only per interface, so the filtering has to be done at the wouter level.

  1. Connect to CSIM router.

  2. List the existing MAC filtering rules withe the command:
    show firewall name SMCOutFilter
    
    The result should look like:
    IPv4 Firewall "SMCOutFilter":
    
     Active on (eth3.13,IN)
    
    rule  action   proto     packets  bytes                                   
    ----  ------   -----     -------  -----                                   
    1     accept   all       119375   14027513                                
      condition - MAC xx:xx:xx:xx:xx:xx
    
    2     accept   all       2817     597981                                  
      condition - MAC yy:yy:yy:yy:yy:yy
    
    10000 drop     all       1902     121212                                  
    
    vyatta@vyatta:~$
    
  3. Enter Vyatta configuration mode, use the command:
    configure 
    
  4. Choose the next available rule, in the above example rule 3 and add a new rule:
    set firewall name SMCOutFilter rule 3 action accept 
    set firewall name SMCOutFilter rule 3 source mac-address xx:xx:xx:xx:xx:xx
    set firewall name SMCOutFilter rule 3 description "name of the devicce"
    

    Where xx:xx:xx:xx:xx:xx is the MAC address of the device and name of the devicce is a short description of the device, it;s owner, etc.

  5. Verify the new rule:
    show firewall name SMCOutFilter
    
    You expect to see the following in the listing:
    +rule 3 {
    +    action accept
    +    description "name of the devicce"
    +    source {
    +        mac-address xx:xx:xx:xx:xx:xx
    +    }
    

    The symbol + at the begining of the line means the rule has been added but is not yet committed.

  6. Commit the new rule:
    commit
    
    And save the configuration:
    save
    
  7. Tules are deleted with the command:
    delete firewall name SMCOutFilter rule n
    

    Where n is the number of the rule to be deleted.

    This had to be committed and saved too.


Posted by Olivier | Permanent link

Fri Oct 4 11:29:42 +07 2019

VMware monitor Dell hard disks

Starting with ESXi 6.x, VMware could not monitor Dell hard disks properly on PowerEdge 1950, 2950, R510 and R710: some specific driver is not included anymore and an error message is saying that The Small Footprint CIM Broker Daemon (SFCBD) is running, but no data has been reported. You may need to install a CIM provider for your storage adapter.

The article Monitor storage for a Dell PERC H710 Mini on vSphere 6.5 by Wil van Antwerpen suggest to install the driver directly drom Broadcom-LSI.

I downloaded the latest driver https://www.broadcom.com/support/download-search/?pg=Storage+Adapters,+Controllers,+and+ICs&pf=RAID+Controller+Cards&pn=MegaRAID+SAS+9266-8i&pa=Management+Software+and+Tools MegaRAID StorCLI MR 6.6 Version: 1.14.12 StorCLI User Guide https://support.hpe.com/hpsc/doc/public/display?docId=a00048288en_us VMware this package must be used on ESXi 4.x servers. VMware-MN This package must be used on ESXi 6.x servers and onwards when the driver used is a legacy MegaRAID SAS Device Driver. VMware MN-NDS This package must be used with MR driver, lsi_mr3, which is a native driver.


Posted by Olivier | Permanent link

Fri Sep 20 15:41:21 +07 2019

Upgrading OpenLDAP

When setting a test server, the binding IP address for OpenLDAP is defined in /etc/rc.conf.

Posted by Olivier | Permanent link | File under: administration, freebsd

Fri Sep 20 15:10:00 +07 2019

Upgrading freeradius

We run freeradius in a chroot'ed environement; as such, libraries from /usr/local/lib/freeradius-x.y.z must be copied manually in the equivalent directory under /var/chroot/freeradius.

When launching a temporary test server, the IP address used to bind freeradius is defined in /usr/local/etc/raddb/sites-enabled/default (in two places).


Posted by Olivier | Permanent link | File under: administration, freebsd

Fri Sep 20 13:51:50 +07 2019

Upgrading of clamav

  1. clamav must be build with gcc:
    setenv CC /usr/local/bin/gcc
    
  2. On mail serveur, clamav must run as user kluser:vscan, after each update, you must change the ownsership of /var/run/clamav and /var/db/clamav (they are overwritten by the install procedure).
    You must also install the proper /usr/local/etc/rc.d/clamav-clamd script that allows to run two versions of in parallel.

Posted by Olivier | Permanent link | File under: administration, freebsd

Tue Sep 17 14:23:41 +07 2019

Dealing with Let's Encrypt certificates

First usage

  1. Install acme.sh
    git clone https://github.com/Neilplan/acme.sh
    
  2. Install nsupdate (on FreeBSD you need to install bind-tools)
  3. Set the environment variable as follow:
    NSUPDATE_SERVER=dns.cs.ait.ac.th
    NSUPDATE_KEY=/usr/local/etc/acme.sh/update.key
    NSUPDATE_ZONE=cs.ait.ac.th
    
    Note that NSUPDATE_KEY has to be copied from Olivier's account on banyan
    These variables are only needed on the first use of acme.sh script, then they are saved in the configuration file but the update.key file needs to be present any time acme.sh will be run in the future.
  4. On the DNS server, update the file /var/chroot/named/etc/namedb/named.conf to add the line
    grant acme-sh-update name _acme-challenge.xxx.cs.ait.ac.th TXT;
    
    then reload DNS.

Generate a certificate

Notr: it does not work on mail, it is a communication problem. until it is fixed, generate the certificate on ufo.

With the DNS API, generating the Let's Encrypt certificate with DNS validation can be done in a single command:

~/.acme.sh/acme.sh --issue --dns dns_nsupdate --force --signcsr --csr /usr/local/ssl/csr/xxx.cs.ait.ac.th.csr -d xxx.cs.ait.ac.th

The parameter --force is required because we use sudo.

Install the certificate

For Radius, the certificate must be installed in the chrooted directory.

  1. Change directory to ~/.acme.sh/xxx.cs.ait.ac.th
  2. Check the validity of the new certificate:
    openssl x509 -noout -modulus -in xxx.cs.ait.ac.th.cer | md5
    sudo openssl rsa -noout -modulus -in /usr/local/ssl/key/xxx.cs.ait.ac.th.key|md5
    
    Both command should provide the same output, if they are not, something went terribly wrong, abort!
  3. Check the validity of the CA:
    diff ca.cer /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca
    
    The files should be similar (it may have a difference of a blank line).
  4. If it is the first time a certificate is being generated, or if the files differ, make a backup copy of /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca:
    sudo cp /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca.old
    
    and copy the new file ca.cer in place:
    sudo cp ca.cer /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca
    
    Note that we maintain one CA file per certificate, even if all the CA files should be the same.
  5. Change directory to /usr/local/ssl/crt
  6. Make a backup copy of xxx.cs.ait.ac.th.crt:
    sudo cp xxx.cs.ait.ac.th.crt xxx.cs.ait.ac.th.crt.old
    
    and install the new certificate:
    sudo cp ~/.acme.sh/xxx.cs.ait.ac.th/xxx.cs.ait.ac.th.cer xxx.cs.ait.ac.th.crt
    
  7. On mail server, you also need to go to /usr/local/ssl/crt/combined and back up and generate a new mail.cs.ait.ac.th.ketcrt file.
  8. Restart the service xxx (or reboot the mail server).
  9. Take note of the expiry date of the certificate:
    openssl x509 -text -noout -in xxx.cs.ait.ac.th.crt
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                xxx:xxx:xxxx
        Signature Algorithm: sha256WithRSAEncryption
            Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
            Validity
                Not Before: Sep 17 05:29:35 2019 GMT
                Not After : Dec 16 05:29:35 2019 GMT
    
    and record the Not After date to remember updating the certificate before in expires.

Posted by Olivier | Permanent link | File under: administration

Tue Sep 10 10:59:21 +07 2019

SSH tricks on VMware

Adding SSH public key authentication to ESXi

On the ESXi server, edit the file /etc/ssh/keys-root/authorized_keys and add the public keys, one key per line.

Supposedly, the file already exists and is mode 1600.

Suppress the SSH enable message

ESXi servers always complain when SSH is enabled. This warning message can be deactivated with the command:

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

Posted by Olivier | Permanent link | File under: administration, vmware

Wed Aug 21 13:03:34 +07 2019

Transferring old account from guppy to banyan

When a student leaves AIT, there is no automatic procedure to save and removes the files that were created on the local disks of guppy.

Further more, because the students are using some tools to virtualize environment, there are many file duplicates that should be taken into account to reduce the space on the final backup. In some instances, a space reduction of 30% could be achieved by simply removing the duplicated files.

  1. Connect to guppy and become super user;
  2. Compress the files of the user:
    find /home2/user -type f -exec xz -f -9 {} \;
    
  3. Remove duplicates:
    freedups.pl -a /home2/user
    
  4. Create some directory on banyan to receive the files, for example /home/corse/deleted/user/guppy;
  5. Transfer the files:
    cd /home2/user; tar cf - . |ssh someone@banyan sudo -A 'sh -c "cd /home/corse/deleted/user/guppy; tar xfBp -"'
    
  6. Remove the files from guppy.

Posted by Olivier | Permanent link | File under: administration, ubuntu

Fri Aug 16 12:35:12 +07 2019

Make permission based on Active Directory persistent

Starting with ESXi 6.5, under certain conditions, ESXi server will loose the permissions that was defined based on Active Directory. The server is still joint to the domain, but the user will not be able to access their VM anymore. All permissions have to be regenerated by hand.

That affects the permissions set on the VMs but also the permissions set at the server level.

While the cause is not exactly known, I suspect it is linked to a temporary unavailability of the AD (like a reboot): when ESXi server cannot reach the AD, it wipes off any permission corresponding to that AD. This is critical with the server hosting students VMs, especially the VMs for WAE course that has two or three students per VM.

To solve this problem, I have created a collection of tools that will automatically back-up the permissions based on AD and restore them if they disappear. The tools have been bundled in a VMWare VIB (vSphere Installation Bundle) for easy deployment.

Download and install the VIB

Get the VIB from /home/pc-application/WINAPPS/VMware/offline_bundle_persist_perms.zip copy it on the ESXi server. Note that only the offline version of the VIB is available.

Connect to the server shell and run:
esxcli software vib install -f -d /offline_bundle_persist_perms.zip
That's all you have to do. From now on, the AD permissions will persist.

The tools have been tailored for CSIM environment, AD permissions are in the form SMB4\user.

update ESXi

Third parties VIB that are not officially signed by VMware prevent ESXi update. And this is diagnosed quite late in the update process, so precious time can be lost. Note that this is true also for ghettoVCB.

Hence, the VIB must be removed before any ESXi update:

  1. login on the server and copy the file /etc/persist_perms/permissions to some place safe across reboots, for example /vmfs/volumes/datastore1/;
  2. remove the VIB (see the command below) without removing AD permissions;
  3. proceed to ESXi update;
  4. reinstall the VIB (see above) and restore the permissions file.

Remove the last AD permission

When the script detects that there is no more AD permissions, it will automatically restore them from the back-up. So when you remove the last permission, the script will try and restore it.

To prevent that you can remove the VIB before removing the last AD permission:

esxcli software vib remove -f -n persist_perms
Alternatively, you can stop cron:
kill `cat /var/run/crond.pid`
remove the permission, update the file /etc/persist_perms/permissions:
echo  \#\!/bin/sh >/etc/persist_perms/permissions
and restart cron:
/usr/lib/vmware/busybox/bin/busybox crond

What is installed by the VIB

The VIB installs the following files:
  • /opt/persist_perms/test-perms.sh the script that checks that the AD permissions are present, makes a back-up and restores the permissions;
  • /opt/persist_perms/perm.awk and /opt/persist_perms/role.awk the scripts that are used in generating the back-up;
  • /etc/persist_perms/permissions the list of AD permissions, this is a shell script that contains all the commands needed to restore the permissions;
  • /etc/rc.local.d/999.persist_perms.sh the script in charge of setting the crontab when ESXi boots (at boot, the crontab consists of a fixed list of tasks, call to our test needs to be inserted during every boot).

How to construct a VIB

VMWare used to offer the tool vibauthor to package a VIB. This tool is not available on VMWare website, but it has been packaged in a docker container by William Lam. You can follow the instructions on William's page or follow the instructions bellow:

  1. install docker and rpm on an Ubuntu machine, follow the current documentation;
  2. download the docker container from /home/pc-application/WINAPPS/VMware/vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm;
  3. install the container, you must be root to run this command:
    rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
    
  4. run the container:
    docker run --rm -it lamw/vibauthor
    
  5. you should be able to access the command vibauthor

Preparing the files to create a VIB has been inspired from the page by William about Creating Custom VIBs For ESXi 5.0 & 5.1 with VIB Author Fling. All the files needed for this VIB can be downloaded from /home/pc-application/WINAPPS/VMware/persist_perms.tgz.

After extracting the tar file, you should see the following file hierarchy:

$ tree stage
stage
├── descriptor.xml
└── payloads
    └── payload1
        ├── etc
        │   ├── persist_perms
        │   │   └── permissions
        │   └── rc.local.d
        │       └── 999.persist_perms.sh
        └── opt
            └── persist_perms
                ├── perm.awk
                ├── role.awk
                └── test-perms.sh

7 directories, 6 files

Note that:

  • In stage/payloads/payload1 you see the files that will be installed by the VIB;
  • The file descriptor.xml describes the VIB and how it should be installed;
  • The exact hierarchy of stage, payloads, payload1, descriptor.xml seems to be defined by vibauthor and should not be modified;
  • The file permissions must have the sticky bit set, that allows the script to modify the file and ensure that the modifications persist across reboot.
  • The file 999.persist_perms.sh is executed by the VIB installation while other files are simply copied to their destinations. My understanding is that because that file is located in /etc/rc.local.d is is being executed after installation. This serves our purpose as it ensures the cron file is updated immediately and not only after a reboot. So the back-up script is launched shortly and a back-up is made on the next cron execution.

Once the stage hierarchy has been finalized, the VIB may be created with the command:

vibauthor -C -t stage -f -O offline_bundle_persist_perms.zip

Posted by Olivier | Permanent link | File under: administration, vmware