Thu Sep 29 13:03:32 +07 2022

Some stuff related VMware and SSL certificates

Certificates are located in /etc/vmware/ssl rui.key is the private key mod 400 rui.crt is the certificate + the CA mod 644 The private key is installed once for all The certificate is generated/installed by ~on/letsencrypt/install_cert_virtual After installing the new certificates, restart the management agents: /etc/init.d/hostd restart /etc/initt.d/vpxa restart If that does not work, try to restart the agent manually from the console. Or try the command dcui from an ssh connection. I have not tried the solutions below. See also https://www.nakivo.com/blog/how-to-restart-management-agents-on-a-vmware-esxi-host/ 4. Use this command as an alternative, to restart all management agents on the ESXi host. services.sh restart &tail -f /var/log/jumpstart-stdout.log The progress of the VMware agents restart is displayed in the console output. VMware restart management agents with services.sh 5. You can also try to reset the management network on a VMkernel interface: esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0 The vmk0 interface is used by default on ESXi. If you have a different name for the management network interface, use the appropriate interface name in the command. This complex command consists of two basic commands separated by ; (semicolon). The vmk0 management network interface is disabled by the first part of the command. When this part is executed successfully and vmk0 is down, then the second part of the command is executed to enable the vmk0 interface. As a result, the ESXi management network interface is restarted. The authorized_keys file for root is in /etc/ssh/keys_root/authorized_keys It can be used from ufo sudo ssh -i /root/.ssh/id_rsa_virtual root@virtualX Crontab for root is in /var/spool/cron/crontab/root

Posted by Olivier | Permanent link

Thu Sep 29 12:47:49 +07 2022

chown error in slapd start

I would like to report a small inconvenient in the port of openldap26-server on FreeBSD. The startup script /usr/local/etc/slapd incudeds the line: chown -RL "$slapd_owner" "${DBDIR}" around the line 127 My DBDIR is a file system mounted to only contain LDAP data: fbsd63<root>334: mount -p |grep da1p1 /dev/da1p1 /var/db/openldap-data ufs rw 2 2 As such, it contains the file .sujournal that is not changeable" fbsd63<root>335: ll -o /var/db/openldap-data/.sujournal 4 65600 -r-------- 1 root wheel schg,sunlnk,nodump,opaque 33554432 Sep 20 12:02 /var/db/openldap-data/.sujournal The chown in the startup script throws an error: fbsd63<root>336: /usr/local/etc/rc.d/slapd restart Stopping slapd. Waiting for PIDS: 41425. chown: /var/db/openldap-data/.sujournal: Operation not permitted Performing sanity check on slap configuration: OK Starting slapd. The script still work, but it may be confusing for some users who have their DBDIR as an independant filesystem.

Posted by Olivier | Permanent link

Wed Jun 15 11:20:14 +07 2022

Rebuilding the firewall

After the power failure of May 10th, 2022, the firewall would not restart.

In this report, I want to summarize the steps that I have taken to get the firewall working again.

Hard disk failure

One of the hard disk had failed. Because the disks are in a RAID 1 array, it was only a matter of replacing the failed disk with one saved from a other machine. The RAID will take care of the mirroring of the nee disk.

The steps to restore the file systems

When trying to boot the firewall, it would hang at a message about Rebuilding LDAP database. My diagnostic was that some file had been corrupted by the power failure and needed to be restored.

In order to restore the file-systems from Amanda, I need to boot from a live system that offers:

  • mounting the Linux ext partitions;
  • connecting to Amanda server with ssh;
  • extracting the files with tar.

Booting the machine from a USB key

That is the part that took me the longest to solve.

The server would boot from a Freebased installation USB, I could launch a live shell, but the partitions on the hard disk were faulty and if the live shell can mount an ext partition, it is missing the fsck for that type of partition.

To mount an ext partition:

  mount -t ext2fs /dev/something /somewhere

Any other type of USB key would not work. I tried Ubuntu, UBCD, Hiren, Vyatta...

I finally resolved to install Ventoy (ventoy.net) on a USB key; and on that Ventoy key I copied Hiren's BootCD 12.0. Note that Ventoy needs a machine under Windows to first create the USB key.

All the files that I used are under /home/pc-application/ZeroShell

For the next step, I also copied ZeroShell 3.9.5 on the Ventoy key.

Restore the file systems

Booting Ventoy key, I could start Hiren BooCD and choose Mini Linux.

In that mini Linux environment, I could fsck the damaged partitions and restore the file systems.

There are 3 partitions on the disk:

  1. /dev/sda1 10 GB, that is mounted on /boot
  2. /dev/sda2 50 GB, that is mounted on /cdrom
  3. /dev/sda3 75 GB, that is mounted on /DB

Note that / is a RAM disk.

Restoring is just a matter of recovering the data from Amanda, mounting each partition to /mnt on the mini Linux and issuing commands like:

  ssh root@192.41.170.11 tar cf - the recovered partition|(cd /mnt; tar xfBp -)

Note that root must be allowed to connect to Amanda.

Rebooting the firewall

When the 3 partitions had been restored, I tried to reboot the firewall, but the error about rebuilding LDAP persisted.

At the end of my rope, I tried to launch ZeroShell 3.9.5 from the Ventoy key. I was not sure of what to expect, but I knew that ZeroShell could run without being copied to the hard disk. That newer version of ZeroShell managed to rebuild LDAP and finished booting. It was just a matter of rebooting the firewall from its original disk.

So far, I don't know why that Dell PowerEdge R200 is so temperamental about the type of USB it can boot from.

I am not sure what LDAP is being used for by ZeroShell because U think I have disabled it in the configuration. I have added a script that dumps LDAP database every two hours.

It seems that the restoring of the file systems has messed up some of the group ownership of some files and directories. The only real effect I could see is in Amanda in /Database/Links/opt/libexec/amanda/.

When booted from the USB key in ZeroShell 3.9.5, the firewall did work properly, but I had no access through SSH or the web interface. That is why I reverted to the previous version.


Posted by Olivier | Permanent link | File under: administration, firewall, backup

Wed Jun 8 16:06:02 +07 2022

Some things to install Amanda on FreeBSD

Recently I ran into a couple of problems when installing or upgrading Amanda on FreeBSD. The problems may have been present earlier, but I did not saw them then.

Modify /usr/local/etc/pkgtools.conf

Add the following in the file /usr/local/etc/pkgtools.conf so that Amanda is being built with the proper configuration:

  MAKE_ARGS = {
	'misc/amanda-server' => 'AMANDA_USER=amanda AMANDA_GROUP=amanda AMANDA_CONFIG=normal AMANDA_WITHOUTIPV6=yes',
	'misc/amanda-client' => 'AMANDA_USER=amanda AMANDA_GROUP=amanda AMANDA_CONFIG=normal AMANDA_GNUTAR_LISTDIR=/usr/local/var/amanda/gnutar-lists AMANDA_WITHOUTIPV6=yes'
  }

On a client machine, only the amanda-client part is needed.

Most of the configuration choices come from very old installation (pre-FreeBSD) but this is how it works for CSIM.

Modify the location of amanda-security.conf

On the server, the security file for Amanda client is located in /usr/local/etc/amanda-client/amanda-security.conf. If kept in the default place, it will conflict with the ownership and mode of the default directory.

This change can only be made by modifying the Makefile file in /usr/ports/misc/amanda-server. Note that even if that configuration file only pertains to Amanda client, the makefile is located in Amanda server.

Around the 33rd line of the makefile, change the value of --with-security-file to reflect:

--with-security-file=/usr/local/etc/amanda-client/amanda-security.conf
Always build or upgrade the server after building the client

Building Amanda client will change the contents of the file /usr/local/lib/perl5/site_perl/Amanda/Constants.pm to contain the line:

$AMANDA_COMPONENTS = " client amrecover ndmp";

Only after building or upgrading the server will the line be reverted to the correct:

$AMANDA_COMPONENTS = " server restore client amrecover ndmp";

Failing to have the proper line will result of the following error message in Amanda summary email:

taper: FATAL Can't locate object method "new" via package "Amanda::Xfer::Dest::Taper::Splitter" (perhaps you forgot to load
"Amanda::Xfer::Dest::Taper::Splitter"?) at /usr/local/lib/perl5/site_perl/Amanda/Taper/Scribe.pm line 764.

Posted by Olivier | Permanent link | File under: administration, backup

Thu May 19 11:08:40 +07 2022

Upgrade FreeBSD source with git

The following steps are needed to upgrade FreeBSD sources using git.
  1. Install git from the ports.
  2. Run it once to create the proper environment:
    sudo  git clone --branch releng/13.1 https://git.freebsd.org/src.git /usr/src
    
  3. Updates should be run with the command
    cd /usr/src
    sudo git pull
    
  4. To change to a new release
    cd /usr/src
    sudo git checkout releng/13.2
    

Posted by Olivier | Permanent link | File under: administration, freebsd

Fri Feb 18 15:03:42 +07 2022

Amanda looses connection to certain Ubuntu servers

It seems that from time to time amanda will loose connection to some servers running Ubuntu.

In the daily amanda mail report, there will be lines like:

puffer1000 / lev 0 FAILED [Request to puffer1000 failed: timeout waiting for ACK]

and an amcheck -c on that host will return an error.

This can be cleared by restarting inetd on the affected servers:

sudo service inetd restart
or xinetd on firewall:
sudo service xinetd restart

Posted by Olivier | Permanent link | File under: administration, backup

Wed Feb 16 13:26:20 +07 2022

Naming virtual machines

When a virtual machine (VM) is created to be used/managed by a user, it should not persist when the use leaves AIT.

Proper naming of the VMs will allow the scripts that manage the user accounts to automatically back-up and delete the obsolete VMs.

The name of the VMs should be of the form:

  1. In the simplest form, the name of the VM is the username, for example st12345
  2. If one user needs more than one VM, the VM name can start with the username, followed by a space or - or _ and some number or index; for example st123456-2 or st123456 banana
  3. Alternatively, it is possible to set the name of the owner of the VM in the VM notes; the line should be of the form owner=st123456
    Note that the check is very loose on this form, if the string owner=user matches, whatever follows that string is accepted.

Any VM that follow these conventions will be automatically backed-up in the user's homedirectory and deleted from the virtual server when the user's account expires.


Posted by Olivier | Permanent link | File under: administration, vmware

Mon Feb 14 13:53:07 +07 2022

Managing eduroam profile on enterprise-wifi.net

enterprise-wifi.net is a very strong tool to create automatic configuration for eduroam. Once the eduroam profile has been created, auto-configuration scripts can be downloaded for Windows, Mac-OS and Linux (so far, no script is available for Android).

Safely configuring an eduroam profile by hand can be tricky as it requests very specific steps to be executed in a given order, eterprise-wifi.net is a great time and headache saver.

But it comes with some caveats.

Managing the administrators

Only Google accounts can create and manage profiles on enterprise-wifi.net. Furthermore, I could not find a way to assign the master administrator role to someone else: the use who first created the profile is the only one that can add administrators. When that user becomes invalid in Google, a totally new profile must be created from scratch.

The profile used for AIT/CSIM has been created by nicoleo@ait.asia

Managing the certificates

For a PEAP-MSCHAPv2 the profile must include a root certificate authority, even when you use a properly certified, and the infrastructure is able to provide all CA, root and intermediates.

As a consequence, if the CA included to the profile expire, the profile need to be generated anew and every device must install the new profile.

As of February 2022, we are using the following:

The profile will have to be updated by September 2024.

The intermediate CAs are provided by Let's Encrypt when signing or renewing a certificate, the are in the file ca.cert; the root CA, I found it on Let's Encrypt web page about the chain of trust.


Posted by Olivier | Permanent link | File under: administration, wireless

Fri Feb 11 11:25:04 +07 2022

Installing ESXi 6.x on Dell R740

Dell PowerEdge R740 comes with the RAID controller PERC H750 and the drivers for this controller are not on the .iso installer for VMware ESXi 6.x.

I used the following method to install the hypervisor.

  1. Create the virtual disk on the RAID controller;
  2. Download the proper driver from VMware Compatibility Guide;
  3. Connect a spare disk via USB or use an USB key, this will be used to do a temporary installation.
    Note: it is very important that the temporary disk or the USB key are smaller than the RAID volume created at step 1;
  4. Install ESXi on the temporary disk or the USB key;
  5. Install the driver for the RAID controller; it should be done with a command like:
    esxcli software vib install -d filename.zip
  6. Boot the newly installed ESXi from the temporary disk or USB key; you should be able to go to the storage and see the RAID/PERC H750 volume;
    Note: it is very important that you create a different temporary install for each server (steps 4 and 5); if you clone one single image to several ESXi servers, you will have some UUID conflicts later;
  7. Reboot the system with a live Linux;
  8. Copy the system from the temporary disk or USB key to the RAID virtual disk, use a command like:
    dd if=/dev/sdx of=/dev/sdy
  9. Remove the temporary disk or USB and reboot into the newly installed ESXi.

Other solution would be to install PowerCLI to prepare a new installation media that include the RAID driver.


Posted by Olivier | Permanent link | File under: administration, vmware

Fri Jan 21 11:38:12 +07 2022

Create new collection in DSpace

At the beginning of each years, new collections must be added to DSpace. The collections are called: CS Dissertations, CS Resarch reports, CS Theses followed by the year, same for IM and DSAI. For consistency, care must be take for the name of the collections, so they are displayed in the corresct alphabetical order.

1Select the proper community and create a new collction. 
2Then you need to assign the roles.
Click on Create... for Submitters. You will define who is allowed to submit new documents.
3 Enter the group name CSIM_LDAP and click on Groups...
4 Click on Add.
5 Click on Save.
Any user autenticated to CSIM LDAP server will be allowed to submit a new document.
6 Note that the Submitters has changed to COLLECTION iii SUBMIT.
Click on Create... for the Accept/Reject Step.
Repeat the steps 3, for and 5 with the group Office instead of CSIM_LDAP.
That will make sure that the users beloging to the group Office (ie. our secretaries) can accept the documents submitted.
7 Note that the Accept/Reject Step has changed to COLLECTION iii WORKFLOW STEP-1.
Click on Edit authirization policies directly.
8 This page lists the autorizations that have been defined.
By AIT policy, the contents of the thesis and dissertations must not be publicly accessible.
Click on DEFAULT_BITSTREAM_READ.
9 In the pull down menu, select AIT.
That encomprise users connected inside the Institue and users authenticated to CSIM LDAP. Any of these users can read the contents of the document.
Click Save.
10 Note that the DEFAULT BITSTREAM READ has been changed to AIT.
At this point, you can click on Edit near the COLLECTION iii SUBMIT and COLLECTION iii WORKFLOW SETP 1 and check that the correct groups are defined: CSIM_LDAP and Office.
When you are done, click on Return.
The new collection has been created and configured, students can start adding thesis and reports to the new collection.
Reppeat for all the new collections that need to be created at the beginning of each year.

Posted by Olivier | Permanent link | File under: administration, web