CSIM Technical Notes

Starting with ESXi 6.x, VMware could not monitor Dell hard disks properly on PowerEdge 1950, 2950, R510 and R710: some specific driver is not included anymore and an error message is saying that The Small Footprint CIM Broker Daemon (SFCBD) is running, but no data has been reported. You may need to install a CIM provider for your storage adapter.

The article Monitor storage for a Dell PERC H710 Mini on vSphere 6.5 by Wil van Antwerpen suggest to install the driver directly drom Broadcom-LSI.

I downloaded the latest driver https://www.broadcom.com/support/download-search/?pg=Storage+Adapters,+Controllers,+and+ICs&pf=RAID+Controller+Cards&pn=MegaRAID+SAS+9266-8i&pa=Management+Software+and+Tools MegaRAID StorCLI MR 6.6 Version: 1.14.12 StorCLI User Guide https://support.hpe.com/hpsc/doc/public/display?docId=a00048288en_us VMware this package must be used on ESXi 4.x servers. VMware-MN This package must be used on ESXi 6.x servers and onwards when the driver used is a legacy MegaRAID SAS Device Driver. VMware MN-NDS This package must be used with MR driver, lsi_mr3, which is a native driver.

When setting a test server, the binding IP address for OpenLDAP is defined in /etc/rc.conf.

We run freeradius in a chroot'ed environement; as such, libraries from /usr/local/lib/freeradius-x.y.z must be copied manually in the equivalent directory under /var/chroot/freeradius.

When launching a temporary test server, the IP address used to bind freeradius is defined in /usr/local/etc/raddb/sites-enabled/default (in two places).

1. clamav must be build with gcc:

   setenv CC /usr/local/bin/gcc
   

2. On mail serveur, clamav must run as user kluser:vscan, after each update, you must change the ownsership of /var/run/clamav and /var/db/clamav (they are overwritten by the install procedure).
   You must also install the proper /usr/local/etc/rc.d/clamav-clamd script that allows to run two versions of in parallel.

First usage

1. Install acme.sh

   git clone https://github.com/Neilplan/acme.sh
   

2. Install nsupdate (on FreeBSD you need to install bind-tools)
3. Set the environment variable as follow:

   NSUPDATE_SERVER=dns.cs.ait.ac.th
   NSUPDATE_KEY=/usr/local/etc/acme.sh/update.key
   NSUPDATE_ZONE=cs.ait.ac.th
   

   Note that NSUPDATE_KEY has to be copied from Olivier's account on banyan
   These variables are only needed on the first use of acme.sh script, then they are saved in the configuration file but the update.key file needs to be present any time acme.sh will be run in the future.
5. On the DNS server, update the file /var/chroot/named/etc/namedb/named.conf to add the line

   grant acme-sh-update name _acme-challenge.xxx.cs.ait.ac.th TXT;
   

   then reload DNS.

Generate a certificate

Not: it does not work on mail, it is a communication problem. until it is fixed, generate the certificate on ufo.

With the DNS API, generating the Let's Encrypt certificate with DNS validation can be done in a single command:

~/acme.sh/acme.sh --issue --dns dns_nsupdate --force --signcsr --csr /usr/local/ssl/csr/xxx.cs.ait.ac.th.csr -d xxx.cs.ait.ac.th

The parameter --force is required because we use sudo.

Install the certificate

1. Change directory to ~/.acme.sh/xxx.cs.ait.ac.th
2. Check the validity of the new certificate:

   openssl x509 -noout -modulus -in xxx.cs.ait.ac.th.cer | md5
   sudo openssl rsa -noout -modulus -in /usr/local/ssl/key/xxx.cs.ait.ac.th.key|md5
   

   Both command should provide the same output, if they are not, something went terribly wrong, abort!
3. Check the validity of the CA:

   diff ca.cer /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca
   

   The files should be similar (it may have a difference of a blank line).
4. If it is the first time a certificate is being generated, or if the files differ, make a backup copy of /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca:

   sudo cp /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca.old
   

   and copy the new file ca.cer in place:

   sudo cp ca.cer /usr/local/ssl/ca/xxx.cs.ait.ac.th.ca
   

   Note that we maintain one CA file per certificate, even if all the CA files should be the same.
5. Change directory to /usr/local/ssl/crt
6. Make a backup copy of xxx.cs.ait.ac.th.crt:

   sudo cp xxx.cs.ait.ac.th.crt xxx.cs.ait.ac.th.crt.old
   

   and install the new certificate:

   sudo cp ~/.acme.sh/xxx.cs.ait.ac.th/xxx.cs.ait.ac.th.cer xxx.cs.ait.ac.th.crt
   

7. On mail server, you also need to go to /usr/local/ssl/crt/combined and back up and generate a new mail.cs.ait.ac.th.ketcrt file.
8. Restart the service xxx (or reboot the mail server).
9. Take note of the expiry date of the certificate:

   openssl x509 -text -noout -in xxx.cs.ait.ac.th.crt
   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               xxx:xxx:xxxx
       Signature Algorithm: sha256WithRSAEncryption
           Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
           Validity
               Not Before: Sep 17 05:29:35 2019 GMT
               Not After : Dec 16 05:29:35 2019 GMT
   

   and record the Not After date to remember updating the certificate before in expires.

Adding SSH public key authentication to ESXi

On the ESXi server, edit the file /etc/ssh/keys-root/authorized_keys and add the public keys, one key per line.

Supposedly, the file already exists and is mode 1600.

Suppress the SSH enable message

ESXi servers always complain when SSH is enabled. This warning message can be deactivated with the command:

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

When a student leaves AIT, there is no automatic procedure to save and removes the files that were created on the local disks of guppy.

Further more, because the students are using some tools to virtualize environment, there are many file duplicates that should be taken into account to reduce the space on the final backup. In some instances, a space reduction of 30% could be achieved by simply removing the duplicated files.

1. Connect to guppy and become super user;
2. Compress the files of the user:

   find /home2/user -type f -exec xz -f -9 {} \;
   

3. Remove duplicates:

   freedups.pl -a /home2/user
   

4. Create some directory on banyan to receive the files, for example /home/corse/deleted/user/guppy;
5. Transfer the files:

   cd /home2/user; tar cf - . |ssh someone@banyan sudo -A 'sh -c "cd /home/corse/deleted/user/guppy; tar xfBp -"'
   

6. Remove the files from guppy.

Starting with ESXi 6.5, under certain conditions, ESXi server will loose the permissions that was defined based on Active Directory. The server is still joint to the domain, but the user will not be able to access their VM anymore. All permissions have to be regenerated by hand.

That affects the permissions set on the VMs but also the permissions set at the server level.

While the cause is not exactly known, I suspect it is linked to a temporary unavailability of the AD (like a reboot): when ESXi server cannot reach the AD, it wipes off any permission corresponding to that AD. This is critical with the server hosting students VMs, especially the VMs for WAE course that has two or three students per VM.

To solve this problem, I have created a collection of tools that will automatically back-up the permissions based on AD and restore them if they disappear. The tools have been bundled in a VMWare VIB (vSphere Installation Bundle) for easy deployment.

Download and install the VIB

Get the VIB from /home/pc-application/WINAPPS/VMware/offline_bundle_persist_perms.zip copy it on the ESXi server. Note that only the offline version of the VIB is available.

Connect to the server shell and run:

esxcli software vib install -f -d /offline_bundle_persist_perms.zip

That's all you have to do. From now on, the AD permissions will persist.

The tools have been tailored for CSIM environment, AD permissions are in the form SMB4\user.

update ESXi

Third parties VIB that are not officially signed by VMware prevent ESXi update. And this is diagnosed quite late in the update process, so precious time can be lost. Note that this is true also for ghettoVCB.

Hence, the VIB must be removed before any ESXi update:

1. login on the server and copy the file /etc/persist_perms/permissions to some place safe across reboots, for example /vmfs/volumes/datastore1/;
2. remove the VIB (see the command below) without removing AD permissions;
3. proceed to ESXi update;
4. reinstall the VIB (see above) and restore the permissions file.

Remove the last AD permission

When the script detects that there is no more AD permissions, it will automatically restore them from the back-up. So when you remove the last permission, the script will try and restore it.

To prevent that you can remove the VIB before removing the last AD permission:

esxcli software vib remove -f -n persist_perms

Alternatively, you can stop cron:

kill `cat /var/run/crond.pid`

remove the permission, update the file /etc/persist_perms/permissions:

echo  \#\!/bin/sh >/etc/persist_perms/permissions

and restart cron:

/usr/lib/vmware/busybox/bin/busybox crond

What is installed by the VIB

The VIB installs the following files:

• /opt/persist_perms/test-perms.sh the script that checks that the AD permissions are present, makes a back-up and restores the permissions;
• /opt/persist_perms/perm.awk and /opt/persist_perms/role.awk the scripts that are used in generating the back-up;
• /etc/persist_perms/permissions the list of AD permissions, this is a shell script that contains all the commands needed to restore the permissions;
• /etc/rc.local.d/999.persist_perms.sh the script in charge of setting the crontab when ESXi boots (at boot, the crontab consists of a fixed list of tasks, call to our test needs to be inserted during every boot).

How to construct a VIB

VMWare used to offer the tool vibauthor to package a VIB. This tool is not available on VMWare website, but it has been packaged in a docker container by William Lam. You can follow the instructions on William's page or follow the instructions bellow:

1. install docker and rpm on an Ubuntu machine, follow the current documentation;
2. download the docker container from /home/pc-application/WINAPPS/VMware/vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm;
3. install the container, you must be root to run this command:

   rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
   

4. run the container:

   docker run --rm -it lamw/vibauthor
   

6. you should be able to access the command vibauthor

Preparing the files to create a VIB has been inspired from the page by William about Creating Custom VIBs For ESXi 5.0 & 5.1 with VIB Author Fling. All the files needed for this VIB can be downloaded from /home/pc-application/WINAPPS/VMware/persist_perms.tgz.

After extracting the tar file, you should see the following file hierarchy:

$ tree stage
stage
├── descriptor.xml
└── payloads
    └── payload1
        ├── etc
        │   ├── persist_perms
        │   │   └── permissions
        │   └── rc.local.d
        │       └── 999.persist_perms.sh
        └── opt
            └── persist_perms
                ├── perm.awk
                ├── role.awk
                └── test-perms.sh

7 directories, 6 files

Note that:

• In stage/payloads/payload1 you see the files that will be installed by the VIB;
• The file descriptor.xml describes the VIB and how it should be installed;
• The exact hierarchy of stage, payloads, payload1, descriptor.xml seems to be defined by vibauthor and should not be modified;
• The file permissions must have the sticky bit set, that allows the script to modify the file and ensure that the modifications persist across reboot.
• The file 999.persist_perms.sh is executed by the VIB installation while other files are simply copied to their destinations. My understanding is that because that file is located in /etc/rc.local.d is is being executed after installation. This serves our purpose as it ensures the cron file is updated immediately and not only after a reboot. So the back-up script is launched shortly and a back-up is made on the next cron execution.

Once the stage hierarchy has been finalized, the VIB may be created with the command:

vibauthor -C -t stage -f -O offline_bundle_persist_perms.zip

Accessing an MS SQL server from Perl is a topic that lack complete information (or at least information that I can relate to.)

One solution is to use DBD::Sybase module in a compatible way. Better is to use DBD::ODBC in a more native way.

While the file path and name are specific to freeBSD, I am confident that the mechanism is the same fo other Unixes and the relationship between the components should be the same.

Install the needed components

You need to install:

• DBD::ODBC the Perl module
• unixODBC the ODBC library for Unix
• FreeTDS the Microsoft TDS library for Unix

Configure and test FreeTDS

FreeTDS configuration file is located in /usr/local/etc/freetds/freetds.conf and you need to add a section of the form:

# A typical Microsoft server
[SOMENAME]
        host = IP or hostname of MS SQL server
        port = 1433
        tds version = 7.0

You can test the configuration with:

tsql -S SOMENAME -U usernamne -P password -D database

At the prompt >1 you can enter SQL queries, enter go to execute the query.

Configure and test unixODBC

unixODBC configuration files are located in /usr/local/etc/odbcinst.ini and /usr/local/etc/odbc.ini.

The file odbcinst.ini defines the driver used by the database and should have a section of the form:

[some driver name]
Driver = /usr/local/lib/libtdsodbc.so
FileUsage   = 1

The Driver option points to the library in use, in that case, it points to FreeTDS driver.

The file odbc.ini defines the database server and should have a section of the form:

[SOMENAME]
Driver = some driver name (sqame as the section name above)
Database = database name
Server = IP or hostname of MS SQL server
TDS_Version = 7.0
Port = 1433

You can test the configuration with:

isql -v SOMENAME username password

At the prompt, you can enter SQL queries.

Configure and use DBD::ODBC

If everything worked so far, it is as simple as using:

use DBI;
my $dbh = DBI->connect("dbi:ODBC:DSN=SOMENAME", "username", "password");

How to replace the battery in the UPS at CSIM.

EPC powercom 2kVA

Battery: 6x12V 9Ah
Last replacement: May 2018

1.	Pull on the middle front bezel, then slide it to the right to un-hook it.
2.	Remove the screw then slide the retaining bar to the left.
3.	Pull on the battery drawer.
4.	Follow the diagram to install the new bateries: two sets of 3 batteries in serie

APC Smart UPS 3000XL

Power: 3kVA
Battery: 4x12V 18Ah
Last replacement: May 2018

1.	Remove the front bezel, then remove both screws to open the battery compartment.
2.	Follow the diagram to install the new bateries: two sets of 2 batteries in serie

APC Smart UPS 5000

Power: 5kVA
Battery: 16x12V 9Ah
Last replacement: May 2018

1.	Remove the front bezel, then remove both screws to open the battery compartment.
2.	Pull on the cord to disconnect the battery set, then use the tab to extract the battery set.
3.	Follow the diagram to install the new bateries: four sets of 4 batteries in serie&parallel (2+2)