Sattelite Link Performance Analysis Package

This package is composed of two separates modules, one to collect the data and save them in a file, and one to analyze the data from the file and produce the graphics and HTML page.

Further are considerations about running ftp and WWW servers on a BSDi machine.

Data collection `comstream` (new version 3-Dec-99)

New:
3-Dec-99 Fixed the name of the files used to store the data, starting year 2000, the file are of the form Jan-2000. Some major changes about data collection and storage had been introduced (see below the parts in italics).
13-Nov-98 Fix the bug about desyncrhonizing read and write on tty connection (compile command has changed).
16-Feb-98 Fixed the access mode on the data files.
28-Jan-98 The module is testing if another acquisition process is running.

This module is in charge of collecting the data from the satellite modem and store them in files.

The data

Every second this module call to the Comstream modem, using the serial line, and request for the value of Eb/No level and AGC level. An average is calculated every 5 minutes.

The average data are stored in a file name from the month and year; for example the file for January 1998 will be Jan-98 (starting January 2000, the files will be called with 4 digits for the year).

Each line in the data file is in the format: Day Time AGC EbNo

For example:
16 20:48 60 11.4
represent a data collected on the 16th of the month, at 8:48 pm, when the AGC level was 60 and the Eb/No level was 11.4.

The data file represent 180KB/month with one measure every 5 minutes.

By default, the data files are created in /var/spool/ftp/pub/comstream/data/ (further /var/spool/ftp/pub is referenced as BASEFTP, it can be relocated). The files has to be created in a directory accessible via ftp so they can be downloaded.

At the same time, average data are stored in a MySQL database, the value stored are Time Stamp, Avg EB/No, Avg AGC, Min Eb/No, Max Eb/no, Min AGC, Max Agc, Error Eb/No, Error AGC, Interval.

Time Stamp: is the time when the average data was computed;
Avg Eb/No: is the average Eb/No on the last Interval of time;
Avg AGC: is the average AGC on the last Interval of time;
Min Eb/No: is the minimum value encountered for Eb/No on the last Interval of time;
Max Eb/No: is the maximum value encountered for Eb/No on the last Interval of time;
Error Eb/No: represent the numbre of measurement of Eb/No that could not be done on the last Interval of time;
Min AGC: is the minimum value encountered for AGC on the last Interval of time;
Max AGC: is the maximum value encountered for AGC on the last Interval of time;
Error AGC: represent the numbre of measurement of AGC that could not be done on the last Interval of time;
Interval: is the time interval between each computation of average, gien in seconds, by default it is 300 seconds (5 minutes).

The interval for data collection can be adjusted by setting the number of seconds in the INTERVAL macro at the top of the program. As the data collection is separated from it's analyze, the collecting interval can be adjusted independently.

Serial Line Setting

As default, comstream access to the modem via the device /dev/tty00, this can be changed with the macro TTY.

The serial port on the modem must be set to 9600 bauds, and 8bit, no parity. In case this should be changed, the C code would have to be adapted. I suggest the following way to proceed:

Using a terminal emulating tool like tip(1), connect to the modem and find a setting of the line that allow communication;
Escape to the shell and check the configuration of the line, this can be done on BSDi with the command stty -a -f /dev/tty00;
Use the configuration as given by stty(1) to set up the flag of the tty device in the C code.

The module

This module is available from ftp://ftp.cs.ait.ac.th/pub/ai3/comstream/comstream.c.gz.

As default it is installed in /usr/local/comstream (this is further refered under the name BASELOG) and it is called comstream.

It consists in a single C file, that has been compiled and tested on BSDi 2.1 (gcc version 2.7.2).

To compile it, use a command line like:
ai3gate23: cc -O2 test.c -o test -lmysqlclient -lm
the library mysqlclient is required to store the data on MySQL database.

When called, comstream starts a subprocess that will collect the data every 5 minutes. The interval can be adjusted. The pid of the subprocess is stored in BASELOG/lock.

When called with option -q, it stops the subprocess collecting data.

When called with option -i, it starts the interactive mode that allows user exchanging commands with the modem. The interactive mode shares the serial connection to the modem with the subprocess that collects data.

The data collection module should be started at boot time, with an entry in /etc/rc.local that looks like:

# start the login of satellite modem
if [ -f /usr/local/comstream/comstream ]
then
        /usr/local/comstream/comstream
fi

Comstream is made available under the the GNU General Public Licence.

Data analyzis `analyze` (new version 3-Dec-99)

New:
3-Dec-99 Fixed the name of the files used to store the data, starting year 2000, the file are of the form Jan-2000.
02-Sep-99 Fixed errors in the refresh time and in the display of the uptime.
16-Feb-98 Fixed the mark of the 1-Jan on yearly graph.

This module is in charge of reading the data files collected by the comstream module and producing graphs and HTML as shown on the page http://ai3gate.cs.ait.ac.th/comstream/.

The graphs

Four graphs are automatically produced, one sumarizing the data for the last 24 hours, one for the last week, one for the last 30 days, and one for the last year. Each graph is accompagnied with the maximum and minimum value, for each measure, on the regarded period.

Eb/No level is represented in blue line. It is scaled to be ploted only for the values in the range 9..14 dB (this can be adjusted with MINEBNO and MAXEBNO).

AGC level is represented in green impulses. It is scaled to be ploted only for the values in the range 50..70 (this can be adjusted with MINAGC and MAXAGC). The scale for AGC level is located on the right handside of the graph.

The weekly, monthly and yearly graphs are not computed on average data but on the full set of data as it seams important to keep the full details, including the maximum and minimum values that data can reach. Saying that the Eb/No level was average of 11 toward the day, while it was 8.5 for 12 hours and 13.5 for 12 hours, is meaning less, 8.5 is not a workable situation and 13.5 is a very good situation, as a result the link has been working good for12 hours, not working average for 24 hours.

Dayly graph is updated every time the module is run. It is created in BASEWWW/comstream/images/oneday.gif.

Weekly graph is updated every 30 minutes, at 0:00 am, 0:30 am, 1:00 am, etc. or if the previous weekly graph is more than 30 minutes old. It is created in BASEWWW/comstream/images/onweek.gif.

Monthly graph is updated every 2 hours, at 0:00 am, 2:00 am, 4:00 am, etc. or if the previous weekly graph is more than 2 hours old. It is created in BASEWWW/comstream/images/onemonth.gif.

Yearly graph is updated every day, at 0:00 am, or if the previous graph is more than 24 hours old. It is created in BASEWWW/comstream/images/oneyear.gif.

The four graphs may be forced to be computed when using the option initialise (-i).

Each graph requires 5KB of hard disk space.

The HTML file

The HTML file is created each time the module is run. It uses data files created when updating the graphs.

It is created in BASEWWW/comstream/index.html.

The module

This module is available from: ftp://ftp.cs.ait.ac.th/pub/ai3/comstream/analyze.gz.

The module is called analyze, and is installed as default in BASELOG.

It consist in a single Perl script that has been tested under BSDi 2.1, Perl 5.3.

It requires the tools Gnuplot and ppmtogif, both tools are available in BSDi. The exact location for those tools has to be updated if needed.

The location for BASEFTP has to be updated accordingly with the comstream module.

The graphs and the HTML files are created in the directory /var/www/docs/comstream (further, /var/www/docs is referenced as BASEWWW). It created several temporary files in BASELOG, the total size needed for temporary files is up to 2MB (when computing a full year graph). Some temporary graph in pbm format are also created in BASEWWW/comstream/image the size should not exceed 300K.

Temporary files in BASELOG are important and should not be deleted! as they are used to save maximum and minimum values for each graph, those values being used later to create the HTML file.

The analyze module is run through crontab(1). I suggest an entry like:
0-55/5 * * * * /usr/local/comstream/analyze
to run the analyze module every 5 minutes.

The option -i (initialise) is provided for ionteractive run of the module, it will force an update of every of the four graphs, as well as the update of the HTML file.

Analyze is made available under the the GNU General Public Licence.

Input/Output considerations

The data collection module will append 17 bytes to the end of the current log file, doing this every 5 minutes. That is 4,896 bytes pper day.

The data analysis module will consume the following dik I/O:

For the dayly graph, every 5 minutes, read up to one month log file (180K), write one day temporary data file (5.5K), this file being read by Gnuplot (5.5K); total 191K every 5 minutes, 55M/day;
For the weekly graph, every 30 minutes, read up to one month + one week log file (225K), write one day temporary data file (39K), this file being read by Gnuplot (39K); total 303K every 30 minutes, 14.5M/day;
For the monthly graph, every 2 hours, read up to two months log file (360K), write one day temporary data file (165K), this file being read by Gnuplot (165K); total 690K every 2 hours, 8.2M/day;
For the monthly graph, every 24 hours, read up to one year log file (1,800K), write one day temporary data file (1,100K), this file being read by Gnuplot (1,100K); total 5.9M/day;

So we have a grand total of 83.6MB of disk I/O per day.

As the amount of disk I/O is much more important for the analysis module than for the collecting module, I recommend, if both modules run on separate machines, that the data are located on the local dik of the machine computing the analysis.

Other tools

As a result of data analysis, an HTML page is created, and data files are made accessible through ftp.

In order to start up WWW server and ftp server on a BSDi machine, here is a summary of the procedure.

Starting up WWW server on BSDi

When connected as root, do the following:

Edit the file /var/www/conf/httpd.conf and customize the value for ServerAdmin and ServerName;
Run /usr/sbin/config_www, it will start the httpd daemon, and create a starting page /var/www/docs/index.html;
Create a directory entry for the data analysis page: mkdir /var/www/docs/comstream and for the images: mkdir /var/www/docs/comstream/images;
If this server is only meant to serve the data analysis page, go to the directory /var/www/cgi-bin and change the mode of every file to null with the command chmod 0 *.

With GCI scripts disabled, the server is safe.

Starting up `ftp` server on BSDi

When connected as root, run the script config_anonftp that is enough to crerate a safe anonymous ftp server that will not allow uploading.

You then need to create the directories /var/spool/ftp/pub/comstream and /var/spool/ftp/pub/comstream/data.

Comments to: Olivier Nicole

Sattelite Link Performance Analysis Package

Data collection comstream (new version 3-Dec-99)

The data

Serial Line Setting

The module

Data analyzis analyze (new version 3-Dec-99)

The graphs

The HTML file

The module

Input/Output considerations

Other tools

Starting up WWW server on BSDi

Starting up ftp server on BSDi

Data collection `comstream` (new version 3-Dec-99)

Data analyzis `analyze` (new version 3-Dec-99)

Starting up `ftp` server on BSDi