ECESCAN




ECE SUBNETS TIME-ACTIVITY TRACKER



User Documentation





Georgia Institute of Technology
School of Electrical and Computer Engineering









Last updated by ChokSheak Lau on Tuesday, 9 July 2002




TABLE OF CONTENTS





 


SECTION ONE. INTRODUCTION

There are already many network monitoring software available, why do we need to create one more to compare with the rest? By the term network monitoring software, we usually get the idea that the software would periodically check the status of each server or workstation in the network, and if there are any problems that arise, the software will attempt to notify the network administrator by sending him an email message or even by paging for him. True, that is network monitoring. However, that type of network monitoring cannot provide the user with information on how the network is doing over an extended period of time.

The initial motivation to begin this project is to find a way to keep track of how frequently a particular resource in the network is being used. Each of the workstation or host machine has been allocated an IP address, and if there is any particular IP address that has not been used at all, then we can reallocate that IP address for use by another host. This thus allows the user to keep track of the usage level of the IP addresses in the entire subnet that is placed under monitoring by a long-term network monitoring software. This project attempts to provide this facility by monitoring a network over an extended period of time and the user can then compile information on the level of usage of the entire network that is under monitoring.


 


SECTION TWO. OBJECTIVES

The primary objective of this project is to monitor every single IP address in the entire ECE network. This is done in order to obtain factual, compiled information over an extended period of time on which IP addresses are actually not in use, and these IP addresses can then be reclaimed for use for another purpose.

This system can also function to provide a status map on the level of activity of each workstation. This can be used for informational purposes so that the system administrator can keep track of the level of activity of the workstations.

This manual attempts to describe the general structure of the program, but programming-specific details will NOT be covered here in order to keep this documentation manageable. The author has already inserted comments in the source code to make it easily readable. However, it must be understood that reading source code written by others is usually a very time-consuming task.


 


SECTION THREE. DETAILED SYSTEM DESCRIPTION

This section will describe the files organization structure and the operational structure of the system.


3.1 Basic Operational Structure

Ecescan is a perl script that scans the entire subnet of the ECE network. The entire system is built around this network scanning script to utilize it and feedback its functions.

The most general way that this system can be thought to be is a system that checks whether each of the workstation in the ECE network is active or inactive. This is done once every five minutes, 24 hours a day, 7 days a week. The results of the scan will be saved in a database and status reports can be generated by reading from that database.


3.2 Conceptual Directory Structure

A main directory for this system will store all the executable scripts and most other project files. There are a few directories where the other files of this system is stored for other uses:
DirectoryDescription
configstores the config file ecescan.cfg
logstores the system log files
data outputstores all the output files generated by the scripts, especially formattedoutput
WWWstores all files that make up the web interface
WWW outputstores all web-generated reports and other web-related files
CGI-BINnot used, converted from CGI scripts to PHP
database backup        stores all the backup files created by script dbbackup

No files will be copied into any other directories, at least this is not necessary under the system.

The MySQL database file is stored as directed by the perl database interface. It is not stored under the same directory structure as the other files in this system.



3.3 Detailed Information on Database

The database used here is MySQL version 9.38 distribution 3.22.30, for pc-linux-gnu (i686). The database name is ecescan. The database table that stores the records is named as iptable. The columns in this table are as follows:
Column NameDetailed Description
IPtext IP address
string, in '123.456.78.9' human-readable format
Hostnamehostname of IP
string, exactly as returned by Nmap
OSOperating System of IP
string, exactly as returned by Nmap
TimeActiveCountsinternally encoded sequence of counters for each time interval (one counter for each 5-minute segment)
string of digits, padded with dash character (minus sign), 288 x 8 bytes total
each number is represented by 8 bytes / characters
zero is represented as "--------", 10: "10------", 65536: "65536---"
time segment number goes from 0 to 287
first 8 bytes give number count for time segment 0, last 8 bytes for segment 287
time segment 0 corresponds to the time interval from 12:00am to 12:05am
time segment 1 corresponds to the time interval from 12:05am to 12:10am
time segment 287 corresponds to the time interval from 11:55pm to 12:00am
adding each of the 288 numbers together would give the total count
TimeTotalCountsexact same format as timeactivecount
DateInserteddate+time the IP was inserted into the database
LastActivedate+time the IP was last scanned to be active

Derived Fields, updated using updatederiveddata
Activenumber of times that the IP was scanned and detected to be active (up)
4 bytes, network long integer converted from IP address format
Totaltotal number of times that the IP was scanned
4 bytes, network long integer converted from IP address format
PercentActivePercentage of Active / Total
4 bytes, network long integer converted from IP address format
DaysOfflineNumber of days that the IP have been offline (not active)
4 bytes, network long integer converted from IP address format
LastModifieddate+time the IP was last modified, updated only by updatederiveddata
converted to readable form by updatedderiveddata, not true timestamp field

Non-Display Fields, used internally only
_LastModifiedtrue timestamp date and time, follows MySQL data format
updated by MySQL whenever data is written into the database
LongIP4 bytes, network long integer converted from text IP address format
The MySQL username and password can be found in the script ecescan.data.


 


SECTION FOUR. INSTALLING THE SYSTEM

The author regrets to mention that it would take some Unix know-how in order to install the system, even when this system is already designed to be easy to use. This is mainly due to the fact that the system makes use of the Perl DBI, the MySQL server, and the internet web interface. Therefore, the user should have the necessary knowledge on how to install each of these systems before he or she would be able to install the Ecescan system successfully.


4.1 System Requirements

This entire system is based in Linux. However, you can also use it with any platform that feedbacks Perl and also each of the following software.

You need to have the following software in already installed in order to use Ecescan. If not, you can download the software for free from the web URL indicated:
No Software Website
1. Perl version 5 or higher www.perl.com
2. Perl Database Interface (DBI) library www.perl.com
3. Nmap version 2.54 beta 3 or higher www.insecure.org
4. MySQL version 9.38 distribution 3.22.30 or higher www.mysql.com
5. Perl module Spreadsheet::WriteExcel www.cpan.org
6. Web Hosting software (optional) www.apache.org
7. PHP www.php.net



4.2 Installation Instructions

The first step of installation would be to extract the file ecescan.tar.gz into the directory in which you want the system to reside. If the directory is /usr/local/ecescan/, you can use the following commands:
cp ./ecescan.tar.gz /export/home1/supervision/ecescan/
cd /export/home1/supervision/ecescan/
gunzip ./ecescan.tar.gz
tar xf ecescan.tar
Now, you will have all the necessary files and directories copied into the directory.

From here, you need to enable the MySQL server in order to be able to interact with the database.


4.3 Enabling MySQL Server

This is done by executing the file mysqld provided with the MySQL package. You need to have the MySQL server running before the MySQL database can be accessed by any program.

Now you need to create the MySQL database for use by the system.


4.4 Creating the MySQL Database

This subsection will attempt to explain in detail the steps required to create the database and the table in the database that the system will use. The name of the database (ecescan), the name of the database table (iptable), the database username (root) and the database password can all be found in the file ecescan.data. If you have use different values for any of the above items, you need to modify the data in ecescan.data accordingly in order for the system to work correctly. This is especially if you had specified a different username and password for the authorized user of MySQL when it was installed.

In order to create the MySQL database 'ecescan', use the following command line:
/usr/local/packages/mysql/bin/mysql -u username -p
           # runs the MySQL monitor (command prompt shell)
Now, you will be prompted for the password, so enter the password
*******
           # password required for MySQL user
The username and password are determined when the MySQL is installed, and can be different. Now, you will enter the MySQL monitor program. Use the following MySQL command:
create database ecescan;
           # must have the trailing semi-colon
The system will then respond with "Query OK, ...". Now, exit the monitor:
exit
           # out of the MySQL monitor program
Ok, the database is then created. Now you need to create the database table 'iptable'. Go to the directory where Ecescan was installed, then enter:
./createtable override
           # create new database table 'iptable'
After creating the database and database table for the system, you would then need to specify the range of IP addresses to monitor.


4.5 Specifying the Range of IP Addresses

This is done by editing the file ecescan.in. Each line of the file is a range of IP addresses that you want the system to monitor. You can simply use the format 123.123.1-254.1-254 if you want to monitor all the IP addresses that begin with 123.123.

To exclude an IP address from 123.123.123.1-254, say 123.123.123.100, you need to split up the line into 2 separate lines:
123.123.123.1-99
123.123.123.101-254
To exclude an IP address from 123.123.1-254.1-254, say 123.123.123.100, you need to split up the line into 4 separate lines:
123.123.1-122.1-254
123.123.123.1-99
123.123.123.101-254
123.123.124-254.1-254
The final component of the installation is to set up the web interface for the system so that users can access the information collected by the system from the internet.


4.6 Installing the Web Interface

This is optional. If you choose not to install the web interface, you can still generate the same results without the web interface available.

In order to install the web interface, you need to know how to work with a web hosting software such as Apache. If not, please refer to an experienced Unix/Linux user for help. Basically, if you are using Apache, you need to insert the following lines in the file httpd.conf. For each of the "Allow from All" clauses, you might want to change it to require user authentication from the web:
# to specify the alias for http://localhost/ecescan/
Alias /ecescan /export/home1/supervision/ecescan/
Alias /ecescan/ /export/home1/supervision/ecescan/
<Directory "/export/home1/supervision/ecescan/">
         Allow from All
</Directory>
You can also set up user authentication to restrict web access to selected users. The elaborate instructions for doing so can be found on
www.apache.org.


 


SECTION FIVE. USING THE SYSTEM

This section covers the details on starting and stopping the system, and also report generating.


5.1 Creating the Executable Scripts

To create the executable shortcuts scripts for the program, run the script makeexe by entering:
cd /export/home1/supervision/ecescan/src makeexe
This will create shortcut scripts in the directory /export/home1/supervision/ecescan/, and after that you can run the shortcut scripts in the same way that you would run the original scripts. The only thing that the shortcut scripts do is to change the directory to the directory where the source files reside, /export/home1/supervision/ecescan/src, and then execute the program. This is necessary because if you try to execute a script using:
/export/home1/supervision/ecescan/src/scriptname
from another directory, then the program will not run correctly.


5.2 Test-Running the Program

To start the ECE scanning process, use:
ecescan -test -watch
This will do a test-run of the scanning without affecting any of the time-activity counters in the database. In order to run it periodically, you need to setup the crontab file.


5.3 Stopping the Program

If you want to stop the program when it is executing in the foreground in your terminal window, hit Ctrl+C.

If you want to stop the program when it is executing as a process in the background, use the command line: "ps -A | grep ecescan" to find out the PID of the ecescan process. Then do "kill -9 PID", where PID is the process ID that you found by running the ps command previously.

If you want to stop the program running periodically after you have set up the crontab file, use "crontab -e" to edit the crontab file and comment out all the Ecescan-related jobs by putting a pound symbol '#' at the beginning of the line.


5.4 Running Ecescan Periodically

You can let the cron utility run ecescan every 5 minutes or at any interval that you desire.

To set the cron job, enter these commands:
export VISUAL=pico
           # optional, makes pico the default text editor; well, if you want to experiment with user-unfriendly software, try vi
crontab -e
           # edit the existing crontab
You will then enter the screen for entering the cron job specification. Now enter this line exactly, but modifying the directory of ecescan accordingly. This entire line must go in a single line, and cannot span two lines:
*/5 * * * * /export/home1/supervision/ecescan/ecescan
It is necessary to use the cd command because the perl module file ecescan.pm is not installed in the filesystem, and hence ecescan.pm can only be accessed by programs that reside in the same directory as itself. Now exit the text editor. If you are using pico, you can exit by pressing Ctrl+X, then press 'y' when asked to save changes, and press the enter key to save the file and exit.

Crontab is now installed to run Ecescan every 5 minutes, 24 hours a day, 7 days a week. There are some other functionalities that you might want Ecescan to perform on a regular basis, such as report- generating and data backup. You can use the same procedure as above to insert each of the lines given as follows:
	# 24x7, once every 5 minutes
	*/5 * * * * /export/home1/supervision/ecescan/ecescan > /dev/null 2>&1

	# rescan ALL missing OS at 12:00 am everyday
	0 0 * * * /export/home1/supervision/ecescan/rescanos > /dev/null 2>&1

	# Generate the patch index page at 7:30 am everyday
	30 7 * * * /export/home1/www/ece-supervisor/patch_reports/generate_list > /export/home1/www/ece-supervisor/pa$

	# set proper permissions for all files at 12:00 am everyday
	0 0 * * * /export/home1/supervision/ecescan/chmodall > /dev/null 2>&1

	# backup database twice a month, 1st & 15th, 11:00 pm
	0 23 1,15 * * /export/home1/supervision/ecescan/dbbackup > /dev/null 2>&1

	# update all existing os once a month - this process takes a LONG time
	# 25th of every month, 12:00 am
	0 0 25 * * /export/home1/supervision/ecescan/updateos > /dev/null 2>&1

	# quick update all hostnames using gethostbyaddr(3) (Linux)
	# twice a day at 1:00am and 1:00pm ( qu = quick update )
	0 1,13 * * * /export/home1/supervision/ecescan/quhostnames > /dev/null 2>&1

	# refresh all DNS data completely: once a day at 1:30am
	30 1 * * * /export/home1/supervision/ecescan/update_dns.pl > /dev/null

	# update duphosts flag: once a day at 2:00am
	0 2 * * * /export/home1/supervision/ecescan/update_duphosts.pl > /dev/null
Now, you need to know how to work with Ecescan manually if necessary.


5.5 Running Ecescan Manually

The main script ecescan can be invoked manually if necessary. This can be done simply by going to the main directory and entering the command ./ecescan. However, running the program manually would modify the counter records in the database file, and write to the main log file. Therefore you can use some command line arguments to over-ride the default settings:
Command Line ArgumentWhat it Does
-testsuppress the counters update, force program to run even in the time periods that are not under monitoring
-w, -watchenter watch mode, print out descriptive progress status of the script
-i input_file_namespecifies input file containing all IP addresses, default is ecescan.in in the main directory
-l log_file_namespecifies log file to store program execution information, default is ecescan.log in the log files directory
-helpdisplays brief help information on using the command line arguments
Running the script ecescan manually is not recommended unless it is solely for testing purposes. If you run ecescan manually, it is advisable to use the -test switch at all times to avoid modifying the counters and thus affecting the accuracy of the gathered data.

The script ecescan takes about two minutes to complete a single run, scanning and updating a total of about 1450 unique IP addresses. Therefore, you can use "./ecescan &" to run ecescan in the background.



 


SECTION SIX. MAINTAINING THE SYSTEM

6.1 Basic Concept of Maintenance

There are four main areas which require maintenance by the user:
  1. Database
  2. Log file
  3. Generated reports
  4. Future program restarting
There is no fixed rule for maintaining the system in the long run. However, the user can follow the prescribed maintenance system to set up the cronjob as described in
Section Five. Using the System. Therefore, the only thing left for the user to do is to check the generated reports in future to oversee the level of activity of the entire subnet under monitoring.

One point to note about the database size is that the size of the database does not expand with time. The size of the database will depend solely on the number of IP addresses that are found in the database. Therefore, the user does not need to worry about setting aside a huge chunk of hard disk space to cater for this system. The size of the database for a record of about 1550 IP addresses is only about 2.7 MB.

However, the main log file ecescan.log will grow in size daily, and so the user might want to delete the files periodically or move them into archive storage in order to prevent it from taking up too much hard disk space. The author feels that the information found in the log files will probably never be used and it is more for purposes of keeping track of the scan history. Other than that, the log files would not be useful at all.


6.2 Recovering Files

In case of the MySQL database being corrupted, you can use the script dbrestore to restore a database found in the backup directory /export/home/backup_ecescan/. Use the following command line:
./dbrestore /export/home/backup_ecescan/backup_filename override
( e.g. filename - websec.2001-01-19.gz )
Please note that the restoration of the database is irrecoverable. Use with caution.


6.3 Files that Might be Modified

ecescan.cfg
modify this file accordingly for directory settings, or just leave it as it is

ecescan.data
make sure you know what you are doing if you are to modify the contents of this file, otherwise just leave it

ecescan.dns.data
links the ip addresses to the dns file, modify accordingly if there are any changes in the DNS files


6.4 When to Use the Utilities

Utility programs are found in the main system directory /export/home1/supervision/ecescan/. Each of these scripts were designed to be executed from the command shell. Note that each of these scripts has to be executed from the directory at which the script resides. These scripts can be used to make changes to the database directly and perform other functions. A list of perl utility scripts are:
Script What It Does
makeexe creates shortcut scripts in /export/home1/supervision/ecescan/
these shortcut scripts are necessary so that any script can be executed correctly from any directory other than from the directory where the script itself resides. all the shortcut scripts do is to go to the directory where the real scripts reside (in subdirectory "src") and then execute the real script from there
ipcount prints the number of IP addresses in the database table
oscount prints the records with a null OS name and the number of OS types in use
dumptable writes first five records into file out.txt, used for debugging purposes use argument 'all' to dump all records, or specify a number of records to dump
stdquery generates report with standard specifications in stdquery.in
textout dumps entire table in text format only accordingly to specifications in textout.in
this is the original script that precedes formattedoutput
customquery make report with input from customquery.in
formattedoutput main script to make report, this is usually executed through the files stdquery, textout and customquery; needless to run it directly
dbbackup backup the database using a filename specifying the date and time of backup
showip shows details of a single ip
rescanos takes all IP with a missing / null OS name and scan them for the OS name excluding "Unspecified OS", "OS unknown", and "OS info unretrievable"

Meaning of OS fields:
"Unspecified OS"retrieved OS info successful, but OS name is NULL
"OS info unretrievable"IP was active, but did not return OS name
"OS unknown"IP was inactive at time of update, so fill-in-the-blank
rescanhostname rescan the ip addrs with a missing / null hostname and fill in the hostname if possible, excluding "Hostname unknown"
updateos rescan the OS name for ALL ip using OS scan; this process takes a very long time to complete (estimated half-a-day's time)
quhostnames quick update ALL the hostnames using gethostbyaddr(3), there is no need to scan for the hostname using ping scan since gethostbyaddr(3) is very much faster
webquery generates reports based on webquery.in found in the WWW output directory
dns.php finds differences between data in DNS files and Ecescan database, outputs to STDOUT in HTML format
duphostreport finds duplicate hostnames in database, outputs to STDOUT in HTML format
checkwebauthuser prints 1 if given user is authorized to edit the database, prints 0 otherwise
excelreport generates a full MS Excel formatted report as out.xls in $ECESCAN_WWW_OUT_DIR found in ecescan.cfg, currently $ECESCAN_WWW_OUT_DIR = '/export/home1/www/ece-supervisor/ecescan/output';
updatederiveddata updates all derived data fields in the iptable, takes less than half a minute to complete
chmodall sets the correct file access permission and for all files. sets the ownership of ALL files to the root user.

Restricted scripts that causes permanent changes ( USE WITH CAUTION )
resetqutable reset the database table for hostnames quick update log table
resettable as the name implies, this will destroy the database table if it exists and re-create a blank one
resetcounters this script sets all counters in the database table to zero
dbrestore restores a backed-up database
removeip deletes a single IP address
The restricted scripts can only be executed by supplying the command line argument "override". If the script requires other command line arguments, then "override" would be the last argument required.


 


SECTION SEVEN. CONCLUSION

When designing the database that is required for this project, the author took a long time to decide whether it is feasible and useful to save the date and time of whether an IP address is up or not. Saving the date and time would give a much more detailed picture of exactly when a particular IP address was active. However, it was very carefully decided that storing the date and time would make the database very large and also slow down the report generating process tremendously. So that was why the concept of using time interval counters was conceived and finally chosen for implementation.

This system has been mainly designed to be easy to use and maintain. Once the system is installed correctly, it should be able to self-propel with minimum user-interference. This manual is not meant to be all-encompassing and there may be some details that are useful to be included herein. Any questions or comments can be directed to
chok@ece.gatech.edu.


# end of user documentation