Cover V10, I03


Web Hosting: A Migrational Case Study

Ripduman Sohan

Hosting, the act of providing a service on behalf of an individual or company, is a concept that has been around for as long as the Internet. There are many types of hosting services, including Web, mail, and database hosting. However, the most popular and longest-lived hosting service has been Web site hosting.

Many organizations, such as universities, commercial companies, and ISPs, provide this essential service for their users or customers. Today, the Web is the most popular medium for retrieving information from the Internet. To ensure your material in this information cornucopia is readily available, it's essential to configure your end to deal with anything your users may want, without inconveniencing them or annoying you.

In this article, I present a case study migration of a system containing 203 virtual hosts from one server to another, many of which had backend databases. The Web server used was Apache, the database, MySQL, all running on FreeBSD and being transferred to Linux. I intend to show you how simply this can be done and share some of the tricks and pitfalls generally involved with setting up, running, and successfully migrating medium- to large-scale Web sites with this software. I've also included virtual hosting because that's what the original job entailed, and also because I wanted to be as thorough as possible. Nevertheless, almost all of the concepts in this article should be adaptable to single Web sites and different software with little or no tweaking.

The Scenario

The source system was a box running FreeBSD 3 on a Pentium II 266 located in San Francisco. It was connected to the Internet via a 256-KB link and was using Apache 1.3.1. Of the 203 virtual hosts, 30 required databases, so it also had MySQL 2.3 installed. The machine setup was incompetent -- so incompetent that the actual MySQL database was available directly off the Web. It also had no backups. The target machine was a brand new, default installation, Redhat 6.3 machine on a T3 link in New York. I didn't have physical access to either side of the scenario and was working off a satellite link with a 700-ms lag.

The reason for the changeover was twofold. The company was increasingly aware of the insecurity and lack of power of the source machine in relation to their increasing customer base, and they were also getting a better deal with a new co-location provider. My job was to move the whole system, with zero downtime and no loss of client data.

The Move


The move started with the most important thing -- a system backup! I couldn't back up any of the user Web files or databases due to the high load on the system. As soon as I touched any of these, the system became highly erratic and with 2.3 GB of user data on the system and no local means of backup, I wasn't going to transfer all the data via the Internet link. Therefore, I initially backed up just the httpd.conf file and the password and group files. However, before you start a migration, I advise you to check that latest system backup is valid. You do have one, right?


The next move was to download and install analog. This is a very well-known and comprehensive Web log analyzer, available as a package for most platforms. You can get started quickly with the following steps:

1. Install analog. Use your package manager, usually rpm -i analog.rpm in Linux.

2. Edit the analog.cfg, usually in /etc/analog.cfg. Edit the sections LOGFILE to point to where your Web server logfile lives, and the section OUTFILE to point to your output filename.

3. Turn on the hourly report with the command FULLHOURLY ON in the analog.cfg file.

4. Run the binary, usually /usr/bin/analog.

This will create a full hourly breakdown report using your log, which you can view with any browser. I did this to build a time profile so I would know the best time for me to actually log into the machine in order to copy and move the data. Most Web servers go through a daily cycle of use, depending on the time zone of their audience, and it's best to work when load average is lowest to minimize disruption to the system. If you can do the migration with downtime or without affecting the service, go ahead and skip this step.

New System Build

After figuring my optimal timings, I built the new server. If you choose to run a dedicated machine as a server, a little forethought in design can go a long way to prevent problems. Your most important resources on a Web server are memory and disk space. Work on maximizing those. Make your Web data partition as large as possible and put it on a separate drive if necessary. Most default Linux server installations come with setups that are not really optimal to Web servers -- do you really need X Windows? What about gimp? Get rid of all unnecessary software. This usually frees up to 800 MB and makes software conflicts less likely. On most RedHat-compatible Linux distributions, you can use the following commands to work with installed packages:

rpm -qa -- Provides the full list of installed packages

rpm -qi packagename -- Information on a selected package

rpm -e packagename -- Removes the selected package

Next, trim memory usage. Disable (or replace with lighter equivalents) all services that don't have to be running: atd, bind, dhcp, and Sendmail (replaceable by ssmtp) are the usual candidates. You can usually remove the packages or just remove the startup scripts from your startup scripts directory. Also ensure that you have adequate swap space (usually as much as available ram). Swap space, at least in Linux, comes in two variants: partition or file. Use the partition type -- at least if your swap space gets corrupted your system won't.

I then upgraded the "key software". Key software is software on which system functionality depends. This is usually Apache and its related support software. It's worth using the latest available stable version of your key software that you deem fit for consumption. (My personal method of finding out the Apache version to use is by querying by using netcraft.) If you have any modules used by Apache (e.g., PHP), it's worth getting the latest versions. Also, this is the time to install any third-party software you want to use; Professional FTP Daemon (Proftpd) and OpenSSH are popular options in this respect. If you're using a system with a package manager and you don't need any non-standard options (i.e., those requiring a source compile), get the installable package. The reason for this is that any issues with that particular OS and software will have been worked out by the package maintainers, so it's worth using the package.

Performance Tuning

The crux of the matter is configuring your software so it performs well. Although I could write several articles on how to configure your system, I'll only give you the main ideas behind maximizing performance for Apache and, to some extent, the system.

Apache is the machine's interface to the world. Configure this poorly, and your beautiful new server with its oceans of ram and your T3 link won't be worth anything.

To configure it well, first, get rid of all unnecessary Apache modules and add any custom ones you do want. You can do this by editing the httpd.conf file and looking for lines similar to:

LoadModule info_module        modules/
This tells Apache to load the module info_module into memory when it starts. Go through each LoadModule line of the default installation and disable all the modules you'll never need. (This is done by putting a # sign at the front of the line.) Typical modules rarely used on production Web servers are mod_autoindex (creates automatic indexes for directories) and (proxy caching module). You can find the complete module description for each standard module in the Apache documentation. This is done to minimize the memory Apache uses because fewer loaded modules mean less memory allocated to module code. Sometimes, disabling modules can also lead to server speed increases.

If you're using Perl as a scripting language for the server, consider having mod_perl loaded. This will eliminate having to run an instance of Perl every time you start up a script, which means better response time for the server.

Next, I always modify the lines StartServers, MinSpareServers, and MaxSpareServers. These lines go together. To understand this, remember that creating a process on most operating systems is quite expensive in terms of time, and in Apache, each process is known as a server. Hence, you'll want to start a reasonable number of processes when you start up the program (StartServers line), while simultaneously having a reasonable number of processes free to serve any other incoming requests before it creates any additional new ones (MinSpareServers line). Conversely, you don't want to waste memory with spare processes lying about after they've finished doing their work (MaxSpareServers line). I find the values 8, 4, and 10 work well for most setups.

The next lines to modify are the lines MaxClients and MaxRequestsPerChild. MaxClients relates to the maximum number of clients that can connect to the server simultaneously. A larger number means more concurrent connections, but worse performance; a smaller number means the opposite. A good compromise is a value of 200. The line MaxRequestsPerChild relates to the number of requests each process can handle before it is forced to die. This prevents errant processes (e.g., one that leaks memory) from hogging system resources. If you're confident everything works well, you can set this value to zero to provide that little extra boost in performance.

As a trick, you can use the above parameters to provide a limited service while you perform maintenance or migration work. When I migrated my data, I restarted Apache on the source machine with a single server and a MaxClients of 50. This allowed users to still get (some) service while I had a more usable machine.

You can also turn of hostname lookups (HostNameLookups line). This prevents Apache from looking up and logging the DNS record of the connecting client, as opposed to just the IP. Finally, you should avoid providing server-side scripting (.shtml files) because this forces Apache to parse each page it sends and makes them uncacheable.

Regarding your system, there are two things that are consequential to performance. The first is the maximum number of open files you can have at any one time. In Linux, you can modify this parameter by modifying the file /proc/sys/fs/file-max. The command echo 16384>/proc/sys/fs/file-max will increase maximum open files to 16384. For administration ease, you can put this command in one of your startup scripts.

The second thing you can do is to rebuild the kernel, cutting out all the unnecessary drivers. This frees memory and makes the kernel leaner and, therefore, faster. However, if you're working off a remote link, be certain the kernel works before deployment. Having an exact replica locally, in terms of software and hardware, may help here.

Data Migration

Data migration is relatively painless if carried out properly. The first thing to do is to make sure all your user accounts have been duplicated with the correct logins and passwords. If you're migrating between homogeneous machines, just copy the relevant password and shadow files across. Otherwise, you may have to manually migrate accounts, but this is dependent on source and destination systems. Most shadow systems now have interchangeable password files, but check first. I was lucky that all the information my company had for virtual hosting was assigned by them (including password), so I made a text file with the user information in it, and use the Linux newusers command to create all the new users.

Remember when migrating user accounts to make sure that all the accounts on the new server have the same group identifier (GID) and user identifier (UID) as on the old system. This prevents permission and ownership problems when you copy the data across. It may also be beneficial to create separate groups for different account classes. For example, I have separate groups for the Web sites that connect to databases, and for those that don't.

After the users are set up on the system, you need to create or migrate the Apache virtual hosts for each virtual host. An easy way of keeping your virtual host configuration separate is to put it in a separate file (e.g., the directive Include conf/vhosts.conf in your httpd.conf would allow additional configuration directives (the virtual hosts) in the vhosts.conf). This makes a migration easy -- just modify the file for your new configuration and include it in your new setup.

You must ensure that all virtual hosts have their own transfer and error log files. This is handy for the customers, because it allows them to maintain and analyze their own log information. It's handy for you, because it frees you of the same task.

After this, it's smooth sailing. All that's left is archiving the data off the old server and restoring it on the new one. There are several ways to do this. My favorite method requires both machines to have OpenSSH installed. Then use the following command, carried out in the data directory of the source server:

tar -cf - * |(ssh -l username tar -xvpf -)
This archives all the data on the host server and unarchives it at the destination, all in one command. Nevertheless, if you feel so inclined, go through the tar, copy, untar cycle. I recommend you don't change the Web data directories when moving the files, in case of any hard coded paths.

Databases should also be migrated at this point. With MySQL, the process is easy. The basic steps are dumping the data to a text file, copying the file to the new server, creating the database on the new server, and importing the data into the database. A quick, typical example is:

oldserver$ mysqldump dbname >outfile  \
  (dump the database dbname to file outfile)
oldserver$ scp outfile newserver: 
  (copy the outfile to the new server using secure copy)

newserver$ mysqladmin create dbname  \
  (create the database dbname on the new server)
newserver$ mysql dbname <outfile   \
  (import data from file outfile)

After migration, you're almost there -- but do things work? The last thing you want is to change your DNS entries to point to the new server, or create new DNS entries only to realize that things don't work. However, you can't check to see whether things work unless you move the DNS entries!

Fortunately, there are a number of solutions to this problem. The most comprehensive one I use is the following:

1. Create a DNS server on an extra machine with fake records that indicate the new server is the Web server for all the virtual hosts you are hosting on it.

2. Find a set of machines on the same network as the fake DNS server that will be used for testing. Point their primary DNS server to this server.

3. Surf the virtual host sites to see whether they work.

This works because the fake DNS server is programmed to assume that it is the master domain holder for the virtual hosts and, hence, passes the wrong information to the client machines. The client machines then contact the new server and request data from it. The advantage of this is that a number of machines can simultaneously be used to do testing.

A less elaborate but easier option is to have one machine on the network configured to have no DNS server, but have a modified hosts file (/etc/hosts for Linux) to point to the virtual hosts. If your server machine needs to do some sort of host lookup, this option is very useful, because it allows you to check that the whole system works before migrating records.

Once you're satisfied that everything works, change the DNS settings to reflect the new server. The job is now done. Keep the old server active for some time after retiring it, because the DNS propagation takes time (usually a few days, but sometimes up to a month). You can then gradually recycle or destroy it.


You can never be too safe on a production server. A few things to check are:

  • Don't give users access to the system if they don't require it. Make their shells /bin/false.
  • If you allow CGI scripts, vet them to ensure they aren't malicious.
  • If you use a custom Web-based administration tool, make sure it's secure.
  • If the main access to your system is via ftp, consider using ProFTPD. This is an excellent ftp server with lots of security features, including the ability to lock users into their home directory, apply quotas, allow logins without a valid shell, and control the maximum number of concurrent user logins.
  • As much as possible, use OpenSSH to access your system. Disable telnet and any other services you don't need.
  • Periodically check and update your software for any discovered security vulnerabilities.
  • Use TCP Wrappers to control access to services.
  • Periodically monitor your log files for any suspicious activity.

As you know, Web servers are integral but sensitive parts of today's Internet. I hope this article has given you an insight to how important it is to plan ahead when preparing or migrating a Web server. I also tried to show how simple setup or migration can be, with little or no downtime, if the planning is done well.


Analog Weblog analyzer --

SSMTP (Send only Sendmail emulator) --

Netcraft International --

ProFTPD (Professional FTP Daemon) --

OpenSSH --

Ripduman Sohan is currently finishing off a degree in Software Engineering at City University, London. He's originally, and still, based in Kenya and has been using and promoting *nix based systems since he was 14 years old.