Using Depot to Create a Centralized, Flexible, and Manageable Software Repository

Bob Jobsky

Do you wish you had a centralized software repository on one server for all flavors of UNIX? And that you could associate each file with the utility it belongs with? And have multiple versions of the same software on the same system? And thatyour users could install their own software versions without using disk space? For free? These features and more are possible using a utility called depot (http://ftp.andrew.cmu.edu/pub/depot/), which has not changed since version 5.13 in 1995.

Depot alone is not enough, however. In this article, I will outline how to combine depot with the use of NFS, wrappers, and a chrooted environment to easily implement a very flexible and manageable UNIX software repository. Within a day, it's possible to complete the setup and start populating the repository with your favorite packages.

Overview

Depot is central to the success of the software repository. It is used to create a virtual directory tree of symbolic links that point to the actual locations of the software packages that make up your repository. The real repository can be centralized on one file server or distributed across several servers. I describe the one server approach here. Each version of each package for each OS gets its own directory on the server. Each client then has a depot.pref configuration file that instructs depot how to create the symbolic links in its virtual directory tree. Depot superimposes the links for all the software packages into one directory tree. The result is a single entry in the PATH variable, yet each package is in its own "space" on one server making package management much easier.

Figure 1 shows the scalable directory structure of a depot server and the relationship of the virtual repository on a depot client to the depot server. All directories and symbolic links within the virtual repository are managed by the depot utility through entries made in the depot.pref file on the client. In the sample depot.pref file of Figure 1, the third column specifies the top-level directory trees that are mapped into the top level of the virtual repository. To have version 5.8 of vim available for use on Solaris 8 hosts, you first populate the repository on the depot server using one of the methods described in the section "Four Methods to Add a Utility to the Repository" below. If you used method 4, the resulting vim install tree would then get moved to /export/disk1/software/vim/5.8/sol8. Then, on each Solaris 8 depot client, add the entry path vim /depot/depot_server/depot/vim/5.8/sol8 to the /abc/depot/depot/depot.pref file and run the depot utility. All the symlinks will be automatically created and vim will then be available in the /abc/depot/bin directory.

I am currently working for a startup that is almost exclusively Solaris except for a few Linux hosts. Therefore, the details described below are mostly for the Solaris operating system. There will be minor differences with other flavors of UNIX. Since my company does have a few Linux hosts, and plans to add more, I have verified this technique on Linux, though I have not yet rolled it out to the development environment. As long as depot can be compiled, and a chroot environment created, this technique should port to any flavor of UNIX.

Disk Space

The required file server disk space depends on many factors. You should plan on reserving at least 2 GB per OS version. Keep in mind that you want your file server to hold the original downloaded packages and tarballs, the installed directory trees, as well as a chroot directory tree for each version of the OS. My chroot directory trees alone are about 500 MB each for Solaris 2.5, 2.6, 7, and 8. My Linux chroot directory tree is 830 MB, but could be significantly reduced if disk space was more of a concern than the time required to optimize the chroot environment.

Directory Structure

On the repository file server, create a software directory with the top-level directory structure shown in Figure 1. For this article, I assume the software directory is a subdirectory of /export/disk1.

Export this directory to your network. Here is a sample dfstab entry for a Solaris host:

share -F nfs -o \
rw=@10.1.1.0,root=bld_hst_os1.domain.com:bld_hst_os2.domain.com /export/disk1/software

On each client, create the directories /abc/depot/depot and /depot/depot_server where abc is a short abbreviation for your company or organization. Here, /abc/depot will be the top of the virtual directory tree of symlinks created by depot, and /depot/depot_server will be the automount point to the real repository on the file server.

After reading this article, you may be tempted to use /usr/local for the top of the virtual directory tree. I urge you not to use /usr/local or any other path that is a default for precompiled packages. If you do, then you will not be able to install any precompiled Solaris packages independently of the depot.

Now, make the automount entries. Add the following line to /etc/auto_master:

/depot        auto_depot

depot_server        depot_server:/export/disk1/software/depot

Also, on each build host, create a directory /software. Note that it is possible for the file server to also be a client and a build host.

Creating a chroot Environment

Unfortunately, there are some packages that need to be compiled for and installed into their final resting places. They will not work if they are later relocated. Other packages have environment variables that are supposed to allow relocation but occasionally fail to work properly if moved. I had such problems with emacs. For these types of packages, you need to supply the prefix=/abc/depot option to the configure utility. However, since your build host is most likely a client as well, its /abc/depot directory is where all of the depot's symlinks are. Executing make install will superimpose the package's real files into depot's virtual tree.

Unless you can afford the luxury of a dedicated build host for each OS version so the build host does not need the /abc/depot virtual directory tree, you will have to create a chroot environment so you can have an empty /abc/depot directory. I recommend placing these chroot environments on the file server since it probably has plenty of available disk space and any host can become a build host simply by modifying the root access list of the exported share on the depot server.

Unlike a minimal chroot environment for Internet services described in "Jailed Internet Services" by Liam Widdowson (Sys Admin magazine, August 2001), this chroot environment needs to be very robust to compile the variety of apps that are available. Simple scripts are provided below, which create Solaris and Linux chroot environments that are practically a copy of the real OS. You will first need to mount the /export/disk1/software directory on the filer to /software on the build host. The build host will need root privileges on the mount point. The script runs on the build host:

#!/bin/sh
# Make a Solaris 8 chroot environment.
OSVERSION=sol8 ; export OSVERSION
CHROOTHOME=/software/chroot/$OSVERSION ; export CHROOTHOME
DOWNLOADHOME=/software/downloads ; export DOWNLOADHOME
mkdir -p $CHROOTHOME
cd $CHROOTHOME
mkdir -p usr/dt usr/local var/tmp tmp builds abc/depot
ln -s ./usr/bin bin
ln -s ./usr/lib lib
cd /
tar cf - sbin etc platform kernel devices | ( cd $CHROOTHOME ; tar xf - )
ufsdump 0ubf 126 - /dev | ( cd $CHROOTHOME ; ufsrestore rbf 126 - )
rm $CHROOTHOME/restoresymtable
cd /usr
tar cf - bin lib ccs openwin ucb sbin share include | ( cd $CHROOTHOME/usr ; tar xf - )
cd /usr/dt
tar cf - lib include share | ( cd $CHROOTHOME/usr/dt ; tar xf - )

pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/make-3.79.1-$OSVERSION-sparc-local
pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/gcc-2.95.3-$OSVERSION-sparc-local
pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/perl-5.6.1-$OSVERSION-sparc-local
pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/bash-2.05-$OSVERSION-sparc-local
pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/gzip-1.3-$OSVERSION-sparc-local
pkgadd -n -R $CHROOTHOME -d $DOWNLOADHOME/zlib-1.1.3-$OSVERSION-sparc-local

#!/bin/bash
# Make a Linux 2.4 kernel chroot environment
OSVERSION=lnx24 ; export OSVERSION
CHROOTHOME=/software/chroot/$OSVERSION ; export CHROOTHOME
mkdir -p $CHROOTHOME
cd $CHROOTHOME
mkdir -p usr/X11R6 usr/bin usr/share usr/lib usr/libexec usr/sbin usr/include \
  sbin bin lib var/tmp tmp builds abc/depot
cd /
tar cf - sbin bin lib etc var | ( cd $CHROOTHOME ; tar xf - )
dump 0bf 64 - /dev | ( cd $CHROOTHOME ; restore rbf 64 - )
rm $CHROOTHOME/restoresymtable
cd /usr
tar cf - X11R6 bin lib libexec sbin share include | ( cd $CHROOTHOME/usr ; tar xf - )

If your OS does not have prebuilt packages available for GNU tools, you will need to find some other way to get them into your chroot environment. Copying /usr/local or whatever directory tree your tools are currently in is probably the easiest method. Unfortunately, this directory tree may be huge and contain gigabytes of unneeded packages. This is the exact problem depot will solve for you, once it is set up. For now, you may have to build them from scratch into the chroot environment.

Adding the Depot Utility to the Software Depot

The first utility that is added to the software depot is depot itself. Depot is not available as a package for Solaris, so it needs to be compiled. It can be compiled in the chroot environment using method 3 as described below. The GNU make utility and a compiler are required to install depot. Also, gzip is required when running the depot command to update the virtual repository. Because these three utilities are able to be relocated, it is possible to install each into the real repository using one of the four methods below. Each utility's bin directory then needs to temporarily be added to the PATH variable. After depot is installed and the virtual directory updated, entries for the three utilities can be added to the depot.pref file and another update performed. After this update, links for gzip, depot, make, and gcc should exist in /abc/depot/bin so the temporary PATH entries can be replaced with /abc/depot/bin.

Four Methods to Add a Utility to the Repository

There are four basic methods to add a utility to the repository. Before discussing the details, you must make a decision in each of the four methods: Should the utility be added once for an older OS version and then, if possible, used by later OS versions? Or should the utility be added for each version of the OS? Since disk space is not a concern for me, but repository consistency and organization are, I always choose to add a separate instance of the utility even if, for example, the Solaris 2.5 binaries will work on Solaris 8. The only exception is if there is a precompiled version for Solaris 2.5 and not any later version.

Method 1

The easiest and quickest method to install a utility is by using the package-management utilities of the OS on precompiled binary packages. The requirements of this method are:

• The package management utility needs to have relocation options so that the directory where the package will be installed can be specified. Solaris's pkgadd utility achieves this with the -R option. Note that Red Hat Linux's rpm utility has a similar option but I found that every rpm I tried to install in an alternate directory reported that the package could not be relocated.
• Package management utilities have a directory or database in which package information is stored. This must be able to be relocated to allow for installation of multiple versions of the same package for multiple OS versions. Solaris's packadd utility meets this requirement by placing the package information in the var directory under the root directory specified by the -R option.
• The utility itself must be able to be relocated. Many prepackaged utilities are available at http://www.sunfreeware.com. By default, they are compiled and packaged to run under /usr/local. Even though pkgadd can relocate the utility, it may not run from /abc/depot. If this is the case, the utility must be compiled in the chroot environment as described in method 4 below.

To add the precompiled Solaris 2.5 tk package to the repository, execute the following on a Solaris build host. Note that the build host for this method can be any version of Solaris since the downloads directory is located on the file server and is common to all build hosts:

mkdir -p /software/depot/tk/8.3.3/sol25
pkgadd -R /software/depot/tk/8.3.3/sol25 -d /software/downloads/tk-8.3.3-sol25-sparc-local

path    tk              /depot/depot_server/tk/8.3.3/sol25/usr/local

/abc/depot/bin/depot -u -T /abc/depot   # unlock the repository
/abc/depot/bin/depot -T /abc/depot      # Do the update

Method 2

The next easiest method to install a utility is simply an extension to method 1. Basically, some precompiled packages can be relocated only if environment variables are set that override the default location for which the package was previously compiled. In this case, a wrapper script must be created for the executable.

As in method 1, install the package into the repository. Here I use the less utility as an example for Solaris 7. The precompiled less package expects the terminfo directory to be located in /usr/local/share/terminfo. However, if the ncurses package is installed using method 1, its virtual location will be in /abc/depot/share/terminfo. Here are the steps:

mkdir -p /software/depot/less/358/sol7
pkgadd -R /software/depot/less/358/sol7 -d /software/downloads/less-358-sol7-sparc-local

cd /software/depot/less/358/sol7/usr/local/bin
mv less less.pkg
ln -s /abc/depot/wrappers/less.wrapper less

cd /software/depot/wrappers/wrappers
vi less.wrapper
chmod 755 wrapper
cat less.wrapper

#!/bin/sh
TERMINFO=/abc/depot/share/terminfo
export TERMINFO
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/abc/depot/lib
export LD_LIBRARY_PATH
exec /abc/depot/bin/less.pkg "$@"

path    less            /depot/depot_server/less/358/sol7/usr/local

path    wrappers     /depot/depot_server/wrappers

Method 3

The third method to install a utility is used when the utility can be relocated but is not available in a precompiled package format. I seldom use this method because installation is nearly as much work as for method 4; and method 4 will always work whereas there is a chance this method will not. If it turns out the utility cannot be relocated, you will need to repeat the installation using method 4.

The one utility for which I do use this method is depot itself. Because the depot binary can be relocated, it can be installed directly into the real repository, even though it will be run from the virtual directory /abc/depot/bin.

This method must be performed on a build host with the same OS version as the destination repository. In this example, the build host must be Solaris 2.6:

mkdir /software/depot/depot/5.13/sol26
mkdir /builds ; cd /builds
tar xf /software/downloads/depot-5.13.tar
cd depot-5.13
export CC=gcc
./configure --prefix=/software/depot/depot/5.13/sol26 --exec-prefix=/software/depot/depot/5.13/sol26
make
make install

Add the following line to the /abc/depot/depot/depot.pref file on any Solaris 2.6 client that will be using the depot repository:

path    depot           /depot/depot_server/depot/5.13/sol26

/software/depot/depot/5.13/sol26/bin/depot -T /abc/depot

Method 4

The fourth possible method to install a utility is the most universal and also the most time consuming. However, if there is no precompiled binary package, or the utility can't be relocated, or you simply want to be in control of the parameters you compile with, this is the method to use. For some flavors of UNIX, this may be the only method that can be used for all utilities. Keep in mind, though, that after you compile and install once, all clients will have access to the utility just by making the appropriate entry to the depot.pref file. A little work up front will have a big payoff later.

The basic concept for this method is to compile and install the utility in a chroot environment where /abc/depot is an empty directory. Then the install tree is relocated to the real repository outside the chroot environment. When the depot is updated, symlinks will be created under /abc/depot to the real repository. This allows the utility to believe it is running from the location for which it was compiled.

Because the packaged version of cvs on http://www.sunfreeware.com is two-years old, I decided to install the latest version using this method:

cp /software/downloads/cvs-1.11.1p1.tar.gz  /software/chroot/sol8/builds

chroot /software/chroot/sol8 /bin/bash
cd /builds
gunzip cvs-1.11.1p1.tar.gz
tar xf cvs-1.11.1p1.tar
cd cvs-1.11.1p1
./configure --prefix=/abc/depot --without-gssapi
make
make install

exit
mkdir -p /software/depot/cvs/1.11.1p1/sol8
mv /software/chroot/sol8/abc/depot/* /software/depot/cvs/1.11.1p1/sol8

path    cvs             /depot/depot_server/cvs/1.11.1p1/sol8

Documentation

Although brief, this section is as important as all the other sections combined. For every hour you save in administration time by using the software repository, you will save another hour by taking a few minutes to document each and every utility you install. Almost every utility has its installation quirks, whether or not depot is used as described in this article. Even if an install goes cleanly, it is worthwhile to document that fact.

Even though many people like to document by installing within a script session, I like to work in three to four open windows simultaneously. I find it easiest to cut and paste my commands into an editor as I proceed. When I finish installing a utility for one version of Solaris, I simply cut and paste out of my install notes to install the other Solaris versions of the utility, making slight modifications where necessary. Whatever your documentation style is, please create a file "pkgname.version.install" in the /software/notes directory and place your commands in it.

Comments

A common problem when updating your repository's symlinks with the depot command involves errors about unresolved conflicts. These are due to identically named files in two repositories trying to exist in the same virtual directory space. For example, many make installs will place a "dir" file in the info directory. Rename all but the emacs "dir" file to dir.utility, then copy the pertinent lines from "dir.utility" to the emacs "dir". Similarly, some installs put their documentation directly into the doc directory. Create a subdirectory of doc named "utility", then move all the documentation files to it.

Some utilities are replacements for standard OS utilities, but usually more efficient and powerful. If they are placed in the /abc/depot repository and /abc/depot/bin is first in your PATH, some problems may occur by using the replacement version instead of the standard OS version. In this case, create two virtual repositories -- one contains symlinks to utilities needed at the front of the PATH, while the other virtual repository contains symlinks needed at the end of the PATH variable.

Taking the last comment a step further, note that many virtual repositories can be created on the same client using almost no disk space while having only a single real repository on the file server. Each virtual repository might contain different combinations or versions of utilities for use by different user groups on the same client. Also, users can use the depot command to create virtual repositories within their home directories, again using very little space. As the administrator, you can populate the real repository with as many different versions of a utility as you want, then let the users modify their depot.pref file(s) to suit their needs. This only works, however, for utilities that are location-independent or able to be relocated through the use of environment variables.

Documenting the contents of a depot.pref file for a particular build can help with configuration management. It is very easy to return a client to a previously known state simply by restoring the depot.pref file to its previous state, which assumes you leave old versions of utilities in the real repository.

Because of security and availability concerns, it may not be desirable or possible to have NFS mounts within a production environment. However, because of the modular nature of the development repository and the depot utility, a very specific subset of the real depot repository can simply be copied to each production server, then the depot.pref file can be configured to point to the local repository instead of the remote one. To ensure the utilities work properly, keep the virtual repository in the same location as in the development environment, namely /abc/depot.

On a similar note, the entire repository can be replicated to various sites. Each site can pick which versions of which utilities to use simply by maintaining the depot.pref files on the clients.

To centralize the management of the depot.pref file, you can write a script that concatenates or merges network, group, and host-specific pieces of depot.pref together and then pushes them out to the depot clients.

For utilities that have config files within the repository, replace the config file in the real repository with a symlink to a local location, then add the real config file to that location on each client.

In the rare event that a utility needs a regular file and not a symlink to exist in the virtual directory, make an entry in the depot.pref file such as:

specialfile            etc/sudoers

To switch between versions of a utility on a host, simply change the entry in the depot.pref and rebuild the virtual repository with the depot command.

I have set up an email account at depotaid@hotmail.com. Depending on the mail volume and time commitment, I will try to respond to questions concerning this architecture. There are many details I simply cannot cover here.

Conclusion

Combining the depot utility with a chroot build environment for each OS allows an administrator to create a single software repository that is extremely flexible, scalable, portable, and manageable. This power is achieved because each utility is self-contained within its own directory tree. Basically, any version of a utility only needs to be built once on the file server to have it available for every host of that OS type within an organization.

Bob Jobsky has been working as a UNIX systems administrator since 1993. He holds Chemical and Electrical Engineering degrees and currently works at SupplyEdge Inc. in Pasadena, CA.