Cover V10, I04


What to Do When the Server Doesn't Serve -- Using CODA

Brett Lymn

In the previous articles in this series, I looked at NFS failover methods using fairly mainstream free software and the operating system facilities themselves. In some environments, this is probably a good fit for providing some measure of redundancy in your file servers. This time, I shall look at another approach to providing access to files by using a file system called CODA that is specifically designed to handle server outages.


CODA is an intriguing file system being developed by a group headed by M. Satyanarayanan at Carnegie Mellon University (More information on CODA can be found at This file system has been designed to be able to operate when the client has been disconnected from the server without disrupting normal operations. The CODA file system is an experimental file system. The decision to use it in a production environment must be weighed very carefully, because glitches in the software may result in CODA eating your file system and causing loss of data. Having said this, the actual situation is somewhat better; CODA does work. It is not very forgiving in some circumstances, but if CODA is treated reasonably in terms of system resources, it works well. A major advantage of using CODA as a failover file system is that it is, by design, able to gracefully cope with network outages. This makes it ideal for providing failover file serving.

On a CODA client, the files currently being worked on are cached on the local client's disk, and updates to the files are passed between the client and the server to ensure the cached copies of the files are kept in sync with the server. If the server goes down, then the CODA client simply keeps a log of the changes made to the cached files. When the client loses contact with the server, then the client is said to be running in disconnected mode. When the server comes back up, the changes made on the client are reintegrated into the server's copy of the file to bring it up to date. Later, I will look at what CODA does when two clients modify the same file when they are disconnected.

Installation and Use

The instructions for installing CODA can be found on the CODA Web site. The installation of CODA is a bit involved and requires some manual setup, but by carefully following the instructions, you should be able to get a server and client running. Note that you can have a CODA server and client running on the same machine, so you can view the CODA file system on the server itself.

Using CODA is quite simple; the client simply runs the client software, which mounts the CODA volume from the server. Once this is done, the CODA file system looks like any other file system, and the files therein can be treated the same as any other file on the system. Initially, when you mount the CODA volume, the volume is in read-only mode. To modify files on the volume, you must authenticate your session to the CODA server by using the clog command. The clog command contacts the server and acquires an authentication token, which is used by the client during transactions with the CODA server. The token has an expiry time of about 25 hours, after which the client must again authenticate to the server. The status of the authentication token can be viewed by using the ctokens command.


CODA comes with some tools that allow you to monitor the status of the CODA file system. The developers strongly recommend this to ensure the CODA file system is running properly. One of these tools is called codacon. Running this tool produces a running log of the actions performed by the CODA client. Some of the output is cryptic and probably only makes sense to a CODA developer, but it does provide a good monitor of the status of the CODA file system -- alerting you to server disconnections, conflicts, and other important information. Another tool that comes with CODA is the cmon tool, which provides a periodically updating display of the status of the CODA servers including statistics on resource utilization.

The CODA file system can be managed by using the cfs tool. This tool will allow you to change a lot of parameters with the CODA file system. You probably will not use most of the commands available with cfs. One of the things you can do with cfs is force the CODA client to check whether the CODA server is up or not by using the checkserver cfs command like this:

cfs cs
This command is particularly handy when you know the CODA server has just come back up and you want to make the CODA client aware of it. Normally, the CODA client will poll the server periodically when the client is in disconnected mode, and will realize that the server is up when the poll is done; but the checkserver command can be used if you are impatient.

When a CODA client loses contact with the server, the client enters what is called disconnected mode. In this mode, the local cache will be accessible on the client, and modifications will be logged locally. When the server is once again available, the local change log will be integrated back to the server. If CODA detects that another client has modified the file, this is called a conflict, and I will show later how it is handled. An attempt to access a file not in the local cache will result in a file error. This is unfortunate, but there is little that can be done if the server is down and the client does not have a copy of the file to work with. Clearly, it is undesirable to find that the files you need are not in the local CODA cache when the server goes down. To address this, the CODA tool set includes a tool called hoard that allows you to build a database of hints to CODA as to which files you wish to be cached locally. You can assign priorities to a file or set of files that will affect which files are discarded from cache when the cache is full.

In order to use CODA as a failover file system, we want all the shared file system to be available on the client and we would create a hoard command file (say, hoard.conf) with the following contents:

a /coda/file/tree    600:d+
This tells hoard to add the file tree /coda/file/tree with a priority of 600 (the higher the number, the higher the priority). It also says that you want to hoard all subdirectories, including any new subdirectories created after the hoard command is run, under the given root directory. Once the hoard.conf file is set up, you can feed it into hoard like this:

hoard -f hoard.conf
This loads the hoard database with the commands in hoard.conf. The database need only be set once as it persists across restarts of the CODA client. Once the hoard database is set up, you can run:

hoard walk
which starts filling the local CODA client cache from the server with files, as per the specifications in the hoard database. When the hoard command has completed, the cache on the CODA client contains all the files requested on disk so they will be accessible even if the server gets disconnected.

As mentioned before, if the CODA server is down or disconnected for some reason, an awkward situation can occur if multiple clients update the same file. In CODA parlance, the file is said to be in conflict, and the CODA client stops normal operations until the conflict is resolved by a human. When the client is in conflict, the name of the volume that is in conflict is logged to the CODA logs. Using this, an administrator can run the repair tool that presents a list of changes pending on the client. Also in repair mode, the versions of the files on the server are viewable as separate entities, so the server versions of files can be checked against the local copies. These pending changes can either be integrated with the server (if there is no conflict), or if the file conflicts, the administrator can elect to discard the local changes or overwrite the server copy. Once all the conflicts have been resolved, the client can mount the CODA volume again and resume normal operation.

In addition to the client-server model, CODA also supports replication between servers, so you can have multiple CODA servers with the data automatically replicated between the servers. Thus, you can either provide redundancy of the CODA server, or you can have multiple servers separated by a relatively low bandwidth link. Then the CODA clients bind to the local server but still have up-to-date files. Again, if the servers lose contact because of a network outage, the file modifications are logged and reintegrated when the servers regain connectivity. The process for reintegrating server-to-server conflicts is a bit different from client-to-server reintegration. The same repair tool is used, and the process is similar in that the changes are reviewed and one version is accepted as the correct one. But, the process is a bit more complex than a client-server conflict.

By now it should be fairly apparent that the failover when using CODA happens automatically because it is part of the design of the file system. By setting up some CODA servers and clients, and by using the hoard command to fill the local client caches, the clients will be able to continue functioning if they lose contact with the server. By using the server-to-server replication, the chances of clients losing contact with a server can be minimized, as well.

As good as this may sound, CODA does, unfortunately, have some disadvantages. For a start, it is still an experimental file system, which means that most places would not even dream of placing CODA into a production system. Also, CODA is not a solution supported by any vendor, so the support for CODA is totally reliant on answers from a mailing list. These factors mean that CODA will probably not be a serious contender for a while, however, it does show considerable promise. I currently use CODA at home, and although it was a bit challenging to set up initially, I have found it very usable for keeping my home server synchronized with the work I do on my laptop.


This article completes my series on failover, but with a subject this broad, there are bound to be approaches that have not been mentioned. I hope that these articles have given you good food for thought. If you need write access to your files when the server is down, then CODA is the clear winner here. None of the other approaches discussed will allow write access by the clients. If you are limited to implementing vendor-supported solutions, then the cachefs approach discussed in the first article gives a close approximation to what CODA can do, but only with read-only clients and manual cache filling. If you have a large file tree that is fairly static and needs to be accessed in an ad hoc manner, then implementing redundant servers is probably the best way to go. Here Sun's automounter has some fairly clear advantages over AMD in that the failover of the file servers on the client is more elegant. Presumably this is because Sun put some smarts into the kernel allowing the automounter to recover from a server going down part way through a read of a file. Because AMD is purely done at the user level, these smarts are not available; thus, the failover is not as smooth.

Brett Lymn works for a global company helping to manage a bunch of UNIX machines. He spends entirely too much time fiddling with computers but cannot help himself.