Balancing User Flexibility and Administrative Support

Chris Byers

In most corporate and institutional environments today, there is a great deal of pressure to leverage all assets to achieve peak business efficiency. As UNIX systems have become the de facto IT server backbone in most high volume environments, these pressures often can lead to undesirable results when certain users feel that they need more control over shared resources than others, often at the expense of other resources and users. As new users and departments vie for use and control of these resources, UNIX administrators are being pulled into a tug of war between competing interests. To keep their sanity, systems administrators must find a good balance between providing flexibility and managing multiple systems with multiple users. By following some basic guidelines, administrators can often improve both the working environment for users and minimize headaches. Presented here are guidelines that have worked for me.

There are generally six key issues UNIX administrators need to consider when striking a balance between user desires and administrative contol: NFS access, remote shell access, port access, SQL access, bandwidth, and disk capacity. Let's look at each of these areas and see how they affect the balance.

NFS Settings

Shared filesystems are often overlooked as a key architecture consideration when looking at client access. Yet this can often be the worst culprit in performance and security issues. Disk and network contention can play havoc with the system, and too much access leaves the potential for removing key shared files, etc.

Common UNIX tools such as nfsstat and iostat can be used to monitor NFS client loading over time, and it is generally a good idea to set up some sar reporting mechanisms to track utilization peaks and trends. Clear the registers first with nfsstat -csz to start from scratch (see Listing 1). Note that RPC calls are also tabulated. We will look at RPC traffic later in the article.

In /etc/exports, each line contains a mount point and a list of machine or netgroup names allowed to mount the file system at that point, along with access settings. Client access names can be listed as single hosts, netgroups (if using NIS), wildcards, IP networks, and in some instances “=public” for use as a public root directory.

Depending on your setup, any one or a combination of these choices may be right for you. In setups with fewer clients, it is generally easiest to specify single hosts as having access to shared file systems. You can literally add as many as you like, but be careful when using system utilities to modify these settings. Many only use a limited character buffer, which means you run the risk of truncating your exports file.

The most typical (and desired) situation for client access involves different business units requiring access to different shared file systems. As long as each business unit uses host naming conventions (e.g., “act????” for accounting hosts), you can use wildcards for host access. Here is an example:

/acct/bin       -ro=act*.local.domain
/acct/docs      -rw=act*.local.domain

In the first line, clients only have read rights to the /acct/bin directory, as they are using shared executables in this directory. This eliminates accidental overwrites and deletions of shared files. The second line allows all users to read and write documents in that file system at will.

Remote Shell Access

Another area of consideration is remote shell access. As with everything else, the greater the number of connected hosts, the greater the security and capacity concerns. Unless your server is connected to the Internet, it's generally considered “safe” to allow remote shell access, such as rsh, remsh, and rlogin as long as you use .rhosts files instead of using /etc/hosts.equiv.

Why not use /etc/hosts.equiv? Opinions differ on this subject, but it is generally considered to give too low of a granularity for remote access rights. Using .rhosts could become an administrative nightmare with very large numbers of users, but in such situations one should rethink the issue of granting this level of remote access to so many users before thinking of going to hosts.equiv.

Port Access for RPCs

Almost every client request to the server involves a port. NFS, telnet, and rlogin all require a defined port number and protocol. All port numbers below 1024 can only be bound by root, so that clients connecting to services running these ports can be assured of connecting to a standard implementation. For example, telnet uses port 23/tcp. However, you can change this to a different number above 1024, effectively closing off telnet to all except those who know the new port number (i.e., telnet 192.15.347.36:2207). Anything above 1024 is fair game. Many client/server applications used on client machines require a new port entry for RPCs (remote procedure calls).

Database SQL Access from Clients

As another example of RPC port access, sqlnet(tm) requires the following entry:

orasrv  1525/tcp

Probably the most notorious performance robbing RPCs are sqlnet statements, because poorly formed statements can cause network saturation, extremely high disk utilizations, cpu saturations, and worse. Unfortunately, in many instances (at least from the server point of view) there is little that can be done about such situations, except to be aware of which groups of clients are using such ports and how.

Anyone requesting a port that you have never heard of should raise an immediate red flag, and you should demand a great deal of detail and background about the corresponding application thatwould be using the port.

Server Use and Bandwidth

Administrators often have to deal with network delays experienced by a number of users. Just tracking down who's sucking up all the bandwidth is a task in itself.

A good place to start is the netstat command:

netstat -gin 1

In Listing 2, the netstat command with the -gin options will yield a running tally of network packets in, network packets out, and numbers of errors and collisions on each network card. In this instance, I set the interval to 1 to see the packets per second count. The first line contains the cumulative statistics since the card was last initialized.

If you consistently have over 200 packets/sec, or if collisions are consistently over 50/sec, you may want to consider switching to a 100 mbps Ethernet connection, or even FDDI.

With the -na option, netstat yields each connection showing the client IP address and the rpc port number being used (5th column). The last column shows the state that the connection is in. Notice the FIN_WAIT_2 state on 141.2.10.67.514. This indicates a final wait state on a remote shell command (port 514/tcp is the remote shell port). This can show you the number of connections coming in from each client.

If there is consistently one client IP address that seems to be sending most of the RPCs, you can look at the associated port or ports to see which service the client seems to be demanding. This will give you at least an idea of which clients are demanding the most resources.

Now that you know who's using your network bandwidth, do they really need it? If some do and some don't, you may want to think about a firewall for restricting port access. A firewall can give you back a great deal of control over your servers, and probably more sleep at night as well.

Server Use and Disk Capacity

Just about anyone who has done system administration has had to deal with users constantly asking for more disk space. There are some simple things you can do to keep a lid on things.

You may want to consider using disk quotas. The rquotad server is used by quota for a user of a local file system currently mounted by a remote machine by means of NFS. The downside is that you set up quotas for each offending user individually.

To invoke disk quotas on the server, edit the file system table on the server (i.e., /etc/fstab) and add the quota option to the line for the local filesystem. To enable quotas, just run the quotaon command. After turning on quotas, the quotacheck command must be run to update the kernel and <filesystem>/quotas file.

Now the hard part. On most systems (e.g., HP-UX), use the edquota command on each user to enable quotas for them. However, once you have a prototype user, you can run edquota with the -p option and use that user to replicate the settings to many other users at once. Unfortunately, you also have to run the edquota command on each relevant user on each client.

The trade off for disk quotas is managing disk use now or playing catch-up down the road. Users at least get a warning when they have reached their soft limit, and quota usually gives them a week to either clean up old files or ask for more space on their system or on nfs-mounted file systems.

Political Considerations

No matter how much you try to please everyone, there will always be users out there who complain that they don't have enough disk space on the server, or they don't have the required flexibility, or they don't have enough support from the sysadmin group, etc. Let's take a look at some considerations when facing these dilemmas.

Limiting Access

We probably all know someone who, well, knows it all. This can be a rather prickly problem when faced with such a manager who sees no reason why he can't have full root access everywhere. The only thing that really works is to stand your ground--quietly, patiently, and tactfully. If you need to, bring a senior staff member into the conversation on your side. Don't forget that your job often depends on having a stable systems environment, and that's worth a couple of ruffled feathers any day.

On the technical side, make sure that each system stands up to rigorous security standards (e.g., only root access to device files, etc.). Even though most systems are generally safe for non-root users out of the box, there are often hidden holes that users may inadvertantly fall into.

Prioritizing Client Demands

Every department quite naturally thinks that their work is the most important part of the organization, and therefore they should be given preference for system resources and flexibility. The easiest way to prioritize these demands is by considering the following:

• Is the cost justified in terms of time/money (your time, often your department's budget)?

• How mission critical or time sensitive is the request?

• How will this enhance the operation of the organization as a whole?

These may not be easy questions to answer, but they become easier over time. Remember, in most sites your systems are crucial to the operation of the organization as a whole, so careful planning is always called for.

Conclusions

One of the most frustrating and important jobs of systems administration is finding a good balance between giving users flexibility and administering multiple client machines and servers. Achieving a good balance can mean the difference between a clean, efficient, well-architected system environment and an administrative nightmare, where you find yourself fixing the same problems over and over again. Using these simple architecture and political guidelines can help solve some of the most common issues we face. But most of all, remember the advice given to parents about dealing with children: be consistent, and don't be afraid to say no.

About the Author

Chris Byers is a Senior UNIX Consultant in the Philadelphia area. In addition to UNIX consulting, he has also been a contributing author to such books as Unix UnLeashed, System Administrator's Edition and Internet Edition, Unix Unleashed, Third Edition and Linux Unleashed, Third Edition.