 The
Great Solaris Administration Best Practices Debate
Peter Baer Galvin
In the February 2002 Sys Admin magazine, my column covered
"The Golden Rules of Sun Systems Administration" (http://www.sysadminmag.com/documents/s=2293/sam0202j/sam0202j.htm).
This led to a flurry of feedback, point, and counterpoint about
the true Solaris systems administration best practices. This month,
my column includes the points from readers, and my view on them.
So much time, effort, and discussion is spent on specific sys
admin tasks ("how to get this thing to do that thing"),
and relatively little time is spent on the larger issue of the best
ways to install systems, the best ways run systems, and in general,
the best practices of systems administrators. Systems administration
is a journeyman's trade. Most admins don't earn a degree
in systems administration. Most don't even take any courses
in it, except for the occasional "how to" tutorial or
vendor course. Rather, good sys admin practices are learned on the
job, through interactions with other administrators and direct experience
(i.e., "the hard way").
Thus, I believe that a Solaris Sys Admin Best Practices FAQ is
a good and useful thing to create and maintain. This month, I include
new ideas and points of discussion from readers. In the May issue
of Sys Admin, the wisdom from "Golden Rules" and
here, plus your new feedback, will be collected into such a FAQ.
I hope it will continue to evolve and improve over time, again with
reader input.
With no further ado, here is the more salient feedback to the
Golden Rules column. Stewart Dean wrote:
Comments/Additions:
1. PAY ATTENTION: Your world is hostile; "smell" the
wind, listen to the slightest noise, *know* your world. You can
generally deal with problems two ways:
- Early and cheap, if you're paying attention and are proactive
- Late and expensive if you are smashed under the avalanche.
So, pay attention to how things work when they work well; pay
attention when things change. Devise trip wires and reports.
Related to this: pay attention to error messages and clean up the
errors behind them. You ignore them at your peril: the problem will
snowball and devour you at the worst possible time and/or messages
will mount up until they hide a really important one that you miss
because you begin ignoring all messages.
2. STRATEGY & TACTICS: It's essential to learn the difference
between strategy and tactics and learn the place for both. Strategy
is arranging the battlefield so that your chances are maximized:
you may thus be able to win easily or without even a fight. Tactics
refer to hand-to-hand combat. You win by being good at both.
Subrules:
a. An ounce of strategy can be worth a couple of tons of tactics.
See Rule 1.
b. Don't be overly clever with strategy where tactics will
do. Life is short.
3. HELP USERS FIX IT THEMSELVES: If you have reports that tell
you about quota abuse, overly large mail Trash and Sent folders
and the like, extend the exec to send messages to users informing
them of the approaching fubar while they can still deal with it.
Besides, you will have the immense satisfaction of being able to
say to them, "You got a warning message, you turkey, why didn't
you pay attention", when they call up to whine at you.
[Plus, following this rule tends to get users who appreciate and
respect you, and who are your allies later, when you need it. -
PBG]
4. HTML SYSADMIN DOC: Document your system administration by using
basic HTML (Netscape Composer is plenty sufficient). It is "indexed":
grep on the search argument; it hyperlinks to your own stuff
and that of your vendors. You can find the answers to fubars that
happened a year ago (when they recur).
You can access the data from anywhere (although you'll want
to restrict that naturally); you can ftp it all from the server
to your laptop for when you're at a remote site or on vacation
and problems occur. You can share the info and add contributions
easily. You can burn a CD of it to archive with the backups.
[I've frequently kicked myself for not writing down some
solution to some problem that recurred. If I'd taken 10 minutes
to document it, I would have saved 10 hours at the next occurrence.
- PBG]
Another Occam's Razor corollary. I thought I thought it up,
but Dick Paddock seems to have originated it: Never attribute to
malice what can be explained as the result of sheer idiot stupidity.
Re: users, bosses, the guy who left you the mess you're cleaning
up, etc.
[Very true. I learned long ago not to pass judgment on whoever
created or left the mess that I was dealing with. I once criticized
a system configuration in front of a client, and it turned out to
have been done by one of my co-workers! - PBG]
Some minor things:
a. Most cell phones now come with text messaging so it's
now dead easy to have your trip wire execs call your cell phone.
b. Don't just save a backup, also save your system configuration
(with Sun, it's essential, with AIX less so) and your system
admin doc.
That reminds me of one of the more important but hard to understand
tenets I follow: do not be "too" clever. Of course the
challenge is in knowing when "too" is occurring. For example,
a client was devising multiple, complicated ways to use a set of
systems, rather than simplifying the solution by using more equipment.
There are always tradeoffs like that in any facility architecture.
I was concerned by the complexity and advised them of this tenet.
The systems manager in charge responded, "oh, I'm not".
He was, and it ended up costing his job when the facility failed
and failed again. Frequently, implementation and management complexity
is unnecessary and results from "too clever" systems administration.
An anonymous reader is miffed about one of my golden rule statements:
Since you say that putting files in /usr/local/bin is "deprecated",
how about informing the totally clueless where the now standard
location would be?
Well, as with all of the best practice advice, there are really
no hard-and-fast rules. In this case, the direction of Sun, as discerned
by their installation changes over time, is to make /usr
as read-only as possible. Rather, /opt/local is the best
place to make local changes. If users (or administrators) cannot
stand the change, a link from /usr/local to /opt/local
is reasonable.
Ken Stone states:
I read your article on the Golden rules of Sun administration
and can appreciate them all. We have recently begun collecting what
we call principles. Many are similar to what you have documented,
but here are a couple you might find merit in adding to the list:
Completion backward principle -- Never move forward with something
unless you are fully prepared to return the server to the original
starting point. Make images, back stuff up and test, make sure you
have the original CDs, or whatever; just make sure you do it before
you begin.
Gita principle -- When users diagnose their own problems and
provide a solution based upon their limited knowledge of possible
solutions. This is a little tougher to explain. Basically it's
important as an admin to recognize when this is happening while
communicating with users. The example we use is when someone brings
his car to a shop for a tune-up; usually it's because the car
is running badly and he has diagnosed it as a tune-up because that's
all he knows about. In reality, it's something like the wheel
being flat, etc.
Let me add these are not policy of the company I work for or official
in any way, shape, or form. These are just things we use to identify
situations when we commiserate over the finer points of being an
administrator.
Those are wonderful and true Golden Rules.
An anonymous reader contributes:
I enjoyed reading your article in Sys Admin mag. Mostly, I got
quite a few chuckles -- although the suggestions you put forth
are sound in basis, many are impractical for large shops. When in
doubt, reboot? Sometimes a sys admin doesn't have the luxury
to reboot systems every time there is a problem. One of the beauties
of UNIX (above other operating systems that I shall refrain from
naming) is that it separates user- and system-level functions cleanly
and frequent reboots to solve problems are not absolutely necessary.
For every time that a corrupted jumbo patch was the culprit in my
professional experiences, there have been ten that rebooting ad
nauseum proved nothing and some real detective work was needed to
resolve a problem. It is true that editing /etc/system typically
requires a reboot (unless you're really brave and are willing
to test your changes by running adb on a live kernel), but
I think you're doing your readers a bit of a disservice by
encouraging them to use the reboot as a crutch rather than as a
last resort; knowing when to reboot -- and when not to --
is one characteristic of a good, experienced admin.
[That is a great point for all of these rules -- knowing when
to apply them is as important as knowing the rule. And I agree that
rebooting is sometimes pointless. That said, I've been surprised
at the number of issues that are still resolved on Solaris by a
reboot. Sometimes, fighting with the machine may waste time, when
a reboot could save time. -PBG]
Similarly, I would take exception with your suggestion that it
is a best practice to "Use Defaults Whenever Possible."
Again, this seems to be a suggestion geared at inexperienced administrators
that don't have the time to immerse themselves in the operating
system enough to become expert at it. For instance, were I to follow
this tenet in my daily life, I would install the entire OEM+ Solaris
distribution on all my machines... only to be nagged by our security
administrator every other week when the latest sadmind, dtspcd,
or rpc.ttdbserverd exploit is released in the wild. By learning
enough about my system to know the minimum setup that it needs to
operate properly, not only do I eliminate security risk, but I reduce
some unnecessary dead weight in my environment. It is especially
disappointing to see this subject treated so lightly since your
bio states that you teach security topics worldwide (actually, I
think I attended one of your talks at SANS '97).
[Rats, I hate it when people use my words against me. But this
is a very good point. If security is a primary concern, then many
of these rules should be used in antithesis. For example, being
too clever is bad, unless you're trying to secure a machine
by adding layers of defenses. Then being too clever is good. The
point I was trying to make is not to change the defaults without
good reason, and security is one of the best reasons. -PBG]
I like many of your suggestions and overall it was a good column
(never change anything on Fridays is an especially insightful point),
but I really think you should have prefaced it with an explanation
of who your target audience is. The administrator that is looking
to make the leap from novice to expert would not be well-served
by taking some of these suggestions as gospel.
Art Kufeldt lets us know:
Another corollary to the "Check the Cables" rule is:
Check the Simple and Obvious Stuff First. A good example is the
user who calls and says, "I can't login anymore."
Rather than reset their password or worse yet, tear into the NIS
server, I just ask them if they have the "Caps Lock" key
on. Probably 70-80% of the time it is something simple like that.
Also certain apps lock up the mouse and keyboard when the "Num
Lock" key is invoked, so rather than kill their processes or
make them reboot, I ask them that question also. Systems administration
can be really time consuming as you are well aware of, but things
like these rules can make your life a lot easier. Excellent article!
Perhaps the most concise (and true) rule comes from Juan Pablo
Sánchez Beltrán:
When I think back at all the time I wasted (mostly on Windows
systems) making just one last change, or one small improvement,
only to cause irreparable harm resulting in prolonged debugging
(or an operating system reinstallation), it makes me wish I had
this rule tattooed somewhere obvious. The beauty of this rule is
that it works in real life, not just for systems administration.
I enjoyed your article "The Golden Rules of Sun Systems Administration".
It should be required reading for all admins.
I'd like to mention a few rules of my own. This one would
have saved a lot of people a lot of anxiety:
Power off before you remove or rebuild -- if you power it
off, you may find out that there are people depending on services
running. If you have already removed the server, replaced hard drives,
or started putting a new OS on, it's much harder to resolve
the unforeseen issues. I've often seen complaints from people
who got the email but thought the change did not affect them. Sometimes
even technical staff will forget to mention a dependency.
Here's a sneaky one to help you get your project management
work done:
Announce big changes far earlier than you really need them done
-- this is an extension to the "All projects take twice
the Estimated Time and Money". Let's say you need to power
off a data center or replace a critical server by October 31. Announce
it for September. People will be much more forthcoming about possible
gotchas as the "deadline" approaches. You can adjust to
"push back" very diplomatically and generously because
your real deadline is not imperiled.
And something philosophical:
The really good stuff is never funny at the time.
Regards, Pete Tamas
Very nice rules to live by. I especially like the idea of announcing
an earlier date. We could also call this the "Scotty rule"
(from Star Trek), but he not only acted like the problem would take
longer than it would, but also that it was harder than it is. This
could make you a hero in the eyes of your users...until they catch
on!
Finally, this from Christopher Jones:
It was the article on "The Golden Rules of Sun System Administration"
that I just got in the mail yesterday. I fully agreed with every
point you made on there, I've seen all of them. I've been
a UNIX admin here at NASA for over five years now. But I wanted
to point out to you (and maybe you're just partial to Sun)
that everything you said applies to all UNIX platforms out there
-- Sun, SGI, HP-UX, AIX, and Linux (and I'm sure I've
missed some). Even though Sun seems to be the big dog out there
these days, there's still plenty of the other operating systems
in play still.
Here where I am, we've got pretty much a 50/50 mix of Sun
and SGI (with most of our high-end high-memory high-number of processor)
systems being the SGI). Even though I've got my favorites,
I try to treat all the flavors equally.
I did enjoy the article though... Keep writing them!
Very true. Given that most of my experience is with Solaris, I
did not want to speak for other operating systems. My experiences
with Linux, Win2K, and even Tops-20 (way back when), indicate that
these are life-long systems administrator rules, regardless of the
operating system being administered.
For another perspective about the sys admin life, you may want
to check out:
http://www.linux.com/enhance/newsitem.phtml?sid=1&aid=11529">linux.com
Conclusions
These comments seem to hit home with long-term systems administration
experiences. Are there any others that we've missed? If so,
send them along to pbg@petergalvin.org, and I'll include
them in The Solaris Sys Admin Best Practices FAQ.
Peter Baer Galvin (http://www.petergalvin.org) is the
Chief Technologist for Corporate Technologies, a premier systems
integrator and VAR. Before that, Peter was the systems manager for
Brown University's Computer Science Department. He has written
articles for Byte and other magazines, and previously wrote
Pete's Wicked World, the security column, and Pete's Super
Systems, the systems management column for Unix Insider.
Peter is coauthor of the Operating Systems Concepts and Applied
Operating Systems Concepts textbooks. As a consultant and trainer,
Peter has taught tutorials and given talks on security and systems
administration worldwide.
| 
