How I Recovered Data I Thought I'd Lost
A few years ago, when I worked in a commercial software development environment, we experienced a disk failure on one of the HP-UX application development machines. This disk held a file system that contained all of the original hand-written source code for a large customer software project. The work there-in was the product of a project team consisting of 3-5 software engineers over weeks of time. If the content of this file system could not be restored, it would be a major setback, but of course we had been performing daily full backups. Still, because of the amount of time estimated to restore the development environment, it was suggested that the engineers go home for the day, and the systems adminstrators began work on the restoration.
Not too long before this incident occurred, we had gone through the effort of formalizing our backup procedures. New scripts were prepared and tested independently by both our systems administrators and Oracle Database administrators. The SAs produced a backup script that would dump(1) each file system, one after the other, to a tape. Likewise, the DBAs prepared a backup script to preserve the Oracle database contents as a cpio(1) archive to a tape. The two scripts were subsequently integrated, so that the entire system backup would be performed nightly to a single high-density 4mm DDS tape, which was the state-of-the-art at the time -- mid 1996.
Unfortunately, after the integration of the entire backup procedure, no one had properly tested the resulting combined script by attempting to restore data from the resulting backup tapes. Although the file system and database portion performed flawlessly as independent backups, it was soon discovered that the database backup, which executed after the file system backups, either began by issuing an mt rew command, which caused the tape to be rewound (this was the method we sometimes used simply to determine whether a tape was loaded), or it did not access the tape drive by its no rewind device file.
Conventionally, a UNIX tape device file whose name contains the letter n, as in /dev/rmt/0hbn, leaves the tape read/write head at the current position after a tape operation completes. This feature allows the administrator to issue a sequence of backup commands that write multiple backups, one after another, to a single tape. Without this feature, every backup operation would be followed by the tape being rewound automatically, and mean that a subsequent backup would overwrite the previous. In other words, without the no-rewind device, a tape would hold the data from only one backup operation, such as a single file system dump.
As you can imagine, the consequences of this mistake were disastrous. When the SA attempted to fast forward the tape, using mt fsf n, to the position of the lost file system's backup, the tape drive encountered the End of Data marker and stopped. The relatively small database backup had overwritten one or more of the relatively large file system backups, and then the tape drive dutifully wrote a new End of Data marker just after the database backup, but before the position of file system backup that we needed to access. Essentially, the tape drive itself was keeping us from reaching our data, which unfortunately was located past the End of Data marker.
With some understanding of tape markers, and past experience with tapes of many sorts, the SAs laid out their options, each of which were pursued to some degree. At the same time, some nosy system programmers -- myself included -- who had in the past been responsible for some sys admin tasks, discussed the merits of each of these pursuits. We independently followed our own tack, in part, because of a division of responsibilities within the growing company and because some areas were off limits to us unless deemed absolutely necessary. I'll describe each of the various approaches.
Send for Help
An initial thought was that we could send the broken disk to a firm specializing in data recovery. If memory serves me, a phone call was made and it was determined that it would cost $5,000 minimum, regardless of whether anything could be retrieved. Also, the round-trip time for this process would be at least 2 to 4 days. Since we were game to follow our other leads, and perhaps make some progress in short order, this was not an interesting option to pursue immediately.
You Can't Get There From Here
A number of us thought we could investigate how one might be able to ask the tape drive to skip past the premature End of Data (EOD) marker. Also, the brute-force option of simply physically winding the tape past the marker, and then loading it into the tape drive, was considered.
Alas, these were short paths to nowhere. Apparently the DDS specification dictates the characteristics of the EOD marker and the behavior of the drive with regard to it. Neither the mt(1) command nor the driver itself seem to have a feature to skip past EOD. We surmised that it is either impossible for the tape drive hardware to read through or past EOD, or that it is only within the control of the drive firmware.
A brute force method was also tried. We manually wound the tape ahead by inserting the eraser-end of a pencil into one of the tape reel access holes. Unfortunately, upon loading the tape into the drive, the tape drive would automatically rewind the tape before any operations could be performed.
Measure Once, Cut Twice?
Another idea was that we could unwind the tape and splice out the premature End of Data marker -- certainly not elegant, but still a possibility.
Conveniently, the backup operation had been performed in this erroneous way for some time. As a side effect, we had accumulated boxes of similarly botched tapes. The older tapes provided lots of raw material with which to attempt to find a workable recovery procedure by trial-and-error.
For splicing, the tapes unfortunately weren't good ol' reel-to-reel 9-track tapes, but rather they were tiny 4mm Digital Data Storage DDS tapes, just like the Digital Audio Tapes DAT media. Physical manipulation of the media was further impeded by the tape enclosure, which is similar to that of a VHS video tape, only much smaller, with a spring-loaded door that snaps shut to protect the magnetic medium. The tape and tape-drive specifications were examined to determine the length of the tape and data density, and some rough values in meters were decided upon as to where the splice should occur. Before too long, the computer room floor was littered with snippets of tape, every one a reminder of another failed attempt. In the end, none of the spliced tapes was able to be read. A co-worker thought that this may be due to the helical scanning head that is used to write the data to the tape itself -- it is perhaps impossible to splice such a tape amidst the data and preserve its integrity.
While fruitless experimentation continued in the computer room, I turned to my Web browser and visited http://www.deja.com, which was www.dejanews.com at the time, hoping to find the advice of others who had been in this predicament. Deja.com provides a wonderful archive of USENET news, a sort of USENET Library of Congress, but with commercials. Using the site's Power Search feature, I found a thread from the comp.sys.hp.hpux newsgroup with the subject [Q] reading past End Of Tape with HP-UX 9.05. To my surprise and delight, the poster described a method by which one can potentially recover data from a tape trashed in the way I've described. The method involved the following steps:
1. Overwriting the tape again, with just the right amount of data to clobber the premature EOD marker that was keeping us from reaching our backup data.
2. Powering off the tape drive, before it can write another EOD marker, as it is designed to do.
This procedure, while possibly specific to DDS tape technology, is neither HP nor HP-UX-specific, nor is it in principal even UNIX-specific.
Before performing the steps below, which are my interpretation of the steps I discovered, you should practice this using old backup tapes that have been similarly trashed. If you don't have such tapes, I suggest taking the time to produce practice tapes that are similarly overwritten. Remember, this is potentially a destruction process, so you may only have one try with your most recent, best backup tape.
1. Load a clobbered tape into the drive.
2. Position the tape to EOD, using the 'n'o rewind device file:
# mt -f /dev/rmt/0hbn eod
3. Create a compressed sample data file (e.g., /tmp/17secs_of_tape). This file, when written to the tape positioned at EOD, will hopefully result in enough data being written to clobber an EOD marker. Here, we call it 17secs_of_tape because that is the estimated length of time to clobber the 10-second long EOD marker. This marker is relatively large since it must be able to be reliably observed and identified by the tape drive when performing a fast-forward operation. This is similar to how some modern cassette tape players locate the quiet spaces between songs on music tapes.
The following is a quote from Kevin Jones (firstname.lastname@example.org), an HP employee who, in an unofficial capacity, authored and posted a step-by-step procedure from which I worked. Regarding this portion of the procedure he wrote:
On 354x0 drives, with a native transfer rate of 184 Kbytes/sec, the 17secs_of_tape file should be just over 3 Mbytes of compressed data. You can use whatever data you want, not just the HP-UX kernel. On HP C15x3 drives with a native transfer rate of 510 Kbytes/sec, you need about 8 Mbytes of data.
Kevin chose to use the HP-UX kernel itself as a source of data to write, since it was a known quantity to him, and contains sufficiently random content to produce a compressed data file of a given size. If you are not running HP-UX, or are running a version with a kernel very different in size from the 9.05 kernel, it may be best to choose some other data file. If it exists on your system, I suggest using /dev/random as a source of unpredictable content, which is unlikely to be able to be compressed much by the tape drive. In this example, I'll presume you want a file of 8 MB in size:
# ls -l /tmp/17secs_of_tape # determine whether or not it exists
# dd bs=1024 count=8192 if=/dev/random of=/tmp/17secs_of_tape
# ls -l /tmp/17secs_of_tape # should be 8MB in size
Remember that a file containing more data than necessary could result in your accidentally overwriting even more of your precious data just beyond the premature EOD marker. Be conservative with the file size unless you are certain that there is much tape between the EOD and the data you wish to recover.
4. Write the sample data to the tape:
# dd if=/tmp/17secs_of_tape of=/dev/rmt/0hbn
5. Immediately (without performing any other tape operation), power off the tape drive. If the tape drive is internal, consider changing to single-user mode, and be sure to sync(1) the file system(s) and wait for the drive activity to complete before powering off the system.
6. Wait a bit, as you probably should whenever powering equipment off and back on.
7. Fast-forward to just past the first EOF marker (if you have multiple files, or dump(1)s on the tape):
# mt -f /dev/rmt/0hbn fsf 1
or, if you want to have your backup/archive utility attempt to re-sync and potentially even recover data from within the over-written backup, you might try:
# mt -f /dev/rmt/0hbn fsr 100 # adjust \
number of records accordingly
8. Attempt to recover data that was once beyond the premature EOD marker:
# restore if /dev/rmt/0hbn
One complication in our situation was that we were using a compression feature of the tape hardware. On many systems, this feature is used when one accesses the tape drive using a device file with the letter h in the name, as in /dev/rmt/0hbn. This compression means that it is hard to judge just how much sample data (in bytes) one must write to clobber the EOD marker, because the degree of compression achieved depends on the content of the data itself. To deal with this, it is best to use unrepetitive or already compressed data when making your nsecs_of_tape file, because it is repetition in a data stream that enables effective compression.
Another complication was that we did not have access to an external DDS tape drive -- they were all internally mounted within the cabinet of HP-UX machines. We decided to enlist the services of another similarly configured, but currently unused, HP-UX machine and, when the time came, throw its big switch -- thereby powering down the whole machine and the tape drive as a side-effect. While this may seem extremely risky, it was a machine that we could afford to lose and it has been noted that powering off an external drive, if we had one, may have elicited similarly undefined behavior, perhaps resulting in a kernel panic. If you consider doing this, it would be prudent to attempt this recovery operation from single-user mode if possible.
In the end, we had great success with the method described above. The software engineers' work was restored and all returned to normal. This experience reminded us that successful execution of a backup script doesn't necessarily mean that one can recover data from the backup. As a result, backup recovery tests were added to the list of sys admins backup responsibilities. The creative method that was ultimately employed to do the impossible serves as an example of what can be accomplished when admins or engineers take the time to share their experiences with their co-workers and their peers elsewhere.
I would like to thank Kevin Jones (email@example.com) and Frank Slootweg (firstname.lastname@example.org) for their participation in USENET news and, specifically, in assisting with this problem.
About the Author
Dave is a benevolent hacker and UNIX aficionado. He is a Senior Systems Programmer in the Network Engineering Technology group within the Division of Information Technology (DoIT), at the University of Wisconsin, Madison. He can be reached at: email@example.com or via: http://net.doit.wisc.edu/~plonka/.