Cover V10, I11



Tales from the Abyss: UNIX File Recovery

Liam Widdowson and John Ferlito

It is every systems administrator's nightmare -- an important file has been accidentally deleted, falling into the deep abyss where bits and bytes go to die. Typically, this situation presents an inconvenience rather than a tragedy as the files can be recovered from regular backup media.

However, if a systems administrator or developer has significantly modified the file in question since the time of last backup then a large amount of data may be lost. There are also other situations that systems administrators may face, such as incorrectly configured backup scripts, backup hardware failure, or plain old bad luck. Nothing can replace a proper backup strategy, but this article will outline how whole or partial files can be restored directly from the UNIX file system.

The UNIX File System

Files in UNIX file systems are logical containers of data. Each file has an inode (index-node) structure associated with it that contains meta-data such as the physical disk blocks the file is stored on, the file owner, permissions, size, etc. [1]. When files are removed, the inodes are not erased from disk but are marked as free. The actual data contained in the files is still on disk and can potentially be retrieved before being over-written with new data.

A Tale from the Abyss

Some time ago, a graduate in my team was tasked with developing a Perl CGI-based provisioning and support interface to a product we were developing. Earlier on in the project I showed her how she could use UNIX pipes and redirection to debug her Perl script from a shell rather than exclusively through the Web server. Unfortunately, one day she accidentally typed the following:

$ ./ >
That command resulted in the Perl code being turned into a zero-byte file. Unfortunately, she had not checked the file into CVS for a week, which meant around 600 lines of code had been lost. When she told me what had occurred, I explained that it wasn't a problem because we could retrieve the file from last night's backup and she would only have to re-create today's changes.

I phoned our sysadmin and asked him to restore the file in question. However, I did not get the response I hoped for -- the sysadmin said he had never heard of the development server in question and so it was not being backed up.

The graduate sighed and said that it was impossible to restore the deleted file from the UNIX file system, so she would spend the weekend re-coding the script. I told her that it was indeed possible, and that I'd have her file restored in a few hours. Note that this type of bad luck does not only occur to inexperienced administrators. A number of years ago, I accidentally typed crontab -d instead of crontab -e on a Linux system. How many readers back up /var/spool/cron/crontabs?

UNIX File Recovery

As soon as any files have been accidentally erased, all I/O activities on the relevant partition should cease. The partition should be unmounted, or the system should be placed into single-user mode immediately after the incident. If this is not possible (e.g., the root file system cannot be unmounted), then all logged-in users should cease work, and new logins should be disallowed (e.g., with an /etc/nologin file). It is also possible on some UNIX variants to remount the partition read-only. For example:

# mount -o ro,remount -n /home
The key is to prevent other processes from overwriting the disk blocks or inodes previously used by the erased file. This is most pertinent when the partition is almost full, because it is more likely that the deleted inodes will be reused. Alternatively, operations can be performed directly on the partition while mounted read/write but, as mentioned previously, this reduces the chance of recovery.

The following sections outline how files can be restored from UNIX filesystems -- the first section describes how a file with known content can be restored from almost any UNIX variant's file system, while the second section is specific to restoring files from the Linux ext2 filesystem.

Known Content File Recovery Process

Once the system is in a safe state, a raw copy of the partition containing the erased file should be made with the dd(1) command. Consider that the /etc/passwd file has been accidentally removed. The following example illustrates the procedure required to create a copy of the root partition and place it in a file on the /export partition:

# df -k / /export
Filesystem             kbytes     used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0      123231    18413   92495    17%    /
/dev/dsk/c1t0d0s0    17498618 14327901 3170717    82%    /export
# dd if=/dev/dsk/c0t3d0s0 of=/export/recover.dsk
263088+0 records in
263088+0 records out
# ls -l
-rw-r--r--   1 root     other    134701056 Jul  1 16:54 recover.dsk
The target partition must have sufficient space to hold the entire slice (in this case ~128 MB). If sufficient space does not exist on the system in question, then alternative measures, such as NFS mounting a remote partition or performing operations directly on the partition device file, can be considered.

Once a copy of the file system has been made, the system can be brought back into multi-user mode, and normal system operation can continue while the recovery progresses. If a copy of the partition cannot be made, leave the system in a safe state and simply substitute the device name (e.g., /dev/dsk/c0t3d0s0) for the filename (i.e., /export/recover.dsk) as in examples below.

The procedure that follows is more art than science. Consider the example of the erased passwd file. The left-hand side of the following command prints each "line" of the disk with cat(1) (the -n option prints the line number). The output is piped to fgrep(1), which performs a "fast" regular expression search for the "root" entry in the passwd file:

# cat -n recover.dsk | fgrep "root:x:0:1"
200600  root:x:0:1:Super-User:/:/sbin/sh
202098  root:x:0:1:Super-User:/:/sbin/sh
332802  Ë1 root:x:0:1:Super-User:/:/sbin/sh
fgrep may not match the supplied pattern. This probably means that the file contents have been overwritten and that recovery is not possible. Otherwise, the pattern can be further generalized in the hope of increasing the possibility of a match.

In the above example, there are three places on the disk where versions of /etc/passwd file have been stored. The GNU version of grep(1) provides the -A and -B options that allow a number of lines before/after a particular match to be displayed. This assists in retrieving the entire contents of the erased file. If GNU grep(1) is not installed on your system, the C program (seekcat) in Listing 1 is provided to print out the entire file from a particular byte or line offset. The seekcat program takes two arguments: the first is either a -b or -l flag followed by an integer that specifies a byte or line offset where the listing should commence. The second argument is a -f flag followed by the filename, which in this case is the raw disk image. For example:

# fgrep -A 10 "root:x:0:1" recover.dsk > passwd
# cat passwd
The above functionality can be emulated by providing the starting line number to seekcat. The line number argument is derived from the numbers in the "cat -n" output described earlier in this section. Essentially, it instructs seekcat to begin printing the file from where a particular match was found. In the following example, ten lines of the raw disk image will be printed from the first match (line 200,600) onwards. If this did not yield suitable output, then the next match (line 202,098) would be specified:

# seekcat -l 200600 recover.dsk | head -10 > passwd
# cat passwd
Unfortunately, parts of a file sometimes may be scattered in a non-contiguous manner over the entire partition. In that case, the above procedure should be repeated with suitable match patterns to obtain the entire contents of the file. In most situations (including the erased Perl script), the entire file tends to be stored in a contiguous fashion that facilitates easy restoration. The most difficult part is determining which version stored on disk is the latest file. Unfortunately, this can only be determined by visual inspection of the content. Fortunately, CVS or RCS headers that include version numbers, dates, and other well-known contents can aid this process.

Linux File Recovery Process

The following recovery process covers the case where files have been deleted using the rm(1) command on a Linux system. In most cases, this method will result in a perfectly recovered file and does not rely on trial and error. The inherent advantage of this method is that binary files or files with unknown content can be recovered with ease.

The tool used during this process is known as the file system debugger, debugfs(8), which is used to examine and change the state of a file system. The examples below are specific to Linux but should in theory be possible on any UNIX-based file system given a tool with the functionality provided by debugfs(8). If such a tool is not available then the known content method of recovery should be used.

At this point, it is worth mentioning that debugfs is a powerful but extremely dangerous tool. It provides raw access to the file system so care must be taken throughout its use.

debugfs(8) provides a shell-like interface that has three commands that are of interest -- lsdel, cat, and dump. debugfs(8) can be run on the required partition as follows:

# debugfs /dev/hda6
debugfs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
At the prompt, lsdel will give a list of all the deleted inodes on the filesystem. This will take some time as all the inodes on the partition need to be scanned:

debugfs: lsdel
1844 deleted inodes found.
Inode  Owner  Mode    Size    Blocks    Time deleted
749300   1000 100664  27018    2/   7 Tue May  9 19:08:17 2000
749301   1000 100444   1671    1/   1 Tue May  9 19:08:17 2000
......   .... ......   ....    ..   ..........................
944887   1037 100600    597    1/   1 Sat May 26 18:00:00 2001
717281   1000 100400      1    1/   1 Sat May 26 18:08:13 2001
 32605   1000 100644     15    1/   1 Sat May 26 18:09:06 2001
Because the list is typically large, it may be useful to redirect the output to a file that may later be examined with an editor or pager as follows:

# echo lsdel | debugfs /dev/hda6 > /tmp/lsdel-output
From the information given in the lsdel listing, it should be possible to narrow down which deleted inodes are relevant. The most useful fields in the output are -- Owner, Size, Mode, and Time deleted. If no further disk operations occurred after deleting the file(s) then the required inodes should be those at the end of the list. The owner field is the uid of the owner of the file, which can be found in the third field of /etc/passwd. In this example, the file was owned by the user johnf, uid 1000, and contained the string "important_data".

From the listing above, inode 32605 is the last entry and thus a good candidate for initial investigation. The contents of the inode can be listed using the following command:

debugfs:  cat <32605>
The deleted file has been found. The dump command can now be used to write the file out to disk:

debugfs:  dump -p <32605> /tmp/recovered_file
The -p ensures that the same owner, group, and permissions are retained for the file.

This approach is sufficient if a single file has been deleted but can be tedious if a large number of files must be restored. Fortunately, Tom Pycke has written a utility called recover [2] that automates file recovery. The latest version of recover can be retrieved from:
The installation of recover is fairly straightforward:

# tar zxf recover-1.3.tar.gz
# cd recover-1.3
# make
# make install
Recover is installed into the /usr hierarchy by default. Please read the README file for instructions on installing into a different directory hierarchy.

Recover asks a number of simple questions such as:

  • Who is the owner of the files?
  • When were the files deleted?
  • What is the approximate size of the files?

Using this information, it executes debugfs to recover the inodes that match the given criteria and places them in a user-specified directory.

Unfortunately, it is not possible to recover filenames in addition to content and results in a directory full of files named dumpinode-number. If a directory such as /etc was actually removed, this could amount to hundreds of files. At this stage, simple UNIX utilities can be used to sort through the recovered files and the two most useful tools are strings(1) and file(1). strings(1) displays sequences of ASCII characters for a given input file and is useful for extracting text from an otherwise undecipherable binary file. file(1) is able to determine the type of a file (i.e., whether it is a JPEG image or a postscript file). It does this by performing a set of magic number tests to uniquely determine the file type.

For example, the output of file(1) in a directory filled with recovered files is as follows:

# file *
dump34578:    directory
dump98568:    PGP armored text signed message
dump896545:   gzip compressed data, deflated, last modified: Sun 
                 Jan 28 03:31:21 2001, os: Unix
dump78890:    ASCII text dump67245: 'diff' output text
dump76345:    JPEG file dump9723: MPEG 1.0 layer 3 audio stream 
                 data, 128 kBit/s
dump8976:     ASCII C program text
dump57654:    Bourne shell script text executable
dump3463:     troff or preprocessor input text
dump56789:    ELF 32-bit LSB executable, Intel 80386, version 1, 
                 dynamically linked (uses shared libs), stripped
dump9876:     ELF 32-bit LSB shared object, Intel 80386, 
                 version 1, stripped
At this point a simple shell script to sort files of different types by adding file extensions would be useful. For example:

# for i in 'file * | grep "ASCII C program text" | awk -F: '{print $1}'`;
    do mv $i $i.c; done
Once sorted, it is time to begin the process of trying to establish the identity of each file. For text, C, image, and audio files this is a fairly straightforward process as each file can be visually inspected with the appropriate tools and the original file name can be guessed. However, binary files such as executables, libraries, and database files are more difficult to inspect.

In this situation, the strings(1) utility can be used to print out any ASCII text strings in a binary file. For example:

# strings dump45678.bin
From the above output, a guess can be made that the file is the groff(1) executable. Executing the file with a --help argument confirms this. Libraries are a little more difficult because they cannot be executed, however, the objdump(1) command can provide suitable assistance. For example:

# objdump -p dump34756.lib | grep SONAME
The aforementioned method is only useful if the file has been deleted through use of a utility such as rm(1) or the unlink(2) function. If the file has been overwritten, then the inodes that originally contained the data have probably been overwritten. It is still possible that some fragments of the data are held elsewhere on disk. For example, many editors make temporary copies of files that later get deleted, or perhaps not all the inodes in the file have been overwritten. If the inodes are lost, then the known content recovery method already described will be more applicable.


No file recovery method can take the place of thorough, regular, and reliable system backups. The processes described above are designed to provide last-ditch assistance to systems administrators perched on the edge of an abyss.


1. Vahalia, U. 1996, UNIX Internals -- The New Frontiers. Prentice-Hall, Upper Saddle River, NJ

2. Pycke, T., 2001, Recover.

Liam Widdowson is a consultant at Hewlett-Packard. He can be contacted at:

John Ferlito is a senior engineer at Bulletproof Networks. He can be contacted at:

The views expressed in this article do not necessarily represent those of the authors' employer.