extundelete

For many years, I've believed, and have been telling other people, that "Linux does not have an undelete command".

Now, I know that's a pretty ambiguous statement, because it says nothing about which file system you're using, but I generally assume that people either:

  • use EXT2/3/4 (depending on when they set up the partition), or
  • know what they're doing, make an informed decision, and choose EXT, XFS, ZFS, BTRFS, OCFS, or something else as appropriate, in which case they don't need my advice

However, it turns out that I'm wrong about the generalisation regarding EXT2/3/4 having no undelete facility.

If you're using ext3 or ext4, and if you're pretty quick to react after realising you've deleted some files (or directories), there's a good tool out there for getting them back again.

http://extundelete.sourceforge.net/ works by using the journalling feature of ext3 and ext4 (so it doesn't work for ext2, but why would you still be using that these days anyway?) and in my (very limited, I'm pleased to say) experience so far, it does a damned good job of recovering files (pathname plus contents, although not timestamp) so long as you're quick enough after deleting them to unmount the file system in question.

In my case, I managed to set up a fatal combination of rsync jobs in cron.

One had been in place for months (years?), doing a backup (with delete, to create a faithful copy) of a remote server to a local storage device. No problems there.

One day I noticed that another cron job, running on the remote server, and which downloaded a new ~600Mbyte file every week and stored it there, was filling up the disk space, and since I had no reason to keep the files on the remote server (I just needed to keep them somewhere), I decided to implement a new rsync job specifically for these files, which included the --remove-source-files rsync option.

This job downloaded the files to the same local storage as the backup job, and on its first run it did exactly what I wanted - it downloaded the newest file (the others were already present), deleted all the files in that directory on the remote server, and freed up a nice chunk of capacity. I was happy.

The next time the standard backup rsync ran, it saw the empty directory on the remote server, deleted the files on the local storage device (because it's supposed to be a faithful copy of the what's on the remote machine), and I now had no copies of those files, which had built up over several months and couldn't be downloaded again. I was not happy.

Fortunately, my rsync jobs send me an email telling me what they copied and what they deleted, and I noticed all these files having being deleted from the local storage device within about 10 minutes, so I could act quickly enough to recover them.

My first action was to shut down the local storage server (it's a standard Linux machine with a bunch of disks in it in a Raid array). I might have been able to remount the appropriate partition read-only, but since the machine is in the next room from me, and all I knew at the time was that I wanted to stop it doing anything further to that file system, I shut it down with the intention of restarting it from a USB stick or live CD once I had an idea of what to do next.

I booted from a live CD, adjusted /etc/fstab to not mount the partition in question at all (note: this is one more good reason to keep system and data partitions separate) and restarted the machine. I checked that the partition with the deleted files was not mounted and then installed extundelete (I didn't previously know about this tool; I found it in between shutting down the storage server and restarting it once I thought I had a clue what to do next). I then tried it out: 

# extundelete --restore-directory antony/rsync/server3/downloads /dev/md2

Notes:

  • the partition (/dev/md2 in this case) must either not be mounted at all, or be mounted read-only
  • you have to run the command as root (otherwise it can't read the raw partition and find the journal files etc.
  • you specify the directory to be recovered using the path as it would be seen on the mounted file system, not including the mount point
  • the recovered files will (with the command shown above) be placed into a directory structure under RECOVERED_FILES which gets created in the current directory, so make sure there's enough space on the file system you're currently in to write the files
    • there are further options to extundelete which are best found from reading the man page (note: the --help option fails to mention many of these)

In my case this created a directory RECOVERED_FILES/antony/rsync/server3/downloads, with two further directories under it, containing 28 files and 4 files respectively, which were precisely the files which had been deleted. I got the whole lot back (about 21 Gbytes). I was happy again.

I then modified my rsync jobs not to conflict with each other before setting /etc/fstab back to what it was, remounting the data partition, and letting the storage server continue doing its job.

extundelete is available as a Debian / Devuan package, too.

Good luck :)

Go up
Return to main index.

Powered by DokuWiki