*THE* classic Unix horror story

binford2k's picture

Try this in windows!


Unix Recovery Legend

This classic article from Mario Wolczko first appeared on Usenet in 1986.

Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflict such a trauma upon you, and was doing it in jest. So you've probably never experienced the worst of such disasters....

It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15 BST, to be precise, when Peter, an office-mate of mine, leaned away from his terminal and said to me, "Mario, I'm having a little trouble sending mail." Knowing that msg was capable of confusing even the most capable of people, I sauntered over to his terminal to see what was wrong. A strange error message of the form (I forget the exact details) "cannot access /foo/bar for userid 147" had been issued by msg. My first thought was "Who's userid 147?; the sender of the message, the destination, or what?" So I leant over to another terminal, already logged in, and typed

grep 147 /etc/passwd

only to receive the response

/etc/passwd: No such file or directory.

Instantly, I guessed that something was amiss. This was confirmed when in response to

ls /etc

I got

ls: not found.

I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.

When I arrived at his office, his door was ajar, and within ten seconds I realised what the problem was. James, our manager, was sat down, head in hands, hands between knees, as one whose world has just come to an end. Our newly-appointed system programmer, Neil, was beside him, gazing listlessly at the screen of his terminal. And at the top of the screen I spied the following lines:

# cd
# rm -rf *

Oh, shit, I thought. That would just about explain it.

I can't remember what happened in the succeeding minutes; my memory is just a blur. I do remember trying ls (again), ps, who and maybe a few other commands beside, all to no avail. The next thing I remember was being at my terminal again (a multi-window graphics terminal), and typing

cd /
echo *

I owe a debt of thanks to David Korn for making echo a built-in of his shell; needless to say, /bin, together with /bin/echo, had been deleted. What transpired in the next few minutes was that /dev, /etc and /lib had also gone in their entirety; fortunately Neil had interrupted rm while it was somewhere down below /news, and /tmp, /usr and /users were all untouched.

Meanwhile James had made for our tape cupboard and had retrieved what claimed to be a dump tape of the root filesystem, taken four weeks earlier. The pressing question was, "How do we recover the contents of the tape?". Not only had we lost /etc/restore, but all of the device entries for the tape deck had vanished. And where does mknod live? You guessed it, /etc. How about recovery across Ethernet of any of this from another VAX? Well, /bin/tar had gone, and thoughtfully the Berkeley people had put rcp in /bin in the 4.3 distribution. What's more, none of the Ether stuff wanted to know[work?] without /etc/hosts at least. We found a version of cpio in /usr/local, but that was unlikely to do us any good without a tape deck.

Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.

So, what to do? The next idea was to write a program to make a device descriptor for the tape deck, but we all know where cc, as and ld live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and so on, so that /usr/bin/ftp would work. By sheer luck, I had a gnuemacs still running in one of my windows, which we could use to create passwd, etc., but the first step was to create a directory to put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we couldn't rename /tmp to /etc. However, this looked like a reasonable line of attack.

By now we had been joined by Alasdair, our resident UNIX guru, and as luck would have it, someone who knows VAX assembler. So our plan became this: write a program in assembler which would either rename /tmp to /etc, or make /etc, assemble it on another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it (some bright spark had thought to put uudecode in /usr/bin), run it, and hey presto, it would all be plain sailing from there. By yet another miracle of good fortune, the terminal from which the damage had been done was still su'd to root (su is in /bin, remember?), so at least we stood a chance of all this working.

Off we set on our merry way, and within only an hour we had managed to concoct the dozen or so lines of assembler to create /etc. The stripped binary was only 76 bytes long, so we converted it to hex (slightly more readable than the output of uuencode), and typed it in using my editor. If any of you ever have the same problem, here's the hex for future reference:

070100002c000000000000000000000000000000000000000000000000000000
0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
8800040000bc012f65746300

I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved.

So along we trotted to the terminal with the root login, carefully remembered to set the umask to 0 (so that I could create files in it using my gnu), and ran the binary. So now we had a /etc, writable by all. From there it was but a few easy steps to creating passwd, hosts, services, protocols, (etc), and then ftp was willing to play ball. Then we recovered the contents of /bin across the ether (it's amazing how much you come to miss ls after just a few, short hours), and selected files from /etc. The key file was /etc/rrestore, with which we recovered /dev from the dump tape, and the rest is history.

Now, you're asking yourself (as I am), what's the moral of this story? Well, for one thing, you must always remember the immortal words, DON'T PANIC. Our initial reaction was to reboot the machine and try everything as single user, but it's unlikely it would have come up without /etc/init and /bin/sh. Rational thought saved us from this one.

The next thing to remember is that UNIX tools really can be put to unusual purposes. Even without my gnuemacs, we could have survived by using, say, /usr/bin/grep as a substitute for /bin/cat.

And the final thing is, it's amazing how much of the system you can delete without it falling apart completely. Apart from the fact that nobody could login (/bin/login?), and most of the useful commands had gone, everything else seemed normal. Of course, some things can't stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but by and large it all hangs together.

I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way? Answers on a postage stamp to:

Mario Wolczko
------------------------------------------------------------------------
Dept. of Computer Science       ARPA:   miw%uk.ac.man.cs.ux@cs.ucl.ac.uk
The University                  USENET: mcvax!ukc!man.cs.ux!miw
Manchester M13 9PL              JANET:  miw@uk.ac.man.cs.ux
U.K.                            061-273 7121 x 5699
------------------------------------------------------------------------

Hacker's Wisdom: Unix Recovery
Legend

Last modified: Thu Mar 7 13:47:40 EST 1996


http://www.ee.ryerson.ca:8080/~elf/hack/recovery.html

Comments

mchristenson's picture

Re: *THE* classic Unix horror story

We used to outsource some of our support at our data center, and we had multiple rm's from those agents. Luckily, live CD's can get you back rolling to where you can rsync from backups.

# cd
# rm -rf *

If I'm not mistaken, wouldn't this just delete everything in /root ? I'm more of a Linux guy, but I'd presume you'd need the following command to wipe everything.

rm -rf /

-Chris | Editor
Mozy Reviews

binford2k's picture

Re: *THE* classic Unix horror story

It all depends on where root's home dir is. Some unixes do have it set to /

mchristenson's picture

Re: *THE* classic Unix horror story

I never have anything in /root though, all my stuff is in /home so rm -rf * in the root directory wouldn't hurt me too badly.

rbuelke's picture

Re: *THE* classic Unix horror story

It doesn't matter whether you keep anything in /root or /home, if the distribution you are using has the root user's HOME environment variable set to /.

POSIX standard says that, when not supplied with any arguments, the cd command will change to the directory pointed to by HOME (or an implementation based location if HOME is unset or undefined.) If the former, subsequently invoking rm -rf * would then remove both your /root and /home directories.

See http://pubs.opengroup.org/onlinepubs/009695399/utilities/cd.html

scott.allerdice's picture

Re: *THE* classic Unix horror story

The story seems to be terrifying :)

Scott Allerdice
Software Engineer
software development

saitokthx's picture

Re: *THE* classic Unix horror story

sounds scary

Deep's picture

Re: *THE* classic Unix horror story

Good story for future newbie unix users. Well I am pretty sure now every unix course or company using unix sets rm -fr to disable accounts of people trying to run it or stop them from running it and in the current world only the admins have root password. But thanks for sharing the story

jasonla's picture

Re: *THE* classic Unix horror story

Not classic horrible yes, the admins did a good job of killing the process before it made it's way to the rest of the file system. I had a similar problem on windows in the 90's when malware didn't exist and viruses weren't bots but destroyed files. I went to start run telnet and it started searching for telnet.exe i knew something was wrong so went to my windows folder to look for it manually and didn't see it. How strange i thought i just saw a file disappear and another etc. Yup files were disappearing before my very eyes. It was as bad as rm -rf and i had no time to look and see what was doing it so i pulled the plug on my machine. It's what i would do had i seen rm -rf. Pull the power and figure out a plan. For windows i had norton on floppy and was able to clean it out i don't know what i lost but got the system back up and running. Unix wise i've ran redhat, slack, suse, ubuntu, freebsd, and sparc x86 i make it a point to have weekly backups of my drives and with out using recovery CD's or programs i do it almost manually copying files over to the drives so i know they are in a format i can access once recovered. If a friend ever threatened an rm -rf for $5 i would comply however his next trip home would be memorable i would fill his suite case with sex toys and synthetic marijuana and coke powder, it's legal and used to train drug sniffing dogs. It would make for a hell of a trip through TSA screening and possibly a trip to jail court or maybe even drug rehab i'd tell him he can add the $5 to the bail money i'd put up for him.

StevenO's picture

Re: *THE* classic Unix horror story

"If a friend ever threatened an rm -rf for $5 i would comply however his
next trip home would be memorable i would fill his suite case with sex
toys and synthetic marijuana and coke powder, it's legal and used to
train drug sniffing dogs. It would make for a hell of a trip through
TSA screening and possibly a trip to jail court or maybe even drug rehab i'd tell him he can add the $5 to the bail money i'd put up for him."

Ok, this is is hilarious.

I love these rm -rf stories. The 'classic one' in the original post is my favorite.

مدونة's picture

Re: *THE* classic Unix horror story

If one of my friends would do that to my terminal I would kick his ass.
I just can't imagine a rm -rf I couldn't do this to my worst enemy,
nobody deserves a rm -rf it's just pure evil

سيارات

An evil linux command comparable to a sin if you do it to a friend !!!

Editor:  Instead you had to be a gank and post a spam link for me to come break.  You have no right to talk about evil.

ewanm89's picture

Re: *THE* classic Unix horror story

Of course, the modern Linux way of grabbing the RIP, knoppix, mepis, ubuntu, gentoo, etc. CD has never failed for me in such situations.

danthehat's picture

Re: *THE* classic Unix horror story

Heh, ingenious and creative.  I would have given up about half way through (if that)

xargas's picture

Re: *THE* classic Unix horror story

If one of my friends would do that to my terminal I would kick his ass. I just can't imagine a rm -rf I couldn't do this to my worst enemy, nobody deserves a rm -rf it's just pure evil


rm -rf

An evil linux command comparable to a sin if you do it to a friend !!!

Editor:  Instead you had to be a gank and post a spam link for me to come break.  You have no right to talk about evil.

tylerl@drupal.org's picture

Re: *THE* classic Unix horror story

Er... boot disk?

LouisWinthorpe3's picture

Re: *THE* classic Unix horror story

This is without a doubt the most convoluted restore process I’ve ever heard of.  It sounds to me like the operators involved were too naïve when it came to the install process for 4.3BSD.  It really isn’t that involved.  Honestly, you have to wonder who installed their OS, and why didn’t they have access to the manuals?  I mean even if they paid for the tape only, wouldn’t they have printed out the system manuals?

I could have had that box up & running again in under an hour.  First thing to do would be to locate the install tape, and halt the machine.  Boot from tape, and run the copy program to install the miniroot into the swap partition.  Rebooting the VAX again, into the swap partition I would restore the root partition from one week old root backup.

One more reboot, and the VAX would have it’s root back from their best backup, then it would be a matter of restoring other directories as needed.

Oh and yes by default 4.3BSD has separate root partition from the rest of the userland stuff. 

I can only think that these people were absolutely terrified of rebooting their VAX.  I wonder if it was an 11/780 with a borked console tape/floppy.  

But who knows this is over 20 years ago.

Anonymous's picture

Funny

# rm -rf *

Ha ha.

Pretty funny.

Anonymous's picture

No, this is funny :

No, this is funny :

# sudo rm -rf /

Muhahaha

sandalle's picture

Except the "#" (generally)

Except the "#" (generally) denotes you're already root, so the sudo is unneeded. You probably meant

$ sudo rm -rf /

;)

Qasim's picture

Why not use tar

One way to avoid the assembler is to use a combination of tar and uuencode

Basically, if you transfer a tar file with only one directory in it (by manual typing) - you would be able to extract it using

tar xvf etc.tar -C /

Only issue is that the size is a bit larger - but I think if one gives enough thought to this - it could be reduced.

The key to this ofcourse - as you say, is to NOT PANIC.

joeblow1470's picture

Re: Why not use tar

tar is located at /bin/tar as it states in the story(about 1/3 of the way down)(bin was one of the deleted directories)

 

PS great story 

jfwfmt's picture

Re: *THE* classic Unix Horror story

not on Unix, on RT-11 but a close equivalent horror story:

One night I was finishing up the code for the VAX 8600 console. I had finally made the changes to the OS code and to the clock driver to translate the 64Hz interrupt rate to the 60Hz clock on the PDP-11 (with a little studder every 1/4 second) and to transfer the 1KHz clock through the mismatched registers windows to the VAX.

So everything was good and i decided to clean up and leave. I entered:

. delete *,tmp

It took a long time and then came back with:

tmp file not found.

Of course the comma rather than the intended dot made all the difference.

. directory

returned with nothing.

Not to worry, there were men in white coats who religiously took backups of everything every night,

Except they had never done a restore.

Which of course failed.

So I came in the next day and started over.

greffedecheveux's picture

Re: *THE* classic Unix Horror story

Yes i totaly agree with you actualy i can t see it any other way looking forward to see what others have to say about it regards. greffe de cheveux

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.