Delete a million files?
Nov. 19th, 2008 10:24 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
hugmeOne of the devices I admin is an old Solaris 8 device... I was cleaning it up yesterday and found a directory of spam from 2002... good candidate for deletion... I did an `rm -Rf` on the directory and left it alone. Today I go and check to see if it's complete... nope, only 2% done, it had been running for 14 hours. This is odd...
A closer inspection of the directory and I found that there were in fact over 2,000,000 emails in this directory, each in their own directory and file. The system was taking about half a second PER FILE. There has to be a better way.
Some research online I got no result. There are a lot of people with this problem but there weren't any answers which worked. People had tried to script something in perl, php, shell... no result.
So I got to thinking about it, the `rm` command is single threaded... doesn't really use up that much cpu or disk.. so you can run MANY of them at once with out hurting the device (this is a production device btw). BUT if you try to run a rm command for EVERY file in the directory, THAT will hurt the device. What about running a number of remove commands at once, when one stops the next one can start. Sounds like a plan.
It works! here is the command I wrote... in this case the number 5 controls the number of commands you want it to run at the same time... right now I have it set for 50 and I'm deleting 60,000 files about every 2 minutes.
A closer inspection of the directory and I found that there were in fact over 2,000,000 emails in this directory, each in their own directory and file. The system was taking about half a second PER FILE. There has to be a better way.
Some research online I got no result. There are a lot of people with this problem but there weren't any answers which worked. People had tried to script something in perl, php, shell... no result.
So I got to thinking about it, the `rm` command is single threaded... doesn't really use up that much cpu or disk.. so you can run MANY of them at once with out hurting the device (this is a production device btw). BUT if you try to run a rm command for EVERY file in the directory, THAT will hurt the device. What about running a number of remove commands at once, when one stops the next one can start. Sounds like a plan.
It works! here is the command I wrote... in this case the number 5 controls the number of commands you want it to run at the same time... right now I have it set for 50 and I'm deleting 60,000 files about every 2 minutes.
while read X; do while [ "$(jobs | wc -l)" -gt 5 ]; do sleep 1; done; rm -R $X & done< <(ls)
