Find Duplicate Files

Duplicate files take up unnecessary space on the disk. Fortunately, there are tools that automate the search for duplicates.

Fdupes

Install fdupes:

julio@acer ~> sudo pacman -S fdupes

Run fdupes in recursive mode (-r) and redirect the output to a file:

julio@acer ~/Documents/Ebooks> fdupes -r . > dupes1.txt

On my computer, this command took only 7 minutes to analyze 23500 files. The output file, dupes1.txt, had 5714 lines!

julio@acer ~/Documents/Ebooks> fdupes -rf . > dupes2.txt

It took about 7 minutes to analyze 23500 files: dupes2.txt: 3878 lines

Removing blank lines from dupes2.txt using sed -i '/^$/d' dupes2.txt, the file ended up with 2054 lines.

Many of the files it recognized as duplicates were intentionally identical. Examples of programming books are often repeated. Some version control files (git, svn, etc.) were recognized as duplicates, but should not be deleted.

If you want to reduce disk space usage but avoid breaking anything, you can create a script that replaces all duplicate files with hard links.

Delete all duplicate files (be careful with this script):

julio@acer ~/Documents/Ebooks> while read f; do rm "$f"; done < dupes2.txt

Gemini

A good paid alternative for Mac is Gemini, which lists all duplicates in a user-friendly interface and allows you to preview them before sending them to the trash.