Remove duplicate files on Mac using Terminal

I tried to use this link to find and remove duplicates files (mp3’s) but was not successful. I spent days to make it work on Mountain Lion. Finally I got something that is partially working

First go to a subdirectory where you want to identify all the duplicates, copy and past this command (make sure there is no line break in this command – complete command should be in one line)

find . size +20 ! -type d -exec cksum {} ";" | sort | tee ./f.tmp | cut -f 1,2 -d ' ' | uniq -d 

This will create f.tmp file in the current folder inside tmp folder. It will contain all the files links and their md5 value. The list is sorted. You can easily guess how many duplicate files you have by checking the values of first two columns.

Next use the following command to identify the duplicate files only. Note if you have [ or ] characters, remove those from the files first!

cut -f 1,2 -d ' ' f.tmp | uniq | grep -v  -f f.tmp > dup.txt

The list of duplicate files will be dumped (copied) into dup.txt files. You can open it with nano editor or with any editor of your choice.

open -a texteditor dup.txt

You can then supply this file to find to delete the duplicate files.

 

Cut – (cut followed by dash) means read from input stream (that is not a file)

Complete Command in one go

This works on Mountain Lion OSX 10.8.3

find . -size +20 ! -type d -exec md5 -r {} ; | sort | tee ./f.tmp | cut -f 1 -d ' ' | uniq -d | grep - -hif ./f.tmp > dup.txt
Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

5 Responses to Remove duplicate files on Mac using Terminal

  1. adi says:

    Hi the last line does not work on my mountain lion.

    What is the second – in the grep command for?
    Cant get it to work :(…. seems that grep – – hit ./f.tmp > dup.txt does not work.

    Up to uniq -d every thing is fine, if i run it.

    Thanks Adrian

    • admin says:

      The terminal command is different for different version of Mac OS X. I finally gave up on this as it was too complicated. Deleting this files from terminal is even more challenging and risky.

      I would recommend to use a duplicate remove Apps from Apple Appstore. Esp Duplicator Detector which is a free App. Duplicate Dedetive is also another good app but cost $1.99 which is quite affordable.

  2. adi says:

    do have OS X 10.8.5

  3. CatoTShorter says:

    Well I am just going to say that I too had something very similar to what you have, worked very well under 10.6.

    I recently upgraded to 10.9 and now it will not work! The final grep in my version of finding duplicates will not work for some reason. Same as Adrian complained about above.

    At least I am not the only one, obviously Apple changed something fundamental to the grep command and this is very frustrating. Next time, ubuntu!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s