Problem Statement

Is your hard drive filled with backups of holiday pictures or downloaded images? How do you find and delete similar files if they have different formats, resolutions or rotations?
Is your hard drive filled with ripped or downloaded music? How do you find all the duplicates in your collection if they have different bit rates or formats?
How do you find duplicate text files or binary files on your computer?
Do you get a program to handle each case individually or would you rather have one program that does it all?
Here is my solution:

Introducing DuMP3

DuMP3 (derived from Duplicate MP3) is a Java program to find any duplicate or similar file.

It finds files by calculating a fingerprint based on the image, audio or text data for each file and then comparing the fingerprints. It does not compare filenames or even ID3 tags (even though plugin classes could be written that perform these operations). Calculated fingerprints can be stored in a MySQL database so that they do not have to be calculated again.

As an extra bonus DuMP3 will mark the files that can not be read or decoded correctly as corrupt or with a signature mismatch.

DuMP3 can find files that are not exact duplicates:

• Binary files are compared by SHA1 hash (configurable to any MD hash)
• Text files that were changed by addition or deletion (2 fingerprint algorithms available)
• Pictures in different formats, sizes and/or rotations (BMP, GIF, JPEG, JPEG2000, PNG, PNM, RAW, TIFF)
• Audio files that were recorded at different bit rates or saved in different formats (AU, AIF, WAV, MP3, OGG)
• plugin fingerprint classes can be written for any file where inexact matching is needed (fonts, videos, etc)

News

• 4 Apr 2008 DuMP3 0.3.2 released. fixed sql server delete bug [1924684], hotkeys for common functions [1906068], display database connect status in GUI [1906092]
• 14 Mar 2008 DuMP3 0.3.1 released. Stop button. Mark not duplicate fixed.
• 11 Mar 2008 DuMP3 0.3.0 released. database reorg - results are now saved in the database. Split the Debug messages from the display strings to simplify translation.
• 10 Aug 2007 DuMP3 0.2.16 released. SQLServer Integration complete.
• 1 Aug 2007 DuMP3 0.2.15 released. Turkish translation. Refactor so that SQLServer integration can start. Changed version display.
• 27 Dec 2006 DuMP3 0.2.14 is out. Mac OS X fixes. 1614710: build from source. Configuration values bug. MySQL disconnect bug. Text file display.
• 8 Dec 2006 DuMP3 0.2.13 is out. Fix Highlight bug. Fix I18N message bug. Files deleted or moved are now deleted or updated in the database. Indicator on splash screen when new version is available.
• 30 Nov 2006 DuMP3 0.2.12 is out. I18N completed - translators wanted. Highlight non-matching info.
• 26 Oct 2006 DuMP3 0.2.11 is out. Added feature 1581317 - Results layout option.
• 29 Sep 2006 DuMP3 0.2.10 is out. Better image display (gray scale), added buttons that act on both files. See FAQ for update to the database.
• 07 Sep 2006 DuMP3 0.2.9 is out. Fixed bug 1552651 - compare process stops at penultimate file. Added feature 1548866 - Temp trash bin. Some minor enhancements and speed-ups.
• 04 Sep 2006 DuMP3 0.2.8 is out. Fixed bugs 1549667 - Self shown as duplicate, 1548859 - Lost context. Added features 1548861 - Show filename in summary box, 1548860 - Bigger text for percentage match.
• 28 Aug 2006 DuMP3 0.2.7 is out. Better info handling. Sort by column on found extensions.
• 25 Aug 2006 DuMP3 0.2.6 is out. GUI works! Multiple OS builds.
• 10 Jul 2006 DuMP3 0.2.5 is out. Command line can scan multiple directories. Will use multiple processors fully.
• 15 Mar 2006 DuMP3 0.2.3 is out. Bug fix for some invalid MP3s.
• 04 Oct 2005 DuMP3 0.2.2 is out. Performance improvements and minor bug fixes.
• 16 Sep 2005 DuMP3 0.2.1 is out. Bug fix for WAV files. Added extra image formats.
• 05 Jul 2005 DuMP3 0.2.0 is out. Audio comparison is working correctly.
• 27 May 2005 DuMP3 0.1.1 is out. Picture comparison is working correctly.

Known Bugs/Issues

Some valid pictures are marked as corrupt because javax.imageio.ImageIO.read(File) does not understand the format. This is a limitation of Sun's implementation of the (old) JAI image decoders and not of DuMP3. Two examples are: CMYK encoded JPEG and RLE encoded BMP.

Some very large pictures (around 4000x4000 pixels) could cause DuMP3 to run out of memory and crash.

Sometimes GIF files contain JPEG data or vice versa. This occurs usually in images downloaded from websites. These pictures will be marked as having a signature mismatch but fingerprinting will still be attempted.

Limits

DuMP3 is subject to some Java limitations as well as some limitations in the libraries I have chosen. The F.A.Q. covers most of them.