Find duplicate files by content
Read-only
What do risk levels mean?
- Read-only
- Inspects state without changing anything.
- Low risk
- Reversible with a routine follow-up command.
- Medium risk
- Changes state; undo path documented.
- Destructive
- Deletes or overwrites; confirmation required.
Detects files with identical content by hashing them, so you can reclaim space from repeated downloads, copies, and exports.
filesduplicateshashcleanup
Parameters
Directory to scan for duplicates.
Commands
Hash every file and print those whose hash appeared before
find "." -type f -exec shasum -a 256 {} + | sort | awk 'seen[$1]++ {print}'Verification
find "." -type f -exec shasum -a 256 {} + | awk '{print $1}' | sort | uniq -d | wc -lThe number of content hashes that occur more than once; zero means no duplicates.
Undo
Not undoable
This recipe only reads and hashes files; deleting duplicates is a separate, deliberate step.
Pitfalls
- Hashing every file is slow on large trees; narrow the directory or pre-filter by size first.
- The first occurrence of each hash is not printed by the awk filter — every printed file has an unprinted original.
- Hard links to the same inode hash identically but consume no extra space.