File Management Intermediate

Duplicate File Finder

Finds and reports duplicate files in directory. Detects files with same content using MD5 hash.

Published: May 02, 2024

Detailed Information

This script finds duplicate files in a directory. Compares file contents using MD5 hash and detects files with same content.

What Does This Script Do?

This script detects duplicate files:

  • Scans all files in directory
  • Calculates MD5 hash for each file
  • Finds files with same hash
  • Creates detailed report

Why Should You Use It?

Duplicate file finding saves disk space:

  • Disk Savings: Find and delete unnecessary copies
  • Organization: Clean up your file structure
  • Data Integrity: Detect different copies of same files

How to Use

Step-by-Step Usage Guide

1. Create Script File

nano find_duplicates.sh

2. Make Executable

chmod +x find_duplicates.sh

3. Run Script

./find_duplicates.sh /path/to/directory

4. Review Report

cat duplicates_*.txt

Requirements

Requirements

  • md5sum: MD5 hash calculation tool
  • File Access: Read permission to directory

Use Cases

Use Cases

1. Disk Cleanup

Find unnecessary copies and save disk space.

2. File Organization

Clean up and organize your file structure.

Examples

Usage Examples

Example 1: Home Directory Scan

./find_duplicates.sh /home/user

Code

#!/bin/bash

# Duplicate File Finder

if [ -z "$1" ]; then
    echo "Usage: $0 <directory>"
    exit 1
fi

DIR="$1"
REPORT="duplicates_$(date +%Y%m%d_%H%M%S).txt"

if [ ! -d "$DIR" ]; then
    echo "Error: Directory not found: $DIR"
    exit 1
fi

echo "Scanning for duplicate files..."
echo "This may take a while for large directories..."
echo ""

declare -A file_hashes
duplicates_found=0

while IFS= read -r -d "" file; do
    if [ -f "$file" ]; then
        hash=$(md5sum "$file" | cut -d" " -f1)
        if [ -n "${file_hashes[$hash]}" ]; then
            echo "DUPLICATE FOUND:" >> "$REPORT"
            echo "  Original: ${file_hashes[$hash]}" >> "$REPORT"
            echo "  Duplicate: $file" >> "$REPORT"
            echo "  Size: $(du -h "$file" | cut -f1)" >> "$REPORT"
            echo "" >> "$REPORT"
            ((duplicates_found++))
        else
            file_hashes[$hash]="$file"
        fi
    fi
done < <(find "$DIR" -type f -print0)

if [ $duplicates_found -eq 0 ]; then
    echo "No duplicates found!"
else
    echo "Found $duplicates_found duplicate(s)"
    echo "Report saved to: $REPORT"
fi

Usage

chmod +x find_duplicates.sh
./find_duplicates.sh /path/to/directory

Troubleshooting

Troubleshooting

Problem: "md5sum: command not found"

Solution: md5sum is usually installed by default. If missing:

sudo apt-get install coreutils

Tags

duplicate file finder md5 hash

Related Scripts