Windows OS Hub
  • Windows
    • Windows 11
    • Windows Server 2022
    • Windows 10
    • Windows Server 2019
    • Windows Server 2016
  • Microsoft
    • Active Directory (AD DS)
    • Group Policies (GPOs)
    • Exchange Server
    • Azure and Microsoft 365
    • Microsoft Office
  • Virtualization
    • VMware
    • Hyper-V
  • PowerShell
  • Linux
  • Home
  • About

Windows OS Hub

  • Windows
    • Windows 11
    • Windows Server 2022
    • Windows 10
    • Windows Server 2019
    • Windows Server 2016
  • Microsoft
    • Active Directory (AD DS)
    • Group Policies (GPOs)
    • Exchange Server
    • Azure and Microsoft 365
    • Microsoft Office
  • Virtualization
    • VMware
    • Hyper-V
  • PowerShell
  • Linux

 Windows OS Hub / PowerShell / How to Find Duplicate Files Using PowerShell

June 8, 2023

How to Find Duplicate Files Using PowerShell

For one of the projects, I needed a PowerShell script to find duplicate files in the shared network folder of a file server. There are a number of third-party tools for finding and removing duplicate files in Windows, but most of them are commercial or are not suitable for automatic scenarios.

Because files may have different names but identical content, you should not compare files by name only. It is better to get hashes of all files and find the same ones among them.

The following PowerShell one-liner command allows you to recursively scan a folder (including its subfolders) and find duplicate files. In this example, two identical files with the same hashes were found:

Get-ChildItem –path C:\Share\ -Recurse | Get-FileHash | Group-Object -property hash | Where-Object { $_.count -gt 1 } | ForEach-Object { $_.group | Select-Object Path, Hash }

Finding Duplicate Files with PowerShell

This PowerShell one-liner is easy to use for finding duplicates, however, its performance is quite poor. If there are many files in the folder, it will take a long time to calculate their hashes. It is easier to compare files by their size first (it is a ready file attribute that doesn’t need to be calculated). Then we will get hashes of the files with the same size only:

$file_dublicates = Get-ChildItem –path C:\Share\ -Recurse| Group-Object -property Length| Where-Object { $_.count -gt 1 }| Select-Object –Expand Group| Get-FileHash | Group-Object -property hash | Where-Object { $_.count -gt 1 }| ForEach-Object { $_.group | Select-Object Path, Hash }

You may compare the performance of both commands on a test folder using the Measure-Command cmdlet:

Measure-Command {your_powershell_command}

For a folder containing 2,000 files, the second command is much faster than the first (10 minutes vs 3 seconds).

You can prompt a user to select duplicate files to be deleted. To do it, pipe a list of duplicate files to the Out-GridView cmdlet:

$file_dublicates | Out-GridView -Title "Select files to delete" -OutputMode Multiple –PassThru|Remove-Item –Verbose –WhatIf

powershell script for finding duplicate files in windows

A user may select files to be deleted in the table (to select multiple files, press and hold CTRL) and click OK.

Instead of deleting, you can move the selected files to another directory using Move-Item.

Also, you can replace duplicate files with hard links. The approach allows to keep files in place and significantly saves disk space.

param(
[Parameter(Mandatory=$True)]
[ValidateScript({Test-Path -Path $_ -PathType Container})]
[string]$dir1,
[Parameter(Mandatory=$True)]
[ValidateScript({(Test-Path -Path $_ -PathType Container) -and $_ -ne $dir1})]
[string]$dir2
)
Get-ChildItem -Recurse $dir1, $dir2 |
Group-Object Length | Where-Object {$_.Count -ge 2} |
Select-Object -Expand Group | Get-FileHash |
Group-Object hash | Where-Object {$_.Count -ge 2} |
Foreach-Object {
$f1 = $_.Group[0].Path
Remove-Item $f1
New-Item -ItemType HardLink -Path $f1 -Target $_.Group[1].Path | Out-Null
#fsutil hardlink create $f1 $_.Group[1].Path
}

To run the file, use the following format of the command:

.\hardlinks.ps1 -dir1 d:\folder1 -dir2 d:\folder2

The script can be used to find and replace duplicates of static files (that don’t change!) with symbolic hard links.

On Windows Server, you can use the built-in Data Deduplication feature of the File Server role to solve the problem of duplicating files. However, when using deduplication and incremental backup, you will have some troubles while restoring from the backup.

Also, you can use a console dupemerge tool to replace duplicating files with hard links.

2 comments
7
Facebook Twitter Google + Pinterest
PowerShellWindows 10Windows 11Windows Server 2019
previous post
Install and Configure PostgreSQL on Windows
next post
How to Check Who Restarted (Shutdown) Windows Server

Related Reading

Fix: Remote Desktop Licensing Mode is not Configured

August 24, 2023

Wi-Fi (Internet) Disconnects After Sleep or Hibernation on...

March 15, 2024

How to Install Remote Server Administration Tools (RSAT)...

March 17, 2024

How to Find the Source of Account Lockouts...

March 12, 2024

Managing Windows Firewall Rules with PowerShell

March 11, 2024

How to Delete Old User Profiles in Windows

March 15, 2024

Install and Manage Windows Updates with PowerShell (PSWindowsUpdate)

March 17, 2024

Start Menu or Taskbar Search Not Working in...

April 22, 2025

2 comments

karthur November 2, 2022 - 3:17 pm

I’m getting an error when I run this code.
$file_dublicates | Out-GridView -Title “Select files to delete” -OutputMode Multiple –PassThru|Remove-Item –Verbose –WhatIf

Out-GridView : Parameter set cannot be resolved using the specified named parameters.
At File location
+ … ublicates | Out-GridView -Title “Select files” -OutputMode Multiple – …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Out-GridView], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Microsoft.PowerShell.Commands.OutGridViewCommand

Reply
Gauthier June 10, 2024 - 12:55 pm

From Out-GridView docs: The PassThru parameter of Out-GridView lets you send multiple items down the pipeline. The PassThru parameter is equivalent to using the Multiple value of the OutputMode parameter.

Just remove -OutputMode Multiple and it works.

Reply

Leave a Comment Cancel Reply

join us telegram channel https://t.me/woshub
Join WindowsHub Telegram channel to get the latest updates!

Recent Posts

  • Configuring Windows Protected Print Mode (WPP)

    May 19, 2025
  • Map a Network Drive over SSH (SSHFS) in Windows

    May 13, 2025
  • Configure NTP Time Source for Active Directory Domain

    May 6, 2025
  • Cannot Install Network Adapter Drivers on Windows Server

    April 29, 2025
  • Change BIOS from Legacy to UEFI without Reinstalling Windows

    April 21, 2025
  • How to Prefer IPv4 over IPv6 in Windows Networks

    April 9, 2025
  • Load Drivers from WinPE or Recovery CMD

    March 26, 2025
  • How to Block Common (Weak) Passwords in Active Directory

    March 25, 2025
  • Fix: The referenced assembly could not be found error (0x80073701) on Windows

    March 17, 2025
  • Exclude a Specific User or Computer from Group Policy

    March 12, 2025

Follow us

  • Facebook
  • Twitter
  • Telegram
Popular Posts
  • Install and Manage Windows Updates with PowerShell (PSWindowsUpdate)
  • How to Download Offline Installer (APPX/MSIX) for Microsoft Store App
  • Fix: Remote Desktop Licensing Mode is not Configured
  • How to Delete Old User Profiles in Windows
  • How to Install Remote Server Administration Tools (RSAT) on Windows
  • Configuring Port Forwarding in Windows
  • Start Menu or Taskbar Search Not Working in Windows 10/11
Footer Logo

@2014 - 2024 - Windows OS Hub. All about operating systems for sysadmins


Back To Top