Powershell: Inconsistencies between Get-ChildItem and Remove-Item with respect to -Recurse and -Include

[mklement0]

_Update_: Based on @StingyJack's feedback, part of this issue may be related to a longstanding _known Remove-Item problem_ currently documented as:

When it is used with the Include parameter, the Recurse parameter might not delete all subfolders or all child items. This is a known issue.

---

Get-ChildItem allows the following idioms to look for matching files on _all_ levels of the target directory's hierarchy:

Get-ChildItem *.txt -Recurse
Get-ChildItem .  -Recurse -Include *.txt

By contrast, these idioms currently do _not_ work with Remove-Item:

Steps to reproduce

Set-Location ~

Get-ChildItem *.txt -Recurse
Get-ChildItem . -Recurse -Include *.txt

Remove-Item *.txt -Recurse -WhatIf
Remove-Item . -Recurse -Include *.txt -WhatIf

Expected behavior

All 4 commands should output / target the same set of files, across the directory hierarchy.

Actual behavior

The Get-ChildItem commands behave as expected.

The Remove-Item commands do not:

• Remove-Item *.txt -Recurse -WhatIf only targets items _directly_ in the target directory (does not recurse).

• Remove-Item . -Recurse -Include *.txt -WhatIf fails.
  E.g., on macOS: Remove-Item : Cannot remove the item at '/Users/jdoe' because it is in use.; i.e., an attempt is made to blindly remove ., without considering the -Include wildcard pattern.

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on macOS 10.12.4
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)

[StingyJack]

Remove-Item also does not appear to either sort the items for deletion by leaf-first or at least check if the item exists before deleting it.

The example pattern is to use Get-ChildItem and pipe the results

Get-ChildItem -Path "c:\temp\folder\* -Recurse | Remove-Item

but this will get "directory is not empty" and other errors, and you have to run the same thing 2 or more times to actually clean the folder out.

Its also does not carry over the behavior of rd /s or rmdir where a directory and all its contents must be removed. These two causes lead to code like the following being necessary to have a better chance (but not guarantee) of working.

if (Test-Path $folderPath)
{
  $items = Get-ChildItem -Path $folderPath -Recurse
  foreach ($item in $items)
  {
    if (Test-Path $item)
    {
      Remove-Item $item -Recurse -Force -Confirm:$false
    }
  }
  Remove-Item $folderPath -Force -Confirm:$false
}

[mickaeltr]

Hello @StingyJack

It seems that the issue is bigger than that (prior to Windows 10 1903, if I understood correctly)

This video seems to explain well the bug: https://www.youtube.com/watch?v=uhRWMGBjlO8&feature=youtu.be&t=455

And someone made a terrifying script 🤯 about it: https://stackoverflow.com/questions/53553729/cannot-remove-item-the-directory-is-not-empty/53561052#53561052

function Remove-FileSystemItem {
  <#
  .SYNOPSIS
    Removes files or directories reliably and synchronously.

  .DESCRIPTION
    Removes files and directories, ensuring reliable and synchronous
    behavior across all supported platforms.

    The syntax is a subset of what Remove-Item supports; notably,
    -Include / -Exclude and -Force are NOT supported; -Force is implied.

    As with Remove-Item, passing -Recurse is required to avoid a prompt when 
    deleting a non-empty directory.

    IMPORTANT:
      * On Unix platforms, this function is merely a wrapper for Remove-Item, 
        where the latter works reliably and synchronously, but on Windows a 
        custom implementation must be used to ensure reliable and synchronous 
        behavior. See https://github.com/PowerShell/PowerShell/issues/8211

    * On Windows:
      * The *parent directory* of a directory being removed must be 
        *writable* for the synchronous custom implementation to work.
      * The custom implementation is also applied when deleting 
         directories on *network drives*.

    * If an indefinitely *locked* file or directory is encountered, removal is aborted.
      By contrast, files opened with FILE_SHARE_DELETE / 
      [System.IO.FileShare]::Delete on Windows do NOT prevent removal, 
      though they do live on under a temporary name in the parent directory 
      until the last handle to them is closed.

    * Hidden files and files with the read-only attribute:
      * These are *quietly removed*; in other words: this function invariably
        behaves like `Remove-Item -Force`.
      * Note, however, that in order to target hidden files / directories
        as *input*, you must specify them as a *literal* path, because they
        won't be found via a wildcard expression.

    * The reliable custom implementation on Windows comes at the cost of
      decreased performance.

  .EXAMPLE
    Remove-FileSystemItem C:\tmp -Recurse

    Synchronously removes directory C:\tmp and all its content.
  #>
    [CmdletBinding(SupportsShouldProcess, ConfirmImpact='Medium', DefaultParameterSetName='Path', PositionalBinding=$false)]
    param(
      [Parameter(ParameterSetName='Path', Mandatory, Position = 0, ValueFromPipeline, ValueFromPipelineByPropertyName)]
      [string[]] $Path
      ,
      [Parameter(ParameterSetName='Literalpath', ValueFromPipelineByPropertyName)]
      [Alias('PSPath')]
      [string[]] $LiteralPath
      ,
      [switch] $Recurse
    )
    begin {
      # !! Workaround for https://github.com/PowerShell/PowerShell/issues/1759
      if ($ErrorActionPreference -eq [System.Management.Automation.ActionPreference]::Ignore) { $ErrorActionPreference = 'Ignore'}
      $targetPath = ''
      $yesToAll = $noToAll = $false
      function trimTrailingPathSep([string] $itemPath) {
        if ($itemPath[-1] -in '\', '/') {
          # Trim the trailing separator, unless the path is a root path such as '/' or 'c:\'
          if ($itemPath.Length -gt 1 -and $itemPath -notmatch '^[^:\\/]+:.$') {
            $itemPath = $itemPath.Substring(0, $itemPath.Length - 1)
          }
        }
        $itemPath
      }
      function getTempPathOnSameVolume([string] $itemPath, [string] $tempDir) {
        if (-not $tempDir) { $tempDir = [IO.Path]::GetDirectoryName($itemPath) }
        [IO.Path]::Combine($tempDir, [IO.Path]::GetRandomFileName())
      }
      function syncRemoveFile([string] $filePath, [string] $tempDir) {
        # Clear the ReadOnly attribute, if present.
        if (($attribs = [IO.File]::GetAttributes($filePath)) -band [System.IO.FileAttributes]::ReadOnly) {
          [IO.File]::SetAttributes($filePath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
        }
        $tempPath = getTempPathOnSameVolume $filePath $tempDir
        [IO.File]::Move($filePath, $tempPath)
        [IO.File]::Delete($tempPath)
      }
      function syncRemoveDir([string] $dirPath, [switch] $recursing) {
          if (-not $recursing) { $dirPathParent = [IO.Path]::GetDirectoryName($dirPath) }
          # Clear the ReadOnly attribute, if present.
          # Note: [IO.File]::*Attributes() is also used for *directories*; [IO.Directory] doesn't have attribute-related methods.
          if (($attribs = [IO.File]::GetAttributes($dirPath)) -band [System.IO.FileAttributes]::ReadOnly) {
            [IO.File]::SetAttributes($dirPath, $attribs -band -bnot [System.IO.FileAttributes]::ReadOnly)
          }
          # Remove all children synchronously.
          $isFirstChild = $true
          foreach ($item in [IO.directory]::EnumerateFileSystemEntries($dirPath)) {
            if (-not $recursing -and -not $Recurse -and $isFirstChild) { # If -Recurse wasn't specified, prompt for nonempty dirs.
              $isFirstChild = $false
              # Note: If -Confirm was also passed, this prompt is displayed *in addition*, after the standard $PSCmdlet.ShouldProcess() prompt.
              #       While Remove-Item also prompts twice in this scenario, it shows the has-children prompt *first*.
              if (-not $PSCmdlet.ShouldContinue("The item at '$dirPath' has children and the -Recurse switch was not specified. If you continue, all children will be removed with the item. Are you sure you want to continue?", 'Confirm', ([ref] $yesToAll), ([ref] $noToAll))) { return }
            }
            $itemPath = [IO.Path]::Combine($dirPath, $item)
            ([ref] $targetPath).Value = $itemPath
            if ([IO.Directory]::Exists($itemPath)) {
              syncremoveDir $itemPath -recursing
            } else {
              syncremoveFile $itemPath $dirPathParent
            }
          }
          # Finally, remove the directory itself synchronously.
          ([ref] $targetPath).Value = $dirPath
          $tempPath = getTempPathOnSameVolume $dirPath $dirPathParent
          [IO.Directory]::Move($dirPath, $tempPath)
          [IO.Directory]::Delete($tempPath)
      }
    }

    process {
      $isLiteral = $PSCmdlet.ParameterSetName -eq 'LiteralPath'
      if ($env:OS -ne 'Windows_NT') { # Unix: simply pass through to Remove-Item, which on Unix works reliably and synchronously
        Remove-Item @PSBoundParameters
      } else { # Windows: use synchronous custom implementation
        foreach ($rawPath in ($Path, $LiteralPath)[$isLiteral]) {
          # Resolve the paths to full, filesystem-native paths.
          try {
            # !! Convert-Path does find hidden items via *literal* paths, but not via *wildcards* - and it has no -Force switch (yet)
            # !! See https://github.com/PowerShell/PowerShell/issues/6501
            $resolvedPaths = if ($isLiteral) { Convert-Path -ErrorAction Stop -LiteralPath $rawPath } else { Convert-Path -ErrorAction Stop -path $rawPath}
          } catch {
            Write-Error $_ # relay error, but in the name of this function
            continue
          }
          try {
            $isDir = $false
            foreach ($resolvedPath in $resolvedPaths) {
              # -WhatIf and -Confirm support.
              if (-not $PSCmdlet.ShouldProcess($resolvedPath)) { continue }
              if ($isDir = [IO.Directory]::Exists($resolvedPath)) { # dir.
                # !! A trailing '\' or '/' causes directory removal to fail ("in use"), so we trim it first.
                syncRemoveDir (trimTrailingPathSep $resolvedPath)
              } elseif ([IO.File]::Exists($resolvedPath)) { # file
                syncRemoveFile $resolvedPath
              } else {
                Throw "Not a file-system path or no longer extant: $resolvedPath"
              }
            }
          } catch {
            if ($isDir) {
              $exc = $_.Exception
              if ($exc.InnerException) { $exc = $exc.InnerException }
              if ($targetPath -eq $resolvedPath) {
                Write-Error "Removal of directory '$resolvedPath' failed: $exc"
              } else {
                Write-Error "Removal of directory '$resolvedPath' failed, because its content could not be (fully) removed: $targetPath`: $exc"
              }
            } else {
              Write-Error $_  # relay error, but in the name of this function
            }
            continue
          }
        }
      }
    }
}

Stack Overflow

Cannot remove item, The Directory is not empty

When using the Remove-Item command, even utilizing the -r and -Force parameters, sometimes the following error message is returned: Remove-Item : Cannot remove item C:\Test Folder\Test Folder\Ta...

[mklement0]

@mickaeltr Your referring to #8211, which will ultimately only be resolved once the WinAPI file-removal calls become synchronous (you can use the function as a workaround, but it has limitations).

[mickaeltr]

Thanks @mklement0 for the link and explanation.