Conan: [bug] access denied while renaming unzipped folders

Created on 18 Feb 2020  路  14Comments  路  Source: conan-io/conan

This issue is the same than https://github.com/conan-io/conan/issues/5205

Environment Details (include every applicable attribute)

  • Operating System+version: Windows 10
  • Conan version: 1.22.2

  • Python version: 3.5.4

    • Windows indexing is disabled for .conan folder

Steps to reproduce (Include if Applicable)

  • run conan create or conan install such a way it has to build from source.
  • use recipe and version where source code is a big package (in size or number of files), for me it happens all the time with physx and cspice (waiting merge in conan-center-index), sometimes with qt or boost, and with different computers (but always on Windows 10).

Logs (Executed commands with output) (Include/Attach if Applicable)

ERROR: physx/4.1.1: Error in source() method, line 80
        os.rename(extracted_dir, self._source_subfolder)
        PermissionError: [WinError 5] Access denied: 'PhysX-ae80dede0546d652040ae6260a810e53e20a06fa' -> 'source_subfolder'

Adding a try / except + sleep few seconds is a workaround, since this permission denied seems to be ephemeral, or a loop with few retries.

It would be nice if conan could provide a tools.rename in order to handle these kinds of issues on Windows.

look into
Source
picture SpaceIm

Most helpful comment

I've checked again a conan create with physx and antivirus disabled (on my personnal computer, can't do that on corporate desktop), I don't have permission denied error in this case.

picture SpaceIm on 25 Feb 2020
👍2

All 14 comments

Hi @SpaceIm could you please enable CONAN_VERBOSE_TRACEBACK and force that error again? So we will have more information.

set CONAN_VERBOSE_TRACEBACK=1

I would say some service is running in your background and locking PhysX folder. e.g anti-virus.

Also, did you check if os.replace results on the same behavior?

Regards!

uilianries picture uilianries on 18 Feb 2020

@uilianries, here is traceback:

ERROR: Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python37-32\lib\site-packages\conans\errors.py", line 34, in conanfile_exception_formatter
    yield
  File "c:\users\user\appdata\local\programs\python\python37-32\lib\site-packages\conans\client\source.py", line 137, in _run_source
    conanfile.source()
  File "C:\Users\user\.conan\data\physx\4.1.1\_\_\export\conanfile.py", line 85, in source
    os.rename(extracted_dir, self._source_subfolder)
PermissionError: [WinError 5] Access denied: 'PhysX-ae80dede0546d652040ae6260a810e53e20a06fa' -> 'source_subfolder'
SpaceIm picture SpaceIm on 18 Feb 2020

Thanks @SpaceIm

uilianries picture uilianries on 18 Feb 2020

A couple of things:

Windows indexing is disabled for .conan folder

As pointed in the other issue, maybe the AntiVirus could be related as well? Is it excluding the cache?

@uilianries Can you please try to reproduce with those recipes in Windows? The try-sleep sounds too much of a workaround, I'd prefer to know a bit better the reasons of this failure and see if something else could be done.

memsharded picture memsharded on 19 Feb 2020
👍1

that makes me extremely suspicious, is that Windows allows to rename locked file (but doesn't allow to delete them). usually, you can rename running executable, for instance, or rename folder containing that executable as well.

SSE4 picture SSE4 on 20 Feb 2020

I've checked again a conan create with physx and antivirus disabled (on my personnal computer, can't do that on corporate desktop), I don't have permission denied error in this case.

SpaceIm picture SpaceIm on 25 Feb 2020
👍2

This issue is the same than #5205

Environment Details (include every applicable attribute)

  • Operating System+version: Windows 10
  • Conan version: 1.22.2

  • Python version: 3.5.4

  • Windows indexing is disabled for .conan folder

Steps to reproduce (Include if Applicable)

  • run conan create or conan install such a way it has to build from source.
  • use recipe and version where source code is a big package (in size or number of files), for me it happens all the time with physx and cspice (waiting merge in conan-center-index), sometimes with qt or boost, and with different computers (but always on Windows 10).

Logs (Executed commands with output) (Include/Attach if Applicable)

ERROR: physx/4.1.1: Error in source() method, line 80
        os.rename(extracted_dir, self._source_subfolder)
        PermissionError: [WinError 5] Access denied: 'PhysX-ae80dede0546d652040ae6260a810e53e20a06fa' -> 'source_subfolder'

Adding a try / except + sleep few seconds is a workaround, since this permission denied seems to be ephemeral, or a loop with few retries.

It would be nice if conan could provide a tools.rename in order to handle these kinds of issues on Windows.

See my pull request: https://github.com/conan-io/conan/pull/6774

yangcha picture yangcha on 1 Apr 2020

In another context I've observed the permission denied error and there I've tried to isolate the problem. With the following python script I was able to produce the error nearly 100%:

import errno
import os
import platform
import shutil
import stat
import time


def isWindows():
  """ Returns True when running with the native port of Python for Windows,
  False when running on any other platform (including the Cygwin port of
  Python).
  """
  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
  return platform.system() == "Windows"


def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      else:
        raise
  else:
    os.rename(src, dst)


def rmtree(path, ignore_errors=False):
  """shutil.rmtree(path) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  onerror = None
  if isWindows():
    path = path
    onerror = handle_rmtree_error
  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)


def handle_rmtree_error(function, path, excinfo):
  # Allow deleting read-only files
  os.chmod(path, stat.S_IWRITE)
  function(path)


def rename_test():
  if os.path.isdir("old_dir"):
    rmtree("old_dir")
  os.mkdir("old_dir")
  for i in range(100):
    file = "old_dir/test_" + str(i).zfill(3) + ".txt"
    #print("Generate " + file)
    with open(file,"w") as fp:
      for n in range(100):
        fp.write(str(i).zfill(3) + "." + str(n).zfill(3) + ": 01234567890 01234567890 0123456789\n")
  print("Rename old_dir => new_dir")
  rename("old_dir", "new_dir")
  rmtree("new_dir")


# Problem occured for: D:/ma/pkg/baseswp\\.git.tmp' -> 'D:/ma/pkg/baseswp\\.git
for i in range(100):
  print("Rename test " + str(i))
  rename_test()
  • I've run the tests also in a directory where no Windows indexing service is looking into => error still occurs
  • I've checked the Virus Scanner logs, but could not find an entry

=> So my conclusion so far is that maybe the file operations (before the call to os.rename()) return already before the file handle is released (async.).

As a workaround I've implemented a retry loop with waiting 500 ms between retries:

def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      elif e.errno == errno.EACCES:
        # Workaround for the sporadic problem on Windows:
        # PermissionError: [WinError 5] Access denied
        # Retry 3 times to rename with sleeping 500 ms in between
        retry = 3
        while retry:
          #print("*** Retry: " + str(4 - retry))
          time.sleep(0.5)
          try:
            os.rename(src, dst)
            retry = 0
          except OSError as e:
            if retry and e.errno == errno.EACCES:
              retry = retry - 1
            else:
              raise
      else:
        raise
  else:
    os.rename(src, dst)
jokram picture jokram on 21 Jul 2020
👍1

@jokram thanks for your detailed feedback, indeed seems be a problem related to async access, the old sleep / wait is good workaround.

uilianries picture uilianries on 21 Jul 2020

In another context I've observed the permission denied error and there I've tried to isolate the problem. With the following python script I was able to produce the error nearly 100%:

import errno
import os
import platform
import shutil
import stat
import time


def isWindows():
  """ Returns True when running with the native port of Python for Windows,
  False when running on any other platform (including the Cygwin port of
  Python).
  """
  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
  return platform.system() == "Windows"


def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      else:
        raise
  else:
    os.rename(src, dst)


def rmtree(path, ignore_errors=False):
  """shutil.rmtree(path) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  onerror = None
  if isWindows():
    path = path
    onerror = handle_rmtree_error
  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)


def handle_rmtree_error(function, path, excinfo):
  # Allow deleting read-only files
  os.chmod(path, stat.S_IWRITE)
  function(path)


def rename_test():
  if os.path.isdir("old_dir"):
    rmtree("old_dir")
  os.mkdir("old_dir")
  for i in range(100):
    file = "old_dir/test_" + str(i).zfill(3) + ".txt"
    #print("Generate " + file)
    with open(file,"w") as fp:
      for n in range(100):
        fp.write(str(i).zfill(3) + "." + str(n).zfill(3) + ": 01234567890 01234567890 0123456789\n")
  print("Rename old_dir => new_dir")
  rename("old_dir", "new_dir")
  rmtree("new_dir")


# Problem occured for: D:/ma/pkg/baseswp\\.git.tmp' -> 'D:/ma/pkg/baseswp\\.git
for i in range(100):
  print("Rename test " + str(i))
  rename_test()
  • I've run the tests also in a directory where no Windows indexing service is looking into => error still occurs
  • I've checked the Virus Scanner logs, but could not find an entry

=> So my conclusion so far is that maybe the file operations (before the call to os.rename()) return already before the file handle is released (async.).

As a workaround I've implemented a retry loop with waiting 500 ms between retries:

def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      elif e.errno == errno.EACCES:
        # Workaround for the sporadic problem on Windows:
        # PermissionError: [WinError 5] Access denied
        # Retry 3 times to rename with sleeping 500 ms in between
        retry = 3
        while retry:
          #print("*** Retry: " + str(4 - retry))
          time.sleep(0.5)
          try:
            os.rename(src, dst)
            retry = 0
          except OSError as e:
            if retry and e.errno == errno.EACCES:
              retry = retry - 1
            else:
              raise
      else:
        raise
  else:
    os.rename(src, dst)

In another context I've observed the permission denied error and there I've tried to isolate the problem. With the following python script I was able to produce the error nearly 100%:

import errno
import os
import platform
import shutil
import stat
import time


def isWindows():
  """ Returns True when running with the native port of Python for Windows,
  False when running on any other platform (including the Cygwin port of
  Python).
  """
  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
  return platform.system() == "Windows"


def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      else:
        raise
  else:
    os.rename(src, dst)


def rmtree(path, ignore_errors=False):
  """shutil.rmtree(path) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  onerror = None
  if isWindows():
    path = path
    onerror = handle_rmtree_error
  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)


def handle_rmtree_error(function, path, excinfo):
  # Allow deleting read-only files
  os.chmod(path, stat.S_IWRITE)
  function(path)


def rename_test():
  if os.path.isdir("old_dir"):
    rmtree("old_dir")
  os.mkdir("old_dir")
  for i in range(100):
    file = "old_dir/test_" + str(i).zfill(3) + ".txt"
    #print("Generate " + file)
    with open(file,"w") as fp:
      for n in range(100):
        fp.write(str(i).zfill(3) + "." + str(n).zfill(3) + ": 01234567890 01234567890 0123456789\n")
  print("Rename old_dir => new_dir")
  rename("old_dir", "new_dir")
  rmtree("new_dir")


# Problem occured for: D:/ma/pkg/baseswp\\.git.tmp' -> 'D:/ma/pkg/baseswp\\.git
for i in range(100):
  print("Rename test " + str(i))
  rename_test()
  • I've run the tests also in a directory where no Windows indexing service is looking into => error still occurs
  • I've checked the Virus Scanner logs, but could not find an entry

=> So my conclusion so far is that maybe the file operations (before the call to os.rename()) return already before the file handle is released (async.).

As a workaround I've implemented a retry loop with waiting 500 ms between retries:

def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      elif e.errno == errno.EACCES:
        # Workaround for the sporadic problem on Windows:
        # PermissionError: [WinError 5] Access denied
        # Retry 3 times to rename with sleeping 500 ms in between
        retry = 3
        while retry:
          #print("*** Retry: " + str(4 - retry))
          time.sleep(0.5)
          try:
            os.rename(src, dst)
            retry = 0
          except OSError as e:
            if retry and e.errno == errno.EACCES:
              retry = retry - 1
            else:
              raise
      else:
        raise
  else:
    os.rename(src, dst)

I have tried this sleep/waiting method, it seems it does not always work.

yangcha picture yangcha on 21 Jul 2020

@yangcha:

I have tried this sleep/waiting method, it seems it does not always work.

=> Do you think improving the no. of retries will help?

jokram picture jokram on 21 Jul 2020

@yangcha:

I have tried this sleep/waiting method, it seems it does not always work.

=> Do you think improving the no. of retries will help?

I don't think so. If you look into Python's os.rename() code, you will see the folders are recursively created and copied and deleted. Unless you can add the waiting into Python's library code, I don't think it will solve the problem entirely.
I have a solution which is robust and reliable: https://github.com/conan-io/conan/pull/6774

yangcha picture yangcha on 21 Jul 2020

@yangcha please, check #6774 people reviewed there but you didn't answer.

uilianries picture uilianries on 21 Jul 2020

@yangcha please, check #6774 people reviewed there but you didn't answer.

Sorry. I thought the PR was rejected. I will answer that and resolve the conflicts .

yangcha picture yangcha on 21 Jul 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Adnn picture Adnn  路  3Comments
mpdelbuono picture mpdelbuono  路  3Comments
rconde01 picture rconde01  路  3Comments
uilianries picture uilianries  路  3Comments
liberforce picture liberforce  路  3Comments