Conan: Occasional errors running: conan install --update

[rconde01]

I've seen this error twice now in our automated build system:

File "", line 10, in
File "", line 6, in run
File "conan\conans\client\command.py", line 1029, in main
File "conan\conans\client\command.py", line 943, in run
File "conan\conans\client\command.py", line 457, in install
File "conan\conans\client\manager.py", line 319, in install
File "conan\conans\client\manager.py", line 185, in _get_graph
File "conan\conans\client\deps_builder.py", line 281, in load
File "conan\conans\client\deps_builder.py", line 325, in _load_deps
File "conan\conans\client\deps_builder.py", line 391, in _create_new_node
File "conan\conans\client\proxy.py", line 151, in get_recipe
File "conan\conans\client\proxy.py", line 117, in _refresh
File "conan\conans\util\files.py", line 122, in rmdir
File "shutil.py", line 247, in rmtree
File "shutil.py", line 247, in rmtree
File "shutil.py", line 247, in rmtree
File "shutil.py", line 247, in rmtree
File "shutil.py", line 256, in rmtree
File "shutil.py", line 254, in rmtree
WindowsError: [Error 145] The directory is not empty: 'C:\cn\mypkg\x.x.x\me\stable\export\dir1\dir2\dir3\dir4'
conan returned -1

[memsharded]

Reported by @KindDragon:

We get this error often in random places:

opencv/3.1.1@******/stable: WARN: Forced build from source
ERROR: [Error 145] The directory is not empty: 'C:\\Users\\WinBuilder\\.conan\\data\\opencv\\3.1.1\\merlin-ext\\stable\\build\\22357e08bed3f975c745816eb96079b8fc475506\\opencv\\.git\\logs\\refs\\remotes'

grpc/0.15.0@******/stable: WARN: Forced build from source
ERROR: [Error 145] The directory is not empty: u'C:/.conan\\f8jip9\\1\\grpc\\tools\\dockerfile'

We run conan test_package --build=package_name --build=missing
Please help

[memsharded]

Seems that errors are quite related to running in CI servers, and Windows. Can you please @rconde01 , @KindDragon report on this?

• CI agent (jenkins?)
• CI Windows agent setup: Using some AntiVirus?
• Seen error sometime in regular developer machine?

Thanks very much for the feedback.

[KindDragon]

CI agent (jenkins?)

Yes

CI Windows agent setup: Using some AntiVirus?

No

Seen error sometimes in regular developer machine?

conan install works fine for me

[KindDragon]

This bug happens to me several times in the row:

grpc/0.15.0@*****/stable: WARN: Forced build from source
ERROR: [Error 145] The directory is not empty: u'C:/.conan\\2y2jxk\\1\\grpc\\third_party\\protobuf\\objectivec\\Tests\\CocoaPods\\iOSCocoaPodsTester\\iOSCocoaPodsTester.xcodeproj'

Maybe it related to path length? String is more than 129 characters long, 129 > 256/2. Maybe somewhere wrong path length check.

[KindDragon]

IDK, It's build without issues locally

[memsharded]

Hi @rconde01 , can you also give feedback about your automated build system? Is it Jenkins?

This jenkins issue (https://issues.jenkins-ci.org/browse/JENKINS-15331) has some explanation about Windows affecting the deletion of files. Windows can be quite annoying sometimes in these kind of things.

Which OS are you (@KindDragon, @rconde01) using for Windows jenkins agent? Is it a normal Windows 7/8/10, or is it a Windows Server? It might be some service that keeps analyzing/indexing files, and open them without releasing for normal conan operation.

I will check if conan could implement some kind of retry policy for removals, but in my experience, that doesn't help too much to solve the problem.

[memsharded]

Hi again,

I would also need to know the installation of conan into the Win build machines. pip, win-installer?

I haven't been able to replicate it here. It would be very interesting if you could try something in your build machines:

• First try to run from sources, not the installer. Just a routine check, but necessary for the next step
• Replace the conans/utils/files.py function rmdir and use the following code:

def rmdir(path):
    '''Recursive rm of a directory. If dir not exists
    only raise exception if raise_if_not_exist'''
    for _ in range(10):  # Max 2 seconds
        try:
            shutil.rmtree(path, onerror=_change_permissions)
            return
        except OSError as err:
            if err.errno == ENOENT:
                return
            if err.errno == ENOTEMPTY:
                with open("conan_fail_removal.txt", "a") as handle:
                    handle.write("Removal of %s failed, retrying\n" % path)
                time.sleep(0.2)
            else:
                raise

Then run the servers for a while, see if it works. check for the logged file with error traces.

[KindDragon]

Which OS are you (@KindDragon, @rconde01) using for Windows jenkins agent? Is it a normal Windows 7/8/10, or is it a Windows Server?

It normal Windows 8.
At the moment I temporarily workaround the problem by removing the cache each time

rmdir /s/q c:\Users\******\.conan\data\
rmdir /s/q c:\.conan\

[memsharded]

Thanks for the info @KindDragon. Will you be able to try the solution I posted above? Anything I can do to help in the process? Cheers

[memsharded]

I would like to have the feedback also from @claasd here, as from https://github.com/conan-io/conan/issues/904 I see that you are using Jenkins, and also the short_paths feature (so Windows should be):

Are you experiencing any of these problems in the removal of conan folders during Jenkins Win slave builds?

[claasd]

Hi,
I did not encounter this error yet.
We use Jenkins 2 with pipelines, Windows Server 2012 build nodes and conan installed via pip with python 3.

But then, we always use a fresh conan layout for each build, deleting the whole workspace with Jenkins commands (jenkins pipeline deleteDir command) before initializing conan with all environment variables pointing to subdirs in the workspace.
We only use the command to delete local sources and builds, without any problems yet:

conan remove * --builds --src --force

If you want me to try a special setup, let me know.

[memsharded]

Thanks @claasd for the feedback, that definitely sounds like a very good setup for CI.
No need that you try any special setup, the snippet above was only for those who are experiencing problems at folder removal, if you are not, it is not necessary at all.

[KindDragon]

@memsharded I reproduce error after several minor fixes for snipet:

Removal of C:\Users\WinBuilder\.conan\data\openni2\2.2\merlin-ext\stable\package\ca33edce272a279b24f87dc0d4cf5bbdcffbc187 failed, retrying

[memsharded]

So the error appeared and an Exception was raised? Or the fix captured the failure, retried and finally succeed, and only there was a trace in the log file? Would this then be a possible fix to the issue?

[KindDragon]

It failed after this because errno.EACCES error happens:
ERROR: [WinError 5] Access is denied: 'C:\Users\WinBuilder\.conan\data\openni2\2.2\merlin-ext\stable\package\ca33edce272a279b24f87dc0d4cf5bbdcffbc187\include'
I'll try again

[memsharded]

Yes, maybe extend the hack to cover that failure too:

if err.errno == ENOTEMPTY or err.errno == EACCES:

Or use different ifs, and produce different logs. Thanks for trying, you are doing a great job.

[KindDragon]

I have successfully completed building project twice with following new error on log:

Removal of C:/.conan\zj7b85\1 failed ENOTEMPTY, retrying

[memsharded]

So then, it seems confirmed:

• It is the Windows OS, keeping file handles open more time than it should
• Re-trying the removal for a while fixes it

Have you been running with the fix for a while, and everything seems good then?

So the question is how to integrate this in the conan codebase. For me, it reads like a bit ugly hack, but should be harmless in any case. Probably conditionally define the rmdir function, so the retry is an opt-in (maybe enabled with some environment variable, that the CI builder can set)

[KindDragon]

Have you been running with the fix for a while, and everything seems good then?

Yes

Probably conditionally define the rmdir function, so the retry is an opt-in (maybe enabled with some environment variable, that the CI builder can set)

I think at-least 2-3 retries should be by default.

[claasd]

Any update on this? We are running into this error frequently now, especially with large dependency trees.

[memsharded]

Hi @claasd,

Yes, it seems that the fix proposed to @KindDragon was kind of working. The culprit seemed Windows OS resource releasing (or the lack of it). Can you please confirm that this happens also in Windows only? Would you be able to apply the above patch to check if things improve?

To summarize it:

Replace the conans/utils/files.py function rmdir and use the following code:

def rmdir(path):
    '''Recursive rm of a directory. If dir not exists
    only raise exception if raise_if_not_exist'''
    for _ in range(10):  # Max 2 seconds
        try:
            shutil.rmtree(path, onerror=_change_permissions)
            return
        except OSError as err:
            if err.errno == ENOENT:
                return
            if err.errno == ENOTEMPTY or err.errno == EACCES:
                with open("conan_fail_removal.txt", "a") as handle:
                    handle.write("Removal of %s failed, retrying\n" % path)
                time.sleep(0.2)
            else:
                raise

This was kind of hacky, so we didnt include it, but if it is fully confirmed, we would add it to the ongoing release.
Thanks!

[claasd]

Hi,

I am indeed working on Windows only.
A workaround I came across is to change the path variable in conan.conf to a different disc.

[storage]
# This is the default path, but you can write your own
path = D:/.conandata

I will test your fix and give you feedback.

[memsharded]

That is very surprising that changing the unit will fix it. But who knows what windows is doing behind the curtains. We also found that some recent CMake versions also hold some reference and keep generating tlog files even after the CMake process has ended. Not sure what is happening, but happening on Windows only too.

Looking forward your feedback.

[memsharded]

Hi @claasd

Any news/feedback on this? We want to release a 0.23 RC for testing asap, because there are some features to check, and if this is confirmed, I would like to add it to the RC. Please tell me. Thanks!

[claasd]

Hi, the fix alone is not working, as my exception occures in _change_permissions; This does not throw an OSError but a plain Exception:

Traceback (most recent call last):
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\shutil.py", line 385, in _rmtree_unsafe
    os.rmdir(path)
OSError: [WinError 145] Das Verzeichnis ist nicht leer: 'C:\\Users\\cldieder\\.conan\\data\\CMakeMacros\\0.3.0-build.37\\cab\\testing\\export\\__pycache__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\cldieder\AppData\Local\Programs\Python\Python35\Scripts\conan.exe\__main__.py", line 9, in <module>
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\conan.py", line 6, in run
    main(sys.argv[1:])
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\command.py", line 1047, in main
    error = command.run(args)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\command.py", line 862, in run
    raise exc
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\command.py", line 840, in run
    method(args[0][1:])
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\command.py", line 310, in install
    no_imports=args.no_imports)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\manager.py", line 260, in install
    deps_graph = graph_builder.load(conanfile)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\deps_builder.py", line 288, in load
    loop_ancestors)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\deps_builder.py", line 348, in _load_deps
    new_options, new_loop_ancestors)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\deps_builder.py", line 360, in _load_deps
    new_options, new_loop_ancestors)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\deps_builder.py", line 345, in _load_deps
    new_node = self._create_new_node(node, dep_graph, require, public_deps, name)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\deps_builder.py", line 415, in _create_new_node
    reference=requirement.conan_reference)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\loader.py", line 71, in load_conan
    result = load_conanfile_class(conanfile_path)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\loader_parse.py", line 16, in load_conanfile_class
    loaded, filename = _parse_file(conanfile_path)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\client\loader_parse.py", line 60, in _parse_file
    rmdir(pycache)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\util\files.py", line 123, in rmdir
    shutil.rmtree(path, onerror=_change_permissions)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\shutil.py", line 387, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\util\files.py", line 115, in _change_permissions
    raise Exception()
Exception

[claasd]

When I change the thrown Exception in _change_permisions, the following error occurs:

File "c:\users\cldieder\appdata\local\programs\python\python35\lib\site-packages\conans\util\files.py", line 128, in rmdir
    if err.errno == ENOTEMPTY or err.errno == EACCES:
NameError: name 'ENOTEMPTY' is not defined

running python 3.5
EDIT: They were not included yet, ignore this comment

[memsharded]

Ok, ignoring latest comment. But the above trace is indicative, it seems a different failure of what @KindDragon was experience. This failing removal is the one done to remove the old "__pycache__" directory for python 3 when loading and parsing package recipes. Do you know if this is the cause of the failure in your case, always? Or is it failing also while removing build or package folders while creating packages? I am having a new look with this data, thanks!

[claasd]

Hi,
After adding the necessary includes and removing the throw Exception in the _change_permissions, I can confirm that the fix works for me.

[memsharded]

That is very weird, that your failure is in the _change_permissions. I will try to find the cause, fix it, and propose it for the release. Please keep the fix and keep me updated if the failure happens again, or maybe tomorrow after some builds check and report the contents of "conan_fail_removal.txt" file. Thanks very much!

[claasd]

No, I think its a functional bug.
If shutil.rmtree fails, you assume that it is a permission problem.
If permission is good (os.access(path, os_W_OK)) returns true, you just throw a plain exception.
I changed it the following way:

def _change_permissions(func, path, exc_info):
    import stat
    if not os.access(path, os.W_OK):
        os.chmod(path, stat.S_IWUSR)
        func(path)

[memsharded]

Submitted PR https://github.com/conan-io/conan/pull/1295. I have kept the raising of the Exception, but not a plain Exception, but the one causing the failure. Such exception is necessary for other scenarios of error handling.

If someone with a better Windows-fu, knows what is happening and any more elegant solution (this one is ugly), that would be great.

[claasd]

I think the problem is induced by the Windows Search indexer, a background program that indexes all files in certain directories for faster search. By Default, the full user folder (e.g. C:\Users\unsername) is indexed.
The indexer opens files, scans their content and closes them again. I remember having this problem on windows with svn in the user directory.

Once I turned off indexing for the subfolder .conan the problems disappeared.

To go around this problem, you could change the data folder to go into the AppData sub-directory (automatically excluded from windows search indexing) , or go to any other location on the system that is not below C:\Users.

Googling "turn off windows indexing for specific folder" or just "turn off windows indexing" for tons of pages that explain how to manually disable indexing for specific folders, you might add that to the documentation.

Another way could be to change this programmatically at the first conan run, some suggestions about how to do it are here:
https://social.technet.microsoft.com/Forums/Lync/en-US/bf99c428-7fbc-4cd5-9a4a-0c6b69e906ab/disable-allow-files-on-this-drive-to-have-contents-indexed-in-addition-to-file-properties?forum=ITCG

I hope this helps.