Conan: [question] issue around conan locks - unable to proceed without clearing local cache.

[fulara]

@memsharded I think this is caused by me using your local branch.
It seems the file:
xmlparser/2.41.3/_/_/metadata.json.lock..
EDIT:
It seems that even doing something like:

rm -rf $( find $CONAN_USER_HOME/.conan/data/ -iname '*.lock' )

Doesnt help.

Environment Details (include every applicable attribute)

• Operating System+version: centos7
• Compiler+version: gcc 8
• Conan version: 1.26.0-dev - using memshared fix/multi_mode_package_id
• Python version: 3.16.10
  using bundled conan with pyinstaller.

Steps to reproduce (Include if Applicable)

Hard to say..
I was jumping between conan versions back and forth between 1.23.0 1.24.0 and 1.26.0-dev - however I think I was only building my artifacts with the 1.24.0 and 1.26.0-dev.

Logs (Executed commands with output) (Include/Attach if Applicable)

conan create . --build missing
...
xmlparser/2.41.3 is locked by another concurrent conan process, wait...
If not the case, quit, and do 'conan remove --locks'
...
$ conan remove --locks
Cache locks removed

doesnt help - ends up in the same issue.

[fulara]

EDIT:
it seems unrecoverable altogether...

$ rm -rf  $CONAN_USER_HOME/.conan/data/ 
$ conan create . --build missing
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
Exporting package recipe
...
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=RelWithDebInfo
compiler=gcc
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os.version=rh7
os_build=Linux
os_build.release=rh7
[options]
[build_requires]
[env]

xmlparser/2.41.3 is locked by another concurrent conan process, wait...
If not the case, quit, and do 'conan remove --locks'

doing rm -rf $CONAN_USER_HOME/.conan seems to moved things forward.

[memsharded]

Yes, those locks that can be removed with conan remove --locks are those to protect the packages in the cache, and they live in <home>/.conan/locks. This is why removing only the data folder, doesn't fix it.

That lock of the metadata.json file is not removed by this command, and it is the first situation that we see it causing a block. Only readers-writer locks cause the issue that requires the conan remove --locks and the metadata one is a mutex one.

Besides that, as you said, very difficult to know what could be happening...

[fulara]

I think I got into this bad state because I ctrl+c after one of my builds failed, or maybe I have ctrl+c'd into it.

Anyway I am closing the ticket - just wanted to inform you that this sort of thing has hasppened! :)

[fulara]

Just a final comments here, @memsharded
You can get into this state by pressing 'Ctrl+C'.
This is unrecoverable state it seems - I tried to:

$ cd $CONAN_USER_HOME/.conan
$ rm -rf $(find -iname '*\.lock')

But it still didnt help.

There is no 'locks' directory or the locks file.

[fulara]

Please reopen this @memsharded .

THis is critical issue that caused productio issue for us.

Easily reproducible with pressing of ctrl+c.

I can provide you a detailed scednario

[memsharded]

Hi @fulara

Sure, please let me know: Is the issue that some interrupted installation leaves the locks there, or that conan remove --locks doesn't work as expected and does not clean the locks correctly?

[fulara]

Thanks for prompt response as usual @memsharded

I was trying to reproduce the issue for last 10 minutes, but no success. I'll try during next week.

How I think I came across this issue:

invoke conan install
press ctrl+C
....
invoke conan install

You will end up with:

**is locked by another concurrent conan process, wait...**
**If not the case, quit, and do 'conan remove --locks'**
And doing the conan remove --locks: 
**$ conan remove --locks**
**Cache locks removed**

Does not help.
The only solution I found is to remove the whole conan directory $CONAN_USER_HOME and reinstall it.
As mentioned in the original post I am using bundled conan on redhat 6.

OFFTOP not related to this post:
different_conan_ctrl_c_issue.txt

I am now trying to reproduce with conan 1.26.0 - if something has been changed in locks / ctrl+c logic since then please tell me.

I will try to learn a way to reproduce this and come back to you,
But I essentially think that you have internal locks in databases or something like that and ctrl+c cause you not to delete them.

[memsharded]

Hi @fulara

Ok, that makes sense. The conan remove --locks should make the cache usable again, no matter what. If this is not happening, this is a bug and must be fixed. Lets have a look, please keep us posted with any new info you find.

No, up to my knowledge nothing has changed with locks recently.

Many thanks for reporting!

[fulara]

I have done over 200 hundred ctrl+cs now, came across 3 different issues but no one of those was the lock based.
I must be doing something wrong because i got it often in the past, anyway will get back to you as soon as i get corrupted cache.

[jgsogo]

I'm trying to reproduce the issue without success 😞 It is easy to block Conan and get a xyz is locked by another concurrent conan process, wait...... but I'm always able to work around it running conan remove --locks which would be the real issue.

Blocking Conan is as easy as terminating the process while Conan is building a package (Ctrl+C usually is not enough, the contextmanager exit functions are called), this can happen for different reasons: docker container is stopped, running an IDE plugin, the IDE decides to kill the process... But then, conan remove --locks from the console always works or raises an error. The implementation of this command is quite straightforward:

1. It computes the path to every package in the cache: https://github.com/conan-io/conan/blob/c829789c2cd96a888494e6b994de37b52d26e707/conans/client/cache/cache.py#L222

2. It removes the files related to locks: https://github.com/conan-io/conan/blob/c829789c2cd96a888494e6b994de37b52d26e707/conans/util/locks.py#L38

   If the files are owned by a different user, if the running process cannot remove them, at least in MacOS I always get an exception from os.remove that is propagated. Not the best error, but at least something useful:

   Traceback (most recent call last): File "/Users/jgsogo/dev/conan/conan/conans/client/command.py", line 2103, in run method(args[0][1:]) File "/Users/jgsogo/dev/conan/conan/conans/client/command.py", line 1116, in remove self._conan.remove_locks() File "/Users/jgsogo/dev/conan/conan/conans/client/conan_api.py", line 94, in wrapper return f(api, *args, **kwargs) File "/Users/jgsogo/dev/conan/conan/conans/client/conan_api.py", line 1081, in remove_locks self.app.cache.remove_locks() File "/Users/jgsogo/dev/conan/conan/conans/client/cache/cache.py", line 226, in remove_locks Lock.clean(conan_folder) File "/Users/jgsogo/dev/conan/conan/conans/util/locks.py", line 41, in clean os.remove(folder + ".count") PermissionError: [Errno 13] Permission denied: '/Users/jgsogo/.conan/data/fmt/7.0.3/_/_.count'

[jgsogo]

Ok, there is a way to reproduce it, but I think it is not a reachable status using only Conan commands:

If we run, for example, a conan create fmt/7.0.3@conan/stable and interrupt the process so we get the file /Users/jgsogo/.conan/data/fmt/7.0.3/conan/stable.count with a value different from 0. Now we are in a status where every command involving fmt/7.0.3@conan/stable will fail and Conan asks us to run conan remove --locks first.

Now, we remove manually the folder containing the package: rm -fr /Users/jgsogo/.conan/data/fmt/7.0.3/conan/stable/. This creates a wrong status in the Conan cache: running conan remove --locks won't notice about the existence of <cache>/fmt/7.0.3/conan/stable.count and <cache>/fmt/7.0.3/conan/stable.count.lock because the folder <cache>/fmt/7.0.3/conan/stable doesn't exist. The files won't be removed and we are locked forever.

I cannot imagine how to reach this status...

[jgsogo]

Hi! It's been quite a few weeks since we had a look at this issue. Even though I was able to get to the issue reported state, it was running several manual operations forcing the failure.

Conan v2.0 will implement a new cache and the locking mechanism will be refactored. It is not worth the effort to refactor the locking mechanism in 1.x without evidence that the error is hitting several users consistently.

Thanks for reporting, we expect Conan 2.0 to have a bulletproof locking mechanism 🚀