Conan: scm "url" = "auto" vs GitLab CI runner

[db4]

My CI scripts simultaneously build Windows and Linux binaries on an internal GitLab server using Gitlab runners. Both Windows and Linux runner uses docker if that matters. And most the time one of them fails to upload a package to an internal Artifactory CE repo:

Remote manifest:
Time: 2018-11-26 11:45:30
conanfile.py, MD5: 3dd4f4769e515e72d79466d548dd2d32

Local manifest:
Time: 2018-11-26 11:44:54
conanfile.py, MD5: d7535264b9001b772fdf0317e1f95aef

Local 'conanfile.py' using '\n' line-ends
Remote 'conanfile.py' using '\n' line-ends

----------------------------------------
Traceback (most recent call last):
  File "build.py", line 44, in <module>
    main()
  File "build.py", line 41, in main
    builder.run()
  File "/usr/local/lib/python2.7/dist-packages/cpt/packager.py", line 443, in run
    self.run_builds(base_profile_name=base_profile_name)
  File "/usr/local/lib/python2.7/dist-packages/cpt/packager.py", line 518, in run_builds
    r.run()
  File "/usr/local/lib/python2.7/dist-packages/cpt/runner.py", line 88, in run
    self._upload, package_id)
  File "/usr/local/lib/python2.7/dist-packages/cpt/uploader.py", line 43, in upload_packages
    retry=int(self._upload_retry))
  File "/usr/local/lib/python2.7/dist-packages/conans/client/conan_api.py", line 88, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/conan_api.py", line 818, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 68, in upload
    integrity_check, policy, remote_name, recorder)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 90, in _upload
    remote_manifest = self._check_recipe_date(conan_ref, recipe_remote)
  File "/usr/local/lib/python2.7/dist-packages/conans/client/cmd/uploader.py", line 165, in _check_recipe_date
    (remote_recipe_manifest.time, local_manifest.time))
conans.errors.ConanException: Remote recipe is newer than local recipe: 
 Remote date: 1543232730
 Local date: 1543232694

If I rerun the failing job manually, it succeeds. Any Idea how to fix/work around this? Both Windows and Linux docker instances are running on the same physical server, so time sync issues are hardly the problem. Even if they are, can I somehow prevent conan from detecting that difference and failing with an exception?

[db4]

My initial analysis was totally wrong. Here is an updated one. I have a recipe with

class BuildConan(ConanFile):
    scm = {
        "type": "git",
        "url": "auto",
        "revision": "auto",
        "submodule": "recursive"
    }

that I build with GitLab CI. "url": "auto" is supposed to be replaced by the real git repo url. But when GitLab runner executes CI job git url contain a unique job token needed to access a git repo: http://gitlab-ci-token:[email protected]/user/repo.git. This token changes each time CI job is run. So concurrent jobs generate different conanfile.py and I see the above output.

Replacing auto with the real url http://git.host.local/user/repo.git (gitlab-ci-token stripped) seems to help. Maybe conan should do that itself? (or at least issue a warning if it sees something@ in the scm.url?)

[lasote]

Thanks for the explanation. I think we cannot assume that Conan should clear always the credentials, it could happen a user actually wants them. Maybe we could add a new argument to the scm, something like clear_url_credentials: True/False (Default True?).

[jgsogo]

I think that those url credentials should be removed always... if the user wants to save inside the recipe the username and password, there are fields in the scm dict dedicated to that info.

[db4]

Yes, a new argument looks good to me. But maybe also check for some well known CI credentials like gitlab-ci-token:xxxxxxxx@ and strip them unconditionally (maybe with a warning)?

[db4]

Sorry, missed @jgsogo comment. For me, "removing always" (with a warning that url credentials have been removed) is even better.

[AliAskari]

@db4 Do you happen to have a link to the Dockerfile you used to create the Docker image for Windows? I have been trying to test it but it looks way more complicated than the Linux docker file, getting and installing msvc compiler is just a pain...

[db4]

@AliAskari You can find my Docker file here: https://github.com/conan-io/conan-docker-tools/issues/50#issuecomment-419881449

[lasote]

@jgsogo ok, we can remove them always. We can add the flag later if it is requested.

[lasote]

@AliAskari about docker windows, there is a WIP here. Please try to avoid mixing topics in the same issue if they are not related.

[lasote]

Hi! I think #3183 feature from @jgsogo could help with this issue. You could write a tinny function now to remove the credentials from the URL. I think that the mechanism is generic and good enough. Could this issue be closed when merged?

e.g:

import os
from conans import ConanFile
from conans.client.tools.scm import Git


def get_url():
    here = os.path.dirname(__file__)
    git = Git(here)
    tmp = git.get_remote_url()
    # REPLACE HERE YOUR URL
    return tmp

class MyLib(ConanFile):
    name = "issue"
    version = "3831"
    scm = {'type': 'git', 'url': get_url(), 'revision': "auto"}

[db4]

@lasote I would still prefer automatic credential removal by conan, proposed by @jgsogo. Yes, your function is tiny, but one should not forget to copy/paste it each time a new conan package is created. But if it's too hard, of course explicit git.get_remote_url() is a way to go.

[lasote]

Thanks for the feedback

[tonka3000]

Hey @lasote ,

I have exactly the same problem with gitlab and it already hurt alot across my projects because CI get really noisy.

I also think that a directl url cleanup in conan would be great, because copy paste the get_url it into every recipe is not that pratical. But I think you can make that opt-in, so f.e.

class BuildConan(ConanFile):
    scm = {
        "type": "git",
        "url": "auto",
        "revision": "auto",
        "submodule": "recursive",
        "credential_removal" : True # <= new option
    }

Maybe you can call it remove_sensitve_data or something like that.

The nice thing is that it is an opt-in feature, so every recipe author can decide if removal is required or not. It also wont fail in older conan version,but not removal would happen.

If you don't like the option, it would be nice to have a "ready to use" function in tools.

[lasote]

Will be released into 1.12. Thanks guys!