Conan: Travis, Bintray and Remotes

[jacmoe]

I am using a couple packages from Bintray in my project, and it works "on my machine" because I have added the remote to Conan.

But I can't seem to figure out how to add the remote to Conan on Travis..

I have this in my install.sh :

pip install conan_package_tools # It install conan too
conan remote add dimi https://api.bintray.com/conan/dimi309/conan-packages
conan user -p $BINTRAY_APIKEY -r dimi jacmoe

I tried just with conan user , but then I added my Bintray API key to Travis ...

Am I not able to use other people's packages on Travis?
Or, why does the remote get ignored?

Thanks :)

[jacmoe]

Is that the right way to log into Conan/Bintray?

Do I need to log in in order to just use the packages?

[jacmoe]

Probably related to #1380 ?

[memsharded]

Yes, very related to #1380. Yesterday I managed to upload to bintray from CI with:

appveyor.yml:

build: false

environment:
    PYTHON: "C:\\Python27"
    PYTHON_VERSION: "2.7.8"
    PYTHON_ARCH: "32"

    CONAN_UPLOAD: 1
    CONAN_REMOTE: myrepo 
    CONAN_REFERENCE: "HelloCi/0.1"
    CONAN_USERNAME: "memsharded"
    CONAN_CHANNEL: "testing"
    VS150COMNTOOLS: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Common7\\Tools\\"

    matrix:
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
          CONAN_CURRENT_PAGE: VisualStudio_15_x86_64
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
          CONAN_CURRENT_PAGE: VisualStudio_15_x86
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_CURRENT_PAGE: VisualStudio_14_x86_64
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_CURRENT_PAGE: VisualStudio_14_x86
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_CURRENT_PAGE: VisualStudio_12_x86_64
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
          CONAN_CURRENT_PAGE: VisualStudio_12_x86

install:
  - set PATH=%PATH%;%PYTHON%/Scripts/
  - pip.exe install conan_package_tools # It install conan too
  - pip.exe install conan --upgrade
  - conan remote add myrepo https://api.bintray.com/conan/memsharded/myrepo
  - conan user # It creates the conan data directory

test_script:
  - python build.py

Now, everything is a bit messy, we are working on it to stabilize, also adding some functionality into conan-package-tools. Thanks!

[jacmoe]

When I call conan remote add <stuff> it is totally ignored.
Luckily, it is not happening on a official Conan package, so it's okay. :)
I can live with a failing Travis build status, for now.

[memsharded]

Ignored? There are a few other things that could be:

• Added CONAN_REMOTE: myrepo?
• In travis.yml I had to add the conan remote add in a .travis/install.sh script, otherwise it was failing in OSX in travis.
• What are the message? is it possible than the reference, user, channel or password is not specified?

[jacmoe]

It is not ignored.
I tested that by adding it twice. :)
Then Conan complained that it already exists.

Strange thing is that it is not shown when running conan remote list and not taken into consideration when searching for packages.

[memsharded]

It is weird indeed.
Maybe you might want to check this repo: https://github.com/drodri/hello_ci
It has published to bintray succesfully for both appveyor and travis (linux & osx). There might be some tips there.

[jacmoe]

The message:

glfw/3.2.1@dimi309/stable: Not found, looking in remotes...
glfw/3.2.1@dimi309/stable: Trying with 'conan-center'...
glfw/3.2.1@dimi309/stable: Trying with 'conan-transit'...

ERROR: Unable to find 'glfw/3.2.1@dimi309/stable' in remotes

[jacmoe]

Thanks, I will take a look :)

[jacmoe]

My problem is that my package depends on other packages, not made by me, and not in either conan-center or conan-transit.

When I change CONAN_USERNAME, this interesting thing happens:

nasl/0.4@dimi309/stable export: Copied 13 '.h' files
nasl/0.4@dimi309/stable export: Copied 15 '.c' files
nasl/0.4@dimi309/stable export: Copied 1 '.txt' files: CMakeLists.txt
nasl/0.4@dimi309/stable: The stored package has not changed
glfw/3.2.1@dimi309/stable: Not found, looking in remotes...
glfw/3.2.1@dimi309/stable: Trying with 'conan-center'...
glfw/3.2.1@dimi309/stable: Trying with 'conan-transit'...

ERROR: Unable to find 'glfw/3.2.1@dimi309/stable' in remotes

It is interesting that my library is found, even though my username is 'jacmoe' :)

[memsharded]

Got it. This might require some extra variable or something in conan-package-tools to easily add other people remotes. cc/@lasote

[memsharded]

In the meantime, I think it is necessary to add the dimi309 user repo to your ci build scripts.

[lasote]

@dimi309 Did you deleted your packages from the conan.io before the migration right? It will take some time to review your include requests 😞 We are a little overwhelmed.

@jacmoe Try adding the @dimi309 remote https://api.bintray.com/conan/dimi309/conan-packages and tell us if there is any problem.

We are trying to prepare 0.24 and new conan package tools with improvements for CI, including easily adding remotes for other users etc.

[jacmoe]

That's what I did; or thought I did. See the first half of this issue's comment section where I was stumped that Conan ignored my conan remote add command :)

I will try and add it directly in the main Travis YAML file, because it does not get picked up by Conan if I do it in .travis/install.sh

I am using 3 CONAN_DOCKER tasks, and no matrix besides that, if that is relevant.

Anyway, I'll leave you to it 👍

And, @dimi309 has made a couple of great packages!
They work great anywhere but on Travis ;p

[ghost]

Hi @lasote,

( @jacmoe thank you for your kind words)

I didn't delete the conan.io packages. They are under the coding3d user which was my username:

https://www.conan.io/coding3d

However, I already started deploying the latest version directly to bintray before you migrated there, using a little hack:

https://twitter.com/dimi309/status/869415715455074305

On bintray however, I am using the dimi309 user, so that it matches my github username:

https://bintray.com/dimi309/conan-packages

The latest published commits worked as far as I know on travis & appveyor:

https://travis-ci.org/dimi309/conan-packages/builds/239140941?utm_source=github_status&utm_medium=notification

https://ci.appveyor.com/project/coding3d/conan-packages/build/1.0.90

I've recently switched glfw to build the master version rather than 3.2.1 (I needed that for some experiments with vulkan) and that does indeed crash on travis, but turning the conanfile back to v3.2.1 and updating the remotes, now that there has been a migration, should fix it.

Anyway, there is no hurry for this on my side. I just signaled that I'm taking good care of these packages and would like to make them available through conan-center if and when that will be possible.

[ghost]

Hi guys,

I think my package builds are crashing by something related perhaps to the issue you are discussing. Notice the error on travis ci at the end of this log:

https://travis-ci.org/dimi309/conan-packages/jobs/242278780

ERROR: Error while trying to get recipe sources for bzip2/1.0.6@lasote/stable. No remote defined

This wasn't happening last week... I can't figure out why, or if I'm doing something wrong...

[lasote]

@dimi309 It is weird, I'll take a look asap.

[lasote]

@jacmoe Are you using Docker? right? There is a problem with Docker and additional remotes, because the new remotes are not being configured in the container, We are trying to release a new package_tools with easy support for the remotes etc.

[jacmoe]

Yes, I am using Docker :)
Looking forward to the new package_tools.
Meanwhile I will experiment with the Travis build matrix 👍

[jacmoe]

Okay that works:

glfw/3.2.1@dimi309/stable: Not found, looking in remotes...
glfw/3.2.1@dimi309/stable: Trying with 'conan-center'...
glfw/3.2.1@dimi309/stable: Trying with 'conan-transit'...
glfw/3.2.1@dimi309/stable: Trying with 'dimi'...

Kind of:

WARN: Can't find a 'glew/2.0.0@dimi309/stable' package for the specified options and settings:

• Settings: arch=x86, build_type=Debug, compiler=apple-clang, compiler.version=8.1, os=Macos
• Options: shared=False
  ERROR: Missing prebuilt package for 'glew/2.0.0@dimi309/stable'
  Try to build it from sources with "--build glew"

How do I pass the "--build missing" argument to Conan?

[ghost]

You can pass it to the ConanMultiPackager constructor, like this:

https://github.com/dimi309/conan-packages/blob/master/small3d/build.py

[jacmoe]

That did the trick, thanks:)
Now, at least it builds when I don't use Docker.

[ghost]

You're welcome! Enjoy it while it lasts before I add vulkan support and everything goes to hell again :rofl:

[ghost]

... but I'm only joking. It suffices to install the LunarG sdk and force glfw to build locally.

[jacmoe]

*whew*
Well, in any case, I hope that you release new versions/packages when you add shiny, new features :)

[ghost]

I didn't write these libraries obviously :smile: apart from small3d, so I can't really add features to them. I only package them because I find it very efficient to pick them up from conan when I need them. But indeed, I try to use the latest version always, so the packages get updated accordingly. I don't use versioning on the github repositories though. I just make sure that the conanfiles can deploy any new (or older) version at any time.

[jacmoe]

I know that you didn't 😄
I used system installed glfw/glew before Conanizing my projects using your packages.
What I do (for Assimp) is to name the Conan version after the Assimp version, so when I switch to a new Assimp version, I release a new package.

[ghost]

That's pretty much what I do, if I have understood correctly. The conan version is the library version in effect and, in my case, there is no package source (i.e. conanfile) version.

[ghost]

I'm happy to hear that there's an assimp conan package by the way!

[lasote]

Well, I think I know what's the problem with both issues, @dimi309 and @jacmoe
We need to share the remotes registry between docker containers and host, so not only the remotes, we need the association between the recipes and it's remotes. I hope to have it fixed and released today in package tools.

[lasote]

Hi! I've released the conan package tools 0.4.0.
A couple of important changes:

• If you want to publish to your repository you need to declare the CONAN_UPLOAD variable but with the URL of the remote.
• If you need additional remotes, like the @dimi309 for the @jacmoe project, add it (multiple separated with ",") to CONAN_REMOTES environment variable.
• Remove any manual "conan remote add". It's not needed anymore.

Please give me feedback!

.travis.yml example using two bintray repositories, one to upload and an additional one:

env:
   global:
     - CONAN_REFERENCE: "lib/1.0"
     - CONAN_USERNAME: "lasote"
     - CONAN_CHANNEL: "stable"
     - CONAN_UPLOAD: "https://api.bintray.com/conan/luisconanorg/fakeconancenter"
     - CONAN_REMOTES: "https://api.bintray.com/conan/luisconanorg/conan-testing"

...

[jacmoe]

How do I authenticate when CONAN_UPLOAD is set ?

I have already added a Travis environment variable with my BINTRAY_APIKEY and set it to not show it's value in the build log.

Is there a CONAN_PASSWORD variable to set?

[lasote]

Yes, CONAN_PASSWORD with the api key.
I'm going to document it better in the readme.

[jacmoe]

The new Conan package tools does indeed work! 😄

Now I need to fix my own code too 😈

Solved 👍

[ghost]

Thanks @lasote !
Just a little suggestion: I think it would be good if the old functionality by which the upload was just ignored if no CONAN_PASSWORD is entered in travis or appveyor should be maintained. Now it seems to crash in that case:

https://ci.appveyor.com/project/coding3d/conan-packages/build/1.0.102/job/9s17wuhdgd09d4hg

[ghost]

The good news is that, indeed, beyond that, the builds work fine now 😃

https://travis-ci.org/dimi309/conan-packages/builds/242939122

https://ci.appveyor.com/project/coding3d/conan-packages/build/1.0.103