Conan: Support Azure CI templates

[uilianries]

Hi!

During this weekend @Croydon and I started to use Azure DevOps to build Conan packages. As Azure is growing with multi platform support it could nice to have it on CI templates using conan new

• [X] I've read the CONTRIBUTING guide.
• [ ] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[Croydon]

This could look like this: https://github.com/bincrafters/conan-templates/blob/6ae52eab9a9d8167ffbf368ec420ad8fffcc61b0/azure-pipelines.yml

[Croydon]

Which can be seen in action here: https://dev.azure.com/cr0ydon/conan-ogg/_build/results?buildId=31&view=logs

[kaylangan]

I'm a Program Manager for Azure Pipelines. Let me know if you have any questions or suggestions. I'm happy to help!

[Croydon]

Hi @kaylangan!
We have successfully created a Linux container matrix for our purpose, however, we can't see a way right now to create a mass building matrix for macOS with different xcode/apple-clang versions.

There is the xcode task but it doesn't look useful for our purpose. It allows to set a specific xcode version, but wants to execute a command directly on a xcode workspace, which won't work for us as we don't have a workspace file at this point of the build.

I'm guessing we would require something similar to the UsePythonVersion task: A UseXcodeVersion task, which allows us to select a specific xcode version, but then let's us run any arbitrary script (e.g. python scripts) which can make use of this xcode version.

[davidstaheli]

Hi @Croydon. I'm a Program Manager working with Kayla. A UseXcodeVersion task is a good idea. In lieu of that, would it work for you to run a script that dynamically sets the Xcode version you need? The concept is described here, but we should update the other Xcode docs you found to have this info too.

It would look something like this:

- job: macOS
  pool:
    vmImage: 'macOS-10.13'
  steps:
  - script: |
      /bin/bash -c "sudo xcode-select -s /Applications/Xcode_$(xcodeVersion).app/Contents/Developer"
      # Do something with Xcode/apple-clang here
  strategy:
    matrix:
      xcodeVersion8:
        xcodeVersion: 8.3.3
      xcodeVersion9:
        xcodeVersion: 9.4.1
      xcodeVersion10:
        xcodeVersion: 10

FYI @madhurig

[Croydon]

@DavidStaheli Thank you! This works great for Apple Clang 8.1, 9.0 and 9.1: https://github.com/bincrafters/conan-templates/commit/988950e4ea4cf2fc05752ca7bd68177271b45cc7

However, there seems to be no Apple Clang 10 available. From my understanding XCode 10 comes via default with Apple Clang (which is also backed up by this very handy list: https://gist.github.com/yamaya/2924292). Our build tools detect Apple Clang 9.1 when XCode 10 is getting selected however.

Is this an expected result? Is there a way right now to use Apple Clang 10 or is this work in progress?

Edit: This might be also a problem in our compiler (version) detection, I can't tell for sure right now.

[Croydon]

Turned out this was a simple typo (xcode version 10.0 instead of 10). macOS works now completely. Thanks for the help! 🎉

@DavidStaheli Is there a way to restart individual jobs? Our builds can take many hours. If single jobs fail because of e.g. network problems then it would be nice to be able to restart only them.

[davidstaheli]

Hi @Croydon - that's great news that you got it working! I'll see if we can make it easier to avoid the first issue that you hit (10 vs. 10.0).

About restarting jobs...
We're currently on 2 ways for you to do this. First, you'll be able to re-run jobs in the Azure Pipelines UI. I'm hoping that's available by the end of this year. Second, you'll be able to re-run a job from GitHub's Checks UI. We're working on that over the next 3-4 weeks and barring any complications, I'm hoping it's available to you in the first week of December. Sorry it's not available sooner!

[Croydon]

Sounds good!

I'm guessing we can complete the Azure templates as soon as we have working Windows docker containers in our build tools. We have working containers for MSVC 14 & 15. However, we would like to continue supporting MSVC 12 for now, but failed to setup a Docker container for it so far.

I know this is kinda off-topic and out of the scope of Azure, but by any chance do you know working MSVC 12 containers or could point us to useful resources @DavidStaheli?

[davidstaheli]

We could try creating a container image that has MSVC 12 on it. I see that old version can be downloaded here. Did you get the MSVC 14 and 15 containers from a public place? Their Dockerfiles might be a good way for us to get started with this. I can also track down someone from the MSVC team for their insight.

[uilianries]

Hi @DavidStaheli @kaylangan !

Both MSVC 14 and 15 are here:

https://github.com/uilianries/conan-docker-tools/blob/feature/docker-windows/msvc_14/Dockerfile
https://github.com/uilianries/conan-docker-tools/blob/feature/docker-windows/msvc_15/Dockerfile

They are simple images, but each image could reach 10GB.

You could check our logs here:
https://dev.azure.com/conanio/conan-docker-tools/_build/results?buildId=54&view=logs

The error is the same:

2018-11-08T17:29:30.9031783Z re-exec error: exit status 1: output: time="2018-11-08T17:29:30Z" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\979c52ffb1c9b7db682f8a1c34d95382bef036578a30f092c86b7bcd1510bbb7 flavour=1 folder=C:\\ProgramData\\docker\\tmp\\hcs120180935"
2018-11-08T17:29:30.9032406Z hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\?\C:\ProgramData\docker\windowsfilter\979c52ffb1c9b7db682f8a1c34d95382bef036578a30f092c86b7bcd1510bbb7 flavour=1 folder=C:\ProgramData\docker\tmp\hcs120180935

I have tried some solutions, as adding --isolation=hyperv but didn't work.

The Docker recipes are on my github account because I opened a PR:
https://github.com/conan-io/conan-docker-tools/pull/61

Do you know any way to solve this?

[uilianries]

Related issue: https://github.com/conan-io/conan-docker-tools/pull/61