Conan: Build procedure looking for %CL% as source file on appveyor?

[ghost]

Hello,

I got the following error while rebuilding my fork of SDL2_ttf on appveyor:

c1 : fatal error C1083: Cannot open source file: '%CL%': No such file or directory [C:\Users\appveyor\.conan\data\SDL2_ttf\2.0.14\coding3d\ci\build\4124c994fe89dd8db9389d5e3304fc536dd00df6\SDL2_ttf-2.0.14\VisualC\SDL_ttf.vcxproj] 13 Warning(s) 1 Error(s) Time Elapsed 00:00:01.41 SDL2_ttf/2.0.14@coding3d/ci: ERROR: Package '4124c994fe89dd8db9389d5e3304fc536dd00df6' build failed SDL2_ttf/2.0.14@coding3d/ci: WARN: Build folder C:\Users\appveyor\.conan\data\SDL2_ttf\2.0.14\coding3d\ci\build\4124c994fe89dd8db9389d5e3304fc536dd00df6 ERROR: SDL2_ttf: Error 1 while executing call "%vs120comntools%../../VC/vcvarsall.bat" amd64 && cd SDL2_ttf-2.0.14\VisualC && SET LIB=%LIB%;"C:\Users\appveyor\.conan\data\SDL2\2.0.4\lasote\stable\package\269437c061f87eb35361f278ac7b07b3de6f3bfc\lib";"C:\Users\appveyor\.conan\data\freetype\2.6.3\lasote\stable\package\f99bde292d3e2081fa54ca8bf8a0775e6f518f71\lib";"C:\Users\appveyor\.conan\data\bzip2\1.0.6\lasote\stable\package\c4918833eab9bf878573de52c2369fa4fdcbd8f0\lib";"C:\Users\appveyor\.conan\data\libpng\1.6.21\lasote\stable\package\71216f8a3ce0e621eed5c053c417fdcab8134684\lib";"C:\Users\appveyor\.conan\data\zlib\1.2.8\lasote\stable\package\e0a02d496bbb652b6295152dfce0d3937acc0b56\lib" && SET CL=%CL% /I"C:\Users\appveyor\.conan\data\SDL2\2.0.4\lasote\stable\package\269437c061f87eb35361f278ac7b07b3de6f3bfc\include" /I"C:\Users\appveyor\.conan\data\freetype\2.6.3\lasote\stable\package\f99bde292d3e2081fa54ca8bf8a0775e6f518f71\include" /I"C:\Users\appveyor\.conan\data\bzip2\1.0.6\lasote\stable\package\c4918833eab9bf878573de52c2369fa4fdcbd8f0\include" /I"C:\Users\appveyor\.conan\data\libpng\1.6.21\lasote\stable\package\71216f8a3ce0e621eed5c053c417fdcab8134684\include" /I"C:\Users\appveyor\.conan\data\zlib\1.2.8\lasote\stable\package\e0a02d496bbb652b6295152dfce0d3937acc0b56\include" && msbuild SDL_ttf.sln Error while executing: call "%vs120comntools%../../VC/vcvarsall.bat" amd64 && conan test_package . -s arch="x86_64" -s build_type="Release" -s compiler="Visual Studio" -s compiler.runtime="MD" -s compiler.version="12" -o SD2_ttf:shared="True" --build missing

So for some reason, it seems that the system thinks that %CL% is a source file?
I am not 100% sure that the error got introduced by the latest conan version, but it did happen after the new version announcement.

[memsharded]

I am fine re-building other packages with conan 0.13.1, so might be something related to that package.

[ghost]

It's only on Windows that it happens. And it happened again in another one (small3d) but when it tried to build sdl2_ttf, as it detected that it was missing. Linux and Macos builds are ok. I'll let you know if I find out something further...

[ghost]

Of course I might just drop sdl2_ttf, since it has given me a lot of grief in more ways than one, and start using freetype directly. 😈

[memsharded]

Thanks for the feedback, will try to have a look if possible, now I am a bit busy with Opencv :)

[ghost]

GLFW for the win! ✌️ 😬 Maybe I'll try to upload it.
(Don't worry, it's not serious, I'm just playing around with my game-making. I'm just letting you know of what I encounter, in case it helps discover bugs! 😄 )

[ghost]

sdl2_ttf has many hidden traps for the package creator. Proceed with care! ☠️

[memsharded]

There are already some packages for GLFW. https://www.conan.io/search?q=glfw

I have used/collaborated @GavinNL package successfully

[ghost]

Cool! I might try it. I am not sure yet but I was thinking of switching to GLFW anyway. I'm reading images directly with libpng and if I also read fonts straight through freetype, there's not much SDL will be doing for me other than window management and input, which GLFW can do just fine.

[ghost]

I've fixed it. I just wrote in the conanfile.py:

env_line = env.command_line.replace("%CL%", '')

In effect, where the conan-created env.command_line says "set CL= %CL% etc etc", adding values to the CL variable but trying to keep the existing ones by starting with %CL%, I basically tell it not to try to keep existing info. It's a hack, but it works. The error however could signify that there is indeed a problem somewhere. I'm just not sure what or where it is...

[ghost]

I guess that the appveyor environment just doesn't recognise the %CL% variable (most likely imho) or something that's in it is not right, due to the specific package, or one of its dependencies... But it does say "Cannot open source file: '%CL%'" so it looks like it thinks that %CL% is not a variable but an actual literal I suppose...

[memsharded]

Yes, seems a bug, even if it is not defined, the behavior should be better that that.

[ghost]

... and the question is, whose bug? Conan's? Appveyor's?

[ghost]

Travis is very slow tonight. Maybe the whole conan audience has attacked it with multiple rebuilds 😎

[memsharded]

Been able to replicate it locally in my Win machine. Definitely conan bug.

[ghost]

It did just start happening after the upgrade... Well at least we know, and until there is a fix, there's a way to hack around it. :smile_cat:

[memsharded]

Yes, sure. Happy to hear that, one of the main design goals of conan was that it was easily hackable and almost any problem could be workarounded by users. Investigating, but it is not obvious for me right now...

[ghost]

The goal has been achieved! There are never any blocking issues.

[ghost]

Hi @memsharded

I've noticed there have been some fixes for this, but I think there are still problems. Trying to build my conan-vorbis library on Windows, initially I got an error "\Microsoft was unexpected at this time" at the point where the LIB and CL variables are set. I thought that the backslashes in the paths were perhaps causing the issue, so I added this to my conanfile:

env_line = env.command_line.replace("\\", "/'')

That helped the build pass the point of setting env, but then when building the VS projects I got an error:

error MSB6001: Invalid command line switch for "link.exe". Length cannot be less than zero.

The vcxproj files looked ok on a quick inspection, but I guess that some variable was not quite right, as the error suggested.

I also tried this sort of thing in the vorbis conanfile:

env_line = env.command_line.replace('%ENV%', '"%ENV%"')

Again, this passed the \Microsoft unexpected point without error, but I got the "link.exe" error again.

So I suppose that the way LIB and CL are set is still not perfect... I have found another one of my hacks that allows my package to build, which is removing the %CL% and %LIB% variables completely (thus not linking to pre-existing values):

env_line = env.command_line.replace("%CL%", '')
env_line = env.command_line.replace("%LIB%", '')

So it's not a problem for my package, but this is a solution that specifically works with vorbis because it only directly links to OGG. So you might still want to investigate how conan sets the env variables on Windows. If you would like to use my package for tests, note that I am making these changes only in my develop branch for now:

[memsharded]

Hi!
Thanks very much for such a detailed report. It is true, we are having problems with ConfigureEnvironment in Windows for Visual Studio, which is used by just a very few, but it is causing problems.

It happens that the Win command line and batch files are incredible tricky, not to say a nightmare. We have been working the last two days, made some improvements, but still not able to release and fix the issue. Hopefully tomorrow or the day after we will be able to release a minor with these fixes.

Thanks again for the pointers and your package for testing, we will use them to check everything. Cheers!

[ghost]

I can imagine it's complicated. I prefer to use cmake but ogg & vorbis don't, unfortunately 😆

[ghost]

I've made a mistake; with the latest version of conan, this is enough for a workaround:

env_line = env.command_line.replace("%LIB%", '')

(In any case the first of the two replacement lines I mention in my previous message has no effect, since I am replacing in env.command_line on both of them :| )

[memsharded]

Hi Dimitri,

There is a working fix in https://github.com/conan-io/conan/pull/547/files, in case you want to run it from sources and check if it is working. We have added more comprenhensive testing of ConfigureEnvironment, but if you wan to test with those packages, that would be great. If not, don't worry, we will try to release a minor soon. Thanks!

[ghost]

Hi James,

Success!!!! 🎉

I've made a branch of conan-vorbis in which I'm using your fix_cl2 conan branch from source in appveyor and have removed my workaround from the package's conanfile. Things are looking good indeed! https://ci.appveyor.com/project/coding3d/conan-vorbis/build/1.0.84

Whenever you guys release, ping me and I'll remove my hack from all my packages and rebuild everything. That will be a good final test 🤓

[ghost]

I've relaunched all my builds without the hack. Everything's ok now :+1:

This one produced a problem initially:

https://ci.appveyor.com/project/coding3d/conan-sdl2-ttf/build/1.0.21/job/06go1lya4whp8f8i

However, it was because vcvars was empty so I removed it from the conanfile:

https://github.com/dimi309/conan-sdl2_ttf/commit/96e3c47ef4109c5c1851b3daab444b4d97d9ecf6

I'm not sure this is a problem because I wasn't using vcvars in my other packages. (I've forked this package from @lasote ). If it is, I suppose we can open another issue.

[memsharded]

Nop, the idea is to leave it! Without it, the package will not compile if not using conan-package-tools, that set the vcvars environment. Just check for it having a value:

cmd = vcvars_command(self.settings)
if cmd:
    cmd = cmd + " && "

With that, if vcvars is already set by conan-package tools, it will do nothing, if not, it will set it.

[memsharded]

Thanks very much for trying BTW! Glad that everything worked fine :)

[ghost]

Cool! I'll do it like that. ( after beer time :D )

[ghost]

I'm glad too! :D Thanks for all the help!

[ghost]

Ok, I've re-introduced vcvars together with an "emptiness check". Everything builds fine. I'll keep vcvars in mind for other packages as well. Thanks again!