Conan: tool.environment_append makes environment variables upper-case

[SSE4]

calling tools.environment_append in conan makes variables names upper-cased, e.g.:

ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)

becomes:

PROGRAMFILES=C:\Program Files (x86)
PROGRAMFILES(X86)=C:\Program Files (x86)

it causes troubles for certain build scripts, e.g. for boost 1.65 b2 fails to locate Program Files

[lasote]

environment_append is not uppercasing the variables. How do you find it? I assume you are on Windows. Did you print os.environ to see the uppercased variables? or...

[lasote]

It looks like it's a python thing, the environment variables in Windows are case insensitive, so Python is uppercasing them. In this StackOverflow thread, they recommend using the module nt.environ to get the original values: https://stackoverflow.com/questions/19023238/why-python-uppercases-all-environment-variables-in-windows

Edit: The strange thing is, if the Boost build script asks for "ProgramFiles", the operating system should return the right value, anyway. I need more information about how did you find the issue.

[SSE4]

@lasote
yes, I am running on Windows.
the original issue came up after I am upgraded to boost 1.65 and received build error when running b2.exe under conan, however, exactly same command had no issue when running in clean "Visual Studio Command Prompt"
similar issue reported https://github.com/boostorg/build/issues/230
so, b2 tries to fetch "ProgramFiles", but fails and doesn't go any further.
I was able to track down issue with simple code

env_build = VisualStudioBuildEnvironment(self)
with tools.environment_append(env_build.vars):
    self.run("set")

and compared to the "set" command output from the clean Visual Studio command prompt I have found that conan made all environment variables upper-cased, which was not expected

as for boost, I am not a b2/bjam expert, but I think it works the following way:

• b2 fetches all environment variables into dictionary
• b2 uses case-sensitive search in dictionary for "ProgramFiles"

[memsharded]

After reading https://stackoverflow.com/questions/19023238/why-python-uppercases-all-environment-variables-in-windows, I think that if python made the decision to make a case-insensitive search in the env-vars dictionary to have the same behavior of the OS, maybe it would make sense that b2 would also make a case-insensitive search of env-vars in its dictionary in Windows. What do you think @grafikrobot?

Even if conan could implement some workaround for Windows and use case-sensitive, I am afraid that this could break for existing users relying on this Windows case-insensitiveness.

[grafikrobot]

@memsharded It's not really possible for b2 to take the same approach as Python of upper-casing. The env variables come in as regular b2 variables which are case-sensitive. And making all of those upper-case would cause widespread breakage for b2 users. I'll have to see if there's some other work around in b2 for specific cases.

I personally think this is a Python bug. It's fine to read in and change the case. But changing the case when reflecting env vars back out is changing user state in an "invisible" way. And is something programs shouldn't be doing.

Note, that nothing short of implementing a work around in conan will help the current situation Boost 1.65. As that's a fixed release. And so is Python.

[grafikrobot]

@SSE4 To clarify how b2 works with env vars:

• B2 does not have the concept of dictionaries.
• B2 imports the env vars as regular b2 variables.
• B2 has case-sensitive variables (with almost unlimited allowed character set for variable names).

[memsharded]

Ok, makes sense. I will try to think of some workaround, but so far seems difficult not to make something very very ugly.

[grafikrobot]

@memsharded FYI.. For the particular case this came up in b2 it will be resolved for the next release in December (see https://github.com/boostorg/build/commit/9821622630035c0d7c3f4d7c196f847a335da2b6).

[SSE4]

@grafikrobot thanks a lot for the fix
@memsharded so, as b2 has a fix, what's the plan now, still add workaround to conan or just close the issue?

[danimtb]

I am reopening this issue as we faced another issue with a different recipe here:
https://github.com/conan-io/conan-center-index/issues/394#issuecomment-559739222

This could break other recipes that rely on the environment variables and we should discuss or look into this issue

[lasote]

@danimtb From my point of view, an issue without a fix is not an issue. Python is working like that. For the recipe I would recommend a patch.

[danimtb]

Yes, there is already a patch proposed for the recipe.

I believe that it is a python thing but the motivation of reopening is the fact that there are cases like the ones reported that have proved that commands run from the command line are not behaving the same as the ones run from the recipe.

Additionally, this makes recipes more complex as they have to make patches that are only needed to make the library compile from the recipe.

In this sense, I think it could make sense to improve the situation using nt.environ. However, I understand we would have to evaluate the risk and the value of such a fix.