Conan: [bug] activate.sh/deactivate.sh does not work in cygwin

Created on 11 Mar 2020  ·  17Comments  ·  Source: conan-io/conan

Environment Details (include every applicable attribute)

  • Operating System+version: Windows 10 Enterprise Ver 10.0 Build 17763
  • Conan version: 1.22.2
  • Python version: 3.7.6
  • Cygwin DLL version info:
    DLL version: 3.0.7
    DLL epoch: 19
    DLL old termios: 5
    DLL malloc env: 28
    Cygwin conv: 181
    API major: 0
    API minor: 338
    Shared data: 5
    DLL identifier: cygwin1
    Mount registry: 3
    Cygwin registry name: Cygwin
    Installations name: Installations
    Cygdrive default prefix:
    Build date:
    Shared id: cygwin1S5

Steps to reproduce

  • Open cygwin
  • Create an empty directory

  • In that directory I've created a conanfile.txt with the following content (the package is a simple helloworld.exe to check if the PATH is properly set):

    [build_requires]
helloworld/1.0.0@conti_gen_dev1/testing

[generators]
virtualenv
  • Run conan install .

  • This will generate the following files which might be used in cygwin:

    • activate.sh:

      #!/usr/bin/env sh
export CONAN_OLD_PATH="$PATH"

while read -r line; do
  LINE="$(eval echo $line)";
  export "$LINE";
done < "D:\casdev\_test\conan_poc\conan_test_shell\environment.sh.env"

export CONAN_OLD_PS1="$PS1"
export PS1="(conanenv) $PS1"

    • deactivate.sh:

      !/usr/bin/env sh
export PS1="$CONAN_OLD_PS1"
unset CONAN_OLD_PS1


export PATH="$CONAN_OLD_PATH"
unset CONAN_OLD_PATH
    • environment.sh.env:
      PATH="D:\.conan\614de7\1\bin"${PATH+:$PATH}
  • Now in Cygwin when calling source activate.sh the following errors are displayed:
    -bash: $'\r': command not found -bash: activate.sh: line 7: syntax error near unexpected token `done' 'bash: activate.sh: line 7: `done < "D:\casdev\_test\conan_poc\conan_test_shell\environment.sh.env"
  • Similar problem exists for source deactivate.sh

Notes

  • If I call source activate.sh in a Git Bash then it works fine, so it seems to be a problem only for cygwin
  • If I modify the line endings in activate.sh and deactivate.sh from CR LF to LF then it also works in cygwin
  • See also issue #2568

Proposal

  • activate.sh and deactivate.sh should always use LF as line endings independent of the OS
  • This should work in Git Bash, cygwin, Linux
medium medium bug
Source
picture jokram

All 17 comments

Thanks for the detailed report @jokram

I managed to reproduce this issue. The solution would seem at first sight that is easy to solve, but I found this in the write_generators():

def write_generators(conanfile, path, output):
    """ produces auxiliary files, required to build a project or a package.
    """
    for generator_name in conanfile.generators:
        ...
                for k, v in content.items():
                    v = normalize(v) # THIS!!!!!
                    output.info("Generator %s created %s" % (generator_name, k))
                    save(join(path, k), v, only_if_modified=True)

So the forcing to CRLF is done elsewhere, not in the generator, this will make this a bit more complicated to fix quickly. I will tentatively put it to next release, to check if there is some relatively easy way to do this, but depending on the investigation we might need to re-plan.

memsharded picture memsharded on 11 Mar 2020

Hi, yes, this is something we are doing when running in Windows, all files are normalized to \r\n. Here I found two solutions: https://stackoverflow.com/a/14607651

  • user-side: an environment variable SHELLOPTS=igncr in Windows, or it can also be set in .bash_profile.
  • Conan side: if Windows, we can add this magic line (set -o igncr) 2>/dev/null && set -o igncr; # this comment is required to the sh scripts (I haven't tested it)

Otherwise, we can annotate each generator class with the needed behavior and avoid the v = normalize(v) for some of them.

jgsogo picture jgsogo on 12 Mar 2020

Conan side: if Windows, we can add this magic line (set -o igncr) 2>/dev/null && set -o igncr; # this comment is required to the sh scripts (I haven't tested it)

I am going to test this, that could be a good workaround.

Otherwise, we can annotate each generator class with the needed behavior and avoid the v = normalize(v) for some of them.

Yes, this is probably the way to go, but a bit more complex. I'll have a look too.

memsharded picture memsharded on 12 Mar 2020

I've tested the workaround and it seems to work. 😃
Add the following code in the ~/.bash_profile:

#!/usr/bin/bash
os="$(uname -s)"
case "${os}" in
    Linux*)     os=Linux;;
    Darwin*)    os=Mac;;
    CYGWIN*)    os=Cygwin;;
    MINGW*)     os=MinGw;;
    *)          os="UNKNOWN:${os}"
esac
if [ "$os" = "Cygwin" ]; then
    # Ignore line endings
    export SHELLOPTS
    set -o igncr
fi

Now the commands work:

$ source activate.sh
(conanenv)
/d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv)
/d/casdev/_test/conan_test_shell
$ source deactivate.sh

/d/casdev/_test/conan_test_shell
$ helloworld
-bash: helloworld: Kommando nicht gefunden.
jokram picture jokram on 12 Mar 2020

Good! Thanks for reporting it.

I have tried the (set -o igncr) 2>/dev/null && set -o igncr;, but I couldn't make it work reliably, it keeps failing. I am going to try Conan not normalizing to CRLF in Windows those files.

memsharded picture memsharded on 12 Mar 2020

I am trying the CRLF approach, but I am realizing that I also need to do a replace of all \ to / in order to work. Don't you see that issue, @jokram ?

cc/ @jgsogo

memsharded picture memsharded on 12 Mar 2020

@memsharded:

I am trying the CRLF approach, but I am realizing that I also need to do a replace of all \ to / in order to work. Don't you see that issue, @jokram ?

It seems that my cygwin version is able to handle this. But another user with a different cygwin version have had problems also concerning the paths which then must be cygwin compatible to work e.g. PATH="D:\.conan\614de7\1\bin"${PATH+:$PATH} must be converted to PATH="/d/.conan/614de7/1/bin"${PATH+:$PATH}

I just recognized that I've forgot to specify the cywin version => I've added it in the bug description.

jokram picture jokram on 12 Mar 2020

This has already been reported several times (last one https://github.com/conan-io/conan/issues/6412) and according to this link (http://mingw.org/wiki/Posix_path_conversion) rerouting the paths shouldn't be needed. But, as it is happening again (I come across this issue now https://github.com/conan-io/conan/issues/6061) maybe we should consider modifying the paths too, now we have tests.

It probably depends on the version... and probably the most compatible is to use /c/... paths.

jgsogo picture jgsogo on 12 Mar 2020

I am using

cygwin 3.1.2-1

memsharded picture memsharded on 12 Mar 2020

I'm not sure, but maybe it's also important how conan has been installed:

  • Install Python on native Windows and then install conan via pip install conan into the native Windows Python. This will create the conan.exe in linked to the native Windows Python. Then when calling conan inside of cygwin it would call the Windows native Python (this is how we have done)
  • Enter the cygwin (which comes with python and python3) and there install conan. Then when calling conan the python of Cygwin would be used.

=> So conan might behave different when e.g. checking the OS.
Below both commands are executed from within the Cygwin shell. The "python3" is the Python coming with Cygwin, but the second Python is the Windows native installation:

$ python3 -c "import platform; print(platform.system())"
CYGWIN_NT-10.0

$ /C/Python37/python.exe -c "import platform; print(platform.system())"
Windows
jokram picture jokram on 13 Mar 2020
👍1

It seems that my cygwin version is able to handle this. But another user with a different cygwin version have had problems also concerning the paths which then must be cygwin compatible to work e.g. PATH="D:.conan\614de7\1\bin"${PATH+:$PATH} must be converted to PATH="/d/.conan/614de7/1/bin"${PATH+:$PATH}

I've tried to reproduce this, but I couldn't until now.
For tests I've replaced in activate.sh and deactivate.sh CR LF with LF and tested with different Cygwin versions (see below for details):

  • ⚠️ Cygwin 1.7.32 => Warning for the path:

cygwin warning: MS-DOS style path detected: D:\casdev\_test\conan_poc\conan_test_shell\environment.sh.env Preferred POSIX equivalent is: /cygdrive/d/casdev/_test/conan_poc/conan_test_shell/environment.sh.env CYGWIN environment variable option "nodosfilewarning" turns off this warning. Consult the user's guide for more details about POSIX paths: http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
❌ If the path in activate.sh is /d/... then it will return an error:
$ . activate.sh bash: /d/casdev/_test/conan_test_shell/environment.sh.env: No such file or directory

  • ✔️ Cygwin 2.9.0 => OK
    ❌ If the path in activate.sh is /d/... then it will return an error:
    $ . activate.sh bash: /d/casdev/_test/conan_test_shell/environment.sh.env: No such file or directory
  • ✔️ Cygwin 2.10.0 => OK
    ✔️ If the path in activate.sh would be /d/... then it will also work, because now cygwin accepts such paths.
  • ✔️ Cygwin 3.0.7 => OK
    ✔️ If the path in activate.sh would be /d/... then it will also work

Conclusion

  • activate.sh and deactivate.sh should only contain LF (in Windows and Linux)
  • To support older cygwin versions < 2.10 might be problematic as there the /d/... notation is not accepted which is accepted in Git Bash.
  • If it's OK to support only cygwin >= 2.10 then the path should be in the /d/... notation.
  • Otherwise I think the best is to keep the Windows style path, but with forward slashes: D:/... (my personal preference)

Cygwin 1.7.32

$ cygcheck --version
cygcheck (cygwin) 1.7.32
System Checker for Cygwin
Copyright (C) 1998 - 2014 Red Hat, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

~
$ cd /d/casdev/_test/conan_test_shell
bash: cd: /d/casdev/_test/conan_test_shell: No such file or directory

~
$ cd /cygdrive/d/casdev/_test/conan_test_shell

/cygdrive/d/casdev/_test/conan_test_shell
$ . activate.sh
cygwin warning:
  MS-DOS style path detected: D:\casdev\_test\conan_poc\conan_test_shell\environment.sh.env
  Preferred POSIX equivalent is: /cygdrive/d/casdev/_test/conan_test_shell/environment.sh.env
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
(conanenv)
/cygdrive/d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv)
/cygdrive/d/casdev/_test/conan_test_shell
$ . deactivate.sh

/cygdrive/d/casdev/_test/conan_test_shell
$

Cygwin 2.9.0

$ cygcheck --version
cygcheck (cygwin) 2.9.0
System Checker for Cygwin
Copyright (C) 1998 - 2017 Cygwin Authors
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
~
$ cd /d/casdev/_test/conan_test_shell
bash: cd: /d/casdev/_test/conan_test_shell: No such file or directory
~
$ cd /cygdrive/d/casdev/_test/conan_test_shell
/cygdrive/d/casdev/_test/conan_test_shell
$ . activate.sh
(conanenv)
/cygdrive/d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv)
/cygdrive/d/casdev/_test/conan_test_shell
$ . deactivate.sh
/cygdrive/d/casdev/_test/conan_test_shell
$

Cygwin 2.10.0

$ cygcheck --version
cygcheck (cygwin) 2.10.0
System Checker for Cygwin
Copyright (C) 1998 - 2018 Cygwin Authors
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
0
03/13/20  3:05:43 PM ~
$ cd /d/casdev/_test/conan_test_shell
0
03/13/20  3:05:50 PM /d/casdev/_test/conan_test_shell
$ . activate.sh
(conanenv) 0
03/13/20  3:05:57 PM /d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv) 0
03/13/20  3:06:00 PM /d/casdev/_test/conan_test_shell
$ . deactivate.sh
0
03/13/20  3:06:05 PM /d/casdev/_test/conan_test_shell
$

Cygwin 3.0.7

$ cygcheck --version
cygcheck (cygwin) 3.0.7
System Checker for Cygwin
Copyright (C) 1998 - 2019 Cygwin Authors
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

~
$ cd /d/casdev/_test/conan_test_shell

/d/casdev/_test/conan_test_shell
$ . activate.sh
(conanenv)
/d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv)
/d/casdev/_test/conan_test_shell
$ . deactivate.sh

/d/casdev/_test/conan_test_shell
$
jokram picture jokram on 13 Mar 2020

Thanks a lot for sharing your investigation, @jokram, it will be really useful.

jgsogo picture jgsogo on 16 Mar 2020

Otherwise I think the best is to keep the Windows style path, but with forward slashes: D:/... (my personal preference)

Maybe I've to correct here: I've recognized that when in environments.sh.env the path is D:/... (with forward slashes) then the PATH was not respected, but when it is D:\... with backslashes then it works (tested with Cygwin 3.0.7):

❌ environments.sh.env with forward slashes doesn't work: PATH="D:/.conan/614de7/1/bin"${PATH+:$PATH}

/d/casdev/_test/conan_test_shell
$ . activate.sh
(conanenv)
/d/casdev/_test/conan_test_shell
$ helloworld
-bash: helloworld: Kommando nicht gefunden.
(conanenv)
/d/casdev/_test/conan_test_shell
$ . deactivate.sh

✔️ environments.sh.env with back slashes works fine: PATH="D:\.conan\614de7\1\bin"${PATH+:$PATH}

/d/casdev/_test/conan_test_shell
$ . activate.sh
(conanenv)
/d/casdev/_test/conan_test_shell
$ helloworld
Hello World!
(conanenv)
/d/casdev/_test/conan_test_shell
$ . deactivate.sh
jokram picture jokram on 16 Mar 2020

Talking about the line-ending:

I think it is safe to write the .sh always with LF, I haven't read about any Windows util having problems with it (Git bash, WSL, Cygwin). Users should be aware not to open this file in a Windows IDE and saving it again, and probably there would be some issues if these scripts are being added to a Git repository and checkout out. Anyway, up to Conan these files are generated on demand, to be used locally and the intention never is to open and modify them or to commit to a repository.

The alternative would be to document Cygwin+Conan usage and recommend the igncr alternative, but right now there is no evidence we will break someone if we change to LF.

About the paths:

I was really surprised by your last findings: forward slashes not working for a Cygwin version... 😨 and OTOH @memsharded reported that it was not working for him with backward slashes.

jgsogo picture jgsogo on 16 Mar 2020

@jokram , one question: where do you get old Cygwin versions? It looks like it is not as easy as I thought it'll be. Cygwin website always point to the last one and it is discourage to use older versions due to incompatibilities with packages. Should we worry then about other than the last version?

jgsogo picture jgsogo on 16 Mar 2020

@jgsogo We've stored different Cygwin version as conan tool packages in our Artifactory 😁
Only for the latest version (3.0.7) I've used an installer.
For Cygwin 2.9.0 you might check https://bintray.com/bincrafters/public-conan/cygwin_installer%3Abincrafters, but I haven't tested it.

jokram picture jokram on 16 Mar 2020
👍1

PR #6670 fixes only the issue related with the line endings (all the .sh scripts will have only LF). About the one related to the paths, it looks like there is no consensus, if we change them some of you are reporting a failure, if we don't it is not working for everybody.

If this is still an issue or you want to contribute with more information, please, open a new issue and we will try to investigate it further.

Thanks!

jgsogo picture jgsogo on 27 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

db4 picture db4  ·  3Comments
mpdelbuono picture mpdelbuono  ·  3Comments
liberforce picture liberforce  ·  3Comments
claasd picture claasd  ·  3Comments
zlalanne picture zlalanne  ·  3Comments