Homebrew-core: WeeChat fails to build --with-python

Created on 27 Jul 2018 · 48Comments · Source: Homebrew/homebrew-core

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

[x] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
[x] have a problem with brew install (or upgrade, reinstall) a single, official formula (not cask)? If it's a general brew problem please file this issue at Homebrew/brew: https://github.com/Homebrew/brew/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?
[x] ran `brew gist-logs weechat
https://gist.github.com/aeonzh/f4dc299dadc9d4e25198d715729f2899
[ ] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

To help us debug your issue please explain:

What you were trying to do (and why)
Installing WeeChat
What happened (include command output)
Build failed
What you expected to happen
Install WeeChat
Step-by-step reproduction instructions (by running brew install commands)
brew install weechat --with-python --with-perl build failed
brew install weechat poured the bottle successfully
brew install weechat --with-perl builded successfully
brew install weechat --with-python failed

in progress outdated upstream issue

Source

aeonzh

👍2

Most helpful comment

@DOMT4 so you're saying if I use --with-python@2 it'll install on the latest formula? Just tried it and you are correct. It did install. Well, I feel a little silly. I'll go update my post.

kkirby on 7 Aug 2018

❤1 👍1

All 48 comments

@MindTooth @DomT4 hmm this seems to be reproducible.

ilovezfs on 27 Jul 2018

@ilovezfs I assume you're not using a mix of 10.x CLT and 9.x Xcode? That was my initial wonder here. I'll take a look at it.

It definitely worked for me when we added it, and I'm _fairly sure_ I tested it again on the 24th.

DomT4 on 27 Jul 2018

na. tested on 10.11 with Xcode 8.2.1

ilovezfs on 27 Jul 2018

could be new cmake killed it?

ilovezfs on 27 Jul 2018

Yeah, can reproduce locally with 10.13.6 and Xcode 9.4.1 as well. Super.

[ 49%] Building C object src/plugins/irc/CMakeFiles/irc.dir/irc-ignore.o
cd /tmp/weechat-20180727-73871-23o08g/weechat-2.2/build/src/plugins/irc && /usr/local/Homebrew/Library/Homebrew/shims/mac/super/clang -DENABLE_NLS -DHAVE_CONFIG_H -DHAVE_GCRYPT -DHAVE_GNUTLS -DHAVE_ICONV -DHAVE_ZLIB -DWEECHAT_LICENSE=\"GPL3\" -DWEECHAT_VERSION=\"2.2\" -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_LARGE_FILES -Dirc_EXPORTS -I/usr/local/include -I/tmp/weechat-20180727-73871-23o08g/weechat-2.2/build  -Wall -Wextra -Werror-implicit-function-declaration -DNDEBUG -fPIC   -fPIC -fPIC -o CMakeFiles/irc.dir/irc-ignore.o   -c /tmp/weechat-20180727-73871-23o08g/weechat-2.2/src/plugins/irc/irc-ignore.c
ld: -stack_size option can only be used when linking a main executable
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/plugins/python/python.so] Error 1
make[1]: *** [src/plugins/python/CMakeFiles/python.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

DomT4 on 27 Jul 2018

could be new cmake killed it?

Oh, yes, that seems likely 🙃. Someone else mentioned in a PR they were having CMake issues, so perhaps there's more breaking about the recent CMake update than we've yet to realise.

DomT4 on 27 Jul 2018

nope same thing with cmake 3.11.4
odd

ilovezfs on 27 Jul 2018

Maybe it never worked and we were just tricked by the python -> python@2 deprecation.

ilovezfs on 27 Jul 2018

That's looking possible. Was sure I tested when I submitted my PR the other day but perhaps I simply tested whether the dependency remapping issue was gone without actually letting the build wander through until completion.

DomT4 on 27 Jul 2018

problem seems to be PYTHON_LFLAGS in src/plugins/python/CMakeLists.txt

removing that, the build finishes

no idea if python3 stuff then works but probably

ilovezfs on 27 Jul 2018

They're using:

COMMAND ${PYTHON_EXECUTABLE} -c "import sys; from distutils.sysconfig import *; sys.stdout.write(get_config_var('LIBS') + ' ' + get_config_var('LINKFORSHARED'))"

to determine which flags to pass. Python 3.7 returns:

-ldl  -framework CoreFoundation -Wl,-stack_size,1000000  -framework CoreFoundation /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python

So that explains where the -stack_size flag is coming from.

DomT4 on 27 Jul 2018

As did Python 3.6, so apparently not a new addition there.

-ldl  -framework CoreFoundation -Wl,-stack_size,1000000  -framework CoreFoundation /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/Python

DomT4 on 27 Jul 2018

weechat bug? cmake bug? python upstream bug? homebrew python bug?

ilovezfs on 27 Jul 2018

Maybe some upstream naivety? Python2 returns:

-ldl  -framework CoreFoundation -u _PyMac_Error /usr/local/opt/python@2/Frameworks/Python.framework/Versions/2.7/Python

and upstream didn't change the check from:

COMMAND ${PYTHON_EXECUTABLE} -c "import sys; from distutils.sysconfig import *; sys.stdout.write(get_config_var('LIBS') + ' ' + get_config_var('LINKFORSHARED'))"

when they added python3 support.

Although if it's been an issue since 2013 _surely_ someone would've spotted it before us. Upstream doesn't seem to test against macOS based on their Travis config but it's hard to believe there has been absolutely nobody out there on macOS trying to build with that option.

DomT4 on 27 Jul 2018

I can't testify to what might have changed. When I did submit, it compiled fine with Python 3(6|7?), and I could run it w/ plugins.

Just for doing it, I can recompile when I get home, and see if something has changed.

On 27 Jul 2018, at 18:26, ilovezfs notifications@github.com wrote:

weechat bug? cmake bug? python upstream bug? homebrew python bug?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

MindTooth on 27 Jul 2018

If it's reproducible with other Pythons, probably just a misuse of the CMake variable by weechat.

ilovezfs on 27 Jul 2018

I feel like upstream should perhaps be following CMake's lead and using the -config scripts.

Python note that the config scripts aren't guaranteed to work in all UNIX environments but it's apparently good enough for cmake itself.

    # retrieve library
    execute_process (COMMAND "${_${_PYTHON_PREFIX}_CONFIG}" --ldflags
                     RESULT_VARIABLE _${_PYTHON_PREFIX}_RESULT
                     OUTPUT_VARIABLE _${_PYTHON_PREFIX}_FLAGS
                     ERROR_QUIET
                     OUTPUT_STRIP_TRAILING_WHITESPACE)
    if (NOT _${_PYTHON_PREFIX}_RESULT)
      # retrieve library directory
      string (REGEX MATCHALL "-L[^ ]+" _${_PYTHON_PREFIX}_LIB_DIRS "${_${_PYTHON_PREFIX}_FLAGS}")
      string (REPLACE "-L" "" _${_PYTHON_PREFIX}_LIB_DIRS "${_${_PYTHON_PREFIX}_LIB_DIRS}")
      list (REMOVE_DUPLICATES _${_PYTHON_PREFIX}_LIB_DIRS)
      # retrieve library name
      string (REGEX MATCHALL "-lpython[^ ]+" _${_PYTHON_PREFIX}_LIB_NAMES "${_${_PYTHON_PREFIX}_FLAGS}")
      string (REPLACE "-l" "" _${_PYTHON_PREFIX}_LIB_NAMES "${_${_PYTHON_PREFIX}_LIB_NAMES}")
      list (REMOVE_DUPLICATES _${_PYTHON_PREFIX}_LIB_NAMES)

@flashcode Any thoughts here?

DomT4 on 27 Jul 2018

@DomT4: I don't know where is the problem, and I have no way to test with Homebrew…
Any idea why it suddenly breaks? (there was no changes on WeeChat side, except the add of Python 3.7 support recently, but not related to compiler/linker flags).

flashcode on 28 Jul 2018

@flashcode I don't think it's a sudden thing. It seems likely that building with the python3 option on macOS hasn't worked as expected/hoped for a fair while; can't speak for the python3 versions before this without checking but everything between Python 3.4-3.7 throw the same problematic -Wl,-stack_size,1000000 when LINKFORSHARED is checked.

The macOS linker objects to -stack_size unless linking a main executable per the error thrown. I'm wondering whether weechat's usage of:

"import sys; from distutils.sysconfig import *; sys.stdout.write(get_config_var('LIBS') + ' ' + get_config_var('LINKFORSHARED'))"

is deliberate/desired over just checking the *-config scripts for --ldflags given python3 strips out stack_size there.

FWIW to lend credence to the idea that it's a macOS problem rather than a Homebrew one, MacPorts have been manually stripping out the problematic aspect for their weechat with python3 builds, apparently since 2015 although I can't find any reference to it being reported upstream.

DomT4 on 28 Jul 2018

Submitted a bug report. Gladly take some comments on the bug report.

MindTooth on 30 Jul 2018

For those who don't care about Python3 integration, you can still use Python2 by passing the --with-python@2 flag.

~For those wanting a quick fix you can revert the formula to a known working state:~

~cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula~
~git checkout 3bb70fb03c53031a6aa7b269dd9e990fbce06897 weechat.rb~
~HOMEBREW_NO_AUTO_UPDATE=1 brew install weechat --with-python~

kkirby on 2 Aug 2018

👍1

Given the python(3) option is new, and the general build is unaffected, I'm not sure how folks would end up needing a quick fix, unless I'm missing something?

DomT4 on 2 Aug 2018

@DomT4 If you want Python support in WeeChat you have to install an older version, one that doesn't rely on Python 3. Python has always been an option, it just got updated recently to use Python3.

kkirby on 2 Aug 2018

@kkirby The python dependency was moved over to python@2 a fair while back, so it shouldn't be an issue for existing users. I'm guessing you tried to install weechat --with-python expecting it to be python2 rather than python3?

DomT4 on 2 Aug 2018

@DomT4 I installed weechat and got the error described above:

ld: -stack_size option can only be used when linking a main executable

I figured the easiest fix to get it working ASAP for myself was to install an older formula, which worked.

kkirby on 2 Aug 2018

What did you pass when you installed weechat. Just brew install weechat?

DomT4 on 3 Aug 2018

@DomT4 brew install weechat --with-python

kkirby on 3 Aug 2018

Yeah, that'll be the issue. Pre-compile you should have been seeing a warning, for quite some time now, that --with-python wasn't a valid option and it was getting automagically remapped onto --with-python@2 for you.

DomT4 on 4 Aug 2018

Evening using the recommendation from @kkirby, I get the following:

$ cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
git checkout 3bb70fb03c53031a6aa7b269dd9e990fbce06897 weechat.rb
HOMEBREW_NO_AUTO_UPDATE=1 brew install weechat --with-python
==> Installing dependencies for weechat: python@2
==> Installing weechat dependency: python@2
==> Downloading https://homebrew.bintray.com/bottles/[email protected]_1.high_sierra.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring [email protected]_1.high_sierra.bottle.1.tar.gz
Error: An unexpected error occurred during the `brew link` step
The formula built, but is not symlinked into /usr/local
Permission denied @ dir_s_mkdir - /usr/local/Frameworks
Error: Permission denied @ dir_s_mkdir - /usr/local/Frameworks

TechnotronicOz on 4 Aug 2018

brew doctor?

On 4 Aug 2018, at 13:56, Matt Carter notifications@github.com wrote:

Evening using the recommendation from @kkirby, I get the following:

$ cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
git checkout 3bb70fb03c53031a6aa7b269dd9e990fbce06897 weechat.rb
HOMEBREW_NO_AUTO_UPDATE=1 brew install weechat --with-python
==> Installing dependencies for weechat: python@2
==> Installing weechat dependency: python@2
==> Downloading https://homebrew.bintray.com/bottles/[email protected]_1.high_sierra.bottle.1.tar.gz

################################################################## 100.0%

==> Pouring [email protected]_1.high_sierra.bottle.1.tar.gz
Error: An unexpected error occurred during the brew link step
The formula built, but is not symlinked into /usr/local
Permission denied @ dir_s_mkdir - /usr/local/Frameworks
Error: Permission denied @ dir_s_mkdir - /usr/local/Frameworks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

MindTooth on 4 Aug 2018

@DomT4 Yup. I got that error.

kkirby on 4 Aug 2018

Evening using the recommendation from @kkirby, I get the following:

Different issue. Try this:

=> sudo mkdir /usr/local/Frameworks
=> sudo chmod g+rwx /usr/local/Frameworks
=> sudo chmod 755 /usr/local/Frameworks
=> sudo chown $(whoami) /usr/local/Frameworks
=> sudo chgrp admin /usr/local/Frameworks

That mimics the steps the Homebrew installer takes.

@DomT4 Yup. I got that error.

If it was an upgrade after that the option should've been remapped over to python@2 for you and that no longer be an issue. If it was a fresh install you'll just need to adhere to the new option name and pass --with-python@2 instead.

DomT4 on 5 Aug 2018

@DomT4 It was a fresh install and it told me that it was deprecated and automatically remapped to the new option, which I think is the correct functionality given this line in the formula:

 deprecated_option "with-python" => "with-python@2"

I could be wrong. I'm not familiar with formulas and all their possible configurations.

kkirby on 5 Aug 2018

❤2

Yeah. If it's a fresh install users are expected to use the correct current option name, and if it'd been an upgrade that transition would've happened automatically previously.

The hiccup in expected behaviour would've been if someone had managed to not upgrade weechat at all since the deprecated_option line was added until this point, in which case they would've involuntarily been shunted from Python2 to Python3, but installing fresh as you are there's a loose expectation to check & use the right option names essentially.

DomT4 on 7 Aug 2018

👍1

@DOMT4 so you're saying if I use --with-python@2 it'll install on the latest formula? Just tried it and you are correct. It did install. Well, I feel a little silly. I'll go update my post.

kkirby on 7 Aug 2018

❤1 👍1

@DomT4 so brew install weechat --with-python@2 should work?