Azure-pipelines-tasks: Building Python with --enable-shared on Linux

[anapaulagomes]

Environment

• Server - Azure Pipelines

apgomes88/apgomes88 - 20181118.3 Test_Linux Job

• Agent - Hosted

Pool:Hosted Ubuntu 1604 · Agent: Hosted Agent

Issue Description

Hi, people.

I'm building a multi OS desktop application with Python using pyinstaller. Pyinstaller requires building Python source code using the param --enable-shared (or, --enable-framework on Darwin) in order to access all its dependencies. On MacOS and Windows jobs is working fine but I'm having trouble with Linux build. I ran into the same problem with MacOS locally and I solved it downloading Python source code and doing:

./configure --enable-framework
make
make install

Now I have no idea if this is possible here. Can you help me with it?

- job: 'Test_Linux'
  timeoutInMinutes: 5
  pool:
    vmImage: 'Ubuntu 16.04'

  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: 3.7
      architecture: 'x64'

  - script: |
      sudo apt-get update -qy
      sudo apt-get install -y python3-dev python-dev python-pip
      sudo apt-get install -y espeak
      sudo apt-get install xvfb
      python -m pip install --upgrade pip
      pip install -r requirements.txt
      pip install -r requirements_test.txt
    displayName: 'Install prerequisites'

  - script: |
      sudo cp xvfb.init /etc/init.d/xvfb
      sudo chmod +x /etc/init.d/xvfb
      sudo update-rc.d xvfb defaults
      sudo service xvfb start
    displayName: 'Start X Virtual Frame Buffer'

  - script: |
      export DISPLAY=:10
      pytest
    condition: succeededOrFailed()
    displayName: 'Run tests'

  - script: |
      flake8 .
    displayName: 'Run lint tests'

  - script: |
      pyinstaller --clean pardal.spec
    displayName: 'Build'

Task logs

log_24_38.zip

Error logs

2018-11-18T13:36:55.5615673Z 15521 INFO: Python library not in binary dependencies. Doing additional searching...
2018-11-18T13:36:55.5992515Z Traceback (most recent call last):
2018-11-18T13:36:55.5993616Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/bin/pyinstaller", line 11, in <module>
2018-11-18T13:36:55.5994918Z     load_entry_point('PyInstaller==3.4', 'console_scripts', 'pyinstaller')()
2018-11-18T13:36:55.5995834Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/__main__.py", line 111, in run
2018-11-18T13:36:55.5996335Z     run_build(pyi_config, spec_file, **vars(args))
2018-11-18T13:36:55.5997468Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/__main__.py", line 63, in run_build
2018-11-18T13:36:55.5998050Z     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
2018-11-18T13:36:55.5998822Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 838, in main
2018-11-18T13:36:55.6000016Z     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
2018-11-18T13:36:55.6000937Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 784, in build
2018-11-18T13:36:55.6001464Z     exec(text, spec_namespace)
2018-11-18T13:36:55.6001828Z   File "<string>", line 25, in <module>
2018-11-18T13:36:55.6002536Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
2018-11-18T13:36:55.6003020Z     self.__postinit__()
2018-11-18T13:36:55.6003766Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
2018-11-18T13:36:55.6004231Z     self.assemble()
2018-11-18T13:36:55.6004937Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 573, in assemble
2018-11-18T13:36:55.6005499Z     self._check_python_library(self.binaries)
2018-11-18T13:36:55.6006310Z   File "/opt/hostedtoolcache/Python/3.7.0/x64/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 676, in _check_python_library
2018-11-18T13:36:55.6006783Z     raise IOError(msg)
2018-11-18T13:36:55.6007552Z OSError: Python library not found: libpython3.7.so.1.0, libpython3.7mu.so.1.0, libpython3.7m.so.1.0
2018-11-18T13:36:55.6008346Z This would mean your Python installation doesn't come with proper library files.
2018-11-18T13:36:55.6008839Z This usually happens by missing development package, or unsuitable build parameters of Python installation.
2018-11-18T13:36:55.6009120Z 
2018-11-18T13:36:55.6009488Z * On Debian/Ubuntu, you would need to install Python development packages
2018-11-18T13:36:55.6010124Z   * apt-get install python3-dev
2018-11-18T13:36:55.6010842Z   * apt-get install python-dev
2018-11-18T13:36:55.6011662Z * If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)
2018-11-18T13:36:55.6012110Z 
2018-11-18T13:36:55.6831329Z ##[error]Bash exited with code '1'.

[brcrista]

Thanks for your feedback! You are correct, we need to configure the Python installations on this image with this flag. We have an issue tracking this in our image generation repo: Microsoft/azure-pipelines-image-generation#225.

I will try to get to this soon. We will have to roll out a new version of the hosted image before you see the changes, which happens about once per month.

[brcrista]

Closing as a duplicate -- please subscribe to the issue above to track.

[edjubuh]

Is the PR you referenced the correct one? It looks unrelated.

I have a similar issue with not being able to find the Python libraries, but I'm using cx_Freeze instead of pyInstaller. I think a workaround/solution would be to add the libraries to LDPATH on macOS and Linux, but I'm not too thrilled about doing that on my own pipeline since UsePythonVersion should set that up.

[brcrista]

Sorry, fixed the link now ;) thanks!