Pysyft: Error while installing 0.3.0 on linux

[mothukuriv]

Question

Error while installing 0.3.0 on linux

Further Information

I m trying to install 0.3.0 , tried both pip3 install and from source install using setup.py script.

System Information

• OS: [e.g.ubuntu]
• OS Version: [e.g. 18.0.0]
• Language Version: [e.g. Python 3.7

Additional Context

"import setuptools, tokenize;__file__='/tmp/pip-build-eyseu7ed/av/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-w35dvs78-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/env/include/site/python3.6/av" failed with error code 1 in /tmp/pip-build-eyseu7ed/av/

[TTitcombe]

Do you have two versions of python installed? You said you have Python 3.7 but the log you posted references 3.6

--compile --install-headers /home/user/env/include/site/python3.6/av

[madhavajay]

@mothukuriv Can you please run the following and then paste the output for this commands here.

$ which python3
$ python3 --version
$ python3 -m pip --version
$ python3 -m pip install syft

Syft 0.3 supports python 3.6 on Ubuntu and we have it in our CI: https://github.com/OpenMined/PySyft/blob/master/.github/workflows/tests.yml

Usually these kinds of issues relate to an issue with python paths, not using an virtualenv or some Ubuntu system packages needed to compile dependencies.

[mothukuriv]

(env) user@lambda-quad:~/3.0/PySyft$ which python3 /home/user/3.0/PySyft/env/bin/python3 (env) user@lambda-quad:~/3.0/PySyft$ python3 --version Python 3.6.9 (env) user@lambda-quad105535:~/3.0/PySyft$ python3 -m pip --version pip 9.0.1 from /home/user/3.0/PySyft/env/lib/python3.6/site-packages (python 3.6) (env) user@lambda-quad105535:~/3.0/PySyft$ python3 -m pip install syft

`Building wheels for collected packages: av, pylibsrtp
Running setup.py bdist_wheel for av ... error
Complete output from command /home/vmothuku/3.0/PySyft/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ffqu4usp/av/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp3iqgcqjxpip-wheel- --python-tag cp36:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help

error: invalid command 'bdist_wheel'

---

Failed building wheel for av
Running setup.py clean for av
Running setup.py bdist_wheel for pylibsrtp ... error
Complete output from command /home/vmothuku/3.0/PySyft/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ffqu4usp/pylibsrtp/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpjvcay0ogpip-wheel- --python-tag cp36:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help

error: invalid command 'bdist_wheel'

---

Failed building wheel for pylibsrtp
Running setup.py clean for pylibsrtp
Failed to build av pylibsrtp
Installing collected packages: av, pylibsrtp, aiortc, typing-extensions, websockets, python-dateutil, pytz, pandas, chardet, urllib3, idna, certifi, requests, pillow, torchvision, nest-asyncio, forbiddenfruit, pyparsing, packaging, immutables, contextvars, aiocontextvars, loguru, typeguard, protobuf, dpcontracts, syft
Running setup.py install for av ... error
Complete output from command /home/vmothuku/3.0/PySyft/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ffqu4usp/av/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tv71fji3-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/vmothuku/3.0/PySyft/env/include/site/python3.6/av:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/av
copying av/__main__.py -> build/lib.linux-x86_64-3.6/av
copying av/__init__.py -> build/lib.linux-x86_64-3.6/av
copying av/deprecation.py -> build/lib.linux-x86_64-3.6/av
copying av/datasets.py -> build/lib.linux-x86_64-3.6/av
creating build/lib.linux-x86_64-3.6/av/video
copying av/video/__init__.py -> build/lib.linux-x86_64-3.6/av/video
creating build/lib.linux-x86_64-3.6/av/filter
copying av/filter/__init__.py -> build/lib.linux-x86_64-3.6/av/filter
creating build/lib.linux-x86_64-3.6/av/audio
copying av/audio/__init__.py -> build/lib.linux-x86_64-3.6/av/audio
creating build/lib.linux-x86_64-3.6/av/codec
copying av/codec/__init__.py -> build/lib.linux-x86_64-3.6/av/codec
creating build/lib.linux-x86_64-3.6/av/sidedata
copying av/sidedata/__init__.py -> build/lib.linux-x86_64-3.6/av/sidedata
creating build/lib.linux-x86_64-3.6/av/subtitles
copying av/subtitles/__init__.py -> build/lib.linux-x86_64-3.6/av/subtitles
creating build/lib.linux-x86_64-3.6/av/data
copying av/data/__init__.py -> build/lib.linux-x86_64-3.6/av/data
creating build/lib.linux-x86_64-3.6/av/container
copying av/container/__init__.py -> build/lib.linux-x86_64-3.6/av/container
running build_ext
running config
PyAV: 8.0.2 (unknown commit)
Python: 3.6.9 (default, Oct 8 2020, 12:12:24) \n[GCC 8.4.0]
platform: Linux-5.4.0-52-generic-x86_64-with-Ubuntu-18.04-bionic
extension_extra:
include_dirs: [b'include']
libraries: []
library_dirs: []
define_macros: []
runtime_library_dirs: []
config_macros:
PYAV_COMMIT_STR="unknown-commit"
PYAV_VERSION=8.0.2
PYAV_VERSION_STR="8.0.2"
Could not find libavformat with pkg-config.
Could not find libavcodec with pkg-config.
Could not find libavdevice with pkg-config.
Could not find libavutil with pkg-config.
Could not find libavfilter with pkg-config.
Could not find libswscale with pkg-config.
Could not find libswresample with pkg-config.

----------------------------------------

Command "/home/vmothuku/3.0/PySyft/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ffqu4usp/av/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tv71fji3-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/vmothuku/3.0/PySyft/env/include/site/python3.6/av" failed with error code 1 in /tmp/pip-build-ffqu4usp/av/`

[mothukuriv]

I have 3.6.9 version and I have created virtual environment for pysyft

[madhavajay]

@mothukuriv Thanks for the command outputs. 😊
My first thought is that the pip version looks very old. It also looks like it is unable to build PyAV due to some missing AV libraries.

Firstly for pip your version is:

pip 9.0.1

My current version with python 3.6.9 is:

pip 20.2.4

Can you try the following:

$ python3 -m pip install --upgrade pip
$ python3 -m pip --version

Then secondly since we are using aiortc for our WebRTC library which seems to have a dependency on PyAV for its Video functionality (which we are not using), my guess is this ticket relates to your issue: https://github.com/PyAV-Org/PyAV/issues/697

I think the solution will be:

$ apt-get install ffmpeg-dev

After that try installing syft again.

$ python3 -m pip install syft

Then please include the output here again.

[mothukuriv]

Thank you , Just with the update of pip3 I was able to install syft.
(env) user@lambda-quad:~/3.0/PySyft$ pip3 freeze|grep syft syft==0.3.0 (env) user@lambda-quad:~/3.0/PySyft$ python3 -m pip --version pip 20.2.4 from /home/user/3.0/PySyft/env/lib/python3.6/site-packages/pip (python 3.6)

Installation have only one error in log,

ERROR: torchvision 0.8.1 requires torch==1.7.0, but you'll have torch 1.6.0 which is incompatible.

[madhavajay]

@mothukuriv thats great news! I was hopeful that pip would be enough I guess the updated pip knows how to fetch and build the dependencies but just incase i thought I would include the missing ffmpeg stuff as it looked like that was the downstream issue.

The error you see now is a normal error when you install torchvision with a different version of torch. Just uninstall torch or torchvision and pick a compatible version. If you install torchvision it will match the version of torch for you.

We support:

torch==1.5.0
torch==1.5.1
torch==1.6.0
torch==1.7.0

The torchvision versions which match the above are:

torchvision==0.6.0
torchvision==0.6.1
torchvision==0.7.0
torchvision==0.8.1 (0.8.0 was patched quickly after release)

You can read more here:
https://pypi.org/project/torchvision/

[mothukuriv]

Thank you @madhavajay