Web3.py: Tests not working in docker image & Python 3.8 missing in image

[evowilliamson]

• Version: x.x.x
• Python: 2.7/3.4/3.5
• OS: osx/linux/win
• pip freeze output
  alabaster==0.7.12
  apipkg==1.5
  appdirs==1.4.3
  atomicwrites==1.4.0
  attrs==19.3.0
  Babel==2.8.0
  base58==2.0.0
  bitarray==1.2.1
  bleach==3.1.5
  bumpversion==0.5.3
  cached-property==1.5.1
  certifi==2020.4.5.1
  chardet==3.0.4
  click==7.1.2
  colorama==0.4.3
  configparser==3.5.0
  contextlib2==0.6.0.post1
  cytoolz==0.10.1
  distlib==0.3.0
  docopt==0.6.2
  docutils==0.16
  eth-abi==2.1.1
  eth-account==0.5.2
  eth-bloom==1.0.3
  eth-hash==0.2.0
  eth-keyfile==0.5.1
  eth-keys==0.2.4
  eth-rlp==0.1.2
  eth-tester==0.2.0b2
  eth-typing==2.2.1
  eth-utils==1.8.4
  execnet==1.7.1
  filelock==3.0.12
  flake8==3.4.1
  flaky==3.6.1
  hexbytes==0.2.0
  hypothesis==5.10.4
  idna==2.9
  imagesize==1.2.0
  importlib-metadata==1.6.0
  importlib-resources==1.5.0
  incremental==17.5.0
  ipfshttpclient==0.4.13.2
  isort==4.3.4
  Jinja2==2.11.2
  jsonschema==3.2.0
  lru-dict==1.1.6
  MarkupSafe==1.1.1
  mccabe==0.6.1
  mock==4.0.2
  more-itertools==8.2.0
  multiaddr==0.0.9
  mypy==0.730
  mypy-extensions==0.4.3
  netaddr==0.7.19
  packaging==20.3
  parsimonious==0.8.1
  pathtools==0.1.2
  pkginfo==1.5.0.1
  pluggy==0.13.1
  protobuf==3.11.3
  py==1.8.1
  py-ecc==1.7.1
  py-evm==0.3.0a1
  py-geth==2.3.0
  py-solc==3.2.0
  pycodestyle==2.3.1
  pycryptodome==3.9.7
  pyethash==0.1.27
  pyflakes==1.5.0
  Pygments==2.6.1
  pyparsing==2.4.7
  pyrsistent==0.16.0
  pytest==4.6.9
  pytest-asyncio==0.10.0
  pytest-forked==1.1.3
  pytest-mock==1.13.0
  pytest-pythonpath==0.7.3
  pytest-watch==4.2.0
  pytest-xdist==1.31.0
  pytz==2020.1
  readme-renderer==26.0
  requests==2.23.0
  requests-toolbelt==0.9.1
  rlp==1.2.0
  semantic-version==2.8.5
  six==1.14.0
  snowballstemmer==2.0.0
  sortedcontainers==2.1.0
  Sphinx==2.4.4
  sphinx-better-theme==0.1.5
  sphinx-rtd-theme==0.4.3
  sphinxcontrib-applehelp==1.0.2
  sphinxcontrib-devhelp==1.0.2
  sphinxcontrib-htmlhelp==1.0.3
  sphinxcontrib-jsmath==1.0.1
  sphinxcontrib-qthelp==1.0.3
  sphinxcontrib-serializinghtml==1.1.4
  toml==0.10.0
  toolz==0.10.0
  toposort==1.5
  towncrier==19.2.0
  tox==3.15.0
  tqdm==4.45.0
  trie==1.4.0
  twine==1.15.0
  typed-ast==1.4.1
  typing-extensions==3.7.4.2
  urllib3==1.25.9
  varint==1.0.2
  virtualenv==20.0.18
  watchdog==0.10.2
  wcwidth==0.1.9
  webencodings==0.5.1
  websockets==8.1
  when-changed==0.3.0
  zipp==3.1.0

What was wrong?

I am trying to setup the development environment in order to start contributing to the project. Locally, all seems to be quite well, except for some minor issues, but I do experience issues with the docker image.

• Python 3.8 missing from docker image

I have problems with using the docker image that is provided. It seems that Python 3.8 is missing, while being mentioned in the tox.ini as a dependency:

(venv) ivo@ivo-ThinkPad-P53:~/projects/web3.py$ sudo docker-compose exec sandbox bash
root@d938cf203ee1:/code# python3.6 --version
Python 3.6.10
root@d938cf203ee1:/code# python3.7 --version
Python 3.7.3
root@d938cf203ee1:/code# python3.8 --version
bash: python3.8: command not found
root@d938cf203ee1:/code# whereis python
python: /usr/bin/python3.7m /usr/bin/python3.7 /usr/bin/python2.7 /usr/bin/python /usr/lib/python3.7 /usr/lib/python2.7 /etc/python3.7 /etc/python2.7 /etc/python /usr/local/bin/python /usr/local/bin/python3.6m /usr/local/bin/python3.6 /usr/local/bin/python3.6m-config /usr/local/bin/python3.6-config /usr/local/lib/python3.6 /usr/local/lib/python3.7 /usr/local/lib/python2.7 /usr/share/python /usr/share/man/man1/python.1.gz

root@d938cf203ee1:/code# head -10 tox.ini
[tox]
envlist=
    py{36,37,38}-ens
    py{36,37,38}-ethpm
    py{36,37,38}-core
    py{36,37,38}-integration-{goethereum,ethtester,parity}
    lint
    docs
    py{36,37,38}-wheel-cli

• Tests failing from docker image

root@d938cf203ee1:/code# tox -e py36-core
Maybe try:

    sudo apt-get install pandoc
See http://johnmacfarlane.net/pandoc/installing.html
for installation options
---------------------------------------------------------------

py36-core develop-inst-noop: /code
py36-core installed: alabaster==0.7.12,apipkg==1.5,appdirs==1.4.3,atomicwrites==1.4.0,attrs==19.3.0,Babel==2.8.0,base58==2.0.0,bitarray==1.2.1,bleach==3.1.5,bumpversion==0.5.3,cached-property==1.5.1,certifi==2020.4.5.1,chardet==3.0.4,click==7.1.2,colorama==0.4.3,configparser==3.5.0,contextlib2==0.6.0.post1,cytoolz==0.10.1,distlib==0.3.0,docopt==0.6.2,docutils==0.16,eth-abi==2.1.1,eth-account==0.5.2,eth-bloom==1.0.3,eth-hash==0.2.0,eth-keyfile==0.5.1,eth-keys==0.2.4,eth-rlp==0.1.2,eth-tester==0.2.0b2,eth-typing==2.2.1,eth-utils==1.8.4,execnet==1.7.1,filelock==3.0.12,flake8==3.4.1,flaky==3.6.1,hexbytes==0.2.0,hypothesis==5.10.4,idna==2.9,imagesize==1.2.0,importlib-metadata==1.6.0,importlib-resources==1.5.0,incremental==17.5.0,ipfshttpclient==0.4.13.2,isort==4.3.4,Jinja2==2.11.2,jsonschema==3.2.0,lru-dict==1.1.6,MarkupSafe==1.1.1,mccabe==0.6.1,mock==4.0.2,more-itertools==8.2.0,multiaddr==0.0.9,mypy==0.730,mypy-extensions==0.4.3,netaddr==0.7.19,packaging==20.3,parsimonious==0.8.1,pathtools==0.1.2,pkginfo==1.5.0.1,pluggy==0.13.1,protobuf==3.11.3,py==1.8.1,py-ecc==1.7.1,py-evm==0.3.0a1,py-geth==2.3.0,py-solc==3.2.0,pycodestyle==2.3.1,pycryptodome==3.9.7,pyethash==0.1.27,pyflakes==1.5.0,Pygments==2.6.1,pyparsing==2.4.7,pyrsistent==0.16.0,pytest==4.6.9,pytest-asyncio==0.10.0,pytest-forked==1.1.3,pytest-mock==1.13.0,pytest-pythonpath==0.7.3,pytest-watch==4.2.0,pytest-xdist==1.31.0,pytz==2020.1,readme-renderer==26.0,requests==2.23.0,requests-toolbelt==0.9.1,rlp==1.2.0,semantic-version==2.8.5,six==1.14.0,snowballstemmer==2.0.0,sortedcontainers==2.1.0,Sphinx==2.4.4,sphinx-better-theme==0.1.5,sphinx-rtd-theme==0.4.3,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,toml==0.10.0,toolz==0.10.0,toposort==1.5,towncrier==19.2.0,tox==3.15.0,tqdm==4.45.0,trie==1.4.0,twine==1.15.0,typed-ast==1.4.1,typing-extensions==3.7.4.2,urllib3==1.25.9,varint==1.0.2,virtualenv==20.0.18,watchdog==0.10.2,wcwidth==0.1.9,-e git+https://github.com/ethereum/web3.py.git@05bbf57b97f9c21918f29555b5582ce095674622#egg=web3,webencodings==0.5.1,websockets==8.1,when-changed==0.3.0,zipp==3.1.0
py36-core run-test-pre: PYTHONHASHSEED='2950451072'
py36-core run-test: commands[0] | pytest tests/core
ImportError while loading conftest '/code/conftest.py'.
py._path.local.LocalPath.ImportMismatchError: ('conftest', '/home/ivo/projects/web3.py/conftest.py', local('/code/conftest.py'))
ERROR: InvocationError for command /code/.tox/py36-core/bin/pytest tests/core (exited with code 4)
____________________________________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________________________________
ERROR:   py36-core: commands failed

How can it be fixed?

[marcgarreau]

Thanks for the interest in contributing @evowilliamson. Docker is not a part of my development workflow, but I did confirm that 3.8 isn't available within the container. I see the Dockerfile starts with the python 3.6 image, and that docker-compose.yml doesn't specify versions, so I'm not actually sure where v3.7 is coming from. Pinging our resident Docker expert @cburgdorf - Christoph, some direction here would be appreciated.

[cburgdorf]

I see the Dockerfile starts with the python 3.6 image, and that docker-compose.yml doesn't specify versions, so I'm not actually sure where v3.7 is coming from.

It just happens to be the case that the docker image python:3.6 also contains Python 3.7. For Python 3.8, yes you could just change python:3.6 to python:3.8 in the Dockerfile but the environment will only contain Python 3.8 and Python 3.7 then which means running something like tox -e py36-core will fail in such an environment.

If we care about the Docker image supporting all three Python versions side by side then I guess we either have the following options:

1. Find a base image that supports multiple Python versions (e.g. https://hub.docker.com/r/fkrull/multi-python but that is not from the official Python team)

2. Make amendments in the Dockerfile to install Python 3.6 in the python:3.8 image. That would probably boil down to adding some extra sources and running some apt-get fu.

That said, @evowilliamson if all you want is to start contributing you can just start working against a single Python version. In general you usually only want to run specific tests anyway and leave it up to CI to run tests against all possible environments.

[evowilliamson]

@cburgdorf @marcgarreau thanks for the info. The local installation is fine, so will start from there.

Looking forward to picking up some stories, issues. But first, trying to wrap my head around the codebase!