Setuptools: scan_module(): ValueError: bad marshal data (unknown type code)

Created on 23 Jan 2018  路  25Comments  路  Source: pypa/setuptools

This happens with CPython from git master (e768c86ef442ef89004089a8a34ce5909ffb90f2):

Traceback (most recent call last):
  File "setup.py", line 16, in <module>
    install_requires=["appdirs"],
  File ".../python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File ".../python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File ".../python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File ".../python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File ".../python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File ".../python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

I believe this is because PEP 552 has been implemented, which adds extra 4 bytes to the .pyc header; but setuptools is not aware of that yet.

picture jwilk

Most helpful comment

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.

picture smagafurov on 15 Feb 2018
👍4

All 25 comments

See #1263.

benoit-pierre picture benoit-pierre on 25 Jan 2018

Release on v38.4.1 pending...

jaraco picture jaraco on 3 Feb 2018

Same problem occurred w/ Python 3.7.0a4+ on Travis CI (reported as: travis-ci/travis-ci#9181), probably w/o the fix.

jenisys picture jenisys on 5 Feb 2018

@jaraco Same problem
https://travis-ci.org/aio-libs/aiopg/jobs/341326946

smagafurov picture smagafurov on 14 Feb 2018

@smagafurov : did you update setuptools?

benoit-pierre picture benoit-pierre on 14 Feb 2018

setuptools (38.5.1)

smagafurov picture smagafurov on 14 Feb 2018

@smagafurov: I'm sorry but I don't see the setuptools' update in your Travis log and the traceback line numbers don't match 38.5.1. Can you add a call to pip freeze --all before to make sure the right version is used? Just to be clear, you're not relying on setup_requires to update setuptools, right?

benoit-pierre picture benoit-pierre on 14 Feb 2018

@benoit-pierre: I repeat this problem locally

$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
(venv) sergey@magafurov:~/projects/aiopg [master]
$ pip list | grep tool
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
prompt-toolkit (1.0.15)
setuptools (38.5.1)
smagafurov picture smagafurov on 14 Feb 2018

@benoit-pierre: How to repeat:

git clone [email protected]:aio-libs/aiopg.git
cd aiopg
python3.7 -m venv ./venv
source ./venv/bin/activate
sudo apt-get install libpq-dev
pip install -U setuptools
python setup.py install
smagafurov picture smagafurov on 14 Feb 2018
👍2

I can't reproduce the issue:

> git clone https://github.com/aio-libs/aiopg.git
Cloning into 'aiopg'...
remote: Counting objects: 3267, done.
remote: Total 3267 (delta 0), reused 0 (delta 0), pack-reused 3267
Receiving objects: 100% (3267/3267), 666.68 KiB | 2.22 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.
> cd aiopg
> python3.7 -m venv venv
> ./venv/bin/python --version
Python 3.7.0a4
> ./venv/bin/pip install -U pip wheel setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.7/site-packages
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Collecting setuptools
  Using cached setuptools-38.5.1-py2.py3-none-any.whl
Installing collected packages: wheel, setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-38.5.1 wheel-0.30.0
> ./venv/bin/python setup.py -q bdist_egg                                                        
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'tests/**'
  File "build/bdist.linux-x86_64/egg/aiopg/utils.py", line 18
    ensure_future = asyncio.async
                                ^
SyntaxError: invalid syntax

  File "build/bdist.linux-x86_64/egg/aiopg/connection.py", line 109
    self._conn = psycopg2.connect(dsn, async=True, **kwargs)
                                           ^
SyntaxError: invalid syntax

zip_safe flag not set; analyzing archive contents...
benoit-pierre picture benoit-pierre on 14 Feb 2018

And setuptools' master is still ok on Travis with nightly.

benoit-pierre picture benoit-pierre on 14 Feb 2018

@benoit-pierre: |'m sorry. Please clone this instead:

git clone [email protected]:smagafurov/aiopg.git

I am trying to add python3.7 support to 'aio-libs/aiopg.git'

smagafurov picture smagafurov on 14 Feb 2018

I don't think the clone URL matters (works fine with either one). Can you patch setuptools.commands.bdist_egg to show what's being used?

 setuptools/command/bdist_egg.py | 1 +
 1 file changed, 1 insertion(+)

diff --git i/setuptools/command/bdist_egg.py w/setuptools/command/bdist_egg.py
index 423b8187..a66b03c7 100644
--- i/setuptools/command/bdist_egg.py
+++ w/setuptools/command/bdist_egg.py
@@ -417,6 +417,7 @@ def scan_module(egg_dir, base, name, stubs):
         skip = 12  # skip magic & date & file size
     else:
         skip = 16  # skip magic & reserved? & date & file size
+    print(sys.version_info, skip)
     f = open(filename, 'rb')
     f.read(skip)
     code = marshal.load(f)
benoit-pierre picture benoit-pierre on 14 Feb 2018

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

smagafurov picture smagafurov on 14 Feb 2018 
$ python setup.py install
running install
running bdist_egg
running egg_info
writing aiopg.egg-info/PKG-INFO
writing dependency_links to aiopg.egg-info/dependency_links.txt
writing requirements to aiopg.egg-info/requires.txt
writing top-level names to aiopg.egg-info/top_level.txt
reading manifest file 'aiopg.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'tests/**'
writing manifest file 'aiopg.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.13-x86_64/egg
running install_lib
running build_py
copying aiopg.egg-info/PKG-INFO -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/SOURCES.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/dependency_links.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/requires.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
copying aiopg.egg-info/top_level.txt -> build/bdist.macosx-10.13-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16
Traceback (most recent call last):
  File "setup.py", line 63, in <module>
    include_package_data=True)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/sergey/.pyenv/versions/3.7.0a3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/sergey/projects/aiopg/venv/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 423, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)
smagafurov picture smagafurov on 14 Feb 2018

clone URL matters because I fix python3.7 syntax error here 'smagafurov/aiopg.git'

smagafurov picture smagafurov on 14 Feb 2018

That's another issue, irelevant to the setuptools issue.

benoit-pierre picture benoit-pierre on 14 Feb 2018

sys.version_info(major=3, minor=7, micro=0, releaselevel='alpha', serial=3) 16

The first version that implemented PEP 552 was v3.7.0a4.
But setuptools currently does:

elif sys.version_info < (3, 7):
jwilk picture jwilk on 14 Feb 2018
👍2

That's what I was thinking, thanks for confirming.

benoit-pierre picture benoit-pierre on 14 Feb 2018

this works for me

elif sys.version_info < (3, 7, 0, 'alpha', 4):
smagafurov picture smagafurov on 14 Feb 2018

I'm not sure it make sense to explicitely test for 3.7.0a4: if you're running with nightly, then problems are to be expected, and you're also expected to keep current, no?

What I don't understand is why it fails on Travis, or is it another hole in our testsuite?

benoit-pierre picture benoit-pierre on 14 Feb 2018
👍1

@smagafurov: just to be clear, to fix your issues you should:

  • locally: update Python to >=3.7.0a4
  • Travis CI: update setuptools first before installing other requirements (IMHO adding back nightly testing is a good idea)
benoit-pierre picture benoit-pierre on 14 Feb 2018
👍1

@smagafurov: https://travis-ci.org/benoit-pierre/aiopg/jobs/341442846

I'm also confused by the fact that python setup.py install is called before pip install -Ur requirements.txt.

benoit-pierre picture benoit-pierre on 14 Feb 2018

@benoit-pierre: Thank! Updating setuptools in Travis fixes the problem.

smagafurov picture smagafurov on 15 Feb 2018
👍4

Having exact same issue.
Python: 3.7.6
Setuptools: 45.2.0 (just tried to update, no luck)

Nevermind, it was an issue with the source release I downloaded containing pre-existing .pyc files.

golyalpha picture golyalpha on 15 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jaraco picture jaraco  路  5Comments
Themanwithoutaplan picture Themanwithoutaplan  路  5Comments
yaoking picture yaoking  路  4Comments
rossmacarthur picture rossmacarthur  路  3Comments
Gumnos picture Gumnos  路  4Comments