Pytest: 3.7.0 object of type 'Package' has no len()

[bluzi]

Hey,

Our tests kept failing today due to an error inside pytest, reverting to 3.6.4 solved it.

00:00:11.460 [test] ============================= test session starts ==============================
00:00:11.462 [test] platform linux2 -- Python 2.7.14, pytest-3.7.0, py-1.5.4, pluggy-0.7.1
00:00:11.463 [test] rootdir: /app, inifile:
00:00:11.465 [test] plugins: flake8-1.0.1
00:00:11.466 [test] collected 0 items / 1 errors
00:00:11.467 [test] 
00:00:11.468 [test] ==================================== ERRORS ====================================
00:00:11.470 [test] ______________________________ ERROR collecting  _______________________________
00:00:11.471 [test] /usr/lib/python2.7/site-packages/_pytest/runner.py:201: in __init__
00:00:11.473 [test]     self.result = func()
00:00:11.474 [test] /usr/lib/python2.7/site-packages/_pytest/runner.py:261: in <lambda>
00:00:11.475 [test]     call = CallInfo(lambda: list(collector.collect()), "collect")
00:00:11.477 [test] /usr/lib/python2.7/site-packages/_pytest/main.py:475: in collect
00:00:11.478 [test]     for x in self._collect(arg):
00:00:11.480 [test] /usr/lib/python2.7/site-packages/_pytest/main.py:539: in _collect
00:00:11.481 [test]     for y in self.matchnodes(col, names):
00:00:11.482 [test] /usr/lib/python2.7/site-packages/_pytest/main.py:605: in matchnodes
00:00:11.483 [test]     num = len(nodes)
00:00:11.485 [test] E   TypeError: object of type 'Package' has no len()
00:00:11.486 [test] ----------------- generated xml file: /test-reports/pytest.xml -----------------
00:00:11.487 [test] =============================== warnings summary ===============================
00:00:11.489 [test] <undetermined location>
00:00:11.490 [test]   could not create cache path /app/.pytest_cache/v/cache/nodeids
00:00:11.491 [test]   could not create cache path /app/.pytest_cache/v/cache/lastfailed
00:00:11.492 [test] 
00:00:11.493 [test] -- Docs: http://doc.pytest.org/en/latest/warnings.html
00:00:11.495 [test] !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
00:00:11.496 [test] ===================== 2 warnings, 1 error in 0.13 seconds ======================

It's running on Alpine Linux, if you guys need more information feel free to ask.

[lost-a-tooth]

Seeing the same issue on 3.7.0 and 3.6.4 fixed it, definitely a regression.

[nicoddemus]

Thanks for the report, both of you. This is definitely a problem with the recent addition of package-scoped fixtures.

The current workaround is indeed to pin to !=3.7.0 until 3.7.1 is released with a fix. 👍

[nicoddemus]

Can any one of you test the nicoddemus:collect-file-bug branch to see if this fixes the issue?

It can be installed with:

pip install git+https://github.com/nicoddemus/pytest@collect-file-bug

[bluzi]

Looks like 3.7.1 is bugged too, this is the error now:

00:00:15.413 [test] ============================= test session starts ==============================
00:00:15.415 [test] platform linux2 -- Python 2.7.14, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
00:00:15.417 [test] rootdir: /app, inifile:
00:00:15.418 [test] plugins: flake8-1.0.1
00:00:15.420 [test] collected 0 items / 1 errors
00:00:15.672 [test] 
00:00:15.673 [test] ==================================== ERRORS ====================================
00:00:15.675 [test] ______________________________ ERROR collecting  _______________________________
00:00:15.676 [test] /usr/lib/python2.7/site-packages/_pytest/runner.py:201: in __init__
00:00:15.678 [test]     self.result = func()
00:00:15.679 [test] /usr/lib/python2.7/site-packages/_pytest/runner.py:261: in <lambda>
00:00:15.681 [test]     call = CallInfo(lambda: list(collector.collect()), "collect")
00:00:15.682 [test] /usr/lib/python2.7/site-packages/_pytest/main.py:475: in collect
00:00:15.683 [test]     for x in self._collect(arg):
00:00:15.685 [test] /usr/lib/python2.7/site-packages/_pytest/main.py:538: in _collect
00:00:15.686 [test]     col = root._collectfile(argpath)
00:00:15.688 [test] E   AttributeError: 'list' object has no attribute '_collectfile'
00:00:15.689 [test] ----------------- generated xml file: /test-reports/pytest.xml -----------------
00:00:15.690 [test] =============================== warnings summary ===============================
00:00:15.692 [test] <undetermined location>
00:00:15.693 [test]   could not create cache path /app/.pytest_cache/v/cache/nodeids
00:00:15.695 [test]   could not create cache path /app/.pytest_cache/v/cache/lastfailed
00:00:15.696 [test] 
00:00:15.697 [test] -- Docs: http://doc.pytest.org/en/latest/warnings.html
00:00:15.698 [test] !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
00:00:15.700 [test] ===================== 2 warnings, 1 error in 0.11 seconds ======================

[nicoddemus]

Thanks for confirming @bluzi

[nicoddemus]

@bluzi and @lost-a-tooth, any chance you can provide a snort reproducible example? I'm having trouble to reproduce this. Appreciate it!

[bluzi]

I'll try to find time, need to reproduce that in another environment since I cannot post here the actual code.

[SarLecobee]

Just want to add this seems to be broken also in pytest 3.8.0. Reverting to 3.6.4 fixes it.

====================================================== test session starts ======================================================
platform linux -- Python 3.5.2, pytest-3.8.0, py-1.6.0, pluggy-0.7.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: <cut>, inifile: pytest.ini
plugins: allure-pytest-2.5.1, flaky-3.3.0
collecting 0 items / 1 errors                                                                                                   2018-09-18T17:50:40.244103 - WARNING - No test plan configuration information provided. All tests will be run
collected 0 items / 1 errors

============================================================ ERRORS =============================================================
_______________________________________________________ ERROR collecting  _______________________________________________________
../../../.local/lib/python3.5/site-packages/_pytest/runner.py:201: in __init__
    self.result = func()
../../../.local/lib/python3.5/site-packages/_pytest/runner.py:261: in <lambda>
    call = CallInfo(lambda: list(collector.collect()), "collect")
../../../.local/lib/python3.5/site-packages/_pytest/main.py:477: in collect
    for x in self._collect(arg):
../../../.local/lib/python3.5/site-packages/_pytest/main.py:544: in _collect
    for y in self.matchnodes(col, names):
../../../.local/lib/python3.5/site-packages/_pytest/main.py:609: in matchnodes
    num = len(nodes)
E   TypeError: object of type 'Package' has no len()
 generated xml file: <cut>
===Flaky Test Report===


===End Flaky Test Report===
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================== 1 error in 0.14 seconds ====================================================

[nicoddemus]

Thanks for the ping, unfortunately this has not really been fixed yet. I believe @jonozzz was working on this, but I'm not sure.

[mdmintz]

@nicoddemus @bluzi @jonozzz I have a way to easily reproduce this. Using pytest==3.8.0, if you clone SeleniumBase and cd into the examples folder and run pytest --collect-only *.py, you'll see the following (tested on my MacBook):

(sbase6) DrSeleniums-MacBook-Pro:examples michael$ pytest --collect-only *.py

============================= test session starts ==============================
platform darwin -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: /Users/michael/github/SeleniumBase, inifile: pytest.ini
plugins: xdist-1.23.0, metadata-1.7.0, html-1.19.0, forked-0.2, seleniumbase-1.15.10
collected 0 items / 1 errors                                                   

==================================== ERRORS ====================================
______________________________ ERROR collecting  _______________________________
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/runner.py:201: in __init__
    self.result = func()
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/runner.py:261: in <lambda>
    call = CallInfo(lambda: list(collector.collect()), "collect")
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:477: in collect
    for x in self._collect(arg):
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:544: in _collect
    for y in self.matchnodes(col, names):
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:609: in matchnodes
    num = len(nodes)
E   TypeError: object of type 'Package' has no len()
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.16 seconds ============================

---

In summary, you get: TypeError: object of type 'Package' has no len()

---

If you then install pytest==3.6.4 and run pytest --collect-only *.py again, it works:

(sbase6) DrSeleniums-MacBook-Pro:examples michael$ pytest --collect-only *.py
==================================== test session starts =====================================
platform darwin -- Python 3.7.0, pytest-3.6.4, py-1.6.0, pluggy-0.7.1
rootdir: /Users/michael/github/SeleniumBase, inifile: pytest.ini
plugins: xdist-1.23.0, metadata-1.7.0, html-1.19.0, forked-0.2, seleniumbase-1.15.10
collected 14 items                                                                           
<Module 'examples/basic_script.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_basic'>
<Module 'examples/delayed_assert_test.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_delayed_asserts'>
<Module 'examples/github_test.py'>
  <UnitTestCase 'GitHubTests'>
    <TestCaseFunction 'test_github'>
<Module 'examples/my_first_test.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_basic'>
<Module 'examples/my_test_suite.py'>
  <UnitTestCase 'MyTestSuite'>
    <TestCaseFunction 'test_1'>
    <TestCaseFunction 'test_2'>
    <TestCaseFunction 'test_3'>
    <TestCaseFunction 'test_4'>
<Module 'examples/parameterized_test.py'>
  <UnitTestCase 'GoogleTestClass'>
    <TestCaseFunction 'test_parameterized_google_search_0_pypi'>
    <TestCaseFunction 'test_parameterized_google_search_1_wikipedia'>
    <TestCaseFunction 'test_parameterized_google_search_2_seleniumbase'>
<Module 'examples/proxy_test.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_proxy'>
<Module 'examples/rate_limiting_test.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_rate_limited_printing'>
<Module 'examples/test_fail.py'>
  <UnitTestCase 'MyTestClass'>
    <TestCaseFunction 'test_find_army_of_robots_on_xkcd_desert_island'>

================================ no tests ran in 0.17 seconds ================================

If you need me to help you fix it, I'm happy to help. It's been an issue for a few customers of mine.

[mdmintz]

Upon further investigation, it's the __init__.py file that causes the issues in pytest==3.8.0, but not when using pytest==3.6.4 . With 3.6.4, pytest is smart enough to not try parsing the file, yet with 3.8.0, it fails. Can reproduce more easily with pytest --collect-only __init__.py.

pytest --collect-only __init__.py 
========================================= test session starts =========================================
platform darwin -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: /Users/michael/github/SeleniumBase, inifile: pytest.ini
plugins: xdist-1.23.0, metadata-1.7.0, html-1.19.0, forked-0.2, seleniumbase-1.15.10
collected 0 items / 1 errors                                                                          

=============================================== ERRORS ================================================
__________________________________________ ERROR collecting  __________________________________________
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/runner.py:201: in __init__
    self.result = func()
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/runner.py:261: in <lambda>
    call = CallInfo(lambda: list(collector.collect()), "collect")
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:477: in collect
    for x in self._collect(arg):
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:544: in _collect
    for y in self.matchnodes(col, names):
../../../.virtualenvs/sbase6/lib/python3.7/site-packages/_pytest/main.py:609: in matchnodes
    num = len(nodes)
E   TypeError: object of type 'Package' has no len()

[nicoddemus]

Thanks @mdmintz for trying to reproduce the issue! 👍

Why does pytest fail to parse the __init__ file? An import error, syntax error, something else? Can you provide a minimal example that reproduces the issue?

[mdmintz]

@nicoddemus I think all you need is an empty__init__.py file. Just make one and put it anywhere. Then run pytest __init__.py with 3.8.0 (where it fails), and with 3.6.4 (where it properly ignores it).

[nicoddemus]

Oh I see, passing an __init__.py explicitly indeed reproduces the problem, thanks!

[nicoddemus]

PR opened: #4010

[mdmintz]

Awesome! Looks like the travis-ci tests just passed. I hope the next pytest release can follow the PR merge so that everyone can get back to using the latest version (3.8.1?) and not have to downgrade to an earlier version. I'll also update seleniumbase to use the newer pytest.

[nicoddemus]

Yes, we will release 3.8.1 as soon as #4010 is merged. 👍

[SarLecobee]

This is great news, indeed. Thanks for fixing this!