Coveragepy: Latest 5.0 version raises CoverageException: file is not a database

[ellbosch]

Our coverage report is failing as of this weekend. It's failing on Python 2 and 3.

Stack trace below. Thanks for the help!

Traceback (most recent call last):
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 1025, in execute
    return self.con.execute(sql, parameters)
sqlite3.DatabaseError: file is not a database

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/bin/pytest", line 8, in <module>
    sys.exit(main())
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 72, in main
    config = _prepareconfig(args, plugins)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 223, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 793, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 999, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py", line 957, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py", line 118, in pytest_load_initial_conftests
    plugin = CovPlugin(early_config.known_args_namespace, early_config.pluginmanager)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py", line 164, in __init__
    self.start(engine.Central)
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pytest_cov/plugin.py", line 186, in start
    self.cov_controller.start()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/pytest_cov/engine.py", line 187, in start
    self.cov.load()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/control.py", line 387, in load
    self._data.read()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 741, in read
    with self._connect():       # TODO: doesn't look right
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 296, in _connect
    self._open_db()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 264, in _open_db
    self._read_db()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 268, in _read_db
    with self._dbs[get_thread_id()] as db:
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 1008, in __enter__
    self._connect()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 996, in _connect
    self.execute("pragma journal_mode=off").close()
  File "/Users/ellbosch/Repos/mssql-cli/.tox/py37/lib/python3.7/site-packages/coverage/sqldata.py", line 1027, in execute
    raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
coverage.misc.CoverageException: Couldn't use data file '/Users/ellbosch/Repos/mssql-cli/.coverage': file is not a database
Command '['pytest', '-l', '--cov', 'mssqlcli', '--doctest-modules', '--junitxml=junit/test-6d62ae30-2031-11ea-80ec-3af9d39e6caa-results.xml', '--cov-report=xml', '--cov-report=html', '--cov-append', '-o', 'junit_suite_name=pytest-3.7.4', '-s', '-m', 'not unstable', 'tests/jsonrpc']' returned non-zero exit status 1.
ERROR: InvocationError for command /Users/ellbosch/Repos/mssql-cli/.tox/py37/bin/python build.py unit_test (exited with code 1)

[ellbosch]

Following up--I've isolated this issue to the latest 5.0 release.

[nedbat]

Can you provide me with instructions to reproduce the issue?

[ellbosch]

I only seem to run into this error when attempting to run tox with coverage reports. You should be able to repro this with our repo here. tox.ini shows what's run on tox. The reference to coverage is in the unit_test method inside build.py.

Interestingly, it seems to work fine when calling the unit_test method without tox.

[bhearsum]

I can also reproduce in https://github.com/mozilla-releng/tooltool/ by running:

git clone https://github.com/mozilla-releng/tooltool/
cd tooltool/api
tox -e py37
coverage xml

[willienillie]

I saw a similar issue when running pytest-cov and coverage through tox. It was resolved by pinning coverage to back to 4.5.4.

raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, exc))
INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/my/repo/location/.coverage': Safety level may not be changed inside a transaction

[nedbat]

@mozbhearsum Thanks for the steps. When I try them, it doesn't fail, I thought because you have already pinned coverage==4.5.4 on your master. But when I move back a few commits (to ac7a85d), requirements/test.txt still has coverage pinned. I tried hand-editing the file to use coverage==5.0, but it needs hashes so it wouldn't install.

Do you mind giving me a specific hash to check out from your repo that will show the problem?

[bhearsum]

@mozbhearsum Thanks for the steps. When I try them, it doesn't fail, I thought because you have already pinned coverage==4.5.4 on your master. But when I move back a few commits (to ac7a85d), requirements/test.txt still has coverage pinned. I tried hand-editing the file to use coverage==5.0, but it needs hashes so it wouldn't install.

Do you mind giving me a specific hash to check out from your repo that will show the problem?

Yeah, that's right. ac7a85d073fae891aa9166253069f46bb8b30c3e should reproduce.

[nedbat]

I tried ac7a85d. It pins coverage==4.5.4: https://github.com/mozilla-releng/tooltool/blob/ac7a85d073fae891aa9166253069f46bb8b30c3e/api/requirements/test.txt#L22

[Urth]

On our project this was caused by mixing versions of coverage.py
I think it would help users if coverage.py 5 would check if the .coverage file format is from version 4 and report it as a error.

$ head -c63 .coverage 
!coverage.py: This is a private format, don't read it directly!
$ coverage xml
Couldn't use data file '/home/harm/tooltool/api/.coverage': file is not a database
$ coverage --version
Coverage.py, version 5.0 with C extension
Full documentation is at https://coverage.readthedocs.io

[cjolowicz]

On our project this was caused by mixing versions of coverage.py

Same here. The test suite used a pinned version (4.5.4) of coverage.py to generate coverage data. The session generating an XML report for upload to Codecov used the latest version (5.0), resulting in said error message.

Aligning the versions fixes the issue:

https://github.com/cjolowicz/hypermodern-python/commit/32674c2fcff270c52b5920368a4e014a48bc1683

[nedbat]

I've made the error message specific in 0d6a8784. Now it will say "Couldn't use data file '.coverage': Looks like a coverage 4.x data file. Are you mixing versions of coverage?"

If you are encountering this error, I strongly encourage you not to simply pin coverage==4.5.4, but instead to figure out where the mixed versions are coming from, and upgrade everything to 5.x.

[nedbat]

This is now available in coverage==5.0.1

Please upgrade. If the problem persists, let's debug why. :)

[lzkelley]

I'm getting this new error message after upgrading coverage from 4.5.4 to 5.0.4, when running unit tests on a package that previously used coverage 4. Is there a procedure for moving from 4 --> 5? Is there a way to have it update the .coverage file? Thanks!

[nedbat]

@lzkelley can you provide me with a way to reproduce your problem?