Readthedocs.org: Warning about using sqlite 3.26.0 for development

[raulcd]

Details

Hi,
I was trying to build locally as specified on the developer documentation and I've faced an issue while loading the test_data fixtures:

$ python manage.py loaddata test_data
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: no such table: projects_project__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 115, in loaddata
    connection.check_constraints(table_names=table_names)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 286, in check_constraints
    column_name, referenced_column_name,
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: Problem installing fixtures: no such table: projects_project__old

There was a change of behaviour of sqlite 3.26.0 which is the new default with python 3.6. Django had some issues that were fixed on Djando 2.1.5 (I think) and 2.2 see issue here: https://code.djangoproject.com/ticket/29182

Expected Result

Test fixtures are loaded when using python3.6 with the default sqlite:

$ echo $(python -c "import sqlite3; print(sqlite3.version); print(sqlite3.sqlite_version)")
2.6.0 3.26.0

Actual Result

Fixtures fail to load

Reproduce easily with the following Dockerfile:

FROM python:3.6-alpine

WORKDIR /usr/src/app

COPY requirements.txt .
COPY requirements ./requirements

RUN apk update && \
  apk add --no-cache git build-base libxml2 libxml2-dev libxslt-dev && \
  pip install --no-cache-dir -r requirements.txt

COPY . .

RUN python manage.py migrate
RUN echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', '[email protected]', 'pass')" | python manage.py shell
RUN python manage.py collectstatic
RUN python manage.py loaddata test_data

CMD ["python", "manage.py", "runserver"]

run:
docker build -t readthedocs

[raulcd]

I've been able to make it run by downgrading the python3.6 version to 3.6.7 instead of 3.6.8. Which basically uses a lower version of sqlite3. New Dockerfile working:

FROM python:3.6.7-alpine

WORKDIR /usr/src/app

COPY requirements.txt .
COPY requirements ./requirements

RUN apk update && \
  apk add --no-cache git build-base libxml2 libxml2-dev libxslt-dev && \
  pip install --no-cache-dir -r requirements.txt

COPY . .

RUN python manage.py migrate
RUN echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', '[email protected]', 'pass')" | python manage.py shell
RUN python manage.py collectstatic
RUN python manage.py loaddata test_data

CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

and the version of sqlite that uses is lower than 3.26.0:

docker run -it readthedocs /bin/sh
/usr/src/app # python
Python 3.6.7 (default, Dec 21 2018, 03:29:53) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.24.0'

[stsewd]

Please see https://github.com/rtfd/readthedocs.org/issues/5385#issuecomment-468860491

[humitos]

I think we should add a note/comment/warning in our developers guide that talks about how to start because it's our official guide to start contributing and all new contributors will hit this issue.

This won't be fixed in our project until we upgrade our Django version.

[saadmk11]

@humitos we can add a warning pointing the user to the ticket and also provide them with the fix. ill add a PR for this.

[davidfischer]

I just wanted to add a note that I saw this same issue with sqlite 3.27.1 as well.

[raulcd]

The issue happens with sqlite >=3.26.0

[humitos]

5491 was merged but it does not provide a solution to the developer and the warning note is not 100% accurate since it could happen that even using 3.6.7 you get a wrong libsqlite version and our project won't work.

[saadmk11]

@humitos I think we can tell the users something like this.

Read the Docs is currently using Django version 1.11 and it has a bug which breaks database migrations if you are using sqlite 3.26.0 or Newer. So, we recommend using sqlite < 3.26.0 to run Read the Docs properly on your machine.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

Marking the issue as Accepted since it's something that still needs to be fixed.

[saadmk11]

@humitos I'll add a PR on this.