Netbox: ImportError exception with django-rq v2.3.1 and rq 1.4.0

[tyler-8]

Environment

• Python version: 3.6.8
• NetBox version: 2.7.12

Steps to Reproduce

1. Be on version v2.6.12
2. Upgrade to Netbox v2.7.12 (upgrade.sh or Ansible playbook in my case)
3. Attempt migrations (./manage.py migrate either manually or automatically as part of step 2)

Based on the below errors and the inclusion of django-rq in the traceback, I went looking through the release history of django-rq and it's dependency, rq to see if anything stood out. The latter has seen 4 moderately sized releases since the release of NetBox v2.7.12 in December. I had a hunch that may have something to do with it, so I installed rq==1.1.0 (the most recent version at the time 2.7.12 was released) manually into my virtualenv, then attempted migrations and they worked!

I'm not entirely certain what is causing the problem but downgrading rq definitely solved it. I think this goes back to https://github.com/netbox-community/netbox/issues/4198 and that NetBox releases should be pinning all requirements used in the development environment. This would've been a non-issue if the full dependency chain was frozen.

It's worth noting that v1.4.0 of rq was released several hours before my upgrade attempt - which would explain why my previous tests of upgrading my environment (several days ago) went without any issues. It's possible rq v1.3.0 would work but haven't had a chance to test all the versions after 1.1.0.

And to be clear, this upgrade was done using a fresh venv in an isolated/versioned folder for the v2.7.12 install. I even wiped the whole folder & venv and ran my playbook again - same results.

Expected Behavior

Migrations complete successfully.

Observed Behavior

Migration fails with output below

Traceback (most recent call last):
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/utils.py\", line 66, in __getitem__
    return self._engines[alias]
KeyError: 'django'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/backends/django.py\", line 121, in get_package_libraries
    module = import_module(entry[1])
  File \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File \"<frozen importlib._bootstrap>\", line 994, in _gcd_import
  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load
  File \"<frozen importlib._bootstrap>\", line 955, in _find_and_load_unlocked
  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked
  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module
  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django_rq/templatetags/django_rq.py\", line 4, in <module>
    from rq.exceptions import UnpickleError
ImportError: cannot import name 'UnpickleError'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"./manage.py\", line 10, in <module>
    execute_from_command_line(sys.argv)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line
    utility.execute()
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/__init__.py\", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/base.py\", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/base.py\", line 361, in execute
    self.check()
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/base.py\", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/commands/migrate.py\", line 65, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/management/base.py\", line 377, in _run_checks
    return checks.run_checks(**kwargs)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/core/checks/registry.py\", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/contrib/admin/checks.py\", line 79, in check_dependencies
    for engine in engines.all():
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/utils.py\", line 90, in all
    return [self[alias] for alias in self]
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/utils.py\", line 90, in <listcomp>
    return [self[alias] for alias in self]
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/utils.py\", line 81, in __getitem__
    engine = engine_cls(params)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/backends/django.py\", line 25, in __init__
    options['libraries'] = self.get_templatetag_libraries(libraries)
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/backends/django.py\", line 43, in get_templatetag_libraries
    libraries = get_installed_libraries()
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/backends/django.py\", line 108, in get_installed_libraries
    for name in get_package_libraries(pkg):
  File \"/srv/netbox/venv/lib64/python3.6/site-packages/django/template/backends/django.py\", line 125, in get_package_libraries
    \"trying to load '%s': %s\" % (entry[1], e)
django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'django_rq.templatetags.django_rq': cannot import name 'UnpickleError'

[Ben0it]

Same error here while upgrading from 2.7.11 to 2.8.3.

Adding rq==1.2 in requirements.txt just before line django-rq==2.3.1 allowed the upgrade to complete successfully.

[arnoldthebat]

For ref: rq==1.3.0 worked for my build (currently on 2.8.3). 1.1.0 still failed with other dependencies : ImportError: cannot import name 'ScheduledJobRegistry'

[jeremystretch]

This has been fixed upstream in django-rq v2.3.2. We'll update our pinned dependency for the next NetBox release. In the meantime, the workaround is simply pip install django-rq==2.3.2. (You'll want to keep rq at version 1.3.0 or later.)