Netbox: Custom Fields Migration error 2.10.1

[rebortg]

Environment

• Python version: 3.6.9
• NetBox version: migration from 2.9.4 to 2.10.1

Steps to Reproduce

1. have some custom fields in 2.9.4
2. ./upgrade.sh

Expected Behavior

Observed Behavior

Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, dcim, extras, ipam, secrets, sessions, taggit, tenancy, users, virtualization
Running migrations:
  Applying extras.0051_migrate_customfields...Traceback (most recent call last):
  File "netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/netbox-2.10.1/venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox-2.10.1/netbox/extras/migrations/0051_migrate_customfields.py", line 69, in migrate_customfieldvalues
    cf_data['custom_field_data'][cfv.field.name] = deserialize_value(cfv.field, cfv.serialized_value)
TypeError: 'NoneType' object is not subscriptable

i insert some prints and cf_data is None from this DB Query:
SELECT "dcim_rack"."custom_field_data" FROM "dcim_rack" INNER JOIN "dcim_site" ON ("dcim_rack"."site_id" = "dcim_site"."id") LEFT OUTER JOIN "dcim_rackgroup" ON ("dcim_rack"."group_id" = "dcim_rackgroup"."id") LEFT OUTER JOIN "dcim_site" T4 ON ("dcim_rackgroup"."site_id" = T4."id") WHERE "dcim_rack"."id" = 1 ORDER BY "dcim_site"."_name" ASC, T4."_name" ASC, "dcim_rackgroup"."name" ASC, "dcim_rack"."_name" ASC, "dcim_rack"."id" ASC

this field produce the error.

[jeremystretch]

Could you please update your post to include the complete stack trace? It appears that the exception has been cut off.

[rebortg]

oh yes sorry, updated above.

[jeremystretch]

It appears that you have one or more orphaned CustomFieldValue instances which are assigned to non-existent objects. (This should never happen in practice, though the database does allow for it due to the use of GenericForeignKey for assignment.)

https://github.com/netbox-community/netbox/blob/4ce7dfa55edc0ddc1229c22ad837bcb959c4ec60/netbox/extras/migrations/0051_migrate_customfields.py#L67

If cf_data is None, it means that no parent object was found. In this case, it would seem safe to simply ignore the CustomFieldValue instance. Are you able to confirm this is what's happening? To test, you can modify the line above to something like the following:

parent_object = model.objects.filter(pk=cfv.obj_id).first()
if not parent_object:
    print(f"CFV {cfv.pk} assigned to missing object: {model} {cfv.obj_id}")
    pass

Running the migration again will skip any invalid CFVs, and should complete successfully. (Do note that once all migrations finish, any skipped CFVs will be deleted, so be sure to make a backup of your database first.)

[csallen1204]

Experiencing the same issue. Upgraded from 2.9.11 to 2.10, ran the upgrade script, and Netbox won't start:

Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m Traceback (most recent call last):
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/opt/netbox/netbox/./manage.py", line 10, in <module>
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m 🧬 loaded config '/etc/netbox/config/configuration.py'
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m 🧬 loaded config '/etc/netbox/config/configuration.py'
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m 🧬 loaded config '/etc/netbox/config/extra.py'
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m ▶️ Running the startup script /opt/netbox/startup_scripts/000_users.py
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m ▶️ Running the startup script /opt/netbox/startup_scripts/010_groups.py
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m ▶️ Running the startup script /opt/netbox/startup_scripts/020_custom_fields.py
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     execute_from_command_line(sys.argv)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     utility.execute()
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     self.fetch_command(subcommand).run_from_argv(self.argv)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 330, in run_from_argv
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     self.execute(*args, **cmd_options)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     output = self.handle(*args, **options)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 93, in handle
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     exec(sys.stdin.read())
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "<string>", line 1, in <module>
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/runpy.py", line 285, in run_path
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     return _run_code(code, mod_globals, init_globals,
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     exec(code, run_globals)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "../startup_scripts/__main__.py", line 25, in <module>
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     runpy.run_path(f.path)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/runpy.py", line 268, in run_path
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     return _run_module_code(code, init_globals, run_name,
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/runpy.py", line 97, in _run_module_code
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     _run_code(code, mod_globals, init_globals,
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     exec(code, run_globals)
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m   File "/opt/netbox/startup_scripts/020_custom_fields.py", line 1, in <module>
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m     from extras.models import CustomField, CustomFieldChoice
Dec 15 20:25:32 netbox docker-compose: #033[33mnetbox_1         |#033[0m ImportError: cannot import name 'CustomFieldChoice' from 'extras.models' (/opt/netbox/netbox/extras/models/__init__.py)

[jeremystretch]

ImportError: cannot import name 'CustomFieldChoice' from 'extras.models'

@csallen1204 This is an unrelated issue pertaining to code that's not part of the core NetBox project (startup_scripts/020_custom_fields.py). If you're using the community Docker image, please raise a bug against that project.

[csallen1204]

ImportError: cannot import name 'CustomFieldChoice' from 'extras.models'

@csallen1204 This is an unrelated issue pertaining to code that's not part of the core NetBox project (startup_scripts/020_custom_fields.py). If you're using the community Docker image, please raise a bug against that project.

Thanks @jeremystretch I found the script issue, fixed my version and reported it here https://github.com/netbox-community/netbox-docker/issues/373 in case anyone else lands here thinking it's the same issue.

[rebortg]

Yes, this is a very old installation back to 2.2 or 2.3 maybe there some artifacts in the DB. i used your code snippet in the 0051_migrate_customfields.py file and the migration run without an error. At the moment i don't see any issues with this custom field after the migration.

[jeremystretch]

I'm going to close this out as it appears to be resolved.