Peeringdb: PeeringDB sync fails on MacOS

Created on 28 Jul 2020  路  32Comments  路  Source: peeringdb/peeringdb

Sync is failing. It fails on IX 3233 which looks like a tunnel IX. Maybe #782 should be rethought? 

Traceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 287, in sync_row
    B.clean(obj)
  File "/usr/local/lib/python3.8/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
    obj.full_clean()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1171, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'ix': ['Internet Exchange instance with id 3233 does not exist.']}
%
picture mahtin
👍3

Most helpful comment

I'll point out that my instance doesn't fail with a "virtual" IXP, and in the network objects. Without digging into the code, I think we can safely rule out that as a possible cause.

picture mcmanuss8 on 3 Aug 2020
👍2

All 32 comments

the brigading continues

job picture job on 28 Jul 2020

Can you provide more debugging information, such as what version you are using, or when the problem started.

pip freeze
job picture job on 28 Jul 2020

The full info:

$ pip3 freeze
certifi==2020.6.20
cfu==1.5.0
chardet==3.0.4
click==7.1.2
Django==1.8.17
django-countries==6.1.2
django-handleref==0.5.0
django-inet==0.4.0
django-peeringdb==2.2.0
future==0.18.2
idna==2.10
munge==1.0.0
peeringdb==1.0.0
protobuf==3.12.3
PyYAML==5.3.1
requests==2.24.0
six==1.15.0
twentyc.rpc==0.4.0
urllib3==1.25.10
$

$ peeringdb sync 
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 7
Ignoring object updated after sync began: (org-26777)
Ignoring object updated after sync began: (org-26825)
Ignoring object updated after sync began: (org-26712)
Ignoring object updated after sync began: (org-26824)
Ignoring object updated after sync began: (org-26183)
Ignoring object updated after sync began: (org-26826)
Ignoring object updated after sync began: (org-26827)
Fetching & updating all: fac
Updates to be processed: 0
Fetching & updating all: net
Updates to be processed: 29
Ignoring object updated after sync began: (net-23126)
Ignoring object updated after sync began: (net-23835)
Ignoring object updated after sync began: (net-4536)
Ignoring object updated after sync began: (net-19200)
Ignoring object updated after sync began: (net-857)
Ignoring object updated after sync began: (net-21576)
Ignoring object updated after sync began: (net-23538)
Ignoring object updated after sync began: (net-23616)
Ignoring object updated after sync began: (net-23617)
Ignoring object updated after sync began: (net-23618)
Ignoring object updated after sync began: (net-23917)
Ignoring object updated after sync began: (net-23958)
Ignoring object updated after sync began: (net-24025)
Ignoring object updated after sync began: (net-719)
Ignoring object updated after sync began: (net-24077)
Ignoring object updated after sync began: (net-23613)
Ignoring object updated after sync began: (net-23918)
Ignoring object updated after sync began: (net-23789)
Ignoring object updated after sync began: (net-24078)
Ignoring object updated after sync began: (net-23864)
Ignoring object updated after sync began: (net-10837)
Ignoring object updated after sync began: (net-6394)
Ignoring object updated after sync began: (net-9966)
Ignoring object updated after sync began: (net-23374)
Ignoring object updated after sync began: (net-10282)
Ignoring object updated after sync began: (net-21944)
Ignoring object updated after sync began: (net-24079)
Ignoring object updated after sync began: (net-4977)
Ignoring object updated after sync began: (net-24080)
Fetching & updating all: ix
Updates to be processed: 1
Ignoring object updated after sync began: (ix-3233)
Fetching & updating all: ixfac
Updates to be processed: 0
Fetching & updating all: ixlan
Updates to be processed: 817
Traceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 287, in sync_row
    B.clean(obj)
  File "/usr/local/lib/python3.8/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
    obj.full_clean()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1171, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'ix': ['Internet Exchange instance with id 3233 does not exist.']}
$
mahtin picture mahtin on 28 Jul 2020

Thanks, does the issue persist after dropping tables to start fresh?

job picture job on 28 Jul 2020

That WAS a fresh table.

mahtin picture mahtin on 28 Jul 2020
😄1

Curious, with an empty database I'd expect the Updates to be processed number to be higher, like so:

(test) job@bench test$ peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 19124
Fetching & updating all: fac
Updates to be processed: 3787
Fetching & updating all: net
Updates to be processed: 19618
Fetching & updating all: ix
Updates to be processed: 818
Fetching & updating all: ixfac
Updates to be processed: 2188
Fetching & updating all: ixlan
Updates to be processed: 817
Fetching & updating all: ixpfx
Updates to be processed: 1568
Fetching & updating all: netfac
Updates to be processed: 30937
Fetching & updating all: netixlan
Updates to be processed: 33829
Fetching & updating all: poc
Updates to be processed: 8780
(test) job@bench test$

Can you upgrade to Django 2.2.9? Or maybe actually empty your database?

job picture job on 28 Jul 2020

Can't reproduce

job picture job on 30 Jul 2020

I'm having the exact same issue.

(venv) % rm peeringdb.sqlite3 
(venv) % peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 19154
Ignoring object updated after sync began: (org-8734)
Ignoring object updated after sync began: (org-26683)
Ignoring object updated after sync began: (org-26855)
Ignoring object updated after sync began: (org-26856)
Ignoring object updated after sync began: (org-26857)
Fetching & updating all: fac
Updates to be processed: 3795
Traceback (most recent call last):
  File "/usr/local/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/usr/local/lib/python3.8/site-packages/peeringdb/_update.py", line 287, in sync_row
    B.clean(obj)
  File "/usr/local/lib/python3.8/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
    obj.full_clean()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1203, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'org': ['Organization instance with id 8734 does not exist.']}
(venv) %

if i wait a few hours, i'll get an error on a different org.

tpaseka picture tpaseka on 31 Jul 2020

Tom, can you show pip freeze and are you on django 2.2.9?

job picture job on 31 Jul 2020

Yes, moved to django (followed this ticket).

% pip3 freeze certifi==2020.6.20 cfu==1.5.0 chardet==3.0.4 click==7.1.2 Django==2.2.9 django-countries==6.1.2 django-handleref==0.5.0 django-inet==0.4.0 django-peeringdb==2.2.0 future==0.18.2 idna==2.10 munge==1.0.0 peeringdb==1.0.0 protobuf==3.12.3 pytz==2020.1 PyYAML==5.3.1 requests==2.24.0 six==1.15.0 sqlparse==0.3.1 twentyc.rpc==0.4.0 urllib3==1.25.10

tpaseka picture tpaseka on 31 Jul 2020

Thanks!

For completeness: what OS? What python version?

job picture job on 31 Jul 2020 
% python3 -V
Python 3.8.5

OSX 10.15.6

tpaseka picture tpaseka on 31 Jul 2020
👍1

I have been running a longitudinal test the last 12 hours, no reproduction so far.

Long shot: do you also see this on non-mac platforms?

job picture job on 1 Aug 2020

I peeringdb sync every 5 minutes for 32+ hours and could not produce any error.

while true; do (time peeringdb sync 2>&1) 2>&1 | tee "log-$(date +%s)"; echo; echo "$(date) - sleeping"; sleep 300; done on openbsd 6.7

Any further help or description on what exactly you do, when you do it to reproduce the issue would be welcome.

Can you run the same test on a non-mac platform as an experiment?

job picture job on 2 Aug 2020

I built and installed on MacOS (Catalina 10.15.5). Ran into a different (but similar) issue with a network not existing, that clearly does: https://www.peeringdb.com/net/2425

```
% /usr/local/bin/pip3 freeze | grep -i django
Django==2.2.9
django-countries==6.1.2
django-handleref==0.5.0
django-inet==0.4.0
django-peeringdb==2.2.0
% /usr/local/bin/pip3 freeze | grep -i peering
django-peeringdb==2.2.0
peeringdb==1.0.0
% head -n 1 /usr/local/bin/peeringdb

!/usr/local/opt/python/bin/python3.7

% /usr/local/opt/python/bin/python3.7 --version
Python 3.7.6
% /usr/local/bin/peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 19175
Ignoring object updated after sync began: (org-19478)
Ignoring object updated after sync began: (org-26881)
Fetching & updating all: fac
Updates to be processed: 3795
Fetching & updating all: net
Updates to be processed: 19673
Ignoring object updated after sync began: (net-2425)
Ignoring object updated after sync began: (net-10335)
Ignoring object updated after sync began: (net-14796)
Ignoring object updated after sync began: (net-15745)
Ignoring object updated after sync began: (net-16245)
Ignoring object updated after sync began: (net-19151)
Ignoring object updated after sync began: (net-23047)
Ignoring object updated after sync began: (net-24132)
Ignoring object updated after sync began: (net-24133)
Ignoring object updated after sync began: (net-24137)
Fetching & updating all: ix
Updates to be processed: 815
Ignoring object updated after sync began: (ix-1896)
Fetching & updating all: ixfac
Updates to be processed: 2188
Fetching & updating all: ixlan
Updates to be processed: 814
Ignoring object updated after sync began: (ixlan-1896)
Fetching & updating all: ixpfx
Updates to be processed: 1566
Ignoring object updated after sync began: (ixpfx-2463)
Ignoring object updated after sync began: (ixpfx-2464)
Fetching & updating all: netfac
Updates to be processed: 31017
Ignoring object updated after sync began: (netfac-43093)
Ignoring object updated after sync began: (netfac-43094)
Ignoring object updated after sync began: (netfac-43092)
Traceback (most recent call last):
File "/usr/local/bin/peeringdb", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/peeringdb/cli.py", line 62, in main
return handler(config=cfg, *vars(options))
File "/usr/local/lib/python3.7/site-packages/peeringdb/commands.py", line 20, in _wrapped
r = func(a, *k)
File "/usr/local/lib/python3.7/site-packages/peeringdb/commands.py", line 219, in handle
client.update_all(rs)
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in update_all
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 78, in _atomic_update
sync_func()
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
File "/usr/local/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
return loop.run_until_complete(func(a, **k))
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
item = gen.send(r)
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 287, in sync_row
B.clean(obj)
File "/usr/local/lib/python3.7/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
obj.full_clean()
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 1203, in full_clean
raise ValidationError(errors)
django.core.exceptions.ValidationError: {'net': ['Network instance with id 2425 does not exist.']}

And a subsequent run:
% /usr/local/bin/peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 2
Ignoring object updated after sync began: (org-19478)
Ignoring object updated after sync began: (org-26881)
Fetching & updating all: fac
Updates to be processed: 0
Fetching & updating all: net
Updates to be processed: 11
Ignoring object updated after sync began: (net-16245)
Ignoring object updated after sync began: (net-15745)
Ignoring object updated after sync began: (net-23047)
Ignoring object updated after sync began: (net-19151)
Ignoring object updated after sync began: (net-14796)
Ignoring object updated after sync began: (net-10335)
Ignoring object updated after sync began: (net-24132)
Ignoring object updated after sync began: (net-24133)
Ignoring object updated after sync began: (net-24137)
Ignoring object updated after sync began: (net-21889)
Fetching & updating all: ix
Updates to be processed: 1
Ignoring object updated after sync began: (ix-1896)
Fetching & updating all: ixfac
Updates to be processed: 0
Fetching & updating all: ixlan
Updates to be processed: 1
Ignoring object updated after sync began: (ixlan-1896)
Fetching & updating all: ixpfx
Updates to be processed: 2
Ignoring object updated after sync began: (ixpfx-2463)
Ignoring object updated after sync began: (ixpfx-2464)
Fetching & updating all: netfac
Updates to be processed: 31017
Ignoring object updated after sync began: (netfac-43093)
Ignoring object updated after sync began: (netfac-43094)
Ignoring object updated after sync began: (netfac-43092)
Traceback (most recent call last):
File "/usr/local/bin/peeringdb", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/peeringdb/cli.py", line 62, in main
return handler(config=cfg, *vars(options))
File "/usr/local/lib/python3.7/site-packages/peeringdb/commands.py", line 20, in _wrapped
r = func(a, *k)
File "/usr/local/lib/python3.7/site-packages/peeringdb/commands.py", line 219, in handle
client.update_all(rs)
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in update_all
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 78, in _atomic_update
sync_func()
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
File "/usr/local/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
return loop.run_until_complete(func(a, **k))
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
item = gen.send(r)
File "/usr/local/lib/python3.7/site-packages/peeringdb/_update.py", line 287, in sync_row
B.clean(obj)
File "/usr/local/lib/python3.7/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
obj.full_clean()
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 1203, in full_clean
raise ValidationError(errors)
django.core.exceptions.ValidationError: {'net': ['Network instance with id 15745 does not exist.']}

mcmanuss8 picture mcmanuss8 on 3 Aug 2020
👍1

I'll point out that my instance doesn't fail with a "virtual" IXP, and in the network objects. Without digging into the code, I think we can safely rule out that as a possible cause.

mcmanuss8 picture mcmanuss8 on 3 Aug 2020
👍2 
django.core.exceptions.ValidationError: {'ix': ['Internet Exchange instance with id 1522 does not exist.']}

deleting the db, and starting again, i get error on a new network/ix/fac. i'm going to spin up a VM now to test.

tpaseka picture tpaseka on 3 Aug 2020
👍1

I'll point out that my instance doesn't fail with a "virtual" IXP

Please don't feed the trolls, @mcmanuss8 ;-) ... all ix look equal from our pov :)

arnoldnipper picture arnoldnipper on 3 Aug 2020 
(venv) root@debian:~/peeringdb# peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 19175
Ignoring object updated after sync began: (org-19478)
Ignoring object updated after sync began: (org-26881)
Fetching & updating all: fac
Updates to be processed: 3795
Fetching & updating all: net
Updates to be processed: 19673
Ignoring object updated after sync began: (net-2425)
Ignoring object updated after sync began: (net-10335)
Ignoring object updated after sync began: (net-14796)
Ignoring object updated after sync began: (net-15745)
Ignoring object updated after sync began: (net-16245)
Ignoring object updated after sync began: (net-19151)
Ignoring object updated after sync began: (net-21889)
Ignoring object updated after sync began: (net-22868)
Ignoring object updated after sync began: (net-23047)
Ignoring object updated after sync began: (net-24112)
Ignoring object updated after sync began: (net-24130)
Ignoring object updated after sync began: (net-24132)
Ignoring object updated after sync began: (net-24133)
Ignoring object updated after sync began: (net-24137)
Fetching & updating all: ix
Updates to be processed: 815
Ignoring object updated after sync began: (ix-1896)
Fetching & updating all: ixfac
Updates to be processed: 2188
Fetching & updating all: ixlan
Updates to be processed: 814
Ignoring object updated after sync began: (ixlan-1896)
Fetching & updating all: ixpfx
Updates to be processed: 1566
Ignoring object updated after sync began: (ixpfx-2463)
Ignoring object updated after sync began: (ixpfx-2464)
Fetching & updating all: netfac
Updates to be processed: 31017
Ignoring object updated after sync began: (netfac-43093)
Ignoring object updated after sync began: (netfac-43094)
Ignoring object updated after sync began: (netfac-43092)
Traceback (most recent call last):
  File "/root/venv/bin/peeringdb", line 8, in <module>
    sys.exit(main())
  File "/root/venv/lib/python3.7/site-packages/peeringdb/cli.py", line 62, in main
    return handler(config=cfg, **vars(options))
  File "/root/venv/lib/python3.7/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
  File "/root/venv/lib/python3.7/site-packages/peeringdb/commands.py", line 219, in handle
    client.update_all(rs)
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in update_all
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 78, in _atomic_update
    sync_func()
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in <lambda>
    self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped
    return loop.run_until_complete(func(*a, **k))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped
    item = gen.send(r)
  File "/root/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 287, in sync_row
    B.clean(obj)
  File "/root/venv/lib/python3.7/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean
    obj.full_clean()
  File "/root/venv/lib/python3.7/site-packages/django/db/models/base.py", line 1203, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'net': ['Network instance with id 2425 does not exist.']}
(venv) root@debian:~/peeringdb#

in a brand new, clean VM.

tpaseka picture tpaseka on 4 Aug 2020

The issue appears to be in how the scripts identify the timezones, or lack of.

In europe, you're lucky to be ahead of UTC, so you're in the future of any changes that block the updates, in the US, we are behind, so updates get held back.

to experiment, 

(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# date
Wed 05 Aug 2020 06:25:01 PM PDT
# peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 50
Ignoring object updated after sync began: (org-26847)
Ignoring object updated after sync began: (org-26909)
Ignoring object updated after sync began: (org-26908)
Ignoring object updated after sync began: (org-26910)
Ignoring object updated after sync began: (org-26718)
Ignoring object updated after sync began: (org-26911)
Ignoring object updated after sync began: (org-26912)
Fetching & updating all: fac
Updates to be processed: 3
Ignoring object updated after sync began: (fac-9006)
Fetching & updating all: net
Updates to be processed: 129
Ignoring object updated after sync began: (net-19080)
Ignoring object updated after sync began: (net-4459)
Ignoring object updated after sync began: (net-24164)
^C
(venv) root@debian:~/peeringdb# TZ='Australia/Sydney'; export TZ
(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# date
Thu 06 Aug 2020 11:25:09 AM AEST
(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# 
(venv) root@debian:~/peeringdb# rm peeringdb.sqlite3 
(venv) root@debian:~/peeringdb# peeringdb sync
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 19205
Fetching & updating all: fac
Updates to be processed: 3797
Fetching & updating all: net
Updates to be processed: 19702

after changing timezone, all records correctly import and sync.

tpaseka picture tpaseka on 6 Aug 2020

ran it again.

(venv) root@debian:~/peeringdb# TZ='America/Los_Angeles'; export TZ (venv) root@debian:~/peeringdb# (venv) root@debian:~/peeringdb# rm peeringdb.sqlite3 (venv) root@debian:~/peeringdb# peeringdb sync Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 19206 Ignoring object updated after sync began: (org-26718) Ignoring object updated after sync began: (org-26911) Ignoring object updated after sync began: (org-26912) Fetching & updating all: fac Updates to be processed: 3797 Fetching & updating all: net Updates to be processed: 19703 Ignoring object updated after sync began: (net-2284) Ignoring object updated after sync began: (net-5433) Ignoring object updated after sync began: (net-10342) Ignoring object updated after sync began: (net-11363) Ignoring object updated after sync began: (net-16784) Ignoring object updated after sync began: (net-21895) Ignoring object updated after sync began: (net-22177) Ignoring object updated after sync began: (net-24165) Ignoring object updated after sync began: (net-24166) Ignoring object updated after sync began: (net-24167) Traceback (most recent call last): File "/root/peeringdb/venv/bin/peeringdb", line 8, in <module> sys.exit(main()) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 66, in <lambda> self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped return loop.run_until_complete(func(*a, **k)) File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete return future.result() File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped item = gen.send(r) File "/root/peeringdb/venv/lib/python3.7/site-packages/peeringdb/_update.py", line 287, in sync_row B.clean(obj) File "/root/peeringdb/venv/lib/python3.7/site-packages/django_peeringdb/client_adaptor/backend.py", line 150, in clean obj.full_clean() File "/root/peeringdb/venv/lib/python3.7/site-packages/django/db/models/base.py", line 1203, in full_clean raise ValidationError(errors) django.core.exceptions.ValidationError: {'org': ['Organization instance with id 26718 does not exist.']} (venv) root@debian:~/peeringdb# (venv) root@debian:~/peeringdb# (venv) root@debian:~/peeringdb# (venv) root@debian:~/peeringdb# TZ='Australia/Sydney'; export TZ (venv) root@debian:~/peeringdb# rm peeringdb.sqlite3 (venv) root@debian:~/peeringdb# peeringdb sync Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 19206 Fetching & updating all: fac Updates to be processed: 3797 Fetching & updating all: net Updates to be processed: 19703 Fetching & updating all: ix Updates to be processed: 815 Fetching & updating all: ixfac Updates to be processed: 2190 Fetching & updating all: ixlan Updates to be processed: 814 Fetching & updating all: ixpfx Updates to be processed: 1566 Fetching & updating all: netfac Updates to be processed: 31029 Fetching & updating all: netixlan Updates to be processed: 33934 Fetching & updating all: poc Updates to be processed: 8839 (venv) root@debian:~/peeringdb#

so this is absolutely to do with TZ, sadly, I wouldn't know where to look in the code nor have the understanding of python for where it could be.

tpaseka picture tpaseka on 6 Aug 2020

Is the TZ environment variable set at all normally? What do you see from this?

echo $TZ
shane-kerr picture shane-kerr on 6 Aug 2020

The issue appears to be in how the scripts identify the timezones, or lack of.

Wasn't this issue not dealt with in https://github.com/peeringdb/peeringdb-py/issues/36 already?

arnoldnipper picture arnoldnipper on 6 Aug 2020

@arnoldnipper looks like it is still broken.

tpaseka picture tpaseka on 6 Aug 2020

@shane-kerr 

% echo $TZ

% 
% TZ='America/Los_Angeles'; export TZ
% echo $TZ                           
America/Los_Angeles
% peeringdb sync
Syncing to https://www.peeringdb.com/api
<...snip>
django.core.exceptions.ValidationError: {'org': ['Organization instance with id 26441 does not exist.']}
%

so, still broken

tpaseka picture tpaseka on 6 Aug 2020

so, still broken

What happens when you set TZ to UTC, @tpaseka?

arnoldnipper picture arnoldnipper on 6 Aug 2020

Is this still open, @job

arnoldnipper picture arnoldnipper on 6 Sep 2020

yes, still broken when TZ is left of UTC. When TZ is right of UTC, it works.

tpaseka picture tpaseka on 9 Sep 2020

@20c and/or @job do we have no idea how to fix it?

arnoldnipper picture arnoldnipper on 9 Sep 2020

I don't have the time to debug which function barks on the timezone.

The workaround for now is to run the tool in UTC

job picture job on 9 Sep 2020

I don't see the fix in the 1.0.0 release, but it is in the 1.1.0 release. Can you try updating the the peeringdb package and see if it is fixed?

mngan picture mngan on 19 Sep 2020

Should be fixed in 1.1.0, yes. @tpaseka could you confirm when you get a chance please?

grizz picture grizz on 26 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

vegu picture vegu  路  7Comments
gordon-shumway-net picture gordon-shumway-net  路  9Comments
netravnen picture netravnen  路  14Comments
netravnen picture netravnen  路  4Comments
stillwaxin picture stillwaxin  路  10Comments