Peeringdb: IX-F JSON importer indication of conflict rather than deletion of data

[ccaputo]

Extracted from https://github.com/peeringdb/peeringdb/issues/518 to avoid confusion with its proposal:

In the past, part of the idea with the IX-F JSON importer is that once a network departs an IXP, the IXP can cause PeeringDB to be automatically updated so that the network is no longer listed.

BUT, this is a problem if the IXPs exporter is buggy or not current, since BGP provisioning is happening more and more based on PeeringDB data.

Thus, regardless of allow_ixp_update setting, I suggest the most PeeringDB should do is have a visible indicator which indicates whether the currently populated data is not valid-per-ixp. Could be as simple as coloring the IP addresses red and/or making them italic if they don't match the IX-F JSON data. This flag could also be in the PeeringDB database & REST API so that provisioning software can examine it and decide whether to consider the IXP IX-F JSON data authoritative or not. Choice in the user's hands.

Thus I suggest the following course of action:

1) PeeringDB immediately stop removing data when allow_ixp_update == no. We can disable the cron job on the production site with ease.

2) Revise the IX-F JSON importer to no longer remove data when allow_ixp_update == no.

3) Have the Product Committee scope out a new flag, possibly named "valid-per-ixp" and drive it to implementation both in the GUI and database/REST API. Off the top of my head, the flag settings could be {true|false|undetermined}.

If the above is adopted, https://docs.peeringdb.com/ix-f-json-import-rules/ could be revised as follows:

Remove: "Generally, if a network does not have an entry in the IX member list, the network IX entry is removed."

Change: "allow_ixp_update: no - If a network has an IX entry with differing (asn, ipaddr4, ipaddr6), the network IX entry is removed"

to be: "allow_ixp_update: no - If a network has an IX entry with differing (asn, ipaddr4, ipaddr6), the conflict will be indicated on the website and in the API."

Question for all: Would setting to red and/or italic those IP addresses which are in conflict with IX-F JSON data, on both ix and net pages, be considered overstepping into the realm of user-controlled data?

Feedback received so far...

From @nbakker:

@ccaputo I like it.

From @fkorsback:

Im aligned with @ccaputo here. Flagging the data in a feasible way is definately a good approach to this, make it red, make it italic, make a tickbox next to it. Im cool with either option really. I dont want any auto-deletion of data, even after 120 hours or 120 days. If people don't delete their own wrong data after several reminders and constant api+ui flagging id say its up to a human in the admin-group to determine if the entry should be deleted.

And for those that do accept IX-F updates, then keep the operations as it is today.

As a user of "do not allow IXPDB updates" i fully expect that the only ones that can modify ourrecord is approved admins from our organisation, and perhaps someone from the committee if a resolution cant be self-serviced within appropriate time.

This should solve both problems at once really?

[grizz]

+1

I think this can replace #518, what do you think @arnoldnipper

[arnoldnipper]

I think this can replace #518, what do you think @arnoldnipper

I'm still in favour of auto-deleting records as we also have discussed this with ASN. And #518 could easily be modified to take into account ideas of this proposal.

What we've not yet talked about in detail is the uniqueness of IP addresses. That would have to be abandoned for disputed addresses e.g.

So, what about introducing an attribute-value pair marked_for_deletion for any record. Its value would be a Unix timestamp initially set to a time in the future. If that value is in the past, the record will be deleted. Furthermore, when set uniqueness is no longer needed. In the GUI a marked-for-deletion record could e.g. be set in _italics_

Thoughts?

[durkovic]

If peeringdb is going this way, it should be IMHO possible to show all these states:

1) network's record in sync with IX-F json record
2) network's record exists, but no matching record in IX-F json
3) record present in IX-F json but no matching network's record exists
4) both records exist, but they don't match

Our experience is that manually entered records by networks are much more inaccurate (completely missing, outdated etc.) than automated IX-F json export generated directly from IXP's provisioning database.

[ccaputo]

If peeringdb is going this way, it should be IMHO possible to show all these states:

1. network's record in sync with IX-F json record
2. network's record exists, but no matching record in IX-F json
3. record present in IX-F json but no matching network's record exists
4. both records exist, but they don't match

Our experience is that manually entered records by networks are much more inaccurate (completely missing, outdated etc.) than automated IX-F json export generated directly from IXP's provisioning database.

Interesting ideas and I am certainly open to these additions if there is further support for them.

[fkorsback]

We are still getting hit by this constantly and its seriously affecting our usage of PeeringDB.

Raising this to stewards@ mailing-list so we can get a short and long-term solution going.

[arnoldnipper]

@ccaputo would you mind looking at #585 where part of this issue is addressed? The other part will be addressed in the Data Ownership TF. If you still think this issue addresses a slightly different issue, please let us know. Otherwise, please close it.

[ccaputo]

Closing, even though "indication of conflict" may be a solution adopted in the future.

[ccaputo]

627