Peeringdb: [beta] IX-F importer: test all configured IX-F JSON URLs prior to going to production

[ccaputo]

Prior to engaging the IX-F importer in production, on beta we need to enable the importer flag for all IXes that have a IX-F JSON URL configured, along with testing the full importer against all of these IXes.

[arnoldnipper]

Why? Only compliant JSONs will be able to be imported. If there are semantical errors they will be resolved (e.g. multiple (asn, ipaddr4, ipaddr6) tuples)

[ccaputo]

Why? Only compliant JSONs will be able to be imported. If there are semantical errors they will be resolved (e.g. multiple (asn, ipaddr4, ipaddr6) tuples)

Because this is simple stress testing that must be done before production. Prima facie. It may also reveal other issues that haven't yet been caught, including the provision of the knowledge of how long the importer takes to fully run. As is, the importer takes several minutes per large IX.

[arnoldnipper]

I'm happy to validate all IX-F JSONs and in case they are not compliant to contact the exchange.

[arnoldnipper]

including the provision of the knowledge of how long the importer takes to fully run

Didn't @vegu do that already several times?

[ccaputo]

I'm happy to validate all IX-F JSONs and in case they are not compliant to contact the exchange.

Feel free, but this is not about syntactic validation. It is about testing the importer not the exports.

including the provision of the knowledge of how long the importer takes to fully run

Didn't @vegu do that already several times?

With the new importer, I don't know. An issue with one IX cropped up today. I intend to test this myself, once ready, by performing and reset-all and reviewing the results.

[arnoldnipper]

I've run through all URLs and notified the IX in case there were issues

• the URL did not exist
• the content was not JSON
• there were issues when validating

Please note that an IX-F JSON has not to be fully compliant to being loadable for PeeringDB. NetIX also was the only exchange with duplicate entries.

[ccaputo]

$ time chronic flock -x -w 240 /tmp/pdb_process_admin_tool_command.lock /home/pdb/bin/pdb-container pdb_ixf_ixp_member_import --reset --commit

real    55m18.028s
user    0m0.737s
sys     0m0.219s

[arnoldnipper]

we import 126/245 potential IX-F JSON (List of IX-F JSON Urls taken from IXPDB)

[ccaputo]

we import 126/245 potential IX-F JSON (List of IX-F JSON Urls taken from IXPDB)

Per https://beta.peeringdb.com/cp/peeringdb_server/ixlanixfmemberimportlog the import count was 107, as a result of @grizz's update peeringdb_ixlan set ixf_ixp_import_enabled=1 where ixf_ixp_member_list_url != '';.

Keeping in mind the current config ignores speed & is_rs_peer differences, and not including "SIX Seattle" since I used that for testing in advance of this run, there were:

• 20 IX-F JSONs could not be processed: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Could+not+process.%24 (107 + 20 comes close to @arnoldnipper's 126)

• 147 DeskPRO tickets immediately created due to a lack of email contacts to send emails to: https://beta.peeringdb.com/cp/peeringdb_server/deskproticket/?q=%5E.IX-F%5D.Conflict.%24&all= and https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Conflict+resolution.*%24

• 86 IXes would have been sent an email reporting conflicts: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.one+or+more+networks.%24

• 1,021 networks would have been sent an email reporting conflicts: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.one+or+more+IXPs.%24

• As I understand things, 2,370 conflicts/differences/hints were identified: https://beta.peeringdb.com/cp/peeringdb_server/ixfmemberdata/

[arnoldnipper]

All exchanges have contact addresses now. We could minimise the number of DeskPRO tickets immediately created when we take into account all poc for a network. Even if they only have a sales address. If no poc left, email the exchange about this as they are/have been a customer of the exchange. Thoughts?

[ccaputo]

All exchanges have contact addresses now. We could minimise the number of DeskPRO tickets immediately created when we take into account all poc for a network. Even if they only have a sales address. If no poc left, email the exchange about this as they are/have been a customer of the exchange. Thoughts?

I defer to your or other's judgment on this.

[ccaputo]

Second run was much faster:

$ time chronic flock -x -w 240 /tmp/pdb_process_admin_tool_command.lock /home/pdb/bin/pdb-container pdb_ixf_ixp_member_import --commit

real    15m41.001s
user    0m0.284s
sys     0m0.055s

Same for 3rd run.

[arnoldnipper]

• 20 IX-F JSONs could not be processed: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Could+not+process.%24 (107 + 20 comes close to @arnoldnipper's 126)

@vegu, if an exchange only provides asn in its member_list you don't import the IX-F JSON. Why not? E.g. NaMeX

[ccaputo]

• 20 IX-F JSONs could not be processed: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Could+not+process.%24 (107 + 20 comes close to @arnoldnipper's 126)

@vegu, if an exchange only provides asn in its member_list you don't import the IX-F JSON. Why not? E.g. NaMeX

@vegu ?

[ccaputo]

• 20 IX-F JSONs could not be processed: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Could+not+process.%24 (107 + 20 comes close to @arnoldnipper's 126)

@vegu, if an exchange only provides asn in its member_list you don't import the IX-F JSON. Why not? E.g. NaMeX

@arnoldnipper, I looked at Namex more closely. The error reported is:

There was an issue when we attempted to parse the IX-F data for Namex Rome IXP

No entries in any of the vlan_list lists, aborting.

When looking at the member list, there are no IP addresses present. For example:

        {
            "asnum": 20940,
            "member_since": "2008-01-01T00:00:00Z",
            "url": "https:\/\/www.akamai.com",
            "name": "Akamai",
            "peering_policy": "open",
            "member_type": "peering",
            "connection_list": [
                {
                    "ixp_id": 1
                }
            ]
        },

Thus I don't see how the IX-F Importer can do anything with the data except report an error. Thoughts?

[ccaputo]

I performed a full import run on beta 2.21.0.6. First run run-time was 31m39.364s.

Errors noted:

Processing KleyReX (123)                                                                                                                                                          
ERROR: Network matching query does not exist.     
...
Processing LocIX Frankfurt (2084)
ERROR: Network matching query does not exist.

@vegu, running the importer on these ixlans with --debug -v 3 revealed no further info. Can you dig into this?

[arnoldnipper]

Thus I don't see how the IX-F Importer can do anything with the data except report an error. Thoughts?

I don't concur. Of course the import will not help on fixing address mismatches. But it still will help to see whether a network is connected to an exchange or not. No?

[ccaputo]

Thus I don't see how the IX-F Importer can do anything with the data except report an error. Thoughts?

I don't concur. Of course the import will not help on fixing address mismatches. But it still will help to see whether a network is connected to an exchange or not. No?

So what would you like the course of action for the Importer to be here?

Some ideas:

1) Inform the IX that their IX-F JSON data is missing IP addresses assignments necessary for it to be useful for the PeeringDB Importer?

2) Inform any networks listed as being at the IX, which are not yet listed at the IX, as needing to update PeeringDB with their IP address assignment - even though the Importer can't suggest those IPs since it lacks the data?

Other ideas?

[grizz]

Those are both good ideas. I agree that without IP addresses there's nothing to import. While it might be useful to some to know that there is a network presence at an exchange, it also violates our data validation. I'm guessing that in most, if not all, cases, the exchange would prefer to be notified and fix their code.

[arnoldnipper]

Skip 1. The IX knows about it. And IP addresses are not required to build a netixlan. Sad to say, but PC voted on this.

For 2. No need to suggest Ip addresses. Just suggest adding the network

1. Notify (net,ix) if net's asn is not in ix's IX-F JSON

[arnoldnipper]

it also violates our data validation.

What data validation is violated. Please note that for having a netixlan record an asn field is sufficient. There don't have to be ipaddr{4,6} fields at all. Unfortunately. See #480 and #697.

I'm guessing that in most, if not all, cases, the exchange would prefer not to be notified. They build their exports on purpose.

[vegu]

I performed a full import run on beta 2.21.0.6. First run run-time was 31m39.364s.

Errors noted:

Processing KleyReX (123)                                                                                                                                                          
ERROR: Network matching query does not exist.     
...
Processing LocIX Frankfurt (2084)
ERROR: Network matching query does not exist.

@vegu, running the importer on these ixlans with --debug -v 3 revealed no further info. Can you dig into this?

This is caused by bad data left overs from #168 where we started requiring netixlan.asn to be the same as the owning network's asn. On beta there are currently 2 netixlans in the database that are status="ok" and have a bad local asn value pointing to an asn different than the owning network.

netix = NetworkIXLan.objects.exclude(asn=F("network__asn")).filter(status="ok")
ip4="193.189.82.65" ip6="2001:7f8:33::a120:9152:1" netixlan.asn=209152 network.asn=44326 (KleyReX)
ip4="185.1.166.152" ip6="2001:7f8:f2:e1:0:a120:9152:1" netixlan.asn=209152 network.asn=44326 (LocIX)

Both of those are for network Initq (AS 44326), and fixing their local asns corrected the issue with the importer. As for why these bad entries still existed, my best guess is that we could not automatically correct it during the rollout of #168 since the asn 209152 did not exist as network object in peeringdb at the time (still doesn't).

The way i fixed it on beta was by setting the local asn to match that of the network (209152 to 44326) however on production maybe the appropriate fix would be to create the missing network and move those netixlans to that, i am not sure.

Correcting the bad data fixed this issue with the importer, and it's theoretically not possible (through exposed means) to introduce new bad data entries like that since we are now validating that netixlan.asn be the asn of the parent network.

However, strengthening the importer to be able to deal with this issue would be a quick fix and am happy to do so.

[ccaputo]

This is caused by bad data left overs from #168 where we started requiring netixlan.asn to be the same as the owning network's asn. On beta there are currently 2 netixlans in the database that are status="ok" and have a bad local asn value pointing to an asn different than the owning network.

netix = NetworkIXLan.objects.exclude(asn=F("network__asn")).filter(status="ok")
ip4="193.189.82.65" ip6="2001:7f8:33::a120:9152:1" netixlan.asn=209152 network.asn=44326 (KleyReX)
ip4="185.1.166.152" ip6="2001:7f8:f2:e1:0:a120:9152:1" netixlan.asn=209152 network.asn=44326 (LocIX)

Both of those are for network Initq (AS 44326), and fixing their local asns corrected the issue with the importer. As for why these bad entries still existed, my best guess is that we could not automatically correct it during the rollout of #168 since the asn 209152 did not exist as network object in peeringdb at the time (still doesn't).

The way i fixed it on beta was by setting the local asn to match that of the network (209152 to 44326) however on production maybe the appropriate fix would be to create the missing network and move those netixlans to that, i am not sure.

Correcting the bad data fixed this issue with the importer, and it's theoretically not possible (through exposed means) to introduce new bad data entries like that since we are now validating that netixlan.asn be the asn of the parent network.

However, strengthening the importer to be able to deal with this issue would be a quick fix and am happy to do so.

It seems like the best fix is to correct the production data, rather than adjust the importer to handle something that won't happen again if this one instance is corrected. @peeringdb/ac can you work with Initq to correct this?

[vegu]

• 20 IX-F JSONs could not be processed: https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=%5E.Could+not+process.%24 (107 + 20 comes close to @arnoldnipper's 126)

@vegu, if an exchange only provides asn in its member_list you don't import the IX-F JSON. Why not? E.g. NaMeX

Sorry, had missed this completely. As was brought up further down the thread, yes we are currently skipping ix-f imports with no ips entirely. This is in some parts left over logic from the old importer version, which we could remove, however we would also need to define some protocol for the scenarios allowing no-ip imports creates.

1. Should an IX-F import with missing ips ever cause a deletion of netixlans with ips?
2. Should an IX-F import with missing ips create a new netixlan if netixlan with ips exists?
3. Are IX-F imports with partial data (ips in some connection, missing from others) be treated specially ? ignored ?
4. How do we handle multiples of netixlans with no ips set vs multiples of connection with no ips set. We currently use the ips and the asn to correlate entries to each other on both ends, taking ips out of the question leaves us with the asn only, resulting in a consolidation to one netixlan always.

For 4. specifically - i remember Arnold raising this question during the specification process when determining if using (asn,ip4,ip6) as an identifier would cause issues when no ips were set. Initially i thought it would be ok, as i incorrectly assumed that in the the cases of no ips set, we could expect there to be only one connection entry per asn in the IX-F export.

However revisiting this now, looking at the IX-F schema i currently see no good way to correlate multiple connections for an asn with no ips in the IX-F export data to multiple netixlans for an asn with no ips on the peeringdb side.

Right now - on beta and production - we are still requiring that a netixlan has either ip4 or ip6 set, so this depends on #480 to be implemented, and until then i think the importer right now is doing the only thing it can, which is to ignore connections without ips.

[arnoldnipper]

@peeringdb/ac can you work with Initq to correct this?

done

[arnoldnipper]

@vegu

• ad 1: yes
• ad 2: no
• ad 3: no
• ad 4: what has to be correlated there? What might happen? That the IX-F JSON has n entries and PeeringDB has m!=n entries. What to do? Delete/create netixlans until there are m==n entries. Or am I missing something?

[vegu]

@arnoldnipper values for operational status and later on speed and is_rs_peer

[arnoldnipper]

@arnoldnipper values for operational status and later on speed and is_rs_peer

Take these values into account and apply the above-given algorithm. Makes sense?

[vegu]

You're right, my main concern was that updates to those values could be pushed to existing netixlans in a stable manner, but since they are all essentially unidentified anyways, creating / deleting until peeringdb matches the ix-f source would work.

[ccaputo]

I wonder if someone that groks the remaining issues, could extract them to a fresh ticket and close this one?

[ccaputo]

@arnoldnipper and @peeringdb/ac : In advance of the upcoming production release and Importer production testing, it might make sense to reach out to the IXPs whose IX-F Data is failing to be imported. The current list is at https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/?q=Could+not and included in each entry is the reason for the import failure.

Once an IX-F export is fixed, a reimport can be initiated at https://beta.peeringdb.com/cp/peeringdb_server/commandlinetool/prepare/ or I can assist. (Please don't perform a Reset unless that is intended.)

[arnoldnipper]

@ccaputo and @peeringdb/oc : it looks like ixf_ixp_import_enabled has been set to true even if ixf_ixp_member_list_url == "". E.g. https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/125/change/?_changelist_filters=q%3DCould%2Bnot

Please fix the settings

[arnoldnipper]

@ccaputo and @peeringdb/20c some exchanges only provide a list of asn w/o any ipaddr{4,6} data. This is fine according to the specification, however, was implemented wrongly. E.g. https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/114/change/?_changelist_filters=q%3DCould%2Bnot

Please fix!

[ccaputo]

@ccaputo and @peeringdb/20c some exchanges only provide a list of asn w/o any ipaddr{4,6} data. This is fine according to the specification, however, was implemented wrongly. E.g. https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/114/change/?_changelist_filters=q%3DCould%2Bnot

Please fix!

@arnoldnipper , per @vegu above:

Right now - on beta and production - we are still requiring that a netixlan has either ip4 or ip6 set, so this depends on #480 to be implemented, and until then i think the importer right now is doing the only thing it can, which is to ignore connections without ips.

Since #480 has not yet been decided/resolved, I don't see how @peeringdb/20c can follow your suggestion. If you think appropriate/needed, I wonder if you would consider breaking this out into its own issue for further discussion?

[arnoldnipper]

@ccaputo and @20c please make sure to de-duplicate data before importing. According to the IX-F JSON schema, it is valid to have identical entries.

E.g. https://www.sbix.ch/participants.json

[arnoldnipper]

Since #480 has not yet been decided/resolved, I don't see how @peeringdb/20c can follow your suggestion. If you think appropriate/needed, I wonder if you would consider breaking this out into its own issue for further discussion?

@ccaputo and @vegu this issue also came up when we discussed the implementation of DOTF. And @vegu agreed that it makes sense to have asn only. For whatever reason, it was not implemented. It is a bit tedious to have to remind again and again on what has been decided.

[ccaputo]

@ccaputo and @peeringdb/oc : it looks like ixf_ixp_import_enabled has been set to true even if ixf_ixp_member_list_url == "". E.g. https://beta.peeringdb.com/cp/peeringdb_server/ixfimportemail/125/change/?_changelist_filters=q%3DCould%2Bnot

Please fix the settings

I created https://github.com/peeringdb/peeringdb/issues/846 for this.

@ccaputo and @20c please make sure to de-duplicate data before importing. According to the IX-F JSON schema, it is valid to have identical entries.

E.g. https://www.sbix.ch/participants.json

Created as https://github.com/peeringdb/peeringdb/issues/847.

[ccaputo]

Since #480 has not yet been decided/resolved, I don't see how @peeringdb/20c can follow your suggestion. If you think appropriate/needed, I wonder if you would consider breaking this out into its own issue for further discussion?

@ccaputo and @vegu this issue also came up when we discussed the implementation of DOTF. And @vegu agreed that it makes sense to have asn only. For whatever reason, it was not implemented. It is a bit tedious to have to remind again and again on what has been decided.

I am not finding the handling of asn-only in the spec at https://github.com/peeringdb/peeringdb/issues/697#issuecomment-634117381. Regardless, until resolution of #480, which has had more recent discussion, I don't see how it makes sense to adjust the Importer. Even if it has been decided in the past among a few, it looks like an open issue to be figured out by the @peeringdb/pc.

[arnoldnipper]

I am not finding the handling of asn-only in the spec at #697 (comment). Regardless, until resolution of #480, which has had more recent discussion, I don't see how it makes sense to adjust the Importer. Even if it has been decided in the past among a few, it looks like an open issue to be figured out by the @peeringdb/pc.

I'll bring this up in today's call

[arnoldnipper]

@ccaputo and @peeringdb/oc for VSIX (and maybe others) SSL certification check fails. Does it make sense to disable the check?

[arnoldnipper]

In advance of the upcoming production release and Importer production testing, it might make sense to reach out to the IXPs whose IX-F Data is failing to be imported.

done

[vegu]

Since #480 has not yet been decided/resolved, I don't see how @peeringdb/20c can follow your suggestion. If you think appropriate/needed, I wonder if you would consider breaking this out into its own issue for further discussion?

@ccaputo and @vegu this issue also came up when we discussed the implementation of DOTF. And @vegu agreed that it makes sense to have asn only. For whatever reason, it was not implemented. It is a bit tedious to have to remind again and again on what has been decided.

I can appreciate that this is frustrating and the reason as to why #480 was not implemented as part of the DOTF milestone is laid out here: https://github.com/peeringdb/peeringdb/issues/480#issuecomment-679832114

Aside from the exchange the two of us had regarding #480 and how it's implementation would affect the importer, it was never part of the final specifications document nor put in the group of tickets for the DOTF project.

At the time of the discussion i was having an incorrect understanding of #480 and making the assumption that networks would be limited to one netixlan with nulled ipaddresses per exchange. Which is something the importer could handle right now once we implement #480 and allow it to process without ip addresses.

However recent comments in #480 have corrected this assumption and it somewhat complicates the approach the importer will need to take to make this work.

[arnoldnipper]

However recent comments in #480 have corrected this assumption and it somewhat complicates the approach the importer will need to take to make this work.

Afair we did a case analysis and all cases could be handled. So, it would be possible to implement #480. I also understand that we want to have at least ipaddr{4,6} as this is useful operational data. However, if we only are provided with a list of ASN that does already help to clean up data, i.e. to get rid of stale data.

[ccaputo]

@ccaputo and @peeringdb/oc for VSIX (and maybe others) SSL certification check fails. Does it make sense to disable the check?

Wouldn't that open things up to a man-in-the-middle attack?

[ccaputo]

@ccaputo and @peeringdb/oc for VSIX (and maybe others) SSL certification check fails. Does it make sense to disable the check?

Wouldn't that open things up to a man-in-the-middle attack?

Speaking in my IXP-operator role, I would not be comfortable with PeeringDB being susceptible to a MitM attack which could harm the reputation of the IXP I am responsible for or cause operational impact to those networks with "Allow IXP Update" enabled. SSL Certificate verification is thus critically important.

[vegu]

@ccaputo i split out the ip-address issue to #849 - should we move the SSL question to a new ticket as well - otherwise i think this one can be closed.

My advise for the SSL question would be to keep the current implementation as it will require the ix to fix their certificate, which is beneficial to all parties involved.

[ccaputo]

Agreed. Thanks!