Peeringdb: [beta] IX-F importer: prevent Admin Committee overload by initially limiting importer to IXes enabled by AC

[ccaputo]

Copying from an email: [...] of the 231 DeskPRO tickets currently listed in beta, all but 6 are due to my enabling of the SIX IX-F import. (The SIX has 332 ASNs with 380 routers.) This means that if rolled out to production as-is, the Admin Committee will potentially receive a deluge of thousands of tickets, along with thousands of emails going out to the users. I think we need to be really careful here and consider a staged roll-out of an IXP or a few IXPs at a time. Maybe initially, the Admin Committee should have control of the "Enable IX-F Import" flag until this has been fully rolled out to all IXPs with a IX-F URL configured? This will enable to AC to have some control of the flow. Thoughts?

[vegu]

For what it's worth i just did a full import on all enabled exchanges on my local dev environment.

It's running a database synced from beta a month or so ago, and it generated 4700+ tickets.

Implementation of #770 and #771 will cut this number down a fair bit, and i will run it again then to check the difference, but my advise would also to be to either stagger the rollout of this feature somehow or perhaps turn notifications / tickets off for the initial run of the importer.

[ccaputo]

For what it's worth i just did a full import on all enabled exchanges on my local dev environment.

It's running a database synced from beta a month or so ago, and it generated 4700+ tickets.

Implementation of #770 and #771 will cut this number down a fair bit, and i will run it again then to check the difference, but my advise would also to be to either stagger the rollout of this feature somehow or perhaps turn notifications / tickets off for the initial run of the importer.

Excellent info.

If some notifications are turned off at start, how would they be recovered such that the whole thing does not need to be reset?

@peeringdb/ac: Please provide input re overload given the worst case of the above 4700+ tickets. Thanks.

[vegu]

I think we also need to define if this is something we need to throttle for admincom tickets only or for ix / net notifications as well. Since notifications to the exchanges and networks will likely cause some people to reach out to support as well.

In any case, I was imagining something like this (for tickets, but this could easily be expanded to work with ix/net notifications as well)

1. initial reporter run, generate all the suggestions, dont send any tickets
2. have a command that can send out all tickets on a per exchange basis that can be triggered by admincom
3. notifications / tickets are generated from the existance of IXFMemberData objects (eg. suggestion objects) so they could be rendered at any point as long as that object exists.

that would allow networks to start processing suggestions right away, and the ac can gradually work through the tickets.

[funkestefan]

What kind of tickets will be generated? Whats the expected (re)action?

[ccaputo]

What kind of tickets will be generated? Whats the expected (re)action?

Tickets will be generated to aide in conflict resolution per the D.O.T.F recommendations. Please check out: https://github.com/peeringdb/peeringdb/issues/697#issuecomment-634117381

[funkestefan]

I see.
Currently we do about 1000 tickets per month. I guess (as in coffee cup readings) we could do +500 each month.

[arnoldnipper]

• have a command that can send out all tickets on a per exchange basis that can be triggered by admincom

Could it make more sense to have this sent out per net?

Currently we do about 1000 tickets per month. I guess (as in coffee cup readings) we could do +500 each month.

Really depends on what load this really generates on AC. Currently, we expect the involved parties to sort things out themselves. I.e. AC will wait for 6 days and wait for feedback. If no feedback is received we will send a notification and wait for another 3 days. Only if the issue is not resolved until then AC will resolve it.

[vegu]

Could it make more sense to have this sent out per net?

Implementation wise it makes little difference, if it makes more sense to send per net lets do that

[arnoldnipper]

Implementation wise it makes little difference, if it makes more sense to send per net lets do that

As we are always more net-centric I would suggest that a network gets all the issues at once.

[ccaputo]

If @arnoldnipper and the @peeringdb/ac prefers per-net emails, I am happy to support that. The fewer the emails to a network, the more effective this tool will be. Too many emails and it will be ignored, so it is important to get this correct before going to production - hence https://github.com/peeringdb/peeringdb/issues/767 is needed in order to be able to verify these changes on beta - since it logs emails that _would_ have been sent (we aren't sending emails on beta).

One email representing all new (non-dismissed) conflicts resulting from an importer run, makes sense. If on a subsequent importer run, new conflicts are discovered, a new email would make sense to me, highlighting new issues at the top, and old non-dismissed previously reported issues at the bottom.

Note, that implies initial production run for the importer would be for all IXes with IX-F import configured. Thus at some point on beta, after the current bugs and usability issues are completed and tested, that we perform a clean run with all IXes IX-F imports enabled - so we have knowledge of the full result of a rollout to production.

@peeringdb/pc: production blocker - please evaluate and ask questions or support/reject as appropriate.

[arnoldnipper]

One email representing all new (non-dismissed) conflicts resulting from an importer run, makes sense. If on a subsequent importer run, new conflicts are discovered, a new email would make sense to me, highlighting new issues at the top, and old non-dismissed previously reported issues at the bottom.

I do not agree here, as the email has to go to all parties involved. So it has to read one email per net per ix for all new conflicts.

[ccaputo]

One email representing all new (non-dismissed) conflicts resulting from an importer run, makes sense. If on a subsequent importer run, new conflicts are discovered, a new email would make sense to me, highlighting new issues at the top, and old non-dismissed previously reported issues at the bottom.

I do not agree here, as the email has to go to all parties involved. So it has to read one email per net per ix for all new conflicts.

Are you thinking the same email goes to the IX and the network? That has not been my understanding. https://github.com/peeringdb/peeringdb/issues/697 refers to separate emails. @vegu, what is the code doing right now?

[arnoldnipper]

Are you thinking the same email goes to the IX and the network? That has not been my understanding. #697 refers to separate emails. @vegu, what is the code doing right now?

I guess we want to handle that via DeskPRO tickets. No?

[ccaputo]

Are you thinking the same email goes to the IX and the network? That has not been my understanding. #697 refers to separate emails. @vegu, what is the code doing right now?

I guess we want to handle that via DeskPRO tickets. No?

Not sure what you mean. At https://github.com/peeringdb/peeringdb/issues/697#issuecomment-634117381 an example would be:

If local-ixf[as,ip4,ip6] does not exist or is different from remote-ixf[as,ip4,ip6]
   Create/Update local-ixf[as,ip4,ip6]
   Create/Update a ticket for admin com
   Email the network
   Email the ix

So Create/Update a ticket for admin com refers to creation of a DeskPRO ticket, while the separate (in my understanding) emails to network and ix would have a return address of [email protected] (or DeskPRO ticket-specific email addr) so that the network or ix could reply and communicate with the AC as appropriate. An argument for separate emails per ticket, rather than aggregated, is that the emails could reference the actual ticket (ie., in the Subject: [PDB #...]) so that the AC can easily match up emails with particular tickets and/or DeskPRO automagically handling that.

@vegu is the email system using the DeskPRO ticket numbers in this fashion? If not, could it?

[vegu]

The emails are currently not using any deskpro id nor are they sent in a way they could obtain the deskpro ticket id to use it.

Doing so would require a refactor of how and when those emails are sent since tickets are created off main process in a separate job. Alternatively the ix-f importer could push tickets directly without them being offloaded to a separate job i suppose.

Right now the code follows the importer protocol in the sense that when it says

• Email the network - it emails the network
• Email the exchange - it emails the exchange
• Create a ticket - create a ticket object and push it to the queue - ticket will get sent once the cronjob that handles that picks it
  up (usually within a minute)

This is done on a per conflict basis.

Subject is identical for all 3, message body varies a bit. Ticket to AC has some /cp specific links in it for example.

[vegu]

@vegu is the email system using the DeskPRO ticket numbers in this fashion? If not, could it?

Worth mentioning that fixing #770 we need to change how emails are sent out anyhow since we need to consolidate those delete+add actions before we send anything. So this will change in a fashion where all emails are first collected and processed before they are being sent.

If we were to change the ixf-importer to directly post tickets to the api instead of offloading, and make sure that happens before emails are sent, adding the ticket id to the email subject sounds pretty straight forward.

[ccaputo]

@vegu is the email system using the DeskPRO ticket numbers in this fashion? If not, could it?

Worth mentioning that fixing #770 we need to change how emails are sent out anyhow since we need to consolidate those delete+add actions before we send anything. So this will change in a fashion where all emails are first collected and processed before they are being sent.

If we were to change the ixf-importer to directly post tickets to the api instead of offloading, and make sure that happens before emails are sent, adding the ticket id to the email subject sounds pretty straight forward.

I think this is a very worthwhile good idea, on a per {net,ix} tuple basis. @arnoldnipper, can you confirm?

[arnoldnipper]

In a conflict at least two parties are involved (if not three). And imho it makes sense to have all parties in the loop to avoid loss of information. However, AC only has to be involved if the parties are unable to resolve the conflict themselves. So, what about a tiered procedure

• Send a list of all conflicts to the nets and fixes
• wait x days for the nets/ixes to resolve the conflict
• if the conflict still exist after x days involve AC and create a ticket with all parties involved in the conflict

[DarwinCosta]

On 18 Jul 2020, at 23:03, Arnold Nipper [e-mail] wrote:


In a conflict at least two parties are involved (if not three). And imho it makes sense to have all parties in the loop to avoid loss of information. However, AC only has to be involved if the parties are unable to resolve the conflict themselves. So, what about a tiered procedure

Send a list of all conflicts to the nets and fixes
wait x days for the nets/ixes to resolve the conflict
if the conflict still exist after x days involve AC and create a ticket with all parties involved in the conflict
+1. IN with the proposed procedure.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[vegu]

if the conflict still exist after x days involve AC and create a ticket with all parties involved in the conflict

Would this then be on a per conflict basis again, or still a list of all conflicts that meet that condition (age)?

[arnoldnipper]

Would this then be on a per conflict basis again, or still a list of all conflicts that meet that condition (age)?

If the conflict still exists it has to be on a per conflict basis. Otherwise, you may have too many parties involved. E.g. if a network has 10 conflicts with 10 different ixes. Does that make sense?

[vegu]

@arnoldnipper that does make sense.

Just to be clear, we are also still doing the functionality where qualifying conflicts are dispatched as tickets to deskpro on a per network basis by the admincom, so admincom controls the in-flow of those tickets? Is that correct?

cc: @ccaputo

[arnoldnipper]

Initially, there is no need to involve @peeringdb/ac. Just notify the network and the exchange about all conflicts via email. Only if after x (proposal is to set x to 6) days the conflict is still unresolved open a ticket at DeskPRO with all parties involved. The idea behind this approach is that net/exchange will be able to resolve the conflict themselves.

If any party opens a ticket with AC because of any conflict AC still is able to help.

Does that clarify and still make sense, @vegu and @ccaputo?

[ccaputo]

@arnoldnipper that does make sense.

Just to be clear, we are also still doing the functionality where qualifying conflicts are dispatched as tickets to deskpro on a per network basis by the admincom, so admincom controls the in-flow of those tickets? Is that correct?

cc: @ccaputo

@vegu I think this is evolving and that is okay. I think with @arnoldnipper's idea, a DeskPRO ticket would not be created until a conflict is left unresolved after N days (tunable, proposed 6 days initially) from when an email is separately sent to both the network and the ix.

Generally the ix will be authoritative, thus the email to the ix should not be one that evokes confusion by suggesting the ix has something to fix. It could be along the lines of "Data supplied by your {ix_name} is in conflict with that provided by a {network_name}. Please work with {network_name} to resolve this conflict or correct your data as appropriate. The Peering Admin Committee will assist with this conflict if it is not resolved in N days."

And then the email to the network could be along the lines of the flip of that: "Data supplied by your network {network_name} is in conflict with that provided by {ix_name}. Please work with the {ix_name} to resolve this conflict or correct your data as appropriate. The Peering Admin Committee will assist with this conflict if it is not resolved in N days."

Does that clarify and still make sense, @vegu and @ccaputo?

Yes and good.

[arnoldnipper]

Thanks a lot for these well-drafted emails for conflict resolution, @ccaputo

[grizz]

This looks good, +1 from me. Anyone care to write up a full summary? :)

@peeringdb/pc votes please!

[ccaputo]

This looks good, +1 from me. Anyone care to write up a full summary? :)

@arnoldnipper does this work for a spec?:

• An adjustable knob is created for the AC, controllable from the /cp/ interface, which represents the minimum number of days between when a conflict is discovered and a ticket is opened for the AC if the conflict is not resolved. Zero is allowed if immediate ticket creation is desired by the AC. Default is 6 days. Possible name: {ix-f-importer-days-until-ticket}

• An IX-F Importer run aggregates email notifications for new conflicts, grouped by network and by ix. Thus after an IX-F Importer run, a network would receive an email regarding all new conflicts discovered during the run. Same for an ix. This prevents networks and ixes from being repeatedly notified about the same conflicts.
  -- All emails are logged in a table per #767. In beta, emails are only logged. In production, emails are logged and actually delivered.
  -- For ixes, the emails are addressed to the "Technical Email" address if configured, or the "Policy Email" if it exists and "Technical Email" is not configured. An AC ticket is created immediately for each involved conflict, if this list of contact possibilities is exhausted.
  -- For networks, the emails are addresses to the "Technical" address if configured, or "NOC" if no "Technical", or "Policy" if no "NOC" and no "Technical". An AC ticket is created immediately for each involved conflict, if this list of contact possibilities is exhausted.
  -- The subject line of an email to a network will include the network's ASN and the name of each IXP, for example: Subject: PeeringDB: Action May Be Needed: IX-F Importer data mismatch between AS{network_ASN} and one or more IXPs
  -- The body of an email to a network will start with This email details {count} new data mismatch[es] discovered by the PeeringDB IX-F Importer. Please review and correct these mismatches at your PeeringDB network page (https://{instance}/asn/{network_ASN}) or work with the indicated IXPs to correct the data. followed by horizontal divider (ex. 30 dashes), followed by a numbered list of conflicts, each prepended with a conflict-specific: "Data supplied by your network {network_name} is in conflict with that provided by {ix_name}. Please work with the {ix_name} to resolve this conflict or correct your data at https://{instance}/asn/{network_ASN} as appropriate. Your network page will contain hints to assist with this. The Peering Admin Committee will assist with this conflict if it is not resolved in {N} days.". After this should be the conflict details as seen in DeskPRO tickets followed by another horizontal divider.
  -- The subject line of an email to an ix will include the IX name and the ASN of each network, for example: Subject: PeeringDB: Action May Be Needed: IX-F Importer data mismatch between {ix_name} and one or more networks
  -- The body of an email to an IX will start with This email details {count} new data mismatch[es] discovered by the PeeringDB IX-F Importer. Please review and correct these mismatches in your IX-F JSON export ({IX-F Member Export URL}) or work with the indicated networks to correct the data. followed by horizontal divider (ex. 30 dashes), followed by a numbered list of conflicts, each prepended with a conflict-specific: "Data supplied by your {ix_name} is in conflict with that provided by a {network_name}. Please work with {network_name} to resolve this conflict or correct your data as appropriate. The Peering Admin Committee will assist with this conflict if it is not resolved in N days.". After this should be the conflict details as seen in DeskPRO tickets followed by another horizontal divider.

• If the conflict still exists after {ix-f-importer-days-until-ticket} days, and a DeskPRO ticket has not already been created for the conflict, create a ticket for a given conflict for the AC. Log the ticket in the /deskproticket/ table. At this point this ticket creation should result in emails to all of the involved parties, and include the DeskPRO ticket id for easy referencing/updating of the ticket.

[arnoldnipper]

• At this point this ticket creation should _not_ result in emails to the involved parties since doing so will result in a flood of emails, for something the users have already been emailed about

From an AC pov all parties should be notified as it is most likely that AC have to involve them anyhow. Hence pulling together emails and instantly sending emails out still makes sense even if the parties already have been emailed. Ideally conflicts will have been resolved already and hence only a few emails will be generated

Otherwise fine. However, subjects may become larger than 255 chars. And I do not see a real benefit in having all {ASN, IX names} in the subject.

[ccaputo]

• At this point this ticket creation should _not_ result in emails to the involved parties since doing so will result in a flood of emails, for something the users have already been emailed about

From an AC pov all parties should be notified as it is most likely that AC have to involve them anyhow. Hence pulling together emails and instantly sending emails out still makes sense even if the parties already have been emailed. Ideally conflicts will have been resolved already and hence only a few emails will be generated

Ok. Updated https://github.com/peeringdb/peeringdb/issues/772#issuecomment-662564888 accordingly.

Otherwise fine. However, subjects may become larger than 255 chars. And I do not see a real benefit in having all {ASN, IX names} in the subject.

Ok. I've added a 200 char limit.

Thanks!

[arnoldnipper]

Ok. I've added a 200 char limit.

I would completely remove it as it does not give real benefit, esp. when limited to 200 chars

[ccaputo]

Ok. I've added a 200 char limit.

I would completely remove it as it does not give real benefit, esp. when limited to 200 chars

Ok - updated. My concern had to do with searchability in the email table, but end-user experience is more important.

[arnoldnipper]

Let's give it a try then! @peeringdb/ac please read and chime in if you disagree!

[ccaputo]

@vegu can you update the DeskPRO ticket creation to include the dates of emails to contacts and makes explicit when there are no contacts to email and makes explicit the time of ticket creation? For example, right now a network without contacts would show:

Network Contacts


Exchange Contacts

- [email protected]

This doesn't make clear how long ago a network or ix was contacted.

So that AC clearly knows when emails were sent or if there was a lack of contact, please change this to include UTC timestamps.

Examples:

Time of ticket: Wed 29 Jul 2020 08:12:11 PM UTC

Network Contacts

- No contacts to email as of Wed 29 Jul 2020 08:12:11 PM UTC

Exchange Contacts

- [email protected]  (emailed Wed 29 Jul 2020 08:12:11 PM UTC)

or:

Time of ticket: Wed 29 Jul 2020 08:12:11 PM UTC

Network Contacts

- [email protected]  (emailed Wed 29 Jul 2020 08:12:11 PM UTC)

Exchange Contacts

- [email protected]  (emailed Wed 29 Jul 2020 08:12:11 PM UTC)

or:

Time of ticket: Wed 29 Jul 2020 08:12:11 PM UTC

Network Contacts

- [email protected]  (emailed Wed 29 Jul 2020 08:12:11 PM UTC)

Exchange Contacts

- No contacts to email as of Wed 29 Jul 2020 08:12:11 PM UTC

[arnoldnipper]

If a contact is missing a DeskPRO ticket will immediately be opened. And the DeskPRO ticket has a timestamp

[ccaputo]

If a contact is missing a DeskPRO ticket will immediately be opened. And the DeskPRO ticket has a timestamp

Understood. But since the number of days between successful emails and the ticket being opened can vary, I think it makes sense to include in the ticket when an email was sent.

[arnoldnipper]

The number of days between successful emails and the ticket being opened can only vary systemwide. Hence it still doesn't make sense to me from an AC pov. Let's keep it simple and not overburden with not so important informations.

[ccaputo]

The number of days between successful emails and the ticket being opened can only vary systemwide. Hence it still doesn't make sense to me from an AC pov. Let's keep it simple and not overburden with not so important informations.

Ok. Thank you.