Plugins: net/haproxy Rules in backends or frontends are not sortable

[mj84]

Hi there,

I am running OPNsense 19.7.2 with haproxy 2.17.
It seems that it is impossible to define the sort order of linked rules (actions) in frontends and backends.
I can drag and drop the rules in the desired order, but when I hit save and check the order again, it is just as before I did the change.

I have tried manipulating the rules order via API, the behaviour is the same, the API tells me it has saved my change when it has not.

Removing and re-adding a rule from a front- or backend does not help either.

Please advise.

Best regards,
Markus

[mj84]

Ok, I was able to dig a bit deeper into this and now I am even more confused than before :)

• when I change the order of rules in a front- or backend, the order gets saved correctly, at least that's how it appears to me when I reopen the front- or backend
• if I change anything BUT the rules and save, the order is mixed up the next time I open the object
• if I change nothing at all, the order of rules gets mixed up as well

That is for the GUI part, I will now try to investigate through the API

[fraenki]

@mj84 I've never seen this. Which browser (+version) are you using for your tests?

[mj84]

I was able to reproduce this with Firefox 68.0 and Vivaldi 2.7.1628.33 (Chrome/76.0.3809.137), Firefox 69.0 shows the same behaviour.

[fraenki]

OK, tested again on Firefox 69 and OPNsense 19.7.4: still cannot reproduce it, neither on backends nor on frontends. No matter what I try, it does not get mixed up, it's always the sort order that I chose:

[mj84]

I am updating my installation right now to the latest version and will report back

[mj84]

Unfortunately nothing has changed with 19.7.4 .
It seems to me that the creation time of a rule makes a difference, newer rules will move in front of older ones.

First picture shows my desired state, which will also show up after I hit save:

When I then click save again without changing anything, I'm getting this:

[mj84]

On a side note: the order of linked actions does not show up in the API, I suspect this is a limitation of the Core API, is this true?

[fraenki]

the order of linked actions does not show up in the API, I suspect this is a limitation of the Core API, is this true?

No, the API already returns a properly sorted list. Using my previous example, when accessing https://opnsense/api/haproxy/settings/getBackend/<UUID>/ I get the expected list:

[mj84]

No, the API already returns a properly sorted list. Using my previous example, when accessing https://opnsense/api/haproxy/settings/getBackend/<UUID>/ I get the expected list:

Thanks alot for the clarification, it seems that my JSON parser mixes up the order, since the actions are specified in an object, which has no particular order of its keys in JSON.

Were you able to reproduce my issue while trying to move older rules in front of newer ones and performing an empty change afterwards?

[mj84]

I just thought of a theory:
Is it possible, that the order of linkedActions gets mixed up when changing something other than linkedActions, because they are stored in a JSON object and not an array?

As stated before, a JSON object does not honor the order of its keys and depending on the parser, the order of the keys in an object might change (which is what I saw while parsing the order of linkedActions using python requests).

Cheers,
Markus

[fraenki]

Were you able to reproduce my issue while trying to move older rules in front of newer ones and performing an empty change afterwards?

I was not able to reproduce this issue yet. What exactly do you mean by saying "perform an empty change"?

[mj84]

By empty change I mean you have to open the frontend again, and clicking “Save” without changing anything (changing something different than the linked actions triggers the issue as well, as long as you don’t touch the linked actions).

Also for this issue to occur it is necessary to have older rules in front of newer ones.

_{Sent with GitHawk}

[fraenki]

@AdSchellevis Do you have any idea? :(

[AdSchellevis]

@fraenki no, if we can't reproduce it, it's unlikely we're going to solve it. If @mj84 wants to check his theory, he probably best inspect both the data sent to the backend (in the save post) and the request when fetching the same data back.

PHP honours the item order as far as I know (and we've seen in our tests), but that doesn't guarantee the processing library always will obey order as well (it's not a hard guarantee, but we haven't seen it fail sofar)

More data might help to pinpoint the issue. The details really matter here (exact contents being saved and received and if it differs from what's shown on the screen, the order in which they eventually appear). Which is what @fraenki also mentioned in one of the comments above https://github.com/opnsense/plugins/issues/1494#issuecomment-530409491

[mj84]

@AdSchellevis thanks for the advise, I will try to document my steps as good as possible.

First, I tried to reproduce this through plain API using just curl. I always received the correct result.

I have tried getting frontend objects and updating their properties through curl, but always got the same result, so the PHP backend indeed seems to honor the order.

So I focused a bit on the GUI, since I just knew there was a problem.

Check out this behaviour in the GUI, when I'm just clicking "Save" over and over:

Use this direct link to view the GIF in fullscreen: https://user-images.githubusercontent.com/12526366/65080482-b1f38780-d9a1-11e9-9d82-71afee39fe8f.gif

But why was I seeing a strange behaviour through the API as well which I couldn't reproduce with just using curl?

Now it finally makes sense: I was encountering two different problems.
What we're seeing in the GIF is (probably) the JS part of the WebGUI having some trouble with the ordering of the rules because of them being stored in an object, and therefore showing this erratic behaviour when just clicking "Save" over and over again.

While developing my Ansible modules for HAProxy on opnsense (https://github.com/mj84/ansible-opnsense-haproxy), I encountered a similar behaviour while constructing python dictionaries out of the JSON responses using python requests, the order of keys seemed to depend on their uuids.

For my project I was able to implement a workaround by using python OrderedDicts and parsing the JSON manually instead of having my HTTP library (Python Requests) do that for me.

But to conclude, this feels like rather a bug in the model, since JSON objects are not supposed to have a defined order for its subelements :/

What do you guys think?

[fichtner]

Is this a case where we return a list using only a map? { 1: 1, 2: 2 } where instead it should be [ { 1: 1 }, { 2: 2 } ] ? I can't see how order wouldn't be preserved in a list.

[AdSchellevis]

Keys are uuid’s here, please post exchanged json content as requested. @fichtner sorting keys can be problematic indeed (certainly with int types). We’re not going to change the api for these types (I think this is also the only use case where sorting matters, which if I remember correctly I warned about before starting this journey)... but without a reproducible case, there’s not a lot to investigate...

Python ordereddict types are not relevant in this area.

[mj84]

@fichtner I think you are absolutely right.

@AdSchellevis The API always returns the linkedActions in the correct order. But unfortunately in a data structure that is not supposed to support a specific order of its keys.

In JSON, this

"linkedActions": {
  "cf89aa8b-3b4e-4f1a-b5f7-5ab919f88d73": {
    "value": "myrule1",
    "selected": 1
  },
  "8c6936c1-9691-4b96-bba9-12f4a7347815": {
    "value": "myrule2",
    "selected": 1
  }
}

is the same as

"linkedActions": {
  "8c6936c1-9691-4b96-bba9-12f4a7347815": {
    "value": "myrule2",
    "selected": 1
  },
  "cf89aa8b-3b4e-4f1a-b5f7-5ab919f88d73": {
    "value": "myrule1",
    "selected": 1
  }
}

A correct (order-sensitive) data structure for linkedActions would be something like this:

"linkedActions": [
  {
    "uuid": "8c6936c1-9691-4b96-bba9-12f4a7347815",
    "value": "myrule2",
    "selected": 1
  },
  {
    "uuid": "cf89aa8b-3b4e-4f1a-b5f7-5ab919f88d73",
    "value": "myrule1",
    "selected": 1
  }
]

This is what is being POST-ed to the API when saving a frontend repeatedly in the GUI (just as I did in my GIF) :

• linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"
• linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"
• linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"
• linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"

As you can see, the order of linkedActions seems to change every 2nd request.

[mj84]

If you consider the GET requests to the API as well, this is what happens:

• GET:
  "linkedActions":{"894d9ac4-b6e0-48f1-bed5-bc5a51cf1120":{"value":"redirect_acme_challenges","selected":1},"12d9643d-460e-4c5f-8a06-8db69f3f5849":{"value":"myrule","selected":1}}
• POST:
  linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• GET:
  "linkedActions":{"12d9643d-460e-4c5f-8a06-8db69f3f5849":{"value":"myrule","selected":1},"894d9ac4-b6e0-48f1-bed5-bc5a51cf1120":{"value":"redirect_acme_challenges","selected":1}}
• POST:
  linkedActions: "12d9643d-460e-4c5f-8a06-8db69f3f5849,894d9ac4-b6e0-48f1-bed5-bc5a51cf1120"
• GET:
  "linkedActions":{"12d9643d-460e-4c5f-8a06-8db69f3f5849":{"value":"myrule","selected":1},"894d9ac4-b6e0-48f1-bed5-bc5a51cf1120":{"value":"redirect_acme_challenges","selected":1}}
• POST:
  linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"
• GET:
  "linkedActions":{"894d9ac4-b6e0-48f1-bed5-bc5a51cf1120":{"value":"redirect_acme_challenges","selected":1},"12d9643d-460e-4c5f-8a06-8db69f3f5849":{"value":"myrule","selected":1}}
• POST:
  linkedActions: "894d9ac4-b6e0-48f1-bed5-bc5a51cf1120,12d9643d-460e-4c5f-8a06-8db69f3f5849"

[AdSchellevis]

We’re not going to change the structure, it wasn’t really meant to support sorting originally and will break a lot of other things. If, with the same uuids, we can reproduce your issue, we might look into adding an “order” field, so the client can sort it back properly (which it should also send back in that case)... it’s just odd that all other use-cases we have seen sofar seem to obey the order, which isn’t really logical if the same browser would alway sort its keys.

[mj84]

Sure, changing the data structure is not something one would want to do lightly.
If adding a field for the subelements of linkedActions is a possibility, that can be a solid workaround.

As far as reproduction of this bug is concerned:
have a frontend with multiple rules configured an simply open and save it repeatedly without changing anything. Using the F12 network monitor of your browser (Firefox, Chrome, etc.) and inspecting what is being returned from the API and what is being posted should already yield this behaviour.

[AdSchellevis]

I’ll try the same content and generated uuid’s on a box at my end as soon as I can find a bit of time. Maybe @fraenki can try to use the same generated uuid’s on his box as well to see if the exact same content produces the same issue on his end.

[AdSchellevis]

I can't reproduce the issue, used used the exact same uuids and names as provided and tested on Chrome (76.0.3809.132), Firefox (69.0.1), Safari (13.0 [14608.1.49]) and Edge (42.17134.1.0). Maybe some browser plugin playing tricks or specific settings.
Shuffled the options multiple times on all browsers, all seems to be working as expected. Which means at the moment, I can't help out here.

[mj84]

Thanks for the feedback. Did you make sure to reopen the frontend and just click save again multiple times?
I doubt this is related to a browser addon, since I was using a vanilla Firefox without any addon or changed settings.

[AdSchellevis]

yes, shuffled it multiple times, opened in multiple browsers. maybe your browser console shows some issue somewhere..... I don't know, couldn't reproduce it myself.

[mj84]

The important aspect of this bug seems to be that you don't touch the linked actions at all between saves.

The exact procedure is:

• have a frontend with 2 linked actions
• open the frontend, don't change anything, save it, repeat multiple times.

The linked actions should change every 2nd time you hit save.

I was just able to confirm the issue by running a vanilla OPNsense + HAproxy in liveCD mode.

Browsers used:

• Vivaldi/2.7.1628.33 (Chrome/76.0.3809.137)
• Firefox 69.0.1
• Microsoft Edge 44.17763.1.0

All browsers showed the exact same behaviour.

[AdSchellevis]

@mj84 please be more specific next time, solving issues like this cost a lot of time since the report doesn't contain the steps needed to reproduce the issue (which normally means we can't handle it, or has a high risk of people looking at unrelated topics, such as the json sort order in this case). https://github.com/opnsense/core/commit/a4217d8fcccad2676fc6e449d15df3437d814bff should fix the issue when the same content is written multiple times to the widget.

opnsense-patch a4217d8f

[fraenki]

Thanks, @AdSchellevis! :+1:

[mj84]

@AdSchellevis thanks a lot for the quick fix!
Believe me, I was quite confused about this issue, as I saw it happen in the GUI, but at the time did not know too much about OPNsene internals.
Also, I was playing around with the API too and saw some weirdness there as well, I couldn’t quite describe the problem I was seeing.
I’ll try your patch tomorrow, thanks again!