Konga: Compatibility with Kong 1.x

[NikoGrano]

KongProxyController request error { name: 'invalid size',
  message: 'size must be an integer between 1 and 1000',
  code: 9 }

Using Konga 12.2 and 13.

Name        : kong-community-edition
Arch        : noarch
Version     : 1.0.0rc1
Release     : 1
Size        : 46 M
Repo        : installed
From repo   : bintray--kong-kong-community-edition-rpm
Summary     : Kong is a distributed gateway for APIs and Microservices, focused on
            : high performance and reliability.
URL         : https://getkong.org/
License     : ASL 2.0
Description : Kong is a distributed gateway for APIs and Microservices, focused on
            : high performance and reliability.

[NikoGrano]

This is caused by Kong 1.0.0. Please implement support for Kong 1.0.0. Thank You! :1st_place_medal:

[jeremyjpj0916]

Same, testing Kong 1.0rc2 some today and see similar errors on all service/route/consumer/plugins/upstreams/certificates pages in our dev environment.

My guess is they don't return size in the way the admin API did in the past?

[jeremyjpj0916]

@pantsel I know the problem :P , Kong caught onto your max int size tricks and finally now limits you to 1000 per request.

Implement something recursive like such as this with the pagination logic(my java client code), and Konga should be all set!:

MAX_SIZE = 1000

public Consumers getKongConsumers()
    {
        Consumers kongConsumers = new Consumers();

        request = new Request.Builder()
                .header("Authorization", "Bearer " + jwt_token)
                .url(base_admin_url + "/consumers?size=" + MAX_SIZE) 
                .build();
        try
        {
            response = client.newCall(request).execute();
            //Got a valid response body?
            if(response != null && response.body() != null && response.isSuccessful())
            {
                kongConsumers = gson.fromJson(response.body().string(), Consumers.class);

                if(kongConsumers != null && kongConsumers.getOffset() != null)
                {
                    kongConsumers = getKongConsumers(kongConsumers.getOffset(), kongConsumers);
                }
            }
            else if(response != null && "401".equals(Integer.toString(response.code())))
            {
                renewJwtToken();
                return getKongConsumers();
            }
            else
            {
                //Empty response body, how do we want to handle this
            }
        }
        catch(IOException e)
        {
            //Send an email
            System.out.println("GetKongConsumers" + " Error: " + e.getMessage());
        }

        return kongConsumers;
    }

    public Consumers getKongConsumers(String offset, Consumers kongConsumers)
    {
        Consumers paginatedConsumers = new Consumers();
        try {
            request = new Request.Builder()
                    .header("Authorization", "Bearer " + jwt_token)
                    .url(base_admin_url + "/consumers?offset=" + URLEncoder.encode(offset, "UTF-8") + "&size=" + MAX_SIZE)
                    .build();
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        try
        {
            response = client.newCall(request).execute();
            //Got a valid response body?
            if(response != null && response.body() != null && response.isSuccessful())
            {
                paginatedConsumers = gson.fromJson(response.body().string(), Consumers.class);
                kongConsumers.getData().addAll(paginatedConsumers.getData());
                if(paginatedConsumers.getOffset() != null)
                {
                    kongConsumers = getKongConsumers(paginatedConsumers.getOffset(), kongConsumers);
                }
            }
            else
            {
                //Empty response body, how do we want to handle this
            }
        }
        catch(IOException e)
        {
            //Send an email
            System.out.println("GetKongConsumers" + " Error: " + e.getMessage());
        }

        return kongConsumers;
    }

So no more of this in your js 😄
/assets/js/app/routes/00_routes.js

 'ServiceModel', function resolve(ServiceModel) {
                      return ServiceModel.load({
                        size : 4294967295
                      })

[NikoGrano]

Nice! Could you submit PR for this?

[pantsel]

@jeremyjpj0916 , yup, noticed that.
I just need to find some time to deal with the fix.
I believe that I will manage to get to it in the next couple of days

[jeremyjpj0916]

Excited to test it out! Once you get that out I will continue testing to see if there are any other compatibility issues between Kong 1.0 and the 0.14.x series Konga is already compatible with. Hopefully no more surprises hah.

[rsbrisci]

Bumping this for attention, we could also use this feature as Kong 1.0 will be released soon.

[NikoGrano]

Kong 1.0 will be released soon.
Isn't it released already? At least on my server Kong 1.0 is waiting to be updated and due this I need to hold that update and make YUM black magic to exclude it from automatic updates.

[nerdshack604]

Kong 1.0 GA dropped today -> https://konghq.com/blog/kong-1-0-ga/

[wilderchen]

I have encountered the same problem, anyone can make a PR to solve this issue?

[jeremyjpj0916]

@pantsel Did hack around and make Konga work with 1.0 just by forcing the 1000 limit on query in konga(without pagination because we do not have over 1k) as a quick hack. Seemed services/routes loaded. But the plugins pages and loading were messed up because of schema change violations so that will need some rework too I think, correct me if any of this is wrong @rsbrisci (I was not personally testing this because I am on a trip now, only relaying the info).

[wilderchen]

It seems that the konga isn't compatible with kong 1.0 right now. I found there are a lot of "{{humanizeLabel(key)}}" symbols when I trying to add a plugin

[dsteinkopf]

see also #315

[pantsel]

Hello guys, I know I've been absent for a while but I had some RL issues to address while waiting for Kong to finalize their 1.x version.

They've changed a number of things Konga was depending upon, and a s a result, issues like this and #315 started occuring.

The changes that need to be done to the project in order to achieve 1.x compatibility are not trivial and I need your help to verify that things are working as expected.

Kong 1.x compatibility is being worked on the next branch. Feel free try it out and drop some feedback. If you're using docker, pull pantsel/konga:next.

Bare in mind that while there may still be some broken features, the basic Kong API integration should be working just fine.

I'm yet to decide if Konga 0.14.x will be backwards compatible to Kong < 1.x. In case it won't, I will keep a branch of Konga 0.13.x active for the ones that don't want to upgrade their Kong installations just yet.

Cheers.

[jeremyjpj0916]

Cross posting from the gitter chat:
"
@pantsel Starting to look good, pagination seems to work on service/routes/plugins pages. Plugins page opening up the schema and fields seem to load fine. Consumers, their acl's and their creds seem to load fine(acl has been needing pagination and searchability for awhile but thats another issue for another day lol). So far 1 bug I have found. If you go into an individual service or an individual route, right now the plugins tab on either loads ALL plugins and not just the ones meant for that given service or route. Probably a simple fix not querying down on what needs to be.
"

[wilderchen]

Hello guys, I know I've been absent for a while but I had some RL issues to address while waiting for Kong to finalize their 1.x version.

They've changed a number of things Konga was depending upon, and a s a result, issues like this and #315 started occuring.

The changes that need to be done to the project in order to achieve 1.x compatibility are not trivial and I need your help to verify that things are working as expected.

Kong 1.x compatibility is being worked on the next branch. Feel free try it out and drop some feedback. If you're using docker, pull pantsel/konga:next.

Bare in mind that while there may still be some broken features, the basic Kong API integration should be working just fine.

I'm yet to decide if Konga 0.14.x will be backwards compatible to Kong < 1.x. In case it won't, I will keep a branch of Konga 0.13.x active for the ones that don't want to upgrade their Kong installations just yet.

Cheers.

After debugging into the konga project, I found that the kong 1.0 version bring quite a lot break changes. For example, the service schema interface response change the "fields" from dict/map/hash to list. It seems that it may need quite a lot time to follow up the new version of Kong.

[pantsel]

@wilderchen , have you noticed something concerning the services entity that is not working as expected on Konga:next due to that change? It seems to be working fine on my end.

[pantsel]

@jeremyjpj0916 ,

right now the plugins tab on either loads ALL plugins and not just the ones meant for that given service or route

This happens because their API stopped working with query strings for some reason.

/plugins?service_id={service_id} returns all plugins

only the /services/{service_id}/plugins is valid now.

For the exact same reason, the Eligible consumers on routes and services are not returning correct results.

/services/{service_id}/plugins?enabled=true is not working.

That's why it's hard to keep up with this project. They're breaking everything on nearly every release with no apparent reason. Almost feels like they're doing it on purpose.

Anyways, I'll make the needed changes for the plugins but I will have to wait and see if they will bring query string filters back before i start meddling with the eligible consumers logic.

Thanks again for the feedback.

[jeremyjpj0916]

@pantsel Good stuff! Yeah I think they are changing from query parameter to path based because its more "restful" or they consider it to be better taxonomy. I am not entirely sure. Maybe they are doing it just to make more work for yah 😆 . Either way will give your latest next branch commits a go and see if its looking good!

[pantsel]

Well IMHO that's neither more 'restful' nor better taxonomy. Quite the contrary, it's ver un-restful, plus apart from the results you can get through paths, now there's no way for example to only get a list of only enabled plugins for a service/route?

That doesn't make sense to me...

[jeremyjpj0916]

@pantsel

Creating an upstream resource seems to work against Kong 1.0, creating a target(ip:port) does not seem to work against Kong 1.0. Also snapshots does not work(but I consider it to be the lowest of priorities).

Edit - actually unconfirmed on the ^ with targets, my vpn disconnected from the network i was on when i claimed that lol. Will update later.

-Jeremy

[jeremyjpj0916]

@pantsel

Seems my upstream edit throws an interesting error, Images are as follows top / middle / bottom. All I really want is the active healthcheck by itself with no passive healthcheck anyways... Error in red at the top when i tried to save after deleting all the http statuses out of the auto-populated passive healthchecks section after my initial submit:

Edit -

Also Konga is lacking fields active/passive health checks expose(things like validate the certificate for https healthchecks):
https://docs.konghq.com/1.0.x/admin-api/#add-upstream

A few that seem important are:
healthchecks.active.type
healthchecks.active.https_verify_certificate

The actual targets page looks pretty solid in general, I like the heart next to the targets indicating health which i imagine is coming from the upstreams/{name or id}/health api call for the page 👍 . But adding the parameters that will help support https like I mentioned will be good as it seems right now I can't setup an HTTPS targets in Konga because I can't disable the ssl verify in UI(we are using self-signed certs).

Another point to make is Kong's active vs passive healthchecks documentation is confusing.. It seems to basically disable the use of one of them you need to keep tcp/http_failures / timeouts parameter to the default of 0 ?? I wish in the Konga UI I could do a toggle at the high level those sections(active vs passive) of (enable/disable, and disable won't even let the konga UI do the drop down action on that given active or passive path) and you let Konga internally do the heavy lifting to ensure these fields are defaulting properly so active health-checks are only running or say passive health-checks are only running. Maybe thats just a IMO, but right now it feels tricky having to think about the 0's and such, giving me a headache just thinking about it mate 😆 .

[jeremyjpj0916]

@pantsel I see you did some commits on Christmas 👍 , coding Santa over there lol. So new issue. ACL whitelist to a Service or Route does not seem to work at the moment:

EDIT , also seems the add JWT to service/route seems a bit off. Notice the Object in brackets bit for cookie names:

Edit Edit - tried posting too and noticed claims to verify I could not bubble up say the jwt exp claim as a field or anything in the UI, and the error at the top due to this is: "{"uri_param_names":"expected a set","claims_to_verify":"expected a set"} "

You are doing the lords work man, keep going strong towards this 1.0 compatibility as you are on a roll!

[jeremyjpj0916]

Also this page, the consumers accessible services and routes page does not load. Stays with those loading bubbles forever for me and never completes:

Not a priority, but a really nice to have to see what proxies a user can call.

[pantsel]

@jeremyjpj0916 , that was due to the changes on the plugins schema like @wilderchen pointed out.
Just pushed some fixes on the next.

Thanks for the debuging effort man!

Keep it up.

We're close...

[jeremyjpj0916]

@pantsel

Okay -
JWT plugin seems fixed
ACL plugin seems fixed
Upstreams stuff seems fixed(edit/passive/active and newer fields)

Broken -
Rate limiting/Correlation Id plugin on service/route (drop down menu options no longer seem to look right, treated as text field currently)

Also the consumers accessible services and routes page does not load still behaving like my prior screenshot.

Edit - Also removing API(Deprecated) from UI will be good, it no longer exists as a resource in 1.0, as well as the consumers accessible API resources page).

[pantsel]

@jeremyjpj0916 , he consumers accessible services page loads with no probs on my side.

Can you please check the JS console or the server logs for "ninja" errors :)?

If there aren't any, it probably means that it takes a lot of time to do the back and forth calculations with Kong (since you have lots of consumer groups).

If no errors are raised, wait and see if it times out.

[jeremyjpj0916]

Its been up doing the dot spin thing for about 10 mins lol. I don't see why it would be the this slow for 500ish things? Nothing in my js logs or server logs, the terminal window for std out errors has nothing either.

1. Get all routes
2. Get all services
3. Get all plugins
4. Pair plugins(acl/auth plugins(jwt/oauth2) to the service/routes by the service/route id tied to them locally in konga as a map.
5. Get consumer, check if he has oauth/jwt/etc creds and get his acl groups.
6. If consumer lacks the creds or lacks the acl whitelist group(or he has the acl blacklist group, I don't use blacklist personally though) against the service/route maps with their resources then not authorized?

Doing mostly local processing after querying the needed info from Kong on a few calls?

The query for each of those takes a few seconds 2-5 on the host using admin API now but not minutes :p .

I tried snapshots too, didn't seem to work yet either.

[pantsel]

I can't seem to be able to reproduce this one man. I'll try to investigate further and get back to you.

[jeremyjpj0916]

Its not a big deal yo, the other issues are more worthwhile to tackle anyways(that impact usability):

Rate limiting/Correlation Id plugin on service/route (drop down menu options no longer seem to look right, treated as text field currently)

and

Also removing API(Deprecated) from UI will be good, it no longer exists as a resource in 1.0, as well as the consumers accessible API resources page).

For others maybe snapshots are useful too. I do straight C* backups for Kong generally.

Edit -
The request transformation plugin seems broken from how it used to look as well.

All the ^ are more important anyways :P .

[pantsel]

The request transformation plugin also breaks due to the schema changes.

APIs: Im using 1.0.0rc2 and the resource is still there and fully functional

[jeremyjpj0916]

@pantsel Right rc2 was an early release candidate and not fully in the image of Kong 1.0 yet. 1.0 drops all support for API resources since they intend for all active Kong users to migrate to service/route model for the 1.0 release(there is a Kong 0.15 which has most of the 1.0 enhancements with support still for deprecated resources in that last version, it will be the only version to support them any longer as all futher Kong 1.0+ will not).

[jeremyjpj0916]

Updating a service does not work either it seems(I tried changing the name), I assume the same error probably applies for updating routes too. Error thrown is this:

[jeremyjpj0916]

@pantsel
Further info on why some of my plugins can't load or add/enable/disable on 1.0, such as:
https://github.com/Optum/kong-upstream-jwt (I think it has to do with being an "empty" schema that has no fields?)

0.14.1 Konga JSON view:
{
"created_at": 1516388708653,
"config": {},
"id": "79b19c0f-43ec-4353-8bb2-2e1e7ea7eafb",
"name": "kong-upstream-jwt",
"enabled": true
}

$ curl localhost:8001/plugins/schema/kong-upstream-jwt
{"fields":{}}
/ $

---

1.0 Konga JSON view(can't open or disable/enable this plugin under the hood):

{
"created_at": 1544487594.438,
"consumer": null,
"id": "74865976-80ba-4e22-8c56-bd9466841676",
"service": null,
"enabled": true,
"run_on": "first",
"config": {},
"route": null,
"name": "kong-upstream-jwt"
}

/ $ curl localhost:8001/plugins/schema/kong-upstream-jwt
{"fields":{}}
/ $

Error in console when I attempt to disable/enable this plugin:
KongProxyController request error { message: '2 schema violations (created_at: expected an integer; run_on: required field missing)',
name: 'schema violation',
fields:
{ run_on: 'required field missing',
created_at: 'expected an integer' },
code: 2

DELETING it does work if its already present(was there before i upgraded to 1.0)

Now after deleting it I manually added it back via CURL with:
curl -X POST http://localhost:8001/plugins --data "name=kong-upstream-jwt"

Maybe a new way you handle plugins in 1.0 causes a breaking issue with empty schema plugins?

EDIT - Even weirder, after I had deleted it and added the plugin back manually with curl on 1.0 the enable/disable now works with the Konga toggle(but I still can't open it). Idk why that made a difference unless something about the 0.14.1 migration of an empty schema plugin vs adding it ontop of 1.0 yeilds slight differences. But regardless I still can't add or open an empty schema plugin from scratch with Konga right now so that is a bug.

[jeremyjpj0916]

Also for tracking purposes, maybe this is also why the request transformer plugin your Konga app struggles with: https://github.com/Kong/kong/issues/4137 , seems they have some fixes to make of their own after 1.0 regarding that plugin.

[dsteinkopf]

I also have a problem when trying to change the config of one of my own plugins via konga. Output in konga log:

KongProxyController request error { message: 'schema violation (run_on: required field missing)',
name: 'schema violation',
fields: { run_on: 'required field missing' },
code: 2 }

I don't know if it is a problem with my plugin not being upgraded to kong 1.0. It looks like this was the problem - but I haven't found something about this in the plugin documentation.

Should I open a new issue and/or describe the problem here in more detail? Or does anybody have an idea how to solve this?

[pantsel]

@dsteinkopf , I'll try to use your plugin with the changes I'm making for 1.x compatibility. I'll get back to you once I have an answer.

By the way, have you tried with konga:next? @jeremyjpj0916 's custom plugins are tested and working now.

[dsteinkopf]

Thank you very much. If I can do anything myself or help you in some way, just let me know.

Yes, I am running kong and konga via docker: konga image is pantsel/konga:next. (kong 1.0 image seem not to be available officially - so I am using dsteinkopf/kong-dup:latest which in turn uses temporarily dsteinkopf/kong-1.0:2018-12-27)

[pantsel]

@dsteinkopf , I intsalled your plugin localy using the next branch and it seems to be working fine.

Bare in mind that the next branch now holds Konga 0.14.1-rc6. So, If that's not the version you currently have installed, update your installation.

Let me know what happens.

[dsteinkopf]

Yes, you're right (partly): Now I am using "KONGA 0.14.1-rc7" (but I don't know if that makes a difference compared to the version rc4 (?) this morning). Adding the plugin newly works perfectly - I did some tests.

But I am not able to change any existing plugin that was migrated from an old version. Two examples:

ip-restriction (bundled plugin): I just click on the configured plugin, the config editor opens with the existing values. Then I press "Submit" without any changes and get the error:

3 schema violations (only one of these fields must be non-empty: 'config.whitelist', 'config.blacklist'; config.blacklist: required field missing; run_on: required field missing)

When I click on the "eye" symbol to show the raw view I see

{
  "created_at": 1543291174,
  "config": {
    "blacklist": null,
    "whitelist": [
      "87.169.107.181",
      "192.168.40.0/22"
    ]
  },
  "id": "a7fc793b-5760-495f-b9a5-54346fab3656",
  "service": null,
  "enabled": true,
  "run_on": "first",
  "consumer": null,
  "route": {
    "id": "b3cad4a4-b666-4e34-99b2-38c0b7d09c58"
  },
  "name": "ip-restriction"
}

kong-http-to-https-redirect (my own plugin): Again, just open the editor and "submit" results in this error:

schema violation (run_on: required field missing)

When I click on the "eye" symbol to show the raw view I see ..."run_on": "first",....

Any other tests I could to to help?

---

Another small problem is the "created" date show when viewing then plugins via Services -> my service -> Plugin is "Jan 18, 1970". Also when I Services -> my service -> Routes -> my route -> Plugins. I assume a confusion of second an miliseconds.

[pantsel]

Thanks for the feedback @dsteinkopf , I'll get back to you when I check those things out.

[jeremyjpj0916]

Also seeing a problem around request-termination plugin currently, I cannot disable/enable it. I cannot submit to modify it(it originally existed before the migration from 0.14.1 to 1.0). Here is its current JSON as seen by Konga:

{
  "created_at": 1544487241.556,
  "consumer": null,
  "id": "fe7c76be-ffc6-4732-b6ac-9b6b48b5bee1",
  "service": null,
  "name": "request-termination",
  "run_on": "first",
  "config": {
    "status_code": 200,
    "body": null,
    "content_type": null,
    "message": "Success"
  },
  "route": {
    "id": "c358bcf8-16ea-45cc-a1a3-0e0b6bddd006"
  },
  "enabled": true
}

What displays when trying to edit it:

Edit, I also tried to arbitrarily add the plugin to a fresh route, different error:

[dsteinkopf]

@jeremyjpj0916, this seems to be the same problem as mine (see above).

[dsteinkopf]

After manually "PATCH"ing the ip-restriction plugin (from my example above) via curl, this plugin can now be edited via konga. I fetched the plugin's data from kong via curl, removed created_at from it, added "blacklist": []. Then curl --insecure -X PATCH -H "Content-Type: application/json" -H "Accept: application/json" --data $(jq -c . <xxip-restriction.mod) https://kong-ext-zt.steinkopf.net/kong-ext/plugins/a7fc793b-5760-495f-b9a5-54346fab3656 | jq.

[pantsel]

I think it's fixed with the latest commits.
Pull next again and check it out.

Waiting for feedback.

Cheers

[dsteinkopf]

(had to wait for build on docker hub)
Sorry, konga footer shows rc8 but I don't see any difference to before: Still schema violation (run_on: required field missing) on my own and bundles plugins when they are "old" (migrated from before kong 1.0). Did I make a mistake??

What did you change? (How) can we help?

[dsteinkopf]

Good news about the displayed creation date: Seems correct now 👍

[pantsel]

@dsteinkopf , I found a problem with plugin updates that was addressed with commit https://github.com/pantsel/konga/commit/f8a565f6861c5af79037f9b23c0c6cbf04089850 .

Other than that, I cannot reproduce the issue on my env. Maybe because I'm applying the plugins from scratch.

* Long shot: try on different browsers just in case. We stumbled upon a strange cache-related thingy yesterday with @jeremyjpj0916 . *

Also, I have no idea what this run_on field is and why it is required.
Maybe, it has to do with some sort of db shema patching when migrating to Kong 1.x?

What If you delete and and re-apply those plugins.

Everything works OK (create/update)?

[dsteinkopf]

You could try to reproduce the problem by installing a 0.14 version, installing some plugin(s) and then upgrading. Of course, this is much hassle...

At the moment I am not at a PC, but I tried changing a plugin via my Phone. Still the same error about run_on :-(

I read about run_on on the kong pages... Question is: Why is it missing when konga is updating the plugin, while it is there in the json kong which sends when fetching the plugin data. Also konga shows it inthe raw view ...

Newly adding a plugin works fine.

Also I cannot see any difference between migrated and new plugins. Both contain the run_on attributes. At least it looks like this.

Could you (or I myself with your help) build a konga version which does more logging? e.g. the jsons kong gets from and sends to kong?

[jeremyjpj0916]

@dsteinkopf always when you redeploy konga with changes open it in a private browser from scratch. The browser cache has a lasting impact on Konga. Regardless I am going to give the things I saw broken a go now today and see if they are fixed and report back here.

[pantsel]

@dsteinkopf , @jeremyjpj0916
I think I've found the problem while upgrading from 1.0.0rc2 to 1.0.0.

Patched that and pushed a commit on the next (0.14.1-rc9 now).

Fingers crossed!

[jeremyjpj0916]

My update on the progression of this on the next branch:

Everything seems to be great now, have not tested snapshot imports but I see latest issues resolved such as:

1. request termination saves fine now
2. consumer accessible routes page loads properly
3. "bundled" is no longer a required plugins dependency to see the ^ page or to take snapshots

IMO Konga is ready to expose a full release for 1.0 at this stage. The only caveat it seems to me is that at the moment the request transformation plugin is broken on Kong's end, so Konga can't exactly integrate properly with it yet, but once that done I am sure @pantsel can do a quick fix up for that specific plugin.

EDIT -

One more bug lol, go to consumers page and try to create an Oauth2.0 token on consumer, you will see a schema violation under "name" in red. I tried a HS256 JWT creation for a consumer and that seems to work though. I only use oauth/jwt creds on consumers so someone else can give the other boring ones (key auth/hmac/basic auth) a go 😆 .

[pantsel]

@jeremyjpj0916 , that error was missleading.

The actual error if you check the console is : redirect_uris: "required field missing"

According to Kong's docs, the property name is redirect_uri but Kong 1.x needs it to be redirect_uris and expects an array.

That change is not properly documented.

[dsteinkopf]

YESSS. I also did some experiments and my problems with run_on don't exist anymore! :-) Good work. Thank you, btw, for konga at all !

But I am also still experiencing the problem with the request and response transformer plugins (which you seem to know about).

And updating migrated existing plugins ip-restriction and acl still results in schema violation (config.blacklist: field required for entity check when updating). I guess this is a problem with the difference between empty/null/whatever and an empty list.

[pantsel]

And updating migrated existing plugins ip-restriction and acl still results in schema violation (config.blacklist: field required for entity check when updating). I guess this is a problem with the difference between empty/null/whatever and an empty list.

I think that you won't be able to avoid some manual labor like deleting -> re-assigning plugins on this one.

[dsteinkopf]

ok. that would be ugly but acceptable... But aren’t you (konga) able to get the schema from the plugins and check if there are any missing attributes?

[pantsel]

That's what Konga is doing. The attributes are generated dynamically from the schema.

Still, I don't think that the specific issue config.blacklist: field required for entity check when updating has to do with that.

On my env, those plugins are working with no probs whatsoever so I can't reproduce that error even after upgrading Kong 1.0.0rc2 to 1.0.0.

Long shot: Is it possible you had defined config.blacklist before upgrading and now you're trying to delete config.blacklist and setting only config.whitelist?

UPDATE:
Can you post a screenshot so I can better understand the glitch?

[dsteinkopf]

I did some experiments to check your question about existing blacklist.

This is one of my upgraded plugins:

# curl --silent --insecure -X GET https://mykonga/kong-ext/plugins/44c111f6-863a-4a42-a269-097f45b2dc79 | tee xxip-restriction_from_kong.json | jq .
{
  "created_at": 1543148192,
  "config": {
    "whitelist": [
      "192.168.40.5",
      "192.168.40.21"
    ]
  },
  "id": "44c111f6-863a-4a42-a269-097f45b2dc79",
  "service": {
    "id": "5e350633-545d-44dd-bdf7-1ac1becca952"
  },
  "enabled": true,
  "run_on": "first",
  "name": "ip-restriction"
}

As you can see, blacklist is missing here. Kong won't accept this json:

# curl --silent --insecure -X PATCH -H "Content-Type: application/json" -H "Accept: application/json" --data @xxip-restriction_from_kong.json https://mykonga/kong-ext/plugins/44c111f6-863a-4a42-a269-097f45b2dc79 | jq
{
  "message": "schema violation (config.blacklist: field required for entity check when updating)",
  "name": "schema violation",
  "fields": {
    "config": {
      "blacklist": "field required for entity check when updating"
    }
  },
  "code": 2
}

Then I added an empty blacklist to the JSON:

jq -c <xxip-restriction_from_kong.json '.config |= .+ { "blacklist":[] }' | tee xxip-restriction_plus_blacklist.json | jq
{
  "created_at": 1543148192,
  "config": {
    "whitelist": [
      "192.168.40.5",
      "192.168.40.21"
    ],
    "blacklist": []
  },
  "id": "44c111f6-863a-4a42-a269-097f45b2dc79",
  "service": {
    "id": "5e350633-545d-44dd-bdf7-1ac1becca952"
  },
  "enabled": true,
  "run_on": "first",
  "name": "ip-restriction"
}

This will work then:

# curl --silent --insecure -X PATCH -H "Content-Type: application/json" -H "Accept: application/json" --data @xxip-restriction_plus_blacklist.json https://mykonga/kong-ext/plugins/44c111f6-863a-4a42-a269-097f45b2dc79 | jq
{
  "created_at": 1543144592,
  "config": {
    "whitelist": [
      "192.168.40.5",
      "192.168.40.21"
    ],
    "blacklist": []
  },
  "id": "44c111f6-863a-4a42-a269-097f45b2dc79",
  "service": {
    "id": "5e350633-545d-44dd-bdf7-1ac1becca952"
  },
  "enabled": true,
  "run_on": "first",
  "consumer": null,
  "route": null,
  "name": "ip-restriction"
}

Now this plugin seems to be correctly upgraded and I am also able to edit it via konga.

So I think, konga should be able to add the blacklist by itself. - But I also consider this described method as an acceptable way to upgrade the plugins manually (at least for me).

--

Now I have to look for another plugin in my installation with this problem to create screenshots :-)

[dsteinkopf]

This is a screenshot just after pressing "submit":

[pantsel]

Ok so the blacklist field was not required prior to Kong 1.0.0 and it is now.

It would be helpful if they had a required: true property in the plugins schema but instead they got this:

{
    "fields": [
        {
            "whitelist": {
                "elements": {
                    "type": "string"
                },
                "type": "array"
            }
        },
        {
            "blacklist": {
                "elements": {
                    "type": "string"
                },
                "type": "array"
            }
        }
    ]
}

Apart from that, they could easily fill in the required attributes with the defaults when serving the response from their API but I don't think that 3rd party compatibility is something they go for :).

For Konga, there is no way of knowing if a field is required without per-case monkey-patching which is something I generally try to avoid.

[dsteinkopf]

Yes, you're right. I consider this a bug in this plugin :-(

The kong plugin doc says about the required attribute in the schema:

Default: false. If true, the property must be present in the configuration.

So required does NOT say that this field has to be filled (not empty) - it just says it must be present. Should we open a bug report in kong?

[pantsel]

I'll try to handle it somehow, maybe by populating everything there is in the schema regardless of whether the property exists in their reponse or not

[dsteinkopf]

Ok fine. But as you said, this is not the ideal solution...

[pantsel]

We work with what we have :)

[pantsel]

Strange, this is what Konga is sending to Kong on my env.
As you can see there is no config.blacklist defined.

{
  "created_at": 1546179319,
  "id": "ddf25733-116d-4dbc-b5a9-490c98b4d04d",
  "service": null,
  "enabled": true,
  "run_on": "first",
  "consumer": null,
  "route": null,
  "name": "acl",
  "config": {
    "whitelist": [
      "test"
    ],
    "hide_groups_header": false
  }
}

Still, it updates with no probs.

Im using Kong 1.0.0 from https://github.com/Kong/docker-kong/tree/master/alpine

Question, are you by any chance using Casandra as a datastore? Im using Pg here.

[dsteinkopf]

THAT is strange. I am also using kong 1.0.0 which I built myself from https://github.com/Kong/docker-kong (alpine directory) because kong 1.0 isn't yet available from https://hub.docker.com/_/kong.

Did you try to use curl and PATCH this JSON (without blacklist) ?

And no - I am also using pg.

[dsteinkopf]

BTW. What is your knowledge of the transformer plugin problem? (I am currently trying to add a request-transformer without success.)

[pantsel]

That is a Kong issue. The schema is messed up on 1.0.0
https://github.com/Kong/kong/issues/4137

[jeremyjpj0916]

@pantsel Noted on the oauth change here for their docs: https://github.com/Kong/docs.konghq.com/issues/1092 , if I get time later I will fix it myself, thanks for fixing it in Konga!

[jeremyjpj0916]

@pantsel Okay tested OAuth2.0 client cred creation now, seems to be working good. Did some regression testing and it seems the request termination plugin created on a route breaks once again xD -

To apply it to Kong via curl I did this:

curl -X POST http://localhost:8001/routes/c358bcf8-16ea-45cc-a1a3-0e0b6bddd006/plugins --data "name=request-termination"  --data "config.status_code=200" --data "config.message=Success"

And the json the above produces from Konga view:

{
  "created_at": 1546280442,
  "consumer": null,
  "id": "7fb1ff6b-a7c4-4c62-91a9-bc40100d7f7a",
  "service": null,
  "enabled": true,
  "run_on": "first",
  "name": "request-termination",
  "route": {
    "id": "c358bcf8-16ea-45cc-a1a3-0e0b6bddd006"
  },
  "config": {
    "status_code": 200,
    "content_type": null,
    "body": null,
    "message": "Success"
  }
}

I can taste the finish lineeeee.

[jeremyjpj0916]

Confirmed ^ problem was fixed by the commit. Thx.

[jeremyjpj0916]

@pantsel

So I wiped out my 500+service/route keyspace Kong sandbox and now went back to super barebones for some mem leak testing of Kong. I setup a keyspace and added 2 service/route pairs and that was absolutely it. I set the KONG_PLUGINS env field to "bundled" just since that is the Kong default. I have no custom plugins or anything now. Snapshots under these conditions now is throwing a MYSQL error it seems:

[jeremyjpj0916]

@pantsel ^ seems to have fixed the issue. I did try a restore snapshot and I have a oauth2 and jwt cred assigned to a user. I didn't restore it to an empty node but just clicked restore on a kong environment that already had that snapshot data and I see these errors:

I imagine if it successfully checks they are already present it should just count as a success and keep going right and add +1 to the imported row? Since services/plugins/routes/cousmers/acls seem to have taken that same action successfully.

Edit -
I also made a Kong 1.0 node with 5k service/route pairs with oaut2 and acl added to route. Then 5k consumers with jwt/oauth creds and a white-list group to them. I attempted a Konga backup of that and it ran spinning a really long time before stopping, I imagine a built in timeout? Does not seem to have made the backup after I waited minutes though.

[jeremyjpj0916]

Update, now testing 1.0.2, seems the request transformation plugin page works like a charm out of the box 👍 .

[dsteinkopf]

@jeremyjpj0916 , I can confirm this.

[pantsel]

So I guess we're ready to merge next to master?
I'll do some final tests and publish the new release.

Meanwhile, if you guys see anything unexpected, let me know.

[jeremyjpj0916]

@pantsel I would say it is certainly good enough to release, I would test snapshots and consumer eligible routes at scale with plugins and all enabled on the routes(see my earlier comment a few up with a snapshot import that failed).

Thanks!

[amitofile]

@Niko9911 I got same error on every page services, routes, certificates,

KongProxyController request error { name: 'invalid size',
message: 'size must be an integer between 1 and 1000',
code: 9 }

So I changed the value for,
size : 4294967295 to 999 (in assets/js/app/routes/00_routes.js)
itemsFetchSize: 4294967295 to 999 (in assets/js/app/core/services/ListConfigService.js)

This fixed the alert and started showing lists.
I hope this is fine.

Kong v1.0.2
konga v0.13.0

[pantsel]

@amitofile , Konga 0.13.0 is not compatible with Kong 1+. The change you made won't save the day:).

We're working on Kong 1+ compatibility on the next branch.

It's 99.9% ready and fully compatible with Kong 1+.

You might want to try that instead.

[NikoGrano]

@amitofile as pantsel said.... It's not about getting one thing working. If I fix that error, there will be 10 things to fix. Please do not spam this ticket. Let's focus developing full support for Kong 1.0.

Could you contribute to this issue @amitofile by downloading next branch and testing it against some edge cases? Would appropriate it a lot! :1st_place_medal:

[amitofile]

@pantsel @Niko9911
Yes. next branch nailed it :+1:
It also solved this https://github.com/pantsel/konga/issues/315
Thanks.

[pantsel]

Closing this issue due to housekeeping since Konga 0.14.1 is fully compatible with Kong 1,x.

Feel free to reopen if you stumble upon anything weird.