Kong: Migration of custom plugin not run when plugin installed in custom path (kong 1.x)

[pamiel]

Summary

Version 0.x of Kong used to have an issue when running the migration process for custom plugins, when these plugins were located in a custom path (i.e. not in the default /usr/local/share/lua/5.1/kong/plugins directory): migration was indeed not taken into account (see the issue here: https://github.com/Kong/kong/issues/3394).
A good workaround to this issue is to set the LUA_PATH envar before running the migration process, through the following command:

export LUA_PATH="$APP_LIB/?.lua;`luarocks path --lr-path`"

Unfortunately, since version 1.0 and the new migration framework, it looks the migration of the plugins located in a custom path is still not working, and the workaround itself has no more effect: the logs of the kong migrations up process shows nothing related to the execution of the migration of the custom plugin entities, and tables are not created in Postgres.

Running Kong in server mode, we can see the following logs that show that the plugin is found by Kong during startup of Kong in server mode… but execution of the plugin fails later because the tables do not exist.

2019/04/15 15:02:56 [debug] 190#0: [lua] plugins.lua:267: load_plugin_schemas(): Loading plugin: my-key-auth
2019/04/15 15:02:56 [debug] 190#0: [lua] plugins.lua:281: load_plugin_schemas(): Loading custom plugin entity: 'my-key-auth.my_keyauth_credentials'
…
2019/04/15 15:02:56 [debug] 197#0: *5 [lua] base_plugin.lua:13: init_worker(): executing plugin "my-key-auth": init_worker

Steps To Reproduce

1. Create a plugin with new dedicated entities. Implement the migration inside the migrations/000_base_xxx.lua and reference it into the migrations/init.lua file.
2. Manually deploy the whole plugin directory in a kong/plugins directory outside the Kong internal one
3. Run kong migrations up

Logs does not show that migration of the custom plugins has been done.

1. set the LUA_PATH envar as stated above
2. run again the migration

Still nothing into the logs. Migration is not executed.

Additional Details & Logs

• Kong version: 1.0.3
• Operating system: alpine

[hishamhm]

hi @pamiel! I was unable to reproduce it in latest master.

This is what I tried to do:

1. first I created a path that's found in LUA_PATH

$ mkdir ~/.luarocks/share/lua/5.1/kong/plugins

1. now let's go to the kong sources where I run my dev environment

$ cd kong

1. move a plugin with migrations out of the Kong tree

$ mv kong/plugins/oauth2 ~/.luarocks/share/lua/5.1/kong/plugins

1. reset and bootstrap migrations - verify that it still runs the oauth2 migrations

$ bin/kong migrations reset
$ bin/kong migrations bootstrap
bootstrapping database...
[...]
migrating oauth2 on database 'kong'...
oauth2 migrated up to: 000_base_oauth2 (executed)
oauth2 migrated up to: 001_14_to_15 (executed)
oauth2 migrated up to: 002_15_to_10 (executed)
[...]
24 migrations processed
24 executed
database is up-to-date

oauth2 is there, 24 migrations executed.

1. now let's make sure the plugin isn't found anymore in the default bundled plugins list: I renamed "oauth2" to "oauth3".

$ mv ~/.luarocks/share/lua/5.1/kong/plugins/oauth2 ~/.luarocks/share/lua/5.1/kong/plugins/oauth3

1. and try migrations again:

$ bin/kong migrations reset
$ bin/kong migrations bootstrap
bootstrapping database...
migrating core on database 'kong'...
[...no oauth2 to be seen...]
21 migrations processed
21 executed

correct — now it's 21.

1. now let's try to make it use the custom-named plugin, via KONG_PLUGINS (a.k.a. plugins in kong.conf):

$ bin/kong migrations reset
$ KONG_PLUGINS=bundled,oauth3 bin/kong migrations bootstrap
bootstrapping database...
migrating core on database 'kong'...
[...]
migrating oauth3 on database 'kong'...
oauth3 migrated up to: 000_base_oauth2 (executed)
oauth3 migrated up to: 001_14_to_15 (executed)
oauth3 migrated up to: 002_15_to_10 (executed)
[...]
24 migrations processed
24 executed
database is up-to-date

24 migrations executed, the "oauth3" plugin was found and migrated.

Maybe what's missing is that you're not specifying the custom plugin when running kong migrations?

[pamiel]

Ok, I got it:

• plugins in kong.conf is correctly set on my side, but reproducing your test case is not leading to a correct migration
• looking again to the default values of the kong. conf file, I realized that 2 paths were provided (https://github.com/Kong/kong/blob/master/kong.conf.default#L697) ./?.lua;./?/init.lua; while my lua_package_path only has a single one: /my/path/?.lua. So, adding the missing one... but migration is still not called
• So finally tried the initial workaround to set the LUA_PATH envvar with the 2 paths before running the migration, and this time it is working fine !

As a conclusion:

• the 2 path are required (how could I missed that!)
• but still the bug of the lua_package_path is present
• and hopefully the workaround with the LUA_PATH still works (do you have it as well?) !

Thanks a lot for your help.
I let you keep open or close the issue depending on whether you want to keep it for the lua_package_path issue

[hishamhm]

the 2 path are required (how could I missed that!)

Correct!

but still the bug of the lua_package_path is present

Can you try this patch?

diff --git a/kong/cmd/migrations.lua b/kong/cmd/migrations.lua
index 51dc8e589..6109bd9e3 100644
--- a/kong/cmd/migrations.lua
+++ b/kong/cmd/migrations.lua
@@ -84,6 +84,8 @@ local function execute(args)
   conf.cassandra_timeout = args.db_timeout -- connect + send + read
   conf.cassandra_schema_consensus_timeout = args.db_timeout

+  package.path = conf.lua_package_path .. ";" .. package.path
+
   _G.kong = kong_global.new()
   kong_global.init_pdk(_G.kong, conf, nil) -- nil: latest PDK

and hopefully the workaround with the LUA_PATH still works (do you have it as well?) !

Yes, it works (I had the ~/.luarocks paths in my LUA_PATH already).

I let you keep open or close the issue depending on whether you want to keep it for the lua_package_path issue

Let me know if the above tweak does the trick! If so, I'll close this and send a PR with it.

[pamiel]

Yes, this works ! No more need to set the LUA_PATH envvar with it.

Thanks a lot for your help.

[fighterleslie]

the 2 path are required (how could I missed that!)

Correct!

but still the bug of the lua_package_path is present

Can you try this patch?

diff --git a/kong/cmd/migrations.lua b/kong/cmd/migrations.lua
index 51dc8e589..6109bd9e3 100644
--- a/kong/cmd/migrations.lua
+++ b/kong/cmd/migrations.lua
@@ -84,6 +84,8 @@ local function execute(args)
   conf.cassandra_timeout = args.db_timeout -- connect + send + read
   conf.cassandra_schema_consensus_timeout = args.db_timeout

+  package.path = conf.lua_package_path .. ";" .. package.path
+
   _G.kong = kong_global.new()
   kong_global.init_pdk(_G.kong, conf, nil) -- nil: latest PDK

and hopefully the workaround with the LUA_PATH still works (do you have it as well?) !

Yes, it works (I had the ~/.luarocks paths in my LUA_PATH already).

I let you keep open or close the issue depending on whether you want to keep it for the lua_package_path issue

Let me know if the above tweak does the trick! If so, I'll close this and send a PR with it.

package.path = conf.lua_package_path .. ";" .. package.path

This patch worked like charm!

[fighterleslie]

@hishamhm Would you please close this issue and send a PR with your patch?

[Tieske]

@hishamhm just a heads-up: I think I just ran into the same issue, but initiated from a plugin test. In which case the migrations are not started through the CLI, but directly from the test-helpers.
(in which case the fix doesn't work)

[Tieske]

hmmm, switched dev environment from Gojira to Vagrant, now it works. Will post this with the Gojira folks. Sorry for the noise.