Kong: custom_plugins configuration in kong_tests.conf appears to be ignored in helpers.lua

[lavoiedn]

Summary

Setting the custom_plugins configuration in kong_tests.conf will result in the configuration being ignored when running tests using spec/helpers.lua.

I believe it is an oversight, apologies if it is working as intended or a result of improper use of the test helpers.

Steps To Reproduce

1. Write a plugin in a standalone repository, copying bin/busted, spec/helpers.lua and spec/kong_tests.conf from the kong repository as indicated here.
2. Try to execute tests using the bin/busted binary and the spec/kong_tests.conf file, specifying custom_plugins.
3. When running bin/busted, the custom plugin will not be taken into account, custom_plugins will always be equal to: "dummy,cache,rewriter". (https://github.com/Mashape/kong/blob/0.11.0/spec/helpers.lua#L837)

I looked at the template plugin and it passes custom_plugins in the start_kong call, but I still thought that overriding the kong_tests.conf file's configuration was unintended behaviour.

The workaround I found at the moment is to add a condition that checks for conf.custom_plugins before replacing env.custom_plugins with "dummy,cache,rewriter". See here.

I wasn't sure what the intent of that default value was so I didn't want to attempt a different fix without checking if it was actually a bug first.

Additional Details & Logs

• Kong version: 0.11
• Kong debug-level startup logs (test output):

Error → spec/01-plugin-name/001-test_spec.lua @ 9
plugin-name plugin setup
spec/01-plugin-name/001-test_spec.lua:25: 
Error: 
/usr/local/share/lua/5.1/kong/cmd/start.lua:62: /usr/local/share/lua/5.1/kong/cmd/start.lua:51: nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong.lua:159: plugin-name plugin is in use but not enabled
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong.lua:159: in function 'init'
    init_by_lua:3: in main chunk

stack traceback:
    [C]: in function 'error'
    /usr/local/share/lua/5.1/kong/cmd/start.lua:62: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
    /bin/kong:7: in function 'file_gen'
    init_worker_by_lua:40: in function <init_worker_by_lua:38>
    [C]: in function 'xpcall'
    init_worker_by_lua:47: in function <init_worker_by_lua:45>

• Kong configuration (kong_tests.conf):

custom_plugins = plugin-name

• Operating System: Arch Linux (4.13.3-1-ARCH) (kong/openresty/luarocks installed from source as instructed in the installation guide for kong 0.11.0)

[thibaultcha]

Hi @lavoiedn,

Hmm yes indeed, I see what is going on here... Environment variables intently override configuration settings, including when such settings are lists like the custom_plugins setting. Hence, by hard-coding an environment variable in the spec/helpers.lua module when starting test Kong instances, we are ignoring any such value configured in the test configuration.

At this point, your only bet without maintaining your own patch is to override this environment variable yourself via the supported env argument of start_kong():

helpers.start_kong {
  custom_plugins = "dummy,cache,rewriter,plugin-name"
}

A less than ideal solution, but believe me, we are aware of the crucial lack of testing framework for Kong plugins development, and will be working on it in the upcoming months/year. Ideally, such a testing framework does not communicate such custom settings via hard-coded environment variables...

In the meantime, as this is the intended behavior and your question stands answered, I will be closing this issue, keeping track of this remark internally for when the time comes we address our plugins features and testing framework.

Thank you!

[iyedbennour]

Slightly off topic but testing custom plugins out of the Kong source tree is complicated. On have to copy the spec and the bin folder in the custom plugin development folder. One solution might be to move all that code in a kong.testing lua package and make available as luarock maybe. Also testing plugins that need to call into other servers as part of handling the request, is not straightforward: my plugin calls an external server to do its job, I can make that external server as an API behind kong (which is convenient as I don't need to write a mock server from scratch) but for this to work I need to add my own custom location blocks inside the server block mock_upstream. Are there any plans to improve custom plugins testing.

[thibaultcha]

@iyedbennour Yes, we are aware. We are too busy today to provide such a solution (in fact, all of the modules you mentioned in your comment are note officially supported for testing, aka, use them at your own risk), but this is coming up our list of priorities. When we come up with a testing framework, we intend on making it simple, yet powerful. Stay tuned! :)