Kong: Kong 1.2.2 to Kong 1.3 Migrations up informationals/errors:

Created on 21 Aug 2019  ·  15Comments  ·  Source: Kong/kong

Summary

I am building from src and compiling for various reasons so I may be closing this on my own fairly soon if I didn't do certain steps right, migrations up throws these two things of interest:

Warning:

  | 2019/08/21 07:48:03 [warn] 8#0: *2 [lua] _G write guard:12: __newindex(): writing a global lua variable ('luarocks') which may lead to race conditions between concurrent requests, so prefer the use of 'local' variables
-- | --
  | stack traceback:
  | /usr/local/share/lua/5.1/luarocks/loader.lua:46: in main chunk
  | [C]: at 0x7f14ce518540
  | [C]: in function 'pcall'
  | /usr/bin/kong:3: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>, context: ngx.timer

Error:

Error:
  | /usr/local/share/lua/5.1/kong/pdk/service.lua:107: module 'resty.kong.tls' not found:No LuaRocks module found for resty.kong.tls
  | no field package.preload['resty.kong.tls']
  | no file './resty/kong/tls.lua'
  | no file './resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.ljbc'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls/init.ljbc'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.ljbc'
  | no file '/usr/local/openresty/lualib/resty/kong/tls/init.ljbc'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.lua'
  | no file '/usr/local/openresty/lualib/resty/kong/tls/init.lua'
  | no file './resty/kong/tls.lua'
  | no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/resty/kong/tls.lua'
  | no file '/usr/local/share/lua/5.1/resty/kong/tls.lua'
  | no file '/usr/local/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/luajit/share/lua/5.1/resty/kong/tls.lua'
  | no file '/usr/local/openresty/luajit/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/.luarocks/share/lua/5.1/resty/kong/tls.lua'
  | no file '/.luarocks/share/lua/5.1/resty/kong/tls/init.lua'
  | no file '/usr/local/openresty/site/lualib/resty/kong/tls.so'
  | no file '/usr/local/openresty/lualib/resty/kong/tls.so'
  | no file './resty/kong/tls.so'
  | no file '/usr/local/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/openresty/luajit/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/.luarocks/lib/lua/5.1/resty/kong/tls.so'
  | no file '/usr/local/openresty/site/lualib/resty.so'
  | no file '/usr/local/openresty/lualib/resty.so'
  | no file './resty.so'
  | no file '/usr/local/lib/lua/5.1/resty.so'
  | no file '/usr/local/openresty/luajit/lib/lua/5.1/resty.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/.luarocks/lib/lua/5.1/resty.so'
  | stack traceback:
  | [C]: in function 'require'
  | /usr/local/share/lua/5.1/kong/pdk/service.lua:107: in function 'new'
  | /usr/local/share/lua/5.1/kong/pdk/init.lua:294: in function 'new'
  | /usr/local/share/lua/5.1/kong/global.lua:116: in function 'init_pdk'
  | /usr/local/share/lua/5.1/kong/cmd/migrations.lua:90: in function 'cmd_exec'
  | /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>
  | [C]: in function 'xpcall'
  | /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
  | /usr/bin/kong:7: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>

Some of the Docker file to give you an idea of how I am doing things for the build:

##### ---- From Source with Custom PATCH Approach ----- #####
RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 yaml-dev build-deps make gcc g++ libgcc zlib-dev perl pcre pcre-dev unzip tzdata wrk luarocks git

#CONFIG ENV
ENV LUAROCKS_KONG_VERSION=1.3.0rc2-0
ENV RESTY_VERSION=1.15.8.1
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3

#Pull Openresty and patch
RUN wget https://openresty.org/download/openresty-$RESTY_VERSION.tar.gz \
        && tar -zxvf openresty-$RESTY_VERSION.tar.gz -C /tmp \
        && wget https://github.com/Kong/openresty-patches/archive/master.tar.gz \
        && tar -zxvf master.tar.gz -C /tmp \
        && cd /tmp/openresty-$RESTY_VERSION/bundle \
        && wget https://github.company.com/raw/Repo/company-kong-docker/dev/patches/ngx/nginx-1.15.8-cve_2019_9511_cve_2019_9513_cve_2019_9516.patch -P /tmp/openresty-patches-master/patches/$RESTY_VERSION/ \
    && cd /tmp/openresty-$RESTY_VERSION/bundle && for i in /tmp/openresty-patches-master/patches/$RESTY_VERSION/*.patch; do patch -p1 < $i; done

#DROP in 1.15.8 files manually with Kong and Upstream Keepalive Changes (no patching)
RUN rm /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c \
    && rm /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/ngx_http_upstream.c
COPY patches/ngx/nginx-1.15.8-ngx_http_upstream.c /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/ngx_http_upstream.c
COPY patches/ngx/nginx-1.15.8-ngx_http_upstream_keepalive_module.c /tmp/openresty-$RESTY_VERSION/bundle/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c


#Dowload OpenSSL
RUN cd /tmp \
    && curl -fSL https://www.openssl.org/source/openssl-$RESTY_OPENSSL_VERSION.tar.gz -o openssl-$RESTY_OPENSSL_VERSION.tar.gz \
    && tar xzf openssl-$RESTY_OPENSSL_VERSION.tar.gz 

#Dowload 1.3 Kong dependency https://github.com/Kong/lua-kong-nginx-module
RUN mkdir /tmp/kong \
    && cd /tmp/kong \
    && git clone https://github.com/Kong/lua-kong-nginx-module

#Build OpenResty from source
RUN cd /tmp/openresty-$RESTY_VERSION \
        && ./configure -j2 --with-openssl=/tmp/openssl-$RESTY_OPENSSL_VERSION --add-module=/tmp/kong/lua-kong-nginx-module  --with-pcre-jit   --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
        && make -j2 \
        && make install

#Build Luarocks from source
RUN cd /tmp \
        && wget https://luarocks.org/releases/luarocks-$RESTY_LUAROCKS_VERSION.tar.gz \
        && tar zxpf luarocks-$RESTY_LUAROCKS_VERSION.tar.gz \
        && cd luarocks-$RESTY_LUAROCKS_VERSION \
        &&  ./configure --lua-suffix=jit  --with-lua=/usr/local/openresty/luajit  --with-lua-include=/usr/local/openresty/luajit/include/luajit-2.1 \
        && make build \
        && make install

#Needed for luarocks dependency builds to force https vs ssh
RUN git config --global url.https://github.com/.insteadOf git://github.com/

#Install Kong via luarocks
RUN luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/tmp/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/tmp/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=/tmp/openssl-$RESTY_OPENSSL_VERSION/.openssl

#Get the Kong executable resty script
RUN wget https://raw.githubusercontent.com/Kong/kong/master/bin/kong -P /usr/bin
RUN chmod 777 /usr/bin/kong
RUN mv /usr/local/openresty/bin/resty /usr/bin

######------ END OF FROM SOURCE ------######

Maybe you can spot an issue, or can I just do a luarocks install resty.kong.tls to fix this?

Additional Details & Logs

  • Kong version 1.3
picture jeremyjpj0916

Most helpful comment

I also got this problem. It seems like make install does not always install tls.lua. I needed to manually install that even though I had --add-module (and the module itself compiled just fine). I was sure this was going to bite us.

picture bungle on 22 Aug 2019
👀4 👍2

All 15 comments

Interesting, so the resty.kong.tls is actually the module that comes from here:

#Dowload 1.3 Kong dependency https://github.com/Kong/lua-kong-nginx-module
RUN mkdir /tmp/kong \
    && cd /tmp/kong \
    && git clone https://github.com/Kong/lua-kong-nginx-module

So that must mean my --add-module=/tmp/kong/lua-kong-nginx-module failed:

#Build OpenResty from source
RUN cd /tmp/openresty-$RESTY_VERSION \
        && ./configure -j2 --with-openssl=/tmp/openssl-$RESTY_OPENSSL_VERSION --add-module=/tmp/kong/lua-kong-nginx-module  --with-pcre-jit   --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_v2_module \
        && make -j2 \
        && make install

Looks right to me though.

Also potentially related, watching the build process seems the openresty patchwork fails to patch a few times:
https://github.com/Kong/openresty-patches/issues/53
Seems another person reported similar findings in that repo as well.

jeremyjpj0916 picture jeremyjpj0916 on 21 Aug 2019

Hi @jeremyjpj0916,

Such a coincidence, we just opened https://github.com/Kong/kong/pull/4944 to provide more information for how to setup Kong development environment manually. In short, for compiling the OpenResty suite the recommended method is to use the new openresty-build-tools repository which will automatically include any required module and apply patches for you automatically, so the resulted OpenResty will have all the required libraries already included.

In the case for Kong 1.3, the following command should produce a usable OpenResty build:

./kong-ngx-build -p build \
        --openresty 1.15.8.1 \
        --openssl 1.1.1c \
        --luarocks 3.1.3 \
        --pcre 8.43

Let me know if it works for you!

Datong

dndx picture dndx on 22 Aug 2019
👍1

@dndx appreciate the reply, about 10 minutes ago I was just saying that I should scratch the way we do it and used yalls official repo against my alpine 3.10 base for building from source. I will have to change a few things and include more nginx patchwork I run custom with Kong but that's a good approach for my base install process. Will hijack that shell script and invoke it with the args (and add a few of my own things to it). Will report back if changing over works. I still am scratching my head slightly because what I did do should work and worked for 1.2.1(and added mentioned 1.3 dependencies now). Could very well be a path or permissions problem in containers potentially for the new stuff I changed for 1.3 so far.

jeremyjpj0916 picture jeremyjpj0916 on 22 Aug 2019

I also got this problem. It seems like make install does not always install tls.lua. I needed to manually install that even though I had --add-module (and the module itself compiled just fine). I was sure this was going to bite us.

bungle picture bungle on 22 Aug 2019
👀4 👍2

Will hijack that shell script and invoke it with the args (and add a few of my own things to it). Will report back if changing over works.

@jeremyjpj0916 if you need any tweaks to the script to make kong-ngx-build work properly in your environment with your customizations (e.g. loading extra patches from a third-party path), please let us know — or even better, send a PR adding those as optional flags and we'll be happy to take them!

hishamhm picture hishamhm on 22 Aug 2019

I am likely running into . what Mr. Bungle here has described hah. Regardless I am excited to try yalls build tools script so I will give it a go.

@hishamhm Certainly would be cool to think up a pattern that will enable 3rd party patching into the flow with an optional arg(and how does that work when someone has 3rd party patches that all touch the same source c files in different patches? Say Kong and other groups seperate patch files? Can it? I would think not due to the diffs having no awareness to the other diff patch files so line numbers would be out of sync). I am not as git versed as the Kong squad on the whole using patch git diff files for doing code merging yet 😆, I do it newb old school drop and replace files, so what I have are two files that take what kong 1.3 needs AND what I need and I intend to find a spot in the shell script post patch work to simply drop these in as c file replacements prior to the compile(I spent like 2 hours eyeballing and using diff tools to get them right haha):

Edit - uploading these files again as they were wrong, who knew somewhere from NGINX src -> Openresty -> Kong Patches there were deviations to the nginx files prior to the Kong file diffs, rekt me, but these should officially be the keepalive fix for us
nginx-1.15.8-ngx_http_upstream.c.txt
nginx-1.15.8-ngx_http_upstream_keepalive_module.c.txt

with everything needed:

I believe this much needed upstream ssl keepalive logic I and a few others I have seen need will eventually make its way into OpenResty from what yah fearless leader mentioned awhile back 🚀 so I am pretty content with just file dropping until it makes it in.

Hopefully will have the build tools tested in my dockerfile by EOD and maybe even a 1.3 deployed in our dev 👍 !

jeremyjpj0916 picture jeremyjpj0916 on 22 Aug 2019

@hishamhm @dndx found the first problem with the build script running in a standalone fresh container. Seems to expect a localized /work directory to exist but it does not in our container. Maybe folks that have been running it for awhile already had a /work directory present in their test environment. Continuing to work it into our build process but thought to mention that bug.

Had to do:

RUN git clone https://github.com/kong/openresty-build-tools
RUN chmod -R 777 /openresty-build-tools
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p /usr/local/kong \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
        --pcre $PCRE_VERSION \
    --force
RUN ln -s /usr/local/kong/luarocks/bin/luarocks /usr/bin/luarocks
jeremyjpj0916 picture jeremyjpj0916 on 22 Aug 2019

Getting closer, seems to always fail on luarocks installing of Kong now, as if so many dependencies github gets tired of all the luarocks pulls from our source address lol. Always hangs specifically on pgmoon for me, tried 3 separate builds and it failed same way, time to 1 off and install pgmoon prior to luarocks kong install and see if that helps:

kong-lapis 1.7.0.1-1 depends on pgmoon (not installed)
Installing https://luarocks.org/pgmoon-1.10.0-1.rockspec 

Cloning into 'pgmoon'...
fatal: unable to connect to github.com:
github.com[0: 140.82.113.3]: errno=Operation timed out


Error: Failed installing dependency: https://luarocks.org/kong-lapis-1.7.0.1-1.src.rock  - Failed installing dependency: https://luarocks.org/pgmoon-1.10.0-1.rockspec  - Failed cloning git repository.
Removing intermediate container 5e5744dca170
error: build error: The command '/bin/sh -c luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=/usr/local/kong/openssl' returned a non-zero code: 1

Edit-
Yep nope still failed, will probably dig in later tonight. Dinner time:

Installing https://luarocks.org/pgmoon-1.10.0-1.rockspec
--
  | Missing dependencies for pgmoon 1.10.0-1:
  | lpeg (not installed)
  |  
  | pgmoon 1.10.0-1 depends on lpeg (not installed)
  | Installing https://luarocks.org/lpeg-1.0.2-1.src.rock
  |  
  | In file included from lpcap.h:9,
  | from lpcap.c:9:
  | lptypes.h:38: warning: "luaL_newlib" redefined
  | #define luaL_newlib(L,f) luaL_register(L,"lpeg",f)
  |  
  | In file included from lpcap.c:7:
  | /usr/local/kong/openresty/luajit/include/luajit-2.1/lauxlib.h:125: note: this is the location of the previous definition
  | #define luaL_newlib(L, l) (luaL_newlibtable(L, l), luaL_setfuncs(L, l, 0))
  |  
  | In file included from lpcode.c:12:
  | lptypes.h:38: warning: "luaL_newlib" redefined
  | #define luaL_newlib(L,f) luaL_register(L,"lpeg",f)
  |  
  | In file included from lpcode.c:10:
  | /usr/local/kong/openresty/luajit/include/luajit-2.1/lauxlib.h:125: note: this is the location of the previous definition
  | #define luaL_newlib(L, l) (luaL_newlibtable(L, l), luaL_setfuncs(L, l, 0))
  |  
  | In file included from lptree.c:14:
  | lptypes.h:38: warning: "luaL_newlib" redefined
  | #define luaL_newlib(L,f) luaL_register(L,"lpeg",f)
  |  
  | In file included from lptree.c:12:
  | /usr/local/kong/openresty/luajit/include/luajit-2.1/lauxlib.h:125: note: this is the location of the previous definition
  | #define luaL_newlib(L, l) (luaL_newlibtable(L, l), luaL_setfuncs(L, l, 0))
  |  
  | In file included from lpcap.h:9,
  | from lpvm.c:13:
  | lptypes.h:38: warning: "luaL_newlib" redefined
  | #define luaL_newlib(L,f) luaL_register(L,"lpeg",f)
  |  
  | In file included from lpvm.c:11:
  | /usr/local/kong/openresty/luajit/include/luajit-2.1/lauxlib.h:125: note: this is the location of the previous definition
  | #define luaL_newlib(L, l) (luaL_newlibtable(L, l), luaL_setfuncs(L, l, 0))
  |  
  | gcc -O2 -fPIC -I/usr/local/kong/openresty/luajit/include/luajit-2.1 -c lpcap.c -o lpcap.o
  | gcc -O2 -fPIC -I/usr/local/kong/openresty/luajit/include/luajit-2.1 -c lpcode.c -o lpcode.o
  | gcc -O2 -fPIC -I/usr/local/kong/openresty/luajit/include/luajit-2.1 -c lpprint.c -o lpprint.o
  | gcc -O2 -fPIC -I/usr/local/kong/openresty/luajit/include/luajit-2.1 -c lptree.c -o lptree.o
  | gcc -O2 -fPIC -I/usr/local/kong/openresty/luajit/include/luajit-2.1 -c lpvm.c -o lpvm.o
  | gcc -shared -o lpeg.so lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
  | No existing manifest. Attempting to rebuild...
  | lpeg 1.0.2-1 is now installed in /usr/local/kong/luarocks (license: MIT/X11)
  |  
  | Cloning into 'pgmoon'...
  | fatal: unable to connect to github.com:
  | github.com[0: 192.30.253.113]: errno=Operation timed out
  |  
  |  
  | Error: Failed cloning git repository.
  | Removing intermediate container 2cd9187e16d9
  | error: build error: The command '/bin/sh -c luarocks install pgmoon' returned a non-zero code:
1

Edit, yep forgot we fixed this in our custom build, many corp will block git:// proto, so you have to do this since luarocks wanted to retrieve the packages through git:// :

git config --global url."https://".insteadOf git:// solved the problem.

Will likely publish an end2end solution that should work well for folks here shortly since current documentation feels more like a high level look without linking, most people want full plugNplay that will work in a variety of environments.

jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

@dndx Another issue when specifying a -p /usr/local/kong thats not a singular folder is output like this:

    && ./kong-ngx-build -p /usr/local/kong \

Come build time watch the output of the luarocks 3.1.3 installation:

Initializing project 'luarocks-3.1.3' for Lua 5.1 ...
  | -----------------------------------------------------
  |  
  | Checking your Lua installation ...
  | Adding entries to .gitignore ...
  | Preparing ./.luarocks/ ...
  | Wrote .luarocks/config-5.1.lua
  | Preparing ./lua_modules/ ...
  | ./luarocks already exists. Not overwriting it!
  | Preparing ./lua for version 5.1...
  | cp ./build/config-5.1.lua .luarocks/config-5.1.lua
  | mkdir -p "//usr/local/kong/luarocks/bin"
  | install "build/luarocks" "//usr/local/kong/luarocks/bin/luarocks"
  | mkdir -p "//usr/local/kong/luarocks/bin"
  | install "build/luarocks-admin" "//usr/local/kong/luarocks/bin/luarocks-admin"
  | mkdir -p "//usr/local/kong/luarocks/etc/luarocks"

Notice the double // going on? it ends up being simply /usr/local/kong/luarocks in the end but still an output and attempt // problem. Need something in shell script to handle

-p build
vs
-p /full/path

as I also tried running as -p full/path and that fails for us.

jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

Getting close now, hitting path issues:

ERROR: /usr/bin/kong:9: module 'kong.cmd.init' not found:
--
  | no field package.preload['kong.cmd.init']
  | no file './kong/cmd/init.lua'
  | no file './kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.ljbc'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init/init.ljbc'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.ljbc'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init/init.ljbc'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init/init.lua'
  | no file './kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/luajit-2.1.0-beta3/kong/cmd/init.lua'
  | no file '/usr/local/share/lua/5.1/kong/cmd/init.lua'
  | no file '/usr/local/share/lua/5.1/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/lua/5.1/kong/cmd/init.lua'
  | no file '/usr/local/kong/openresty/luajit/share/lua/5.1/kong/cmd/init/init.lua'
  | no file '/usr/local/kong/openresty/site/lualib/kong/cmd/init.so'
  | no file '/usr/local/kong/openresty/lualib/kong/cmd/init.so'
  | no file './kong/cmd/init.so'
  | no file '/usr/local/lib/lua/5.1/kong/cmd/init.so'
  | no file '/usr/local/kong/openresty/luajit/lib/lua/5.1/kong/cmd/init.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | no file '/usr/local/kong/openresty/site/lualib/kong.so'
  | no file '/usr/local/kong/openresty/lualib/kong.so'
  | no file './kong.so'
  | no file '/usr/local/lib/lua/5.1/kong.so'
  | no file '/usr/local/kong/openresty/luajit/lib/lua/5.1/kong.so'
  | no file '/usr/local/lib/lua/5.1/loadall.so'
  | stack traceback:
  | /usr/bin/kong:9: in function 'file_gen'
  | init_worker_by_lua:48: in function <init_worker_by_lua:46>
  | [C]: in function 'xpcall'
  | init_worker_by_lua:55: in function <init_worker_by_lua:53>

Should hopefully fix this up by later today.

jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

Figured out a way to fix it, LUA_PATH helps out with it, random googling of old kong issue solved that:

https://github.com/Kong/kong/issues/3688#issuecomment-446855185

And a little bit of this in the bash script and we should be golden, I figure bundle is where the files post patch live, so probably didn't need to drop in both spots there but was unsure 👍 -

      # ADDING OPTUM CUSTOM KEEPALIVE PATCHES
      cp -f /patches/nginx-1.15.8-ngx_http_upstream.c /openresty-build-tools/work/openresty-1.15.8.1/build/nginx-1.15.8/src/http/ngx_http_upstream.c
      cp -f /patches/nginx-1.15.8-ngx_http_upstream.c /openresty-build-tools/work/openresty-1.15.8.1/bundle/nginx-1.15.8/src/http/ngx_http_upstream.c
      cp -f /patches/nginx-1.15.8-ngx_http_upstream_keepalive_module.c /openresty-build-tools/work/openresty-1.15.8.1/build/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c
      cp -f /patches/nginx-1.15.8-ngx_http_upstream_keepalive_module.c /openresty-build-tools/work/openresty-1.15.8.1/bundle/nginx-1.15.8/src/http/modules/ngx_http_upstream_keepalive_module.c

      make -j$NPROC
      make -j$NPROC install DESTDIR=${OPENRESTY_DESTDIR}
jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

Managed to run migrations up from 1.2 to 1.3 successfully(seemingly), need to run a finish once I get 1.3 nodes setup.

jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

Fixed as far as original issue at this point(as long as migrations finish goes smoothly heh), kong tls module somehow does start making it in when using yalls shell script logic.

jeremyjpj0916 picture jeremyjpj0916 on 23 Aug 2019

Figured I would add some of the logic we used, if anyone else public net runs into problems building from source, hope maybe one of these bits will help:

######------ KONG BUILD TOOLS APPROACH ------######

ENV LUAROCKS_KONG_VERSION=1.3.0-0
ENV RESTY_VERSION=1.15.8.1
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3
ENV PCRE_VERSION=8.43
ENV KONG_PATH=/usr/local/kong

RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 \
    yaml-dev build-deps bash bash-completion automake bash-doc make gcc g++ libgcc zlib-dev curl perl pcre pcre-dev unzip tzdata wrk luarocks git

RUN git clone https://github.com/kong/openresty-build-tools
RUN chmod -R 777 /openresty-build-tools
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
        --pcre $PCRE_VERSION \
    --force

#Symlink to luarocks executable and nginx executable
RUN ln -s $KONG_PATH/luarocks/bin/luarocks /usr/bin/luarocks
RUN ln -s $KONG_PATH/openresty/nginx/sbin/nginx /usr/bin/nginx

#Needed for luarocks dependency builds to force https vs ssh as most corp block git:// proto
RUN git config --global url.https://github.com/.insteadOf git://github.com/

#Install Kong via luarocks
RUN luarocks install kong $LUAROCKS_KONG_VERSION OPENSSL_DIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION OPENSSL_LIBDIR=/openresty-build-tools/work/openssl-$RESTY_OPENSSL_VERSION CRYPTO_DIR=$KONG_PATH/openssl

#Get the Kong executable resty script
RUN wget https://raw.githubusercontent.com/Kong/kong/master/bin/kong -P /usr/bin
RUN chmod 777 /usr/bin/kong
RUN mv $KONG_PATH/openresty/bin/resty /usr/bin

######------ END KONG BUILD TOOLS APPROACH ------######

Another Important point we got stuck on is to set the lua_path, build from source docs lacked info on that env:

export LUA_PATH="${luarocks-prefix}/share/lua/5.1/?.lua;;${luarocks-prefix}/share/lua/5.1/?/init.lua;"
jeremyjpj0916 picture jeremyjpj0916 on 24 Aug 2019

Had to change something recently too, thought to updatte thread here but pcre having issues with the 1.15.8.1 of openresty causing the use of ngx/re problems finding some ffi binding. Looks like thibaultcha added a way to dynamically link so I change modified the build to this:

ENV LUAROCKS_KONG_VERSION=1.3.0-0
ENV RESTY_VERSION=1.15.8.1
ENV RESTY_OPENSSL_VERSION=1.1.1c
ENV RESTY_LUAROCKS_VERSION=3.1.3
#ENV PCRE_VERSION=8.43
ENV KONG_PATH=/usr/local/kong

RUN apk add --no-cache --virtual build-deps wget tar  ca-certificates \
        && apk add  --update --no-cache readline-dev outils-md5 linux-headers bsd-compat-headers m4 \
    yaml-dev build-deps patch bash bash-completion automake bash-doc make gcc g++ libgcc zlib-dev curl perl pcre pcre-dev unzip tzdata wrk luarocks git

RUN git clone https://github.com/kong/openresty-build-tools
RUN chmod -R 777 /openresty-build-tools
RUN cd openresty-build-tools \
    && mkdir work \
    && ./kong-ngx-build -p $KONG_PATH \
        --openresty $RESTY_VERSION \
        --openssl $RESTY_OPENSSL_VERSION \
        --luarocks $RESTY_LUAROCKS_VERSION \
        #--pcre $PCRE_VERSION \
    --force

Another thing that was interesting was the latest alpine 3.10 i had to install patch explicitly because the patch alpine came with wasn't fully featured and one of the CVE patches failed due to unknown operator "forward" .

jeremyjpj0916 picture jeremyjpj0916 on 29 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

DavidTPate picture DavidTPate  ·  35Comments
noamelf picture noamelf  ·  36Comments
sonicaghi picture sonicaghi  ·  39Comments
SunshineYang picture SunshineYang  ·  39Comments
timusketeers picture timusketeers  ·  36Comments