Opencv4nodejs: Support on Lambda

[codyseibert]

I'm trying to use this library on a AWS lambda function, but I'm not sure how to npm install this successfully.

I've been following this tutorial: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/

From the tutorial, I build opencv with the -D BUILD_SHARED_LIBS=NO flag so that it doesn't build shared library files. When I try to do the npm install step:

OPENCV_DIR=~/opencv OPENCV_LIB_DIR=~/opencv/lib npm install --save opencv4nodejs

node-gyp fails.... but if I build opencv using shared libs, it works fine.

Is there something I need to modify to allow this package to be used with static built opencv libs?

[justadudewhohacks]

Hi cody,

Currently opencv4nodejs expects opencv to be build with shared libs. You can not build with shared libs on aws lambda right?

The issue is that in opencv4nodejs/lib/utils.js I hardcoded the library suffix: const libSuffix = process.platform === 'win32' ? '.lib' : '.so';. On linux static libraries have the suffix '.a' I think.

You could try to simply replace '.so' with '.a' in that line and see if it works. Otherwise I am going to try adding support for static libs asap.

[codyseibert]

Awesome, you're the best! I made a PR for now so I can build my package for lambda.

[codyseibert]

hey, so I tried doing what you recommended (you can see in the PR), but now I'm getting a strange error that I haven't seen before.

my lambda code:

const cv = require('opencv4nodejs');

module.exports.handler = function(event, context, cb) {
  const img = cv.imread('image.png');
  console.log('img', img);
  cb(null);
}

the error:

{
  "errorMessage": "/var/task/node_modules/opencv4nodejs/build/Release/opencv4nodejs.node: undefined symbol: ippicviSwapChannels_8u_C4R",
  "errorType": "Error",
  "stackTrace": [
    "Object.Module._extensions..node (module.js:597:18)",
    "Module.load (module.js:487:32)",
    "tryModuleLoad (module.js:446:12)",
    "Function.Module._load (module.js:438:3)",
    "Module.require (module.js:497:17)",
    "require (internal/module.js:20:19)",
    "Object.<anonymous> (/var/task/node_modules/opencv4nodejs/lib/opencv4nodejs.js:7:8)",
    "Module._compile (module.js:570:32)",
    "Object.Module._extensions..js (module.js:579:10)"
  ]
}

is this an error that is caused by it not finding an image or something like that?

[codyseibert]

this might be related https://stackoverflow.com/questions/36961635/node-app-with-c-addon-complains-about-undefined-symbol-at-runtime

[justadudewhohacks]

This seems to make sense.

"undefined symbol: ippicviSwapChannels_8u_C4R" I think this means it can not find a certain binary. Usually once you require a native module it will load all the required binaries.

Looking at the symbol name "ipp" probably is related to some ipp implementation of the opencv SwapChannels, which it can not find. IPPs are the intel performance primitives, which is the stuff that makes opencv so fast.

My guess is that on aws ipp is not supported. Now that you built opencv with static libraries, during the build it probably did not figure out that ipp support is missing, thus you are getting this error on runtime and not during compile time. But I might also be completely wrong with this assumption. :D

Usually when you install opencv via cmake it will detect ipp support and set the WITH_IPP flag. Maybe one has to implicitly disable any IPP or SSE flags.

I will test a static build with opencv4nodejs on linux as soon as I have time, maybe this is just an issue with building opencv statically.

[codyseibert]

Just a note: I have successfully built opencv statically and installed the "opencv" node module to use it in lambda. Then again, this was with opencv 2.4 and not 3.3.1. Let me know if there is anything I can try to do to help; I'd like to get something running in lambda asap with this node module since the other opencv module is lacking.

[justadudewhohacks]

Okay, as I said as soon as I have time(probably this thursday) I will investigate in this problem. The first thing I would try to make sure is, that one can build this with static libraries on a local linux environment, to clarify whether this issue occurs only with AWS lambda or whether it is a general setup issue with opencv3.

[codyseibert]

so I also tried spinning up a centos 7.1 box on vagrant and did the same stuff. I ended up getting the same error:

/code
module.js:597
return process.dlopen(module, path._makeLong(filename));
^

Error: /root/example/node_modules/opencv4nodejs/build/Release/opencv4nodejs.node: undefined symbol: ippicviSwapChannels_8u_C4R

Hope this helps!

[justadudewhohacks]

From a quick google search I found this:
http://answers.opencv.org/question/94302/multiple-opencv-3-ipp-build-issues/

Seems to be a similar issue with statically linking libippicv.a in opencv3.

[codyseibert]

thanks, I'll see if I can follow his solution and keep you posted

[codyseibert]

Some more info:

I tried building opencv with: "-D WITH_IPP=OFF", but I got this error now when trying to require the module:

module.js:597
  return process.dlopen(module, path._makeLong(filename));
                 ^

Error: /root/example/node_modules/opencv4nodejs/build/Release/opencv4nodejs.node: undefined symbol: _ZNK2cv9Algorithm4saveERKNS_6StringE
    at Error (native)
    at Object.Module._extensions..node (module.js:597:18)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/root/example/node_modules/opencv4nodejs/lib/opencv4nodejs.js:7:8)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)

I may just see if I can use the .so files in lambda and not have to build up the static library for now.

[codyseibert]

Alright, so I managed to get this working on lambda by using the dynamic .so files. Feel free to close this issue and my PR. I was under the impression that you need statically built libraries to work on lambda, but I was wrong.

Thanks for the help!

[justadudewhohacks]

Awesome! Did you have to come up with some extra tricks to get it running with dynamic libs on AWS lambda?

[codyseibert]

Not really, I just copied the *3.3.so files into a /lib folder and copied the node_modules/opencv4nodejs folder after building it using the lambda AMI it worked.

[brmk]

Hey @codyseibert

Can you please tell me more about what you copied to which folder? I am struggling with this task for a few days already.
So I let the opencv4nodejs to download and build opencv. I have .so files in node_modules/opencv-build/opencv/build/lib
I already tried to upload the lambda as is and it displayed me next error:

"errorMessage": "libopencv_core.so.3.4: cannot open shared object file: No such file or directory"

Can you help me? :)

[justadudewhohacks]

Maybe this issue helps you. Especially the last comment of @henrydawson, is about how to deal with the .so files:

Copy the required shared libraries from EC2 (in my case libopenblas.so.0 and libgfortran.so.3) into the root folder of your deployment package, and add the following command to the start of the lambda function so lambda knows how to access them. process.env["PATH"] = process.env["PATH"] + ":" + process.env["LAMBDA_TASK_ROOT"]

[brmk]

@justadudewhohacks
yes, thanks. It worked. I moved files to the root and left only .a and .so.3.4 lib files

cp  node_modules/opencv-build/opencv/build/lib/* .
rm -rf node_modules/opencv-build/opencv/build/lib

[jogando]

Hi, I created a project in a EC2 instance, installed opencv4nodejs, zipped it and tried to upload it to Lambda.
The problem is the size of the unzipped folder. I get the following error from Lambda:
"Unzipped size must be smaller than 262144000 bytes"

I checked the node_modules folder and the size of opencv_build was about 1.3gb.
Is there a way to make the module smaller so it can be deployed to lambda?

Thanks

[JonnyBGod]

Can anyone provide some detailed steps to make this work in lambda?
I tried several different ways with no luck. Either too big or lack of files.