Serverless-chrome: Executable permission not preserved when deploying from Windows

[AmarildoK]

Hi,
I deployed the package and for some reason chrome refuses to connect.
I have logged all the steps and this is the output(Added some custom log stepps). Link to pastebin for output.
Do you have any experience with this error? or know how to fix it.

With best regards,
Amarildo

[adieuadieu]

Hi @AmarildoK,

Hm.. I'm not sure what to say. One thing I noticed in your logs is that the function was running for 62267.33 ms which is a long time for nothing to happen. It might help if you could share your code?

[AmarildoK]

Hi @adieuadieu thanks for the response,

I have forked the repo and committed my changes. Haven't changed much expect I get the url directly from event object. I plan on triggering the event trough our own back-end and won't need to extract the variables from queryStringParameters.

I also deploy the code with a windows machine & serverless as described in the readme file, idk if this matters when the code gets build.
Link to the fork

With best regards,
Amarildo

[adieuadieu]

Thanks for sharing the code, @AmarildoK. Indeed, not much changed.

Could you share here your config.js file here, as well as let me know how you invoke the lambda function (since you're not doing it via API Gateway?) What's the event payload that you're invoking the function with?

One thing you can try to help debug is un-comment lines 93 to 105 in chrome.js. This will cause stdout and stderr to get logged to CloudWatch. However, this will cause your function to timeout, because the event loop will never clear, since node will be continuously listening to those events. But, there might be some hints about what's happening with headless Chrome that gets logged.

[AmarildoK]

Hej @adieuadieu,
config file is also bar minimum
export default { logging: true, }

I invoke my functions trough the test button in lambda management and pass a simple object
{"url":"http://www.randomwebsite.com"}

I also un-commented the lines and for some unknown reason stdout and stderr are null. In the logs you can see that by this error
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'on' of null

The full log. I also logged the instance of the chrome variable.

`START RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36 Version: $LATEST
2017-06-15T19:49:23.659Z b9e93f5c-5203-11e7-87f1-377eb8c52f36 CHROME_PATH /var/task/headless-chrome/headless_shell
2017-06-15T19:49:23.684Z b9e93f5c-5203-11e7-87f1-377eb8c52f36
$ ls /tmp
total 8.0K
drwxr-xr-x 21 root root 4.0K Jun 15 16:01 ..
drwx------ 2 sbx_user1070 476 4.0K Jun 15 19:49 .

2017-06-15T19:49:23.690Z b9e93f5c-5203-11e7-87f1-377eb8c52f36
$ ps lx
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 477 1 0 20 0 795036 35256 ep_pol Ssl ? 0:00 /var/lang/bin/node --max-old-space-size=1228 --max-semi-space-size=77 --max-executable-size=154 --expose-gc /var/runtime/node_modules/awslambda/index.js
0 477 12 1 20 0 115080 1612 - R ? 0:00 ps lx

2017-06-15T19:49:23.701Z b9e93f5c-5203-11e7-87f1-377eb8c52f36 no running chrome yet?
2017-06-15T19:49:23.711Z b9e93f5c-5203-11e7-87f1-377eb8c52f36 Is Chrome already running? false
2017-06-15T19:49:23.713Z b9e93f5c-5203-11e7-87f1-377eb8c52f36 instance ChildProcess {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
_closesNeeded: 1,
_closesGot: 0,
connected: false,
signalCode: null,
exitCode: null,
killed: false,
spawnfile: '/bin/sh',
_handle: Process { owner: [Circular], onexit: [Function], pid: 13 },
spawnargs:
[ '/bin/sh',
'-c',
'/var/task/headless-chrome/headless_shell --headless --disable-gpu --window-size=1280x1696 --no-sandbox --user-data-dir=/tmp/user-data --hide-scrollbars --enable-logging --log-level=0 --v=99 --single-process --data-path=/tmp/data-path --ignore-certificate-errors --homedir=/tmp --disk-cache-dir=/tmp/cache-dir --remote-debugging-port=9222' ],
pid: 13,
stdin: null,
stdout: null,
stderr: null,
stdio: [ null, null, null ] }
2017-06-15T19:49:23.724Z b9e93f5c-5203-11e7-87f1-377eb8c52f36 (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'on' of null
END RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36
REPORT RequestId: b9e93f5c-5203-11e7-87f1-377eb8c52f36 Duration: 67.72 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 28 MB`

[adieuadieu]

@AmarildoK Hm.. definitely _something_ isn't right.

Try the following config.js:

import Cdp from 'chrome-remote-interface'

export default {
  logging: true,
  async handler (event) {

    const versionInfo = await Cdp.Version()

    return {
      statusCode: 200,
      body: JSON.stringify({
        versionInfo,
      }),
      headers: {
        'Content-Type': 'application/json',
      },
    }
  },
}

This should print out the Chrome version info.

[AmarildoK]

@adieuadieu I deployed the same code with the addition of your config.js change. From a mac and from a windows machine and for some reason when I deploy from a the mac the script gives me a return
{ "statusCode": 200, "body": "{\"versionInfo\":{\"Browser\":\"HeadlessChrome/60.0.3095.0\",\"Protocol-Version\":\"1.2\",\"User-Agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/60.0.3095.0 Safari/537.36\",\"V8-Version\":\"6.0.184\",\"WebKit-Version\":\"537.36 (@947514553066c623a85712d05c3a01bd1bcbbffc)\"}}", "headers": { "Content-Type": "application/json" } }

and from when I deploy it from a the windows machine it throws a timeout every time I invoke the function.
{ "errorMessage": "2017-06-16T11:40:26.951Z 84567d9d-5288-11e7-ade6-d9a1173a80cc Task timed out after 30.00 seconds" }

Now deploying from a mac machine fixed it for me don't know why really. Both machines have node 7.4.0.

One think I should mention is that the postInstall fails since i have no bash shell installed on my windows machine. I generated the config.js file myself. If I'm not mistaken you just generate a random uuid and make a simple object and write it to the config.js file. Are my assumptions wrong?

Thanks for the help.

[adieuadieu]

@AmarildoK glad to hear you've had some success. To be honest, I've not yet had the chance to test on Windows—but I _do_ want to support that platform.

The random uuid is actually not used anymore—so you don't need it. All you need is a config.js file which specifies your handler function code. (Also, note that: v1.0 of serverless-chrome will remove the config.js and postinstall script entirely.)

Perhaps you could share with me the artefact (zip package) which gets generated by Serverless on Windows: Install serverless globally with npm install serverless -g, then in the project directory run serverless package. This will generate the zip package without actually deploying. You'll find the zip in your-project-directory/.serverless/**something-something**.zip

[AmarildoK]

@adieuadieu you can get the zip on this location. https://transferxl.com/08v1D5jth82yNt

[ilanc]

Thanks @adieuadieu for serverless-chrome. I have the same problem as @AmarildoK - i.e. works when deployed from ubuntu, timesout when deployed from windows. I've done some poking about and not come up with a firm conclusion yet but wanted to share some insights.

TLDR chrome is not spawning correctly inside a lambda when deployed from windows - don't think cloudformation / webpack are the problem. My hunch is that either there's a jszip problem on windows or perhaps its the exec permission not being set correctly. I'm going to fiddle with the later now.

There are 4 processes that happen during deployment:
1. webpack
2. creation of cloudformation .ymls
3. creation of zip
4. anything else serverless might do whilst uploading .serverless/ to AWS

Webpack seems fine:

• my .serverless/serverless-chrome.zip/handler.js is the same on windows and unbuntu - barring hashes and timestamps

Cloudformation .ymls seem fine:

• again no real differences

Zip

• This thread mentions that there may be issues with zipping on windows
• Hmm.. I wonder whether it gets the rwx permissions correct inside the lambda after deployment? This has come up on other forums - can't remember where now... docker image building I think.

Serverless magic
TBH I'dunno what else goes on here but to check I also downloaded the files from the S3 ServerlessDeploymentBucket's (to which serverless uploads files). The only change appears to be whitespace stripping on the cloudformation-template-update-stack.json.

Finally - I tried to get the code working in a local amazonlinux docker image. I pulled amazonlinux, got headless_shell to run inside the shell and changed the src/ code from being "handler-based" to a CLI callable app (so I can run it in node directly). I've managed to get this to work however on my first attempt I forgot to include the spawn code and I got exactly the same ECONNREFUSED error mentioned in the title of this issue (which is how I found it). This squares with @AmarildoK's observations above of chrome not running in the windows deployed lambda.

[adieuadieu]

Hi @ilanc thank you for such a thorough report! I'm away from my computer at the moment but will take a closer look at what you've found tomorrow. However, I quickly wanted to mention: v1.0 is in development in the _develop_ branch and comes with a Serverless plugin: serverless-plugin-chrome And a standalone package @serverless-chrome/lambda. Do you still encounter the issue when using those? You can find more documentation in the develop branch.

On 2017/07/10, at 22:06, Ilan Copelyn [e-mail] wrote:

Thanks @adieuadieu for serverless-chrome. I have the same problem as @AmarildoK - i.e. works when deployed from ubuntu, timesout when deployed from windows. I've done some poking about and not come up with a firm conclusion yet but wanted to share some insights.

TLDR chrome is not spawning correctly inside a lambda when deployed from windows - don't think cloudformation / webpack are the problem. My hunch is that either there's a jszip problem on windows or perhaps its the exec permission not being set correctly. I'm going to fiddle with the later now.

There are 4 processes that happen during deployment:

1. webpack
2. creation of cloudformation .ymls
3. creation of zip
4. anything else serverless might do whilst uploading .serverless/ to AWS

Webpack seems fine:

my .serverless/serverless-chrome.zip/handler.js is the same on windows and unbuntu - barring hashes and timestamps
Cloudformation .ymls seem fine:

again no real differences
Zip

This thread mentions that there may be issues with zipping on windows
Hmm.. I wonder whether it gets the rwx permissions correct inside the lambda after deployment? This has come up on other forums - can't remember where now... docker image building I think.
Serverless magic
TBH I'dunno what else goes on here but to check I also downloaded the files from the S3 ServerlessDeploymentBucket's (to which serverless uploads files). The only change appears to be whitespace stripping on the cloudformation-template-update-stack.json.

Finally - I tried to get the code working in a local amazonlinux docker image. I pulled amazonlinux, got headless_shell to run inside the shell and changed the src/ code from being "handler-based" to a CLI callable app (so I can run it in node directly). I've managed to get this to work however on my first attempt I forgot to include the spawn code and I got exactly the same ECONNREFUSED error mentioned in the title of this issue (which is how I found it). This squares with @AmarildoK's observations above of chrome not running in the windows deployed lambda.

―
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ilanc]

Thanks @adieuadieu I'll have a look. I can confirm that the windows deployed binary doesn't have exec permission set. A quick hack would be to set the exec permission just prior to spawn. I'm sure this has happened elsewhere - there may be a better fix to get the rwx permissions correctly set inside the zip.

[ilanc]

I'll give the _develop_ stuff a whirl tomorrow. I just broke the unix version - by uncommenting the debugging code in chrome.js. - same error as above (i.e. TypeError: Cannot read property 'on' of null). This causes the same behaviour as in this issue - i.e. timeouts rather than CloudWatch log errors. This is a little confusing to the user. I suspect that the try/catch handlers are not functioning as intended in conjunction with various async functions. I've had issues with this in the past too. I'll try find some notes

[adieuadieu]

Looks like the file permission thing on Windows is still an open issue in Serverless. I'll add some code to check for exec permission and chmod if missing.

[ilanc]

Here's my total hack workaround (in chrome.js) - nb:

1. I'm copying the headless_shell into /tmp so that I can change the exec permission
2. you have to copy to /tmp not rename (/tmp is on a different device / volume)
3. I added an extra dependency = node-cp - which you could avoid

var cp = require('cp')

const INSTALLED_CHROME_PATH = path.resolve('./headless-chrome/headless_shell')
const INSTALLED_CHROME_DEP = path.resolve('./headless-chrome/libosmesa.so')
const CHROME_PATH = '/tmp/headless_shell'; // process.env.CHROME_PATH && path.resolve(process.env.CHROME_PATH)
const CHROME_DEP = '/tmp/libosmesa.so'
const HEADLESS_URL = 'http://127.0.0.1:9222'
const PROCESS_STARTUP_TIMEOUT = 1000 * 5

var checkPermission = function (file, mask, cb) {
  fs.stat(file, function (error, stats) {
    if (error) {
      cb(error, false);
    } else {
      cb(null, !!(mask & parseInt((stats.mode & parseInt("777", 8)).toString(8)[0])));
    }
  });
};

function hackExecutableChrome() {
  console.log(INSTALLED_CHROME_PATH, 'exists', fs.existsSync(INSTALLED_CHROME_PATH));
  console.log(INSTALLED_CHROME_DEP, 'exists', fs.existsSync(INSTALLED_CHROME_DEP));
  if (fs.existsSync(CHROME_PATH))
    return;
  try {
    console.log('Move chrome to /tmp: ', INSTALLED_CHROME_PATH);
    cp.sync(INSTALLED_CHROME_PATH, CHROME_PATH);
    cp.sync(INSTALLED_CHROME_DEP, CHROME_DEP);
    // { "errno": -18, "code": "EXDEV", "syscall": "rename", "path": "/var/task/headless-chrome/headless_shell", "dest": "/tmp/headless_shell" }
    // /tmp must be on a different volume: see https://stackoverflow.com/questions/21071303/node-js-exdev-rename-error
    //fs.renameSync(INSTALLED_CHROME_PATH, CHROME_PATH);
    //fs.renameSync(INSTALLED_CHROME_DEP, CHROME_DEP);
    fs.chmodSync(CHROME_PATH, '555');
    fs.chmodSync(CHROME_DEP, '555');
    checkPermission(CHROME_PATH, 4, function(error, stats) { console.log(CHROME_PATH, 'read', stats); });
    checkPermission(CHROME_PATH, 2, function(error, stats) { console.log(CHROME_PATH, 'write', stats); });
    checkPermission(CHROME_PATH, 1, function(error, stats) { console.log(CHROME_PATH, 'exec', stats); });
    checkPermission(CHROME_DEP, 4, function(error, stats) { console.log(CHROME_DEP, 'read', stats); });
    checkPermission(CHROME_DEP, 2, function(error, stats) { console.log(CHROME_DEP, 'write', stats); });
    checkPermission(CHROME_DEP, 1, function(error, stats) { console.log(CHROME_DEP, 'exec', stats); });
  } catch(e) {
    console.log('Error moving chrome to /tmp');
    printJson(e);
  }
}

function printJson(data) {
    console.log(JSON.stringify(data, null, 4));
}

and call it from spawn():

export async function spawn () {
  hackExecutableChrome();

[adieuadieu]

@ilanc thanks for sharing that. Ah you're right. Good point about copying instead of moving: the location of the lambda function code in /var/task is read-only to node.js. Which is super crappy because it means, as you noted, we have to make a copy of the chrome binary to /tmp before we can chmod it. Argh. That sucks.. It's kind of like a punishment of Windows users for deploying from Windows—which is lame.

I guess there are three things to do here:

1. Implement a exec-permission check and if not executable, copy to /tmp and make it executable, then using the new executable when spawning chrome.
2. Figure out how to write a useful unit test for the work-around.
3. Well-document this limitation for Windows users, as it may cost an extra 100ms (that's a guess) on cold-start.

[adieuadieu]

An alternative to this for-Windows hackery may be to encourage Windows users to a Docker image we could provide. Perhaps based on docker-lambda, we create a Docker image that includes Serverless, and build/deploy the deployment package from a Docker container. Would requires that you have Docker installed—which adds some complexity. Does simplify life for people who wanted to include native node modules in their functions, though.

[ilanc]

That's a useful link thanks - I was using https://hub.docker.com/_/amazonlinux/.

Short of adding features to serverless (e.g. in order to support explicit file includes and permissions) and JSZip (i.e. to support explicit permission setting - e.g. in here perhaps) I can't see any other alternative. On the plus such a solution would probably have wide application beyond just serverless-chrome.

I've been a bit sidetracked with a show stopper (for me) which I thought I'd bring to your attention - you can't download files in headless atm :(
https://bugs.chromium.org/p/chromium/issues/detail?id=696481

So for my purposes it looks like openwhisk and non-headless chrome + xvfb is going to be the only solution right now until 696481 is merged.

[ilanc]

Serverless actually user archiver not JSZip - see zipService.js.

You can hack your local node_modules to work byjust setting all files in the zip to have 777 permissions:

# serverless-chrome\node_modules\serverless\lib\plugins\package\lib\zipService.js : 59
            mode: 511 // HACK: 511 === 0777 //stats.mode,

Then make sure you deploy using the hacked serverless:

node_modules/.bin/serverless deploy

[adieuadieu]

@ilanc So it the main problem that fs.statSync doesn't correctly tell us that the file is executable on Windows? Seems like there may be a solution here somewhere that could be PRd on the Serverless repo?

I've engaged with the Serverless community on this issue.

About being able to download files: you could pass the URL you want to download back to Node and do the actual downloading in your Nodejs code. Might even offer you better control, depending on what you want to do with the downloaded file. That might be simpler than openwhisk + headed-chrome + xvfb—but I'm biased in favour of serverless-chrome :-P

[ilanc]

BTW you don't need serverless-chrome.zip\headless-chrome\libosmesa.so with --disable-gpu