Serverless-python-requirements: Invalid Mode /test when deploying using 4.2.5

[adanot]

After running sls deploy invalid mode error appears when using 4.2.5, works fine on 4.2.4.

I think it might be due to #187 adding quotes to the bindpath.

Serverless: Trying bindPath D:/Workspace/analysis-api/.serverless/requirements (run,--rm,-v,D:/Workspace/analysis-api/.serverless/requirements:/test,alpine,ls,/test/requirements.txt)
Serverless: /test/requirements.txt

  Error --------------------------------------------------

  docker: Error response from daemon: invalid mode: /test.
See 'docker run --help'.

Environment Info

     OS: win32(10 pro)
     Node Version: 9.5.0
     Serverless Version: 1.32.0
     Docker Version :  18.06.1-ce-win73 (19507)

[RED1869]

Same here!

[dschep]

arg.. why does windows have to be so weird :disappointed:

Any thoughts @rob-deans @kichik @PatrickBuTaxdoo

[kichik]

Is that the complete log with SLS_DEBUG=*? It should have tried a few more options.

If I had to guess, I would say the D: drive is not shared with Docker.

[adanot]

@kichik That is not the complete log but only where it failed. Everything else ran fine.

D: is definitely shared.

[kichik]

It should print all of the different Docker commands it's trying. It seems like the right Docker command is missing for you because they all fail. Either that, or the command is somehow malformed. It would be helpful if can check which variation is the closest to working and what's exactly missing in it.

I assume, based on the error message, that something like the following should work:

docker run --rm -v /D/Workspace/analysis-api/.serverless/requirements:/test alpine ls /test/requirements.txt

Also, please include more details about your system. Docker for Windows or Docker Toolbox? Any other details could help.

[T-Tmnr]

Same and debugged.
At ver 4.2.5, project directory becomes encorted by double quoat. (at pip.js)
at Ver.4.2.4 -> projectdir/.serverless/requirements
at Ver.4.2.5 -> "projectdir/.serverless/requirements"
It got the bad effect by child_process.spawnSync quoting and escaping process.
Call child_process.spawnSync with option windowsVerbatimArguments = true, this issue does not reproduced.

After fixing it, next calling 'python.exe' on Docker container.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"python.exe\": executable file not found in $PATH": unknown.

I fixed issues on my branch 'fix-issue-274', and it goes well using the serverless command on my pc.

[gbrcmg]

Same here.

Serverless: Docker Image: lambci/lambda:build-python3.6

Error --------------------------------------------------

docker: Error response from daemon: invalid mode: /test.
See 'docker run --help'.

[dschep]

@T-Tmnr @gbrcmg @adanot How are you running sls & docker? Are you using WSL or not. Are you using Docker4Windows, DockerToolbox Or something inside WSL?

[gbrcmg]

@T-Tmnr @gbrcmg @adanot How are you running sls & docker? Are you using WSL or not. Are you using Docker4Windows, DockerToolbox Or something inside WSL?

I'm using Docker for Windows.
Downgrad to 4.2.4 worked for me.

[T-Tmnr]

@T-Tmnr @gbrcmg @adanot How are you running sls & docker? Are you using WSL or not. Are you using Docker4Windows, DockerToolbox Or something inside WSL?

I'm using Docker for Windows too.

[Oldsouldier]

Seeing this as well. Here's a full log:

```$ serverless deploy
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from C:\project_dir\requirements.txt in C:\project_dir.serverless\requirements.txt...
Serverless: Installing requirements from C:\project_dir.serverless\requirements\requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Trying bindPath C:/project_dir/.serverless/requirements (run,--rm,-v,C:/project_dir/.serverless/requirements:/test,alpine,ls,/test/requirements.txt)
Serverless: /test/requirements.txt

Error --------------------------------------------------

docker: Error response from daemon: invalid mode: /test.
See 'docker run --help'.

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Stack Trace --------------------------------------------

Error: docker: Error response from daemon: invalid mode: /test.
See 'docker run --help'.

at dockerCommand (C:\project_dir\node_modules\serverless-python-requirements\lib\docker.js:20:11)
at getDockerUid (C:\project_dir\node_modules\serverless-python-requirements\lib\docker.js:162:14)
at installRequirements (C:\project_dir\node_modules\serverless-python-requirements\lib\pip.js:231:42)
at installRequirementsIfNeeded (C:\project_dir\node_modules\serverless-python-requirements\lib\pip.js:451:3)
at ServerlessPythonRequirements.installAllRequirements (C:\project_dir\node_modules\serverless-python-requirements\lib\pip.js:530:29)

From previous event:
at PluginManager.invoke (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:22)
at PluginManager.spawn (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:408:17)
at Deploy.BbPromise.bind.then.then (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\plugins\deploy\deploy.js:123:50)
From previous event:
at Object.before:deploy:deploy [as hook] (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\plugins\deploy\deploy.js:113:10)
at BbPromise.reduce (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:55)
From previous event:
at PluginManager.invoke (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:22)
at PluginManager.run (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:421:17)
at variables.populateService.then.then (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:157:33)
at runCallback (timers.js:810:20)
at tryOnImmediate (timers.js:768:5)
at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
at Serverless.run (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:144:8)
at serverless.init.then (C:\Users\dev\AppData\Roaming\npm\node_modules\serverless\bin\serverless:43:50)
at

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information -----------------------------
OS: win32
Node Version: 8.11.3
Serverless Version: 1.29.2
```

Docker: 18.06.1-ce-win73
Win10 64: 1803

Same result in GitBash and Cmd.

[cloudnin9]

Hi
I am noob to python & docker so I hope I am not barking up the wrong tree but here goes, after I dug into the code, I came to the same conclusion as @adanot he mentioned #187, I think it is the double quotes in bindPath variable for requirements folder when mounting to image so this is failing. I got here after using a few of console.logs to see what commands where being generated.

docker run --rm -v "C:/Users/nylth/code/python/hellow_flask/.serverless/requirements":/test alpine stat -c %u /bin/sh

The docker process exits with code 125 but if I removed the ", it works and docker process returns 0 as expected

docker run --rm -v C:/Users/nylth/code/python/hellow_flask/.serverless/requirements:/test alpine stat -c %u /bin/sh

I believe the problem lies in lib/pip.js at line 276

dockerPathForWin(options, path)

[dschep]

Thanks for digging in @naingyelin! What if the path has a space in it? IE rename hellow_flask to hellow flask.

[cloudnin9]

@dschep I can confirm this command works for path with spaces, if you enclose the whole mount path in " including the :/test, just tested it :)

docker run --rm -v "C:/Users/nylth/code/python/hellow flask/.serverless/requirements:/test" alpine stat -c %u /bin/sh

[dschep]

Ah. Thanks for that @naingyelin. That gives us a path forward!

[T-Tmnr]

There is maybe some differences between command-line and spawnSync on Windows.
I explored spawnSync behavior on windows by below code.

testcode.js

const { spawnSync } = require('child_process');
const options = ['arg1', '"arg2"', 'arg 3', '"arg 4"'];
// case1 like docker.js:13
spawnSync('reciever.bat', options, { });
// case2 
spawnSync('reciever.bat', options, { windowsVerbatimArguments:true });
// case3 likes pip.js:249
spawnSync('reciever.bat', options, { shell:true });

reciever.bat

echo ALL=%* >> result.txt
echo $1=%1 >> result.txt
echo $2=%2 >> result.txt
echo $3=%3 >> result.txt
echo $4=%4 >> result.txt
echo $5=%5 >> result.txt

Result

rem case1
ALL=arg1 "\"arg2\"" "arg 3" "\"arg 4\"" 
$1=arg1 
$2="\"arg2\"" 
$3="arg 3" 
$4="\"arg 
$5=4\"" 

rem case2
ALL=arg1 "arg2" arg 3 "arg 4" 
$1=arg1 
$2="arg2" 
$3=arg 
$4=3 
$5="arg 4"

rem case3
ALL=arg1 "arg2" arg 3 "arg 4" 
$1=arg1 
$2="arg2" 
$3=arg 
$4=3 
$5="arg 4" 

Case1: double-quate in arg2 and arg4 are escaped and corrupted. But arg3 is escaped well.
Case2 and case3: arg3 is corrupted.

I tryed to fix this issue on my branch. but can not test on Mac.
https://github.com/T-Tmnr/serverless-python-requirements/tree/fix-issue-274

[dschep]

Awesome @T-Tmnr, could a window user test his change? Install it with npm i T-Tmnr/serverless-python-requirements#fix-issue-274

[cloudnin9]

@dschep I have just tested the build and it works, I am now able to deploy my flask app.
I did encounter issue #226 but just had to regenerate the requirements.txt in bash using wsl

Windows 10 Pro v1803 Build 17134.345
node v8.11.4
Serverless 1.32.0
python 3.7.0
Docker version 18.06.1-ce, build e68fc7a

nvm v1.1.7
npm 6.4.1
pip 18.0

[dschep]

That's great news @naingyelin!

@T-Tmnr, could you open a PR for your change? Also, could you make the change from python.exe to python only for when dockerizePip is set?

[Oldsouldier]

FWIW, T-Tmnr's change worked for me as well.

[T-Tmnr]

@dschep, I opened PR with modifying about dockerizePip.

[T-Tmnr]

My information was "ONLY" bat file.
https://github.com/UnitedIncome/serverless-python-requirements/issues/274#issuecomment-438748776

I try to explore behavior with other command-line application.
Another result is there. Windows bat-file has a darkness...

I make simple application "outarg.exe".

#include <fstream>
#include <iostream>
#include <string>

int main(int argc, char* argv[])
{
    std::ofstream ofs("result.txt", std::ofstream::app);
    for (int i = 0; i < argc; i++) {
        ofs << "argv[" << i << "]=" << argv[i] << "\n";
    }
}

And check spawnSync behavior.

---

With options ['-v', 'host path:guest_path']

When default.

argv[0]=outarg.exe
argv[1]=case1
argv[2]=-v
argv[3]=host path:guest_path

It does not corrupt!
Docker can recognize option.

When windowsVerbatimArguments or shell option enabled.

argv[0]=outarg.exe
argv[1]=case2
argv[2]=-v
argv[3]=host
argv[4]=path:guest_path

Parameters are corrupted.

---

With options ['-v', '"host path":guest_path']

When default.

argv[0]=outarg.exe
argv[1]=case1
argv[2]=-v
argv[3]="host path":guest_path

It does not corrupt!
But in this case, docker cannot recognize option correctly.

When windowsVerbatimArguments or shell option enabled.

argv[0]=outarg.exe
argv[1]=case2
argv[2]=-v
argv[3]=host path:guest_path

It does not corrupt and does not have double-quate!
Docker can recognize option.

---

With options ['-v', '"host path:guest_path"']

When default.

argv[0]=outarg.exe
argv[1]=case1
argv[2]=-v
argv[3]="host path:guest_path"

This case is not in current code.
Modify docker.js to do this quoting, but docker cannot recognize option correctly.

When windowsVerbatimArguments or shell option enabled.

argv[0]=outarg.exe
argv[1]=case2
argv[2]=-v
argv[3]=host path:guest_path

It does not corrupt and does not have double-quate too!
Docker can recognize option.

---

In docker.js, calling spawnSync with default option.
In pip.js, calling spawnSync with shell option.
If bindPath has space-char, docker.js wants un-quoted bindPath but pip.js wants quoted bindPath.

My solution on fix-issue-274,
https://github.com/T-Tmnr/serverless-python-requirements/tree/fix-issue-274
bindPath keeps quoted and modify spawnSync option on docker.js.

Another solution on fix-issue-274-another,
https://github.com/T-Tmnr/serverless-python-requirements/tree/fix-issue-274-another
bindPath is not quoted and modify docker option to quote on pip.js.

Both solutions fix this issue.

[dschep]

Thanks for the thorough report @T-Tmnr!

[moneerrifai]

@T-Tmnr @gbrcmg @adanot How are you running sls & docker? Are you using WSL or not. Are you using Docker4Windows, DockerToolbox Or something inside WSL?

I'm using Docker for Windows.
Downgrad to 4.2.4 worked for me.

@gbrcmg what is it that you downgraded to 4.2.4?

[moneerrifai]

@T-Tmnr @gbrcmg @adanot How are you running sls & docker? Are you using WSL or not. Are you using Docker4Windows, DockerToolbox Or something inside WSL?

I'm using Docker for Windows.
Downgrad to 4.2.4 worked for me.

@gbrcmg what is it that you downgraded to 4.2.4?

Nevermind. I realize now that you were referring to the plugin version itself.

[derekhuizhang]

This issue is still not fixed for me on Windows 10. Like the others above, I have to downgrade to 4.2.4 on every single serverless project that I create.

[dnnsthnnr]

@derekhuizhang Since the changes were merged no newer version was released. So the only solution atm is to downgrade.