Serverless-python-requirements: Deploy fails: ValueError: embedded null byte

[fishcrackers]

When I try to deploy I get an "embedded null byte" error. It appears null bytes are inserted when the serverless-python-requirements plugin copies the requirements.txt file to the .serverless directory.

PS> serverless deploy 
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Trying bindPath C:/Users/me/projects/playground/sls/with_rds (run,--rm,-v,C:/Users/me/Dropbox/projects/playground/sls/with_rds:/test,alpine,ls,/test/serverless.yml)
Serverless: /test/serverless.yml

  Error --------------------------------------------------

  Exception:
Traceback (most recent call last):
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 267, in run
    self.name, wheel_cache
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 314, in populate_requirement_set
    wheel_cache=wheel_cache):
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_file.py", line 89, in parse_requirements
    for req in req_iter:
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_file.py", line 156, in process_line
    isolated=isolated, options=req_options, wheel_cache=wheel_cache
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 206, in from_line
    looks_like_dir = os.path.isdir(p) and (
  File "/var/lang/lib/python3.6/genericpath.py", line 42, in isdir
    st = os.stat(s)
ValueError: embedded null byte
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Exception:
Traceback (most recent call last):
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 267, in run
    self.name, wheel_cache
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 314, in populate_requirement_set
    wheel_cache=wheel_cache):
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_file.py", line 89, in parse_requirements
    for req in req_iter:
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_file.py", line 156, in process_line
    isolated=isolated, options=req_options, wheel_cache=wheel_cache
  File "/var/lang/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 206, in from_line
    looks_like_dir = os.path.isdir(p) and (
  File "/var/lang/lib/python3.6/genericpath.py", line 42, in isdir
    st = os.stat(s)
ValueError: embedded null byte
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

    at installRequirements (C:\Users\me\Dropbox\projects\playground\sls\with_rds\node_modules\serverless-python-requirements\lib\pip.js:146:11)
    at ServerlessPythonRequirements.installAllRequirements (C:\Users\me\Dropbox\projects\playground\sls\with_rds\node_modules\serverless-python-requirements\lib\pip.js:242:5)
From previous event:
    at PluginManager.invoke (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:22)
    at PluginManager.spawn (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\plugins\deploy\deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\plugins\deploy\deploy.js:113:10)
    at BbPromise.reduce (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:55)From previous event:
    at PluginManager.invoke (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:390:22)
    at PluginManager.run (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\classes\PluginManager.js:421:17)
    at variables.populateService.then.then (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\lib\Serverless.js:144:8)
    at serverless.init.then (C:\Users\me\AppData\Roaming\npm\node_modules\serverless\bin\serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     win32
     Node Version:           8.11.3
     Serverless Version:     1.29.2

I'm running the latest version of the serverless-python-requirements plugin.
Docker version 18.06.0-ce, build 0ffa825

Contents of ./.serverless/requirements.txt

Contents of ./requirements.txt

asn1crypto==0.24.0
cffi==1.11.5
cryptography==2.3
idna==2.7
pycparser==2.18
PyMySQL==0.9.2
six==1.11.0

I get the same error on my macos box too. The function works when invoked locally.

I'd appreciate help getting to the bottom of this.

[dschep]

We don't have any Windows dev boxes, but I wasn't able to recreate this on linux or mac. Here's what I ran:

[:wat] $ sls create -t aws-python3
Serverless: Generating boilerplate...
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v1.29.0
 -------'

Serverless: Successfully generated boilerplate for template: "aws-python3"
Serverless: NOTE: Please update the "service" property in serverless.yml with your service name
[:wat] $ sls plugin install -n serverless-python-requirements
Serverless: Creating an empty package.json file in your service directory
Serverless: Installing plugin "serverless-python-requirements@latest" (this might take a few seconds...)
Serverless: Successfully installed "serverless-python-requirements@latest"
[:wat] $ cat > requirements.txt << EOF
asn1crypto==0.24.0
cffi==1.11.5
cryptography==2.3
idna==2.7
pycparser==2.18
PyMySQL==0.9.2
six==1.11.0
EOF
[:wat] $ cat >> serverless.yml << EOF
custom:
  pythonRequirements:
    dockerizePip: true
EOF
[:wat] $ sls package
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...

[fishcrackers]

Thanks. Following your steps I successfully packaged a new project. I copied the cat created requirements.txt to the original project and successfully packaged that project.

[dschep]

Glad you got it working! :+1:

[erickpires]

I had this same problem today. After spending a lot of time trying to understand what was really happening I finally found out that the requirements.txt was been saved with UTF16 encoding (probably due to me running pip freeze > requirements.txt). The solution was to use a text editor to save the file with UTF8 encoding.

Hope this information can help the next person that stumbles into this problem.

[jiannmeng]

I had this same problem today. After spending a lot of time trying to understand what was really happening I finally found out that the requirements.txt was been saved with UTF16 encoding (probably due to me running pip freeze > requirements.txt). The solution was to use a text editor to save the file with UTF8 encoding.

Hope this information can help the next person that stumbles into this problem.

Works perfectly! Thanks.

[mattnakatani]

I had this same problem today. After spending a lot of time trying to understand what was really happening I finally found out that the requirements.txt was been saved with UTF16 encoding (probably due to me running pip freeze > requirements.txt). The solution was to use a text editor to save the file with UTF8 encoding.

Hope this information can help the next person that stumbles into this problem.

When I saved with UTF8 this way from windows 10 powershell in my virtualenv
pip freeze | Out-File -Encoding UTF8 requirements.tx

it saved as UTF8 with BOM and then I would get following error when deploying with AWS codebuild on standard ubuntu 2.0 image:
ERROR: Invalid requirement: '·appdirs==1.4.3'

I put cat requirements.txt into my codebuild project and then realized it was saving with a BOM of · from powershell

I was able to resolve this error by opening the requirements.txt file and using vscode with save as encoding: utf8 (without bom)

[agamm]

I got this error because my requirements.txt had corrupted values, try checking its contents.