Serverless-python-requirements: Poetry: missing .serverless/requirements.txt

Created on 19 Mar 2020 · 22Comments · Source: UnitedIncome/serverless-python-requirements

Using poetry, normal pyproject.toml file and poetry.lock. Version 5.1.0.
sls package gives me:

Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir

  Error --------------------------------------------------

  Error: ENOENT: no such file or directory, open '/Users/cyber/dev/jb/project/.serverless/requirements.txt'
      at Object.openSync (fs.js:439:3)
      at Object.readFileSync (fs.js:344:35)
      at getRequirements (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:354:6)
      at filterRequirementsFile (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:381:24)
      at generateRequirementsFile (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:67:5)
      at installRequirementsIfNeeded (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:501:3)
      at ServerlessPythonRequirements.installAllRequirements (/Users/cyber/dev/jb/project/node_modules/serverless-python-requirements/lib/pip.js:635:29)
  From previous event:
      at PluginManager.invoke (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at getHooks.reduce.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:525:24)
  From previous event:
      at PluginManager.run (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at variables.populateService.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/Serverless.js:133:33)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:120:23)
  From previous event:
      at Serverless.run (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/lib/Serverless.js:120:74)
      at serverless.init.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:82:30)
      at /Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:136:16
      at /Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:57:14
      at FSReqWrap.oncomplete (fs.js:141:20)
  From previous event:
      at initializeErrorReporter.then (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:82:8)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:120:23)
  From previous event:
      at Object.<anonymous> (/Users/cyber/.nvm/versions/node/v10.14.2/lib/node_modules/serverless/bin/serverless.js:71:4)
      at Module._compile (internal/modules/cjs/loader.js:689:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
      at Module.load (internal/modules/cjs/loader.js:599:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
      at Function.Module._load (internal/modules/cjs/loader.js:530:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
      at startup (internal/bootstrap/node.js:282:19)
      at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              10.14.2
     Framework Version:         1.66.0
     Plugin Version:            3.5.0
     SDK Version:               2.3.0
     Components Version:        2.22.3

bug confirmed good first PR help wanted

Source

revmischa

Most helpful comment

I also got this error while having a pyproject.toml but not using poetry (since that file is used for other things too).

In code, it should detect the lack of required contents of the pyproject.toml file and warn the user

Why not just look for poetry.lock instead of pyproject.toml?

Also, the workaround is really simple and already documented:

custom:
  pythonRequirements:
    usePoetry: false

brettdh on 8 May 2020

👍4

All 22 comments

I am receiving this error for sls deploy and not just for poetry

BwL1289 on 27 Mar 2020

@BwL1289 Could you post your serverless.yaml somewhere, your requirements.txt, along with info on your OS and version of NodeJS and version of this plugin?

@revmischa Could you also post your serverless.yaml, and requirements.txt, along with your version of this plugin?

Make sure you're using the latest version of this plugin, and post a little more info so we can replicate it and debug/fix this.

AndrewFarley on 27 Mar 2020

Thanks @AndrewFarley!
serverless.yml (i have also slimmed this down to the minimum and still receive the same error)

---
service: space-debris-gui

frameworkVersion: ">=1.34 <2.0.0"

plugins:
  - serverless-python-requirements
  - serverless-plugin-split-stacks
  - serverless-offline
custom:
  pythonRequirements:
    pythonBin: ../../.pyenv/versions/3.7.2/bin/python3.7 
    dockerizePip: true
    zip: true
    slim: true
    slimPatterns:
      - "**/*.egg-info*"
    noDeploy: 
      - pytest
      - tox
  splitStacks:
    nestedStackCount: 20
    perFunction: false
    perType: false
    perGroupFunction: true

provider:
  name: aws
  runtime: python3.7
  stage: dev
  region: us-east-1

  environment:
    MONGO_DB_NAME: ${env:MONGO_DB_NAME}
    MONGO_DB_URL: ${env:MONGO_DB_URL}
  iamManagedPolicies:
    - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

package:
  individually: true
  exclude:
    - node_modules/**
    - venv/**
    - package.json
    - package-lock.json
    - requirements.txt

functions:
  - ${file(app/routes/aircrafts/functions.yml)}

requirements.txt

apscheduler
black
boto3
botocore
Click==7.0
dnspython==1.16.0
docker
dynaconf[all]
flake8
h5py
matplotlib
mypy
pandas
pre-commit
pymongo==3.9.0
requests
setup-cfg-fmt
setuptools
sphinx
termcolor
towncrier
tqdm
wheel

sls deploy

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command dev
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir

  Error --------------------------------------------------

  Error: ENOENT: no such file or directory, open '/Users/benjaminleff/.vscode/space-debris/.serverless/requirements.txt'
      at Object.openSync (fs.js:440:3)
      at Object.readFileSync (fs.js:342:35)
      at getRequirements (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:354:6)
      at filterRequirementsFile (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:381:24)
      at generateRequirementsFile (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:67:5)
      at installRequirementsIfNeeded (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:501:3)
      at ServerlessPythonRequirements.installAllRequirements (/Users/benjaminleff/.vscode/space-debris/node_modules/serverless-python-requirements/lib/pip.js:635:29)
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:510:17)
      at Deploy.<anonymous> (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:115:50)
  From previous event:
      at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:100:30)
      at /usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:55
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at /usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:24
  From previous event:
      at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at /usr/local/lib/node_modules/serverless/lib/Serverless.js:133:33
      at processImmediate (internal/timers.js:439:21)
      at process.topLevelDomainCallback (domain.js:131:23)
  From previous event:
      at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:120:74)
      at /usr/local/lib/node_modules/serverless/bin/serverless.js:82:30
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:136:16
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:57:14
      at FSReqCallback.oncomplete (fs.js:146:23)
  From previous event:
      at /usr/local/lib/node_modules/serverless/bin/serverless.js:82:8
      at processImmediate (internal/timers.js:439:21)
      at process.topLevelDomainCallback (domain.js:131:23)
  From previous event:
      at Object.<anonymous> (/usr/local/lib/node_modules/serverless/bin/serverless.js:71:4)
      at Module._compile (internal/modules/cjs/loader.js:956:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
      at Module.load (internal/modules/cjs/loader.js:812:32)
      at Function.Module._load (internal/modules/cjs/loader.js:724:14)
      at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10)
      at internal/main/run_main_module.js:17:11

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.13.0
     Framework Version:         1.67.0
     Plugin Version:            3.6.1
     SDK Version:               2.3.0
     Components Version:        2.22.3

BwL1289 on 27 Mar 2020

@AndrewFarley For reference, here is the stripped down version of serverless.yml

---
service: space-debris-gui

frameworkVersion: ">=1.34 <2.0.0"

plugins:
  - serverless-python-requirements

custom:
#   stage: ${opt:stage, self:provider.stage}
  pythonRequirements:
    dockerizePip: true # enables plugin to install package in docker  ${self:custom.isCI.${env:CI}, self:custom.isCI.false}

provider:
  name: aws
  runtime: python3.7


functions:
  - ${file(app/routes/aircrafts/functions.yml)}

BwL1289 on 27 Mar 2020

@BwL1289 i tried replicating your problem but wasn't able to. What version of this plugin are you using? I think you're using an old version.

Here's my log running what you have here above...

# serverless package
Serverless: Generated requirements from /root/farley/slstest/requirements.txt in /root/farley/slstest/.serverless/requirements.txt...
Serverless: Installing requirements from /root/.cache/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
Serverless: Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
Serverless: Running docker run --rm -v /root/.cache/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc\:/var/task\:z -v /root/.cache/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z lambci/lambda\:build-python3.7 /bin/sh -c 'chown -R 0\\:0 /var/useDownloadCache && python3.7 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 0\\:0 /var/task && chown -R 0\\:0 /var/useDownloadCache'...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
#

with my package.json...

{
  "name": "test",
  "version": "1.0.0",
  "description": "TEST",
  "author": "Farley <[email protected]>",
  "license": "Private",
  "dependencies": {
    "serverless-python-requirements": "github:unitedincome/serverless-python-requirements#master"
  }
}

Confirmed this works on my Mac, and on Linux (ubuntu 18)

AndrewFarley on 28 Mar 2020

@AndrewFarley appreciate the help. I'm using 5.1.0.

Here is my package.json

{
  "name": "space-debris-gui",
  "description": "",
  "version": "0.1.0",
  "dependencies": {},
  "devDependencies": {
    "serverless": "^1.67.0",
    "serverless-offline": "^6.1.2",
    "serverless-plugin-split-stacks": "^1.9.3",
    "serverless-python-requirements": "^5.1.0"
  }
}

I also tried to changing it to reflect what you have: "serverless-python-requirements": "github:unitedincome/serverless-python-requirements#master" but am receiving the same error.

If it helps, here's the relevant info from package-lock.json

"serverless-python-requirements": {
      "version": "5.1.0",
      "resolved": "https://registry.npmjs.org/serverless-python-requirements/-/serverless-python-requirements-5.1.0.tgz",
      "integrity": "sha512-lJhikc6wJsOYFxYNGK763xtQITrtGPHD2KJvs8qaBkkOET4RVk8E0N3d8wAO22k+Tk7N6a6Jonnvsmo3W8vtCA==",
      "dev": true,
      "requires": {
        "@iarna/toml": "^2.2.3",
        "appdirectory": "^0.1.0",
        "bluebird": "^3.0.6",
        "fs-extra": "^7.0.0",
        "glob-all": "^3.1.0",
        "is-wsl": "^2.0.0",
        "jszip": "^3.1.0",
        "lodash.get": "^4.4.2",
        "lodash.set": "^4.3.2",
        "lodash.uniqby": "^4.0.0",
        "lodash.values": "^4.3.0",
        "rimraf": "^3.0.2",
        "sha256-file": "1.0.0",
        "shell-quote": "^1.6.1"
      }
    },

What other information can I provide that may help to debug this? Again, really appreciate the help as this is a blocker.

BwL1289 on 28 Mar 2020

@BwL1289 I tried 5.1.0 and same, no issue here. Could you rm -Rf node_modules && rm -f package-lock.json && npm install then run serverless package and copy/paste me your output from that last command. (with the modified package.json to use the #master as I had above)

AndrewFarley on 28 Mar 2020

@AndrewFarley update - i had a pyproject.toml file in my root dir that i just removed (it is being used for black configs, towncrier configs, and setuptools configs and it seems to be running as expected (still in progress).

I assume that the plugin doesnt support both files simulteanously (perhaps without additional configuration). is there a way to make them work together?

BwL1289 on 28 Mar 2020

@BwL1289 If you have a pyproject.toml file it uses poetry instead of pip. But definitely copy/paste me your log from serverless package

AndrewFarley on 28 Mar 2020

@AndrewFarley thanks so much!

$ serverless package

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command dev
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/benjaminleff/.vscode/space-debris/requirements.txt in /Users/benjaminleff/.vscode/space-debris/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/benjaminleff/Library/Caches/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
Serverless: Using download cache directory /Users/benjaminleff/Library/Caches/serverless-python-requirements/downloadCacheslspyc
Serverless: Running docker run --rm -v /Users/benjaminleff/Library/Caches/serverless-python-requirements/a630932a3ab4fa1bd748aa36ca92d5b597c8d78f2f6d888cc2bebc1de8a43f89_slspyc\:/var/task\:z -v /Users/benjaminleff/Library/Caches/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.7 python3.7 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage

BwL1289 on 28 Mar 2020

👍2

@BwL1289 Looks good eh?

AndrewFarley on 28 Mar 2020

🎉1

@AndrewFarley yes sir! thanks again for the help. Appreciate you guys and all that you do. Will try to find workarounds for the configs specified in pyproject.toml

BwL1289 on 28 Mar 2020

Yeah @BwL1289 I don't have much experience with poetry, but it sounds like both of you had the same problem, so it appears there is probably a valid bug here somewhere. Hopefully someone with poetry experience can follow up on this.

AndrewFarley on 28 Mar 2020

@AndrewFarley Got it. Have a great weekend!

BwL1289 on 28 Mar 2020

👍1

@BwL1289 Can you share your pyproject.toml file? So I can replicate this possibly?

AndrewFarley on 28 Mar 2020

Sure thing!

```
$ cat pyproject.toml

[build-system]
requires = [
"setuptools >= 40.6.2",
"wheel >= 0.30.0",
]
build-backend = "setuptools.build_meta"

[tool.black]

line-length = 120
target-version = ['py36', 'py37', 'py38']
include = '.pyi?$'
exclude = '''

[tool.towncrier]
package = 'Mars'

BwL1289 on 28 Mar 2020

@BwL1289 Thank you I confirm this bug with that file now. I don't know poetry much, but if I have some free time I'll look into debugging why this fails. Thanks!!!

AndrewFarley on 28 Mar 2020

@AndrewFarley My pleasure!!

BwL1289 on 28 Mar 2020

I dug through the code and eventually discovered the problem was my pyproject.toml was missing:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

You need this otherwise you get the above error.
It is unfortunate because I was using dephell to convert to poetry and it didn't add that stanza to the generated pyproject.toml.

revmischa on 28 Mar 2020

👍2 ❤1

@revmischa Thanks for that. I would suggest if someone wants to PR this I would recommend...

This plugin could have a bit more info in README.md in the blob about Poetry
Have an example project which contains a valid/working pyproject.toml
In code, it should detect the lack of required contents of the pyproject.toml file and warn the user

AndrewFarley on 30 Mar 2020

I also got this error while having a pyproject.toml but not using poetry (since that file is used for other things too).

In code, it should detect the lack of required contents of the pyproject.toml file and warn the user

Why not just look for poetry.lock instead of pyproject.toml?

Also, the workaround is really simple and already documented:

custom:
  pythonRequirements:
    usePoetry: false

brettdh on 8 May 2020

👍4

'course, now I'm running into #449. 😑

brettdh on 8 May 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

WSL Unknown path format

davegravy · 5Comments

Remove boto3 and botocore in noDeploy to adhere to AWS best practices.

tmclaugh · 4Comments

Installing required Python packages step hangs

cdimitroulas · 4Comments

New release with the latest changes planned?

IanTayler · 4Comments

layer: retain: true doesn't seem to work

miketheman · 3Comments