Serverless-python-requirements: Poetry path dependencies and dockerizePip: true

[MarwanDebbiche]

Hi, I am having issue deploying a service with some poetry path dependencies and the option dockerizePip: true.

Here is my project structure:

├── lambdas
│   ├── serverless.yaml
│   ├── handler.py
│   ├── pyproject.toml
│   └── ...
├── project2  # Some other project using common, not using serverless at all
│   └── ...
└── common  # Common libraries
    └── utils
        ├── pyproject.toml
        └── utils
            ├── __init__.py
            └── ...

lambdas/pyproject.toml:

[tool.poetry]
name = "lambdas"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.1.2"
utils = {path = "../common/utils"}

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Note the utils path dependency.

As my project depends on non pure-python libraries (numpy, which is a pandas dependency), I need to use the dockerizePip: true option.

Here is my serverless.yaml:

service: project-lambdas
provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    description: Hello function
    handler: handler.hello

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePoetry: true
    dockerizePip: true

But as the dependency installation runs inside of a docker container that does not know about the common directory, it fails with this error:

  Error: STDOUT: 

  STDERR: ERROR: Invalid requirement: '../common/utils' (from line 1 of /var/task/requirements.txt)
  Hint: It looks like a path. File '../common/utils' does not exist.

I tried mounting the common directory inside of the container with dockerRunCmdExtraArgs: ['-v', '${env:PROJECT_ROOT_DIR}/common:/var/common'], but then I get the following error:

  STDERR: ERROR: Exception:
  Traceback (most recent call last):
    File "/var/lang/lib/python3.8/shutil.py", line 788, in move
      os.rename(src, real_dst)
  OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-ifpetc_3/lib/python/pytz' -> '/var/task/pytz'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 470, in run
      self._handle_target_dir(
    File "/var/lang/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 527, in _handle_target_dir
      shutil.move(
    File "/var/lang/lib/python3.8/shutil.py", line 798, in move
      copytree(src, real_dst, copy_function=copy_function,
    File "/var/lang/lib/python3.8/shutil.py", line 554, in copytree
      return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
    File "/var/lang/lib/python3.8/shutil.py", line 510, in _copytree
      raise Error(errors)
  shutil.Error: [('/tmp/pip-target-ifpetc_3/lib/python/pytz/zoneinfo/America/Argentina/Ushuaia', '/var/task/pytz/zoneinfo/America/Argentina/Ushuaia', '[Errno 5] Input/output error')]

When I set dockerizePip to false, the deployement doesn't fail, but of course the lambda invokation does with the error Importing the numpy C-extensions failed.

Is there a way to make poetry path dependencies work with dockerizePip: true?

Thank you for your help

[daampie]

Having the same issue with local libs although I don't think it's related to poetry. Issue seems to be that the common lib is not mounted properly to the container.

Project structure:

├── README.md
├── services
    ├── common
        ├── poetry.lock
        ├── pyproject.toml
        ├── src
            ├── common
                ├── __init__.py
                ├── common.py
    ├── service1
        ├── pyproject.toml
        ├── handler.py
        ├── serverless.yml
    └── ...

services/service1/pyproject.toml

[tool.poetry]
name = "service1"
version = "0.1.0"
description = ""
authors = ["None"]

[tool.poetry.dependencies]
python = "^3.7"
common = { path = "../common/" }

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

services/service1/serverless.yml

service: project

provider:
  name: aws
  runtime: python3.7

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true
    usePoetry: true

functions:
  project:
    handler: handler.main
    description: handler

Environment information

Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              12.18.3
     Framework Version:         1.51.0
     Plugin Version:            1.3.11
     SDK Version:               2.3.2

services/service1/package.lock

{
  "name": "project",
  "description": "",
  "version": "0.1.0",
  "dependencies": {},
  "devDependencies": {
    "serverless-python-requirements": "^5.1.0"
  }
}

Then when running sls package or sls deploy the following error pops up:

Error --------------------------------------------------

  Error: STDOUT: 

  STDERR: ERROR: Invalid requirement: '../common' (from line 1 of /var/task/requirements.txt)
  Hint: It looks like a path. File '../common' does not exist.
  WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
  You should consider upgrading via the '/var/lang/bin/python3.7 -m pip install --upgrade pip' command.

When I set dockerizePip: false all works fine, but then you run into the issue as described by @MarwanDebbiche

[john-feusi-neuro]

I'm getting the OSError: [Errno 18] Invalid cross-device link error as well. I have usePoetry: false in my serverless file. The problem seems to be intermittent for me.

[john-feusi-neuro]

I turned off "Use gRPC FUSE for file sharing" and I think it fixed the problem for me.

[joelash]

Can confirm that @john-feusi-neuro's fix worked for me, and I'm not using poetry

[gcampionpae]

This trick was a lifesaver, thanks @john-feusi-neuro for finding this, it's been driving me bananas.

[HerrSchroedinger]

DiTo. Here the Tipp from @john-feusi-neuro helped!

[mohoromitch]

Thank you @john-feusi-neuro 💯!
This fixed my issue which was only running a pip install within a Docker container.