Serverless-python-requirements: Error "chown: changing ownership of ‘/var/task’: Operation not permitted" when running on Bitbucket Pipelines

[kenichi-ogawa-1988]

Version: 4.2.4

I got the error below in these days when using with Bitbucket Pipelines.

Command line

serverless deploy --stage prod --profile $AWS_PROFILE --verbose --region $AWS_REGION

Result

Serverless: Generated requirements from /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE/src/client_authorization/requirements.txt in /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE/.serverless/requirements.txt...
Serverless: Copying from /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE/.serverless/requirements.txt into /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE/.serverless/src/client_authorization/requirements.txt ...
Serverless: Installing requirements from /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6

  Error --------------------------------------------------

  Unable to find image 'lambci/lambda:build-python3.6' locally
build-python3.6: Pulling from lambci/lambda
5be106c3813f: Pulling fs layer
e240967675e1: Pulling fs layer
34ad8da3dd72: Pulling fs layer
61ab74a11d97: Pulling fs layer
61ab74a11d97: Waiting
e240967675e1: Verifying Checksum
e240967675e1: Download complete
61ab74a11d97: Verifying Checksum
61ab74a11d97: Download complete
34ad8da3dd72: Verifying Checksum
34ad8da3dd72: Download complete
5be106c3813f: Verifying Checksum
5be106c3813f: Download complete
5be106c3813f: Pull complete
e240967675e1: Pull complete
34ad8da3dd72: Pull complete
61ab74a11d97: Pull complete
Digest: sha256:9a94cf0c2c35137aca45d1ccafe552fd837060b612e317c278c1d1ecaf707e2d
Status: Downloaded newer image for lambci/lambda:build-python3.6
chown: changing ownership of ‘/var/task/requirements.txt’: Operation not permitted
chown: changing ownership of ‘/var/task’: Operation not permitted

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.11.3
     Serverless Version:     1.30.0

On my Mac, it succeeds.
I think chown on some systems is not allowed. (maybe from this tag? https://github.com/UnitedIncome/serverless-python-requirements/blob/v4.2.0/lib/pip.js )
How can I solve this?

Note: 4.1.1 works well with Bitbucket Pipelines.

[duy9403]

I also encounter the same problem.
I figured out that, when BitBucket pipeline is running, your repo is cloned at /opt/atlassian/pipelines/agent/build/, and we're only allowed to work from that location (mkdir, chown ...)

In this file UnitedIncome/serverless-python-requirements/blob/v4.2.0/lib/pip.js, some directories are hardcoded with /var/. I think that's the problem.

[duy9403]

The last time I was able to run serverless deploy is Sep 26th at 12:39 UTC
The time when error happened is Sep 28th at 11:45 UTC
But between these 2 timelines, serverless-python-requirements doesn't release new version, so I don't know why the problem happen.

[dschep]

Hmm. Interesting. Are you specifying a different docker image or a dockerfile? The default docker image should work with those chowns on any system, since they should be getting executed in the docker container.

Maybe the image was updated. I can look into that. Might be transitioning this plugin to it's own docker image instead of that provided by lambci anyway, and definitly will if that's the cause of this problem.

[duy9403]

For me, I don't specify other Docker image or Dockerfile, so lambci/lambda:build-python3.6 is using (for both success and failed build time I mention above).

I checked and there is no new version of lambci/lambda:build-python3.6 released between the timeline I mentioned above, just for your info.

[dschep]

Hmm. Thats odd. Thanks for checking. Is the directory that the source is in in bitbucket pipelines read only and who is it owned by and who are tests running as?

[kenichi-ogawa-1988]

@dschep I tried running small pipelines below.

image:
  name: lambci/lambda:build-python3.6
pipelines:
  branches:
    master:
      - step:
          name: ls
          script:
            - whoami
            - pwd
            - ls -al

And the whole log is

+ umask 000

+ GIT_LFS_SKIP_SMUDGE=1 git clone --branch="master" --depth 50 https://x-token-auth:[email protected]/kenichi-ogawa/bitbucket-pipelines-check-permission.git $BUILD_DIR ; git reset --hard a0a7c4e6c31824bd954c9ac66aba3af23338e327 ; git remote set-url origin [email protected]:kenichi-ogawa/bitbucket-pipelines-check-permission.git
Cloning into '/opt/atlassian/pipelines/agent/build'...
HEAD is now at a0a7c4e Merge branch 'master' of https://bitbucket.org/kenichi-ogawa/bitbucket-pipelines-check-permission

+ chmod 777 $BUILD_DIR

+ whoami
root

+ pwd
/opt/atlassian/pipelines/agent/build

+ ls -al
total 16
drwxrwxrwx. 3 root root 4096 Oct  4 05:08 .
drwxr-xr-x. 8 root root 4096 Oct  4 05:08 ..
-rw-rw-rw-. 1 root root  191 Oct  4 05:08 bitbucket-pipelines.yml
drwxrwxrwx. 8 root root 4096 Oct  4 05:08 .git

Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

[dschep]

Hmm. that looks like it should work. :confounded:

Here's another thought: could you try setting dockerizePip to non-linux so it doesn't use docker? Since the CI is a 64bit linux machine, if your dependencies all have manylinux wheels available, it should work.

[kenichi-ogawa-1988]

I think so too...

I tried setting dockerizePip to false, and it works well.

[kenichi-ogawa-1988]

By the way this problem may disappear with these instructions:

• Use this plugin version 4.2.4
• dockerizePip to true
• dockerImage to lambci/lambda:build-python3.6
• Clear docker cache of my repository from Bitbucket Pipelines

I suggest this problem may cause by 1 or more complicated conditions.
Sorry for confusing you @dschep 🙇 and thank you for supporting!

I guess the directory /var/task will be mounted to /opt/atlassian/pipelines/agent/build/PATH_TO_SERVICE so it can work.

@duy9403 Do you have still problem?

[duy9403]

@dschep Thank you for supporting.
The setting @kenichi-ogawa-1988 listed above is exactly what my setting was before and after the incident (i.e. I didn't change anything). I just rerun the pipeline now and it works.
Still I don't know where is the problem. Anyway, thank you all for collaborating.

.

[duy9403]

@dschep FYI this problem happen again

chown: changing ownership of ‘/var/task/requirements.txt’: Operation not permitted
chown: changing ownership of ‘/var/task’: Operation not permitted

Status: Downloaded newer image for lambci/lambda:build-python3.6
[email protected]
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           10.12.0
     Serverless Version:     1.32.0

[duy9403]

I found a workaround.
As @dschep mentioned,

[Bitibucket] CI is a 64bit linux machine

So in serverless.yml

custom:
  pythonRequirements:
    dockerizePip: non-linux

And bitbucket-pipelines.yml

pipelines:
  branches:
    master:
      - step:
          name: Serverless deploy
          image: node:10-alpine
          caches:
            - cache-node
          script:
            - apk add python3
            - npm install -g serverless
            - npm install serverless-python-requirements
            - serverless deploy -v

definitions:
  caches:
    cache-node: /usr/local/lib/node_modules

I think we can close this issue.