Serverless-python-requirements: pycache conflict

Created on 14 Aug 2017 · 21Comments · Source: UnitedIncome/serverless-python-requirements

I get the following when I try to deploy

Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python3.6...
Serverless: Linking required Python packages...

  Error --------------------------------------------------

  Unable to link dependency '__pycache__' because a file
                             by the same name exists in this service

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Unable to link dependency '__pycache__' because a file
                             by the same name exists in this service
    at fse.readdirSync.map (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless-python-requirements/index.js:162:19)
    at Array.map (native)
    at ServerlessPythonRequirements.linkRequirements (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless-python-requirements/index.js:151:40)
From previous event:
    at Object.before [as hook] (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless-python-requirements/index.js:249:10)
    at BbPromise.reduce (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:236:55)
From previous event:
    at PluginManager.invoke (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:236:22)
    at PluginManager.spawn (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:248:17)
    at Deploy.BbPromise.bind.then.then (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/plugins/deploy/deploy.js:101:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/plugins/deploy/deploy.js:99:10)
    at BbPromise.reduce (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:236:55)
From previous event:
    at PluginManager.invoke (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:236:22)
    at PluginManager.run (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/classes/PluginManager.js:255:17)
    at variables.populateService.then (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/Serverless.js:99:33)
    at runCallback (timers.js:781:20)
    at tryOnImmediate (timers.js:743:5)
    at processImmediate [as _immediateCallback] (timers.js:714:5)
From previous event:
    at Serverless.run (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/lib/Serverless.js:86:74)
    at serverless.init.then (/home/winstonewert/projects/domain-influencer-engine/node_modules/serverless/bin/serverless:39:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.2.1
     Serverless Version:     1.19.0

The situation is caused by both .requirements and my folder containing a __pycache__ folder.

bug

Source

winstonewert

👍10

Most helpful comment

I just tried rm -rf __pycache__ before sls deploy and it fixed the problem, thanx ! That workaround is fine with me but wasn't clear to me when reading the issue comment log. It may be interesting to mention it in the documentation because AFAIU anybody using Python 3.6 is affected by it.

cedricvidal on 29 Sep 2017

👍10

All 21 comments

good catch!

dschep on 14 Aug 2017

Ok, now that I've got the tests running...

My thought is that we should add links to __pycache__ for everything in .requirements __pycache__. Does it make sense?

winstonewert on 14 Aug 2017

I would suggest omiting __pycache__ from the linking process. So just a conditional and a return here.

dschep on 14 Aug 2017

If we want to go down that route, perhaps we should pass '--no-compile' to pip instead to prevent generating the __pycache__ folder in the first case...

winstonewert on 14 Aug 2017

👍2

Oh, I didn't realize that pip is doing the compiling. I like that idea! (since the *.pyc files are probably only useful if dockerizing the install)

dschep on 15 Aug 2017

Why would .pyc not be useful in general?

winstonewert on 16 Aug 2017

Does a pyc made with a Mac OS, or say 32bit linux distro, work on Amazon linux(thus lambda)?

dschep on 16 Aug 2017

Yes, pyc is cross platform (but not compatible between versions of python).
I think we'd hurt performance quite a bit by not including compiled .pyc
files.

On Wed, Aug 16, 2017 at 12:15 PM Daniel Schep notifications@github.com
wrote:

Does a pyc made with a Mac OS, or say 32bit linux distro, work on Amazon
linux(thus lambda)?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/UnitedIncome/serverless-python-requirements/issues/54#issuecomment-322871412,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0Zfe1qwDOOkKryL-mN24ab-sJjt_z2ks5sYz-_gaJpZM4O2sS5
.

winstonewert on 16 Aug 2017

👍2

Ah cool. In that case... I think the easiest might be to copy the original __pycache__ out of the way and then restore it.

dschep on 16 Aug 2017

That would force the user's code to get recompiled instead of the library
code which is probably better but still not ideal.

On Wed, Aug 16, 2017 at 12:29 PM Daniel Schep notifications@github.com
wrote:

Ah cool. In that case... I think the easiest might be to copy the original
__pycache__ out of the way and then restore it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/UnitedIncome/serverless-python-requirements/issues/54#issuecomment-322874736,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0ZffKnW__RjpSCV_LJw3iazd9CDKCIks5sY0MCgaJpZM4O2sS5
.

winstonewert on 16 Aug 2017

True, but the user's code would only be in __pycache__ if they ran it locally anyway, so might want to add a separate step to explicitly generate those (iirc that's possible, but I don't remember how)

dschep on 16 Aug 2017

So we could:

pass --no-compile to pip
link the files
use compileall module to compile everything

But this would leave .pyc files around after everything is unlinked. This
is probably fine for python3, since everything will be inside __pycache__
folder and ignored unless the source is present. But leaving around .pyc
files might cause wierd issues for python2 since it may find them during
the import process.

On Wed, Aug 16, 2017 at 12:41 PM Daniel Schep notifications@github.com
wrote:

True, but the user's code would only be in __pycache__ if they ran it
locally anyway, so might want to add a separate step to explicitly generate
those (iirc that's possible, but I don't remember how)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/UnitedIncome/serverless-python-requirements/issues/54#issuecomment-322877601,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0ZfYpFh_4HJ2vHWYH6KD_BAvH7DQZMks5sY0X6gaJpZM4O2sS5
.

winstonewert on 16 Aug 2017

How about we add --no-compile and add an issue to investigate the usage of compileall to figure out if there's a good way to handle that (and that could maybe go behind a custom option)

dschep on 16 Aug 2017

Possibility 1:

Only libraries distributed as a single python file get added to
.requirements/__pycache__ so if we skip that we only end up with a
handful of modules uncompiled. That's simple and a bit better than just
skipping compiling the external requirements.

Possibility 2:

Pass --no-compile and compileall only for python 3.6, since its only a
problem there and its safe not to cleanup the __pycache__.

Possibility 3:

Rebuild the whole library to generate the zip file itself skiping the
link/unlink dance and override the artifact completely. (Probably the right
approach in theory, but much more work and would break compatibility)

On Wed, Aug 16, 2017 at 1:24 PM, Daniel Schep notifications@github.com
wrote:

How about we add --no-compile and add an issue to investigate the usage
of compileall to figure out if there's a good way to handle that (and
that could maybe go behind a custom option)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/UnitedIncome/serverless-python-requirements/issues/54#issuecomment-322888313,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0ZfQD7TFG_hzTUl8y0c-dUam1HNSUTks5sY1AQgaJpZM4O2sS5
.

winstonewert on 18 Aug 2017

Hi guys, any progress on this issue ?

cedricvidal on 29 Sep 2017

Sorry, nope. We haven't run into this issue our selves (at least not in a way that a quick rm -r __pycache__ doesn't fix) so I haven't had time to prioritize this.

dschep on 29 Sep 2017

cedricvidal on 29 Sep 2017

👍10

oh, my bad! Agreed, that it might be worth documenting until i'ts fixed :+1:

dschep on 29 Sep 2017

Don't be ! Your plugin is awesome

cedricvidal on 29 Sep 2017

❤1

Sorry, did I miss something? Is this issue fixed or why it was closed without a note?