Pip-tools: Pass additional arguments to pip

[31z4]

It would be nice to have an ability to pass some non harmful additional arguments to pip during pip-sync. E. g. --no-cache-dir or --cache-dir <dir>.

[estan]

I would like to be able to pass -I to pip when doing pip-sync. The reason is I'm unfortunately required to use --system-site-packages on my virtualenv, to get access to the distro installed PyQt (which is not on PyPI yet, so not in requirements.txt), but I still want pip-sync to install as much as possible into the virtualenv (and not skip stuff that is already satisfied by the system-wide site-packages).

[estan]

Just want to check, is there still no way to do this?

[tkwon]

+1 on adding options like --no-cache-dir

[merwok]

A nice interface for this could be the command-line end marker: pip-sync requirements.txt -- --pip-option-1 --pip-option-2=no

[scythargon]

+1

[atugushev]

That's a great idea! How to handle -- using click API? I skimmed docs and code and didn't find the solution.

UPDATE: Found it https://click.palletsprojects.com/en/7.x/arguments/#option-like-arguments

[aorumbayev]

@atugushev any updates on this issue ?

[atugushev]

Hello @aorumbayev,

Unfortunately, there are no updates. It's still not clear how to pass the additional arguments.

[atugushev]

@merwok

A nice interface for this could be the command-line end marker: pip-sync requirements.txt -- --pip-option-1 --pip-option-2=no

What if we need to process files containing double-dash? For example:

$ touch -- --requirements--.in
$ pip-compile -- --requirements--.in

[merwok]

@atugushev Not sure I get your question. Recognizing a single token with the exact value of -- shouldn’t interfere with other params containing two dashes.

[atugushev]

@merwok

Not sure I get your question. Recognizing a single token with the exact value of -- shouldn’t interfere with other params containing two dashes.

I mean how to distinguish filenames and options after --? For instance:

$ pip-compile -- --requirements--.in --no-cache-dir

[merwok]

That wouldn’t be valid. -- means «end of processing, so all parameters for pip-compile mist appear before it:

$ pip-compile --some-opt file1.in --more-opt file2--weird.in -- --pip-option1 --pip-option2 arg --pip-option3

[atugushev]

@tkwon

That wouldn’t be valid. -- means «end of processing, so all parameters for pip-compile mist appear before it:

AFAIK, the -- marks that all further parameters are accepted as arguments. This is how it currently works:

$ echo six > --requirements.in

$ ls 
--requirements.in

$ pip-compile -- --requirements.in
six==1.13.0

$ ls
--requirements.in  --requirements.txt

[merwok]

Yes, this would involve changing the meaning of the token.
Are there really many people who decide to use filenames starting with two dashes? That’s asking for trouble IMO. But even so, there is still a way to disambiguate: ./--filename is a path!

[atugushev]

@merwok

Are there really many people who decide to use filenames starting with two dashes?

I think it's close to zero. Anyways, I'm in favor to change the meaning of the marker, but, honestly, have no idea how to do it using click API.

[merwok]

Seems like click automatically accepts --: https://click.palletsprojects.com/en/7.x/arguments/#option-like-arguments

I’ve also found this which may be of interest: https://click.palletsprojects.com/en/7.x/advanced/#forwarding-unknown-options

[AndydeCleyre]

Another thought on the interpretation of --.

I'm a big fan of plumbum, so checked what happens there, and it's this:

• first occurrence of an island -- is removed and the following args are args, not options.
• subsequent island --s are args themselves

I haven't looked closely at click to compare. EDIT: It looks like click's handling is identical.

dashes.py:

#!/usr/bin/env python3
from plumbum import cli


class Dashes(cli.Application):

    nopanic = cli.Flag('no-panic')

    def main(self, *args):
        if self.nopanic:
            print("Don't Panic")
        print('args:', args)


if __name__ == '__main__':
    Dashes()

$ ./dashes.py
args: ()
$ ./dashes.py --no-panic
Don't Panic
args: ()
$ ./dashes.py --no-panic --
Don't Panic
args: ()
$ ./dashes.py -- --no-panic
args: ('--no-panic',)
$ ./dashes.py --uglyfile
Error: Unknown switch --uglyfile
$ ./dashes.py -- --uglyfile
args: ('--uglyfile',)
$ ./dashes.py -- --uglyfile --no-panic
args: ('--uglyfile', '--no-panic')
$ ./dashes.py --no-panic -- --uglyfile -- --elsewhere-bound
Don't Panic
args: ('--uglyfile', '--', '--elsewhere-bound')
$ ./dashes.py -- --for-here
args: ('--for-here',)
$ ./dashes.py -- -- --elsewhere-bound
args: ('--', '--elsewhere-bound')

[AndydeCleyre]

So here's a simple way to get this working.

Notes:

• args/flags to pip install must follow a second -- (not necessarily consecutive)

$ pip-sync <pip-sync arg/flag>... -- -- <pip install arg/flag>...
$ pip-sync <pip-sync arg/flag>... -- --awfully-named-reqs.txt -- <pip install arg/flag>...

• click's existing-path check moved from the decorator into the body, as src_files gets processed manually to split true src_files from pip install args (and the -- itself)
• black and flake8 do not agree about acceptable whitespace formatting:

install_flags = list(src_files[fwd_args_idx + 1 :])  # black style
install_flags = list(src_files[fwd_args_idx + 1:])   # flake8 style

[atugushev]

@AndydeCleyre I like the idea! Let's go for it ;)

[AndydeCleyre]

@atugushev black style vs flake8 style preference?

[atugushev]

@AndydeCleyre

black style vs flake8 style preference?

I'd prefer black style. See https://github.com/psf/black#slices.

[AndydeCleyre]

Hi, just inviting those interested to have a look and comment on #1080 -- please see my latest comment there.

[AndydeCleyre]

Please note that I'm holding off on any continued work on #1080 until we can get feedback on @atugushev's suggestion of using

$ pip-compile --pip-args="--no-cache-dir --no-deps --force-reinstall"

rather than the PR's current state of using

$ pip-compile -- -- --no-cache-dir --no-deps --force-reinstall

Pinging most recent participants here, @merwok @tim-mitchell @aorumbayev and OP @31z4

Thanks for any feedback!

[31z4]

I like the --pip-args version. Maybe there will be a little bit of inconvenience when args must be quoted, though.

[merwok]

Quoted args with --pip-args seem cleaner to me than the double --. Go for it!

[AndydeCleyre]

Thanks for the feedback everyone, I'll start re-implementing soon.

[AndydeCleyre]

Alright, I've pushed the --pip-args implementation to #1080 -- please go ahead and test it!