Poetry: Adding path dependency fails when given absolute path

[simonpercivall]

Version: 1.0.0b9 (e943a4e)

Expected behaviour: I can add path dependencies specified by absolute path.
Actual behaviour: '/abs/path/to/dependency' does not start with '/abs/path/to/poetry-file/parent'

In poetry.console.commands.init:412(_parse_requirements) a call is made topath.relative_to(), which raiseValueErrorwhenpathisn't a relative path tocwd`.

I'm guessing that this was added thinking it behaves like os.path.relpath(), which it doesn't. AFAICT, there isn't any functionality in pathlib that does the same thing as os.path.relpath().

[finswimmer]

Hey @simonpercivall,

I believe "this is a feature and not a bug". But I couldn't find out what's the intention of it.

@sdispater could you tell us something about this?

fin swimmer

[simonpercivall]

Thanks for the quick reply! If it's not documented and prints a ValueError raised from within pathlib, it's not a feature :) That being said, I'd be happy with whatever outcome as long as I can produce a PR to improve the output.

[gitpushdashf]

Any update on this? Would be really nice to do poetry add /some/absolute/path

[francoposa]

Seconding, as we have a pretty straightforward use case.

I would really like this to support absolute paths, so we can reliably set it up the same way inside and outside of a dockerfile.

We have a docker image with Spark already installed at /usr/local/spark, meaning the PySpark is at /usr/local/spark/python. Requirements file used for both local dev and prod just lists the requirement as -e /usr/local/spark/python.

For local development, devs just install spark in the same place on their Mac, and can just pip install the requirements file as-is

I would really like to transition us to Poetry, but this would be a sticking point.

(my-venv) [~/repos/some-pyspark-app]% poetry add /usr/local/spark/python 

[ValueError]
'/usr/local/spark/python' does not start with '/Users/franco/repos/some-pyspark-app'

Using relative path on a local mac, which won't be the same as the Linux docker container when it gets built...

(my-venv) [~/repos/some-pyspark-app]% poetry add ../../../../usr/local/spark/python

It seems that even though it gets declared with a relative path, Poetry ultimately understands it as an absolute path.
When I switched from the local path requirement to just a standard pyspark requirement and ran poetry lock it printed out

- Updating pyspark (3.0.0 /usr/local/spark/python -> 3.0.1)

[francoposa]

Definitely willing to submit a PR for this as long as it would be welcome

[sinoroc]

Seems like it "_works_" when adding the absolute path to the dependency directly in the pyproject.toml, but not via the CLI.

I use quotes, because the absolute path is in the metadata of the distributions (sdist, and wheel), but it shows up as relative path in other locations: the CLI output (poetry show), the lockfile.

[francoposa]

Yeah, for portability to Docker containers and other dev's machines, we need the lockfile to come out with the absolute path too.

[sinoroc]

Definitely willing to submit a PR for this as long as it would be welcome

@francojposa The "bug" label is still up, so I can only assume a PR would be well received.

[francoposa]

Have started working on this - not to say someone won't beat me to it.

Basic approach is to replace

https://github.com/python-poetry/poetry/blob/master/poetry/console/commands/init.py#L441

OrderedDict(
    [
        ("name", package.name),
        ("path", path.relative_to(cwd).as_posix()),
    ]
    + ([("extras", extras)] if extras else [])
)

with

OrderedDict(
    [
        ("name", package.name),
        ("path", requirement),
    ]
    + ([("extras", extras)] if extras else [])
)

Because at that point, requirement (the path provided by cli user) is fully vetted as a valid package. We can just accept the path as provided by the user. Tests bear this out so far

[francoposa]

Update: I managed to get past the basic error being thrown when parsing the requirement, but this looks like it will require a change to poetry-core. These ultimately get modeled as a poetry-core FileDependency which always uses an absolute path, then the absolute path gets re-resolved relative to the project root when the lock file is being created in poetry.packages.locker

Since FileDependency always makes it into an absolute URL, we lose the understanding of whether it was originally intended to be absolute or relative, then we don't know which one to do when it comes to dump to the lock file.

[francoposa]

Update Update: I have no idea what I am doing and I am pretty overwhelmed by the codebase.

[sinoroc]

@francojposa I know the feeling. :)
Patience, one of the maintainers will get to it at some point. Maybe in the mean time, open a draft pull request, so that the work does not get lost. It does not have to be fully functional or anything.