Poetry: Detect pyproject.toml changes and automatically run `poetry install` if needed

[mtkennerly]

When using Stack (Haskell) and Cargo (Rust), if you manually add a new dependency in package.yaml or Cargo.toml respectively, then the next invocation of stack test or cargo test will detect the change and install the new dependency, whereas poetry run pytest could fail on the dependency missing. It would be great if Poetry could auto-detect the change instead of having to manually run poetry install again. Of course, for an individual user, it's easy to go through the poetry add interface instead, but it's more problematic when working on a team since other people won't immediately know after git pull whether there were updates requiring a poetry install.

[cauebs]

Maybe warn the user instead of automatically installing. But I agree that detecting changes would be useful.

[jacebrowning]

I've achieved this behavior with a Makefile:

.PHONY: all
all: install

.PHONY: install
install: .venv/flag
.venv/flag: pyproject.lock
    @ poetry config settings.virtualenvs.in-project true
    poetry develop
    @ touch $@

pyproject.lock: pyproject.toml
    poetry lock

###############################################################################

.PHONY: test
test: install
    poetry run pytest

but I would be in favor of poetry doing this automatically.

[cauebs]

This is more important than I initially thought, as it's causing some very confusing things, e.g.: #292.

• How do we detect changes in pyproject.toml? Check the file's metadata for last modified time? Hash its contents?
• Where do we store these stamps? In pyproject.lock
• Then, whenever the lock file is used, that should be checked, right?

I can give this one a go, but I'll definitely need some pointers from @sdispater.

[ranisalt]

Check the file's metadata for last modified time?

This is faster and less resource-consuming, but there are known issues when depending on modification times.

I think that just as pyproject contains hashes to deps, a hash to pyproject.toml content is fine too.

[sdispater]

Poetry already tracks changes and discrepancies between pyproject.toml and pyproject.lock.
That's why you can sometimes see a warning when executing install.

Here is the corresponding line: https://github.com/sdispater/poetry/blob/master/poetry/installation/installer.py#L164

A similar approach could be used elsewhere if needed.

And I think displaying a warning (like in install) is preferable instead of installing automatically.

[cauebs]

Well, then why didn't he get a warning in #292?

[sdispater]

It seems like a bug to me. We do no track extras to generate the pyproject.toml hash.

See https://github.com/sdispater/poetry/blob/master/poetry/packages/locker.py#L15

We should add extras to this list

[cauebs]

While we're at it, what's the point of the lock command? How is it different to update? Only that it doesn't install anything?

[PavlosMelissinos]

locker.py seems to include extras now, is this issue resolved?

EDIT: nevermind, I mixed this up with #292. Anyway, what is the state of this?

[mtkennerly]

@PavlosMelissinos, the behavior is still the same as of v0.12.17 - no warning or automatic install:

$ poetry new issue-75
Created package issue-75 in issue-75

$ cd issue-75/

$ poetry install
Creating virtualenv issue-75-py3.5 in C:\tmp\issue-75\.venv
Installing dependencies from lock file


Package operations: 11 installs, 0 updates, 0 removals

  - Installing more-itertools (7.2.0)
  - Installing zipp (0.6.0)
  - Installing importlib-metadata (0.23)
  - Installing six (1.12.0)
  - Installing atomicwrites (1.3.0)
  - Installing attrs (19.3.0)
  - Installing colorama (0.4.1)
  - Installing pathlib2 (2.3.5)
  - Installing pluggy (0.13.0)
  - Installing py (1.8.0)
  - Installing pytest (3.10.1)
  - Installing issue-75 (0.1.0)

$ poetry run pytest
============================= test session starts =============================
platform win32 -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\tmp\issue-75, inifile:
collected 1 item

tests\test_issue_75.py .                                                 [100%]

========================== 1 passed in 0.05 seconds ===========================

Then edit files to add a dependency:

$ cat pyproject.toml
[tool.poetry]
name = "issue-75"
version = "0.1.0"
description = ""
authors = ["mtkennerly <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.5"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
requests = "^2.22"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

$ cat tests/test_issue_75.py
from issue_75 import __version__
import requests


def test_version():
    assert __version__ == '0.1.0'

$ poetry run pytest
============================= test session starts =============================
platform win32 -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\tmp\issue-75, inifile:
collected 0 items / 1 errors

=================================== ERRORS ====================================
___________________ ERROR collecting tests/test_issue_75.py ___________________
ImportError while importing test module 'C:\tmp\issue-75\tests\test_issue_75.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_issue_75.py:2: in <module>
    import requests
E   ImportError: No module named 'requests'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.15 seconds ===========================

Compare with Cargo for Rust:

$ cargo new issue-75-rs
     Created binary (application) `issue-75-rs` package

$ cd issue-75-rs/

$ cargo test
   Compiling issue-75-rs v0.1.0 (C:\tmp\issue-75-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.61s
     Running target\debug\deps\issue_75_rs-067f4b5baf064cb5.exe

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Then edit to add a dependency:

$ cat Cargo.toml
[package]
name = "issue-75-rs"
version = "0.1.0"
authors = ["mtkennerly <[email protected]>"]
edition = "2018"

[dependencies]
derive-error = "0.0.4"

$ cargo test
    Updating crates.io index
   Compiling unicode-xid v0.0.4
   Compiling quote v0.3.15
   Compiling case v0.1.0
   Compiling synom v0.11.3
   Compiling syn v0.11.11
   Compiling derive-error v0.0.4
   Compiling issue-75-rs v0.1.0 (C:\tmp\issue-75-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 10s
     Running target\debug\deps\issue_75_rs-b9dbf4a0aa18a11e.exe

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out