Pip: `pip freeze -r requirements.txt > requirements.txt` could mangle requirements.txt

[Flimm]

• Pip version: 9.0.1
• Python version: 3.5
• Operating System: macOS

Description:

Running a command like this is a fairly common practise:

pip freeze > requirements.txt

If you want to preserve comments and so on, you might be tempted to run:

pip freeze -r requirements.txt > requirements.txt

However, there is a risk that pip will write to requirements.txt before all of requirements.txt has been read. To get round this, the correct invocation would be:

pip freeze -r requirements.txt > $tmpfile
mv $tmpfile requirements.txt

Or you could use sponge if you have it installed:

pip freeze -r requirements.txt | sponge requirements.txt

Proposal:

I propose that pip be changed to read the entirety of requirements.txt before emitting any output, so that the command pip freeze -r requirements.txt > requirements.txt is safe, and to document that behaviour.

If this cannot be implemented, then a warning should be documented.

[larsonnn]

Hello @Flimm

the file is not fully read before standard output comes? Or why is there chance to break?
Real Question. :)

[Flimm]

As a user of pip, I do not know whether the file is fully read before any output is emitted. It is not safe for me as a user to assume that.

If pip has already been implemented in such a way as to guarantee that the file is always read before any output on stdout is emitted, then the documentation should state that.

[larsonnn]

Im not sure, but I think the file will be read line by line stored, processed and then comes the stdout.
But a clarification would be nice.

[pfmoore]

I think it's simply a case of "undocumented details". We don't guarantee that we'll read all of requirements.txt before producing any output, so you shouldn't rely on it. If we did do that, and we were willing to guarantee that we'd maintain that behaviour going forward (so we'd have tests that verified it, for example), then we would (should) document that fact. But in the absence of such docs, you shouldn't assume it.

For comparison, sed has a special --inplace flag to guarantee in-place edits, but it's not the default behaviour.

[dstufft]

This is basically the nature of the way that the shell works and there's not a great way for pip to fix this (for instance, Python itself can produce output before pip even gets a chance to run).