Pip: Redesigning pip's output (for install, wheel and download commands)
I have written https://gist.github.com/pradyunsg/6eb63fa4571a9723f4d42166e8abb417 as kind of a proposal. :)
Thoughts @pypa/pip-committers?
pradyunsg
All 31 comments
I think it's understood but stating anyway, because it won't hurt.
If we decide to integrate only some parts聽(or none) of that proposal, I'm obviously fine with that. :)
Looking at this again, "0" looks a little dirty at the top, in the Fetching portion, due to the large amounts of text there.
Would dropping dependency information there be fine? (gem does that)
pradyunsg
on 6 Aug 2017
Also there is no reason why the wheels should not be built in parallel.
ghost
on 8 Aug 2017
@xoviat I don't disagree with that; it's just that no one has had the time to go ahead and implement it yet.
825 is probably where discussions on the parallelisation (of any part IMO) of pip should probably happen. That said, I think it'd be best to defer any discussion on that until someone actually makes a PR for it.
pradyunsg
on 9 Aug 2017
If we go the --debug way, it would be useful to add some debugging related information (versions of pip, setuptools, wheel, python path, pip executable path and stuff like this).
pradyunsg
on 20 Aug 2017
Related: tqdm/tqdm#427.
ghost
on 29 Aug 2017
@xoviat That's for when there are parallel downloads, correct?
pradyunsg
on 30 Aug 2017
The UI problem should be addressed all at once so that the UI is parallel capable and runs on another thread rather than the call_subprocess function.
ghost
on 30 Aug 2017
@xoviat I'll assume you meant yes to that yes/no question I asked. :)
I'm averse to adding that to this discussion because it mixes multiple interests and that makes stuff harder to discuss and keep on track.
My aversion comes from the fact that this probably won't happen very soon and that functionality change is big enough that when it lands a UI change would be justified. I agree that a UI that extends cleanly would be nice.
I don't want to discuss how some future pip with more features should or would look. I want to limit this discussion to the current functionality. If we end up such that a future feature's UI would fit in seemlessly, all good but I don鈥檛 want that to be the talking point. That stuff can (and should) be discussed when there's someone writing code for it or
If you disagree with that, I kindly request you to open a new issue to discuss how a pip with parallel building or downloading should look - please don鈥檛 ask for that discussion in this issue.
pradyunsg
on 31 Aug 2017
+1. Parallel processing will be a pretty big change, so it should be a separate issue/PR of its own. But please don't open another issue for that - as has already been noted, that discussion is ongoing on #825. (Technically #825 is about parallel downloads only, but any other use of parallelism in pip can initially be brought up on that thread as well, and spun off into a separate discussion from there if that seems appropriate).
Until #825 results in a PR and that PR has landed, we should assume pip runs processes serially.
pfmoore
on 31 Aug 2017
that discussion is ongoing on #825.
Oh, yeah. I forgot about that. Oops.
@pfmoore Did you get time to look at the Gist linked above? Thoughts? :)
pradyunsg
on 31 Aug 2017
@pradyunsg I can't look at gists easily as inline content, because access to gists is blocked at work. So I haven't had a chance to look yet.
pfmoore
on 31 Aug 2017
Would pradyunsg.github.io be accessible? Otherwise, I don't mind posting it inline.
pradyunsg
on 31 Aug 2017
Don't worry, I'll get to it in the end... (I just have to remember/have time when I'm at home, which is when I should be looking at this stuff anyway :smile: )
pfmoore
on 31 Aug 2017
Haha. Okay. :)
pradyunsg
on 31 Aug 2017
@xavfernandez Hi!
Sorry for being nosy. Could you please take a look at this? I'd be grateful if you tell me what you think. :)
pradyunsg
on 26 Sep 2017
@pradyunsg Well I really like "Step 0" and its clear "Fetching / Wheeling / Installing" steps split.
Having a level between our current very verbose --verbose option and normal output is also a good (and old ^^) idea.
Concerning Chose 12.0.2 out of 23 versions available, maybe Chose 12.0.2 out of 11 compatible versions (23 available), this should help debug cases like #4646.
The resolver and its backtracking algorithm will also be quite interesting to output ^^
xavfernandez
on 2 Oct 2017
I know this has been here awhile. I do like the changes and I do like the example that @xavfernandez has indicated. One thing that I'm not entirely sure about here is that this output assumes that there are a number of distinct steps where we first download all of the packages, then build wheels, then install them.
That isn't completely correct though, particularly with PEP 517 where we might have to build a wheel to get access to things like dependencies of the project (the hook that allows you to avoid building a wheel is optional).
dstufft
on 3 Oct 2017
Chose 12.0.2 out of 11 compatible versions (23 available)
+1
we might have to build a wheel to get access to things like dependencies of the project
Oh right. That means pip would either have to build a source tree into a wheel or use the hook to get the metadata, immediately after downloading it. Correct?
One way to handle this could be to get rid of the Wheeling section (as @xavfernandez calls it) and still follow the same theme as above the current proposal by having the building spinners on the next line, indented, in the Fetching section instead.
pradyunsg
on 3 Oct 2017
The resolver and its backtracking algorithm will also be quite interesting to output ^^
Yeah. FWIW, I think it'd just come down to nicely showing "no compatible version found, backtracking to change a previous choice" during the downloading phase. I was giving it some thought and have some ideas but honestly, I don't want to talk too much about it until I have something to show for it.
pradyunsg
on 3 Oct 2017
This design looks much better than the current one, but I feel this design is still somewhat verbose.
For example: As for
Fetching packages:
Flask == 0.12.2 - downloaded sdist (83kB)
click == 6.7 [>= 2.0 from Flask] - using cached wheel
Jinja2 == 2.9.6 [from Flask] - downloaded wheel (340kB)
itsdangerous == 0.24 [>= 0.21 from Flask] - downloaded sdist (46kB)
Werkzeug == 0.12.2 [>= 0.7 from Flask] - downloaded wheel (312kB)
MarkupSafe == 1.0 [>= 0.23 from Jinja2] - using cached wheel
it would be better to move dependencies like [>= 2.0 from Flask] to verbose as generally we do not care about that information.
It would also be nice to change downloaded sdist (83kB)/using cached wheel to sdist (83KB)/wheel (cached).
So it becomes:
Fetching packages:
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Jinja2 == 2.9.6 - wheel (340kB)
itsdangerous == 0.24 - sdist (46kB)
Werkzeug == 0.12.2 - wheel (312kB)
MarkupSafe == 1.0 - wheel (cached)
I also created two conceptual designs that may be helpful on this.
futursolo
on 27 Feb 2019
Fetching packages:
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Jinja2 == 2.9.6 - wheel (340kB)
itsdangerous == 0.24 - sdist (46kB)
Werkzeug == 0.12.2 - wheel (312kB)
MarkupSafe == 1.0 - wheel (cached)
There's no way to vertically align this since at this stage, pip doesn't know what the package names are going to be.
pradyunsg
on 27 Feb 2019
it would be better to move dependencies like
[>= 2.0 from Flask]to verbose as generally we do not care about that information.
Yea, I was thinking about this. To be clear, it's about where the requirement "came from", rather than dependencies.
pradyunsg
on 27 Feb 2019
There's no way to vertically align this since at this stage, pip doesn't know what the package names are going to be.
Yes, pip learns more packages as it downloads them. But it doesn't mean that there's no way to align data.
- Use
\033[F(move cursor up a line) and\033[K(clear text to end).
class Stdout:
def __init__(self):
self.active_lines = 0
@staticmethod
def get_twidth():
return shutil.get_terminal_size().columns
@staticmethod
def print_fill_line(s):
print(s + " " * (Stdout.get_twidth() - len(s)))
def moveback(self):
for _ in range(0, self.active_lines):
sys.stdout.write("\033[F")
sys.stdout.flush()
def update_active_lines(self, lines):
if len(lines) < self.active_lines:
raise ValueError
self.moveback()
for line in lines:
self.print_fill_line(line)
self.active_lines = len(lines)
sys.stdout.flush()
def print_permanent_line(self, s):
self.moveback()
sys.stdout.write("\033[K")
print(s.rstrip())
self.active_lines = 0
sys.stdout.flush()
stdout = Stdout()
all_pkgs = [
('Flask', '0.12.2', 'sdist (83kB)'),
('click', '6.7', 'wheel (cached)'),
('Jinja2', '2.9.6', 'wheel (340kB)'),
('itsdangerous', '0.24', 'sdist (46kB)'),
('Werkzeug', '0.12.2', 'wheel (312kB)'),
('MarkupSafe', '1.0', 'wheel (cache)'),
]
stdout.print_permanent_line("Fetching Package(s):")
for i in range(0, len(all_pkgs)):
lens = []
lines = []
now_pkg = all_pkgs[:i + 1]
for pkg in now_pkg:
for index, f in enumerate(pkg):
try:
lens[index] = lens[index] \
if len(f) < lens[index] else len(f)
except:
lens.append(len(f))
for pkg in now_pkg:
fs = []
for index, f in enumerate(pkg):
fs.append(f + " " * (lens[index] - len(f)))
lines.append(" " + fs[0] + " == " + fs[1] + " - " + fs[2])
stdout.update_active_lines(lines)
time.sleep(.3)
prints
Fetching Package(s):
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Fetching Package(s):
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Jinja2 == 2.9.6 - wheel (340kB)
Fetching Package(s):
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Jinja2 == 2.9.6 - wheel (340kB)
itsdangerous == 0.24 - sdist (46kB)
Werkzeug == 0.12.2 - wheel (312kB)
MarkupSafe == 1.0 - wheel (cache)
- Use
\t
Fetching Package(s):
Flask == 0.12.2 - sdist (83kB)
click == 6.7 - wheel (cached)
Jinja2 == 2.9.6 - wheel (340kB)
itsdangerous == 0.24 - sdist (46kB)
Werkzeug == 0.12.2 - wheel (312kB)
MarkupSafe == 1.0 - wheel (cache)
This one has some limitations. Content will be pushed into the next column if the last one is too long. But it still looks better than the original one.
- From the last example we can see that its really the third column that is messing with our eyes and making the entire thing look messy. So why don't we just get rid of it? Why would I care if I am using an egg or a wheel or a source archive as long as it works? And if it's cached, just hide it.
Fetching Package(s):
Flask == 0.12.2
Jinja2 == 2.9.6
itsdangerous == 0.24
Werkzeug == 0.12.2
- Use a
dnf-style table design
- I explained it in the design page here.
================================================================================
Package(s) Version Distribution Size Speed
================================================================================
Flask 5.1.2 sdist 83kB 827KB/s
click 5.1.2 wheel(non-any) cached N/A
Jinja2 5.1.2 wheel(non-any) 340kB 725KB/s
itsdangerous 5.1.2 sdist 46kB 927KB/s
Werkzeug 5.1.2 wheel(non-any) 312kB 918KB/s
MarkupSafe 5.1.2 wheel(non-any) cached N/A
1.
Yea, I'm not super warm to the idea of using ANSI escapes.
That said, I do want us to retain this information, as it is useful information when using pip, to debug how things are happening and reporting of how things are being done.
As of now, I want to keep that column, even though I'm not happy with how it looks myself. ;)
pradyunsg
on 27 Feb 2019
Yea, I'm not super warm to the idea of using ANSI escapes.
Apart from anything else it's not obvious they would work on Windows (colorama has some ANSI support, but I don't know how complete or robust it is).
pfmoore
on 27 Feb 2019
That said, thanks @futursolo for the ideas (and code)!
pradyunsg
on 27 Feb 2019
One thing to consider when doing this -- move warning and error messages to the end of the output instead of being in the middle or something.
(eg -- warnings about broken dependencies as added in #5000 or warnings about not-on-PATH script locations)
pradyunsg
on 19 Nov 2019
Alright. So, one of the "trends" from the UX studies seems to be that users think most of pip's output isn't relevant most of the time. It's a lot of output with not enough relevant information.
I made a thing to make visualizing the CLI stuff easier. Don't @ me for feature requests on that. :P
Let's call this one the "Oct 9 2020 iteration"
This is just a toy example I've made, of what we could output in a hypothetical pip with revamped outputs on the CLI.
What you want to do to see this properly is copy the entire thing below and paste it into the left text box of the page above. Once you click outside of the text box, the other elements will process that and you can press "Play" to see it in action. :)
$ pip install flit
+++
$ pip install flit
Resolving dependencies... (0.0s)
+++
$ pip install flit
Resolving dependencies... (0.1s)
Current task: Looking up "flit" on package index
+++
$ pip install flit
Resolving dependencies... (0.2s)
Current task: Downloading flit-3.0.0-py3-none-any.whl (48 kB)
| | 0 kB 5.8 MB/s
+++
$ pip install flit
Resolving dependencies... (0.3s)
Current task: Downloading flit-3.0.0-py3-none-any.whl (48 kB)
|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅 | 20 kB 5.8 MB/s
+++
$ pip install flit
Resolving dependencies... (0.4s)
Current task: Downloading flit-3.0.0-py3-none-any.whl (48 kB)
|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 48 kB 5.8 MB/s
+++
$ pip install flit
Resolving dependencies... (0.5s)
Current task: Gathering dependency metadata (flit 3.0.0)
+++
$ pip install flit
Resolving dependencies... (0.6s)
Current task: Looking up "pytoml" on package index
+++
$ pip install flit
Resolving dependencies... (0.7s)
Current task: Using cached pytoml-0.1.21-py2.py3-none-any.whl (8.5 kB)
+++
$ pip install flit
Resolving dependencies... (0.8s)
Current task: Gathering dependency metadata (pytoml 0.1.21)
+++
$ pip install flit
Resolving dependencies... (0.9s)
Current task: Looking up "flit-core" on package index
+++
$ pip install flit
Resolving dependencies... (1.0s)
Current task: Using cached flit_core-3.0.0-py2.py3-none-any.whl (8.5 kB)
+++
$ pip install flit
Resolving dependencies... (1.1s)
Current task: Gathering dependency metadata (flit-core 3.0.0)
+++
$ pip install flit
Resolving dependencies... (1.2s)
Current task: Looking up "requests" on package index
+++
$ pip install flit
Resolving dependencies... (1.3s)
Current task: Using cached requests-2.24.0-py2.py3-none-any.whl (8.5 kB)
+++
$ pip install flit
Resolving dependencies... (1.4s)
Current task: Gathering dependency metadata (requests 2.24.0)
+++
$ pip install flit
Resolving dependencies... (1.5s)
Current task: Looking up "docutils" on package index
+++
$ pip install flit
Resolving dependencies... (1.6s)
Current task: Using cached docutils-0.16-py2.py3-none-any.whl (8.5 kB)
+++
$ pip install flit
Resolving dependencies... (1.7s)
Current task: Gathering dependency metadata (docutils 0.16)
+++
$ pip install flit
Resolving dependencies... (1.8s)
Current task: Looking up "urllib3" on package index
+++
$ pip install flit
Resolving dependencies... (1.9s)
Current task: Using cached urllib3-1.25.10-py2.py3-none-any.whl (8.5 kB)
+++
$ pip install flit
Resolving dependencies... (2.0s)
Current task: Gathering dependency metadata (urllib3 1.25.10)
+++
$ pip install flit
Resolving dependencies... (2.0s)
Current task: Gathering dependency metadata (urllib3 1.25.10)
[assume a few more steps like this -- I chose an example with too many dependencies]
+++
$ pip install flit
Resolving dependencies: done in 7.3s
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet... (0.0s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet... (0.1s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet... (0.2s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi... (0.0s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi... (0.1s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi... (0.2s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi... (0.3s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi... (0.4s)
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (1 of 9) pytoml
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (2 of 9) flit-core
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (3 of 9) urllib3
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (4 of 9) idna
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (5 of 9) certifi
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (6 of 9) chardet
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (7 of 9) requests
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (8 of 9) docutils
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installing collected packages... (9 of 9) flit
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installed 9 packages successfully.
+++
$ pip install flit
Resolving dependencies: done in 7.3s
Building local wheels...
chardet: done in 0.2s
certifi: done in 0.4s
Installed 9 packages successfully.
$
Um, yes I like this? When's it going to be ready? 馃檪
More seriously, I think it would be great to implement the console output mechanisms needed to implement this type of output. Iterating on the precise form of the reporting will be relatively easy once we have the machinery in place. And overall, I like the look of the "Oct 9 2020 iteration".
However, we need to be careful not to vendor in a big bunch of of console rendering libraries in pursuit of prettier output - pip is big enough already 馃檨
pfmoore
on 9 Oct 2020
That output looks really good, and I'm another +1 on it. I think ideally it'd support pushing logging messages "above" that, so that all the relevant "Stuff I'm doing" stays at the bottom, and the top is just important logging information like.. deprecation notices or what have you.
dstufft
on 10 Oct 2020
/cc @ei8fdb, since this is the catch-all for the pip-output discussion. IDK if you saw what I posted earlier this week, based off of our discussion. :)
pradyunsg
on 12 Oct 2020
Related issues
dstufft
路
3Comments
wrouesnel
路
3Comments
cjerdonek
路
3Comments
yizhang-zen
路
3Comments
kkom
路
3Comments
Most helpful comment
Alright. So, one of the "trends" from the UX studies seems to be that users think most of pip's output isn't relevant most of the time. It's a lot of output with not enough relevant information.
I made a thing to make visualizing the CLI stuff easier. Don't @ me for feature requests on that. :P
Let's call this one the "Oct 9 2020 iteration"
This is just a toy example I've made, of what we could output in a hypothetical pip with revamped outputs on the CLI.
What you want to do to see this properly is copy the entire thing below and paste it into the left text box of the page above. Once you click outside of the text box, the other elements will process that and you can press "Play" to see it in action. :)