Brew: Completion for `brew` commands in Zsh is really slow

[marlonrichert]

• [x] ran brew update and can still reproduce the problem?
• [x] ran brew doctor, fixed all issues and can still reproduce the problem?
• [x] ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

brew instab⇥

What happened (include command output)

There's a significant delay before this gets completed to brew install. Other completions for brew in Zsh suffer from similar or even worse delays.

What you expected to happen

I expect it to be near instantaneous, as it is for all other completions in Zsh.

Step-by-step reproduction instructions (by running brew commands)

$ cd "$(mktemp -d)"
$ ZDOTDIR=$PWD HOME=$PWD zsh -df
% autoload -Uz compinit && compinit; bindkey '^I' complete-word
% eval $( brew shellenv )
% brew ins

and press tab⇥. ## Output of `brew config` and `brew doctor` commands

❯ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/marlon/.pyenv/shims/python-config
  /Users/marlon/.pyenv/shims/python3-config
  /Users/marlon/.pyenv/shims/python3.8-config

❯ brew config
HOMEBREW_VERSION: 2.4.0-41-g0125c4c
ORIGIN: https://github.com/Homebrew/brew
HEAD: 0125c4cd56b1e9a238206090dbf3f6a24a463c7d
Last commit: 2 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 73122fb58ab5938f8c9a14db5267b30b9b3763ab
Core tap last commit: 5 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.Hiq6bnKqb4/org.macosforge.xquartz:0
HOMEBREW_EDITOR: nano
HOMEBREW_MAKE_JOBS: 4
CPU: quad-core 64-bit kabylake
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
Clang: 11.0 build 1103
Git: 2.27.0 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 1.8.0_144
macOS: 10.15.5-x86_64
CLT: 11.0.33.16
Xcode: 11.5
XQuartz: 2.7.11 => /opt/X11

[MikeMcQuaid]

As with all "this is slow" issues: we do not consider this a bug but will accept a pull request to improve the speed.

[nhooyr]

@MikeMcQuaid If you don't leave these sorts of issues open, it's very difficult for someone looking to contribute to figure out what needs work.

[nhooyr]

I opened a similar issue in the past and debugged the slow line.

See https://github.com/Homebrew/brew/issues/6106 for anyone who visits this in the future and wants to fix it.

[nhooyr]

You can see the slowness with:

~code-server:master*
$ /usr/bin/time brew commands --quiet --include-aliases > /dev/null
        0.78 real         0.43 user         0.32 sys

[nhooyr]

I think it's just due to ruby being slow to startup.

e.g. catting a formula takes a similar amount of time:

~code-server:master*
$ /usr/bin/time brew cat code-server > /dev/null
        0.84 real         0.45 user         0.35 sys

[nhooyr]

@MikeMcQuaid What're your thoughts on changing the completion scripts to instead hard code the list of commands?

[marlonrichert]

An update from my part on this: I tested a bit more and found that if I do zstyle ':completion:*' use-cache true, then caching is used for these completions and they are no longer slow —except for the first time and whenever the cache gets invalidated, which is each time brew updates itself.

zstyle ':completion:*' use-cache is not true by default; it's false. Should brew shellenv perhaps add zstyle ':completion:*:brew:*' use-cache true, so that caching gets enabled for most users?

Also, the caching strategy could be a bit more sophisticated than it is now. At present, the caching policy looks only at the date of brew's git index, which gets updated (and thus invalidates your cache) each time brew updates. This same caching policy is used for __brew_formulae, __brew_all_commands and __brew_all_casks. Since brew updates whenever you use it and most people don't use it that often, the net effect is that most of the times you try to use completion on brew, the cache is going to be invalidated and completion is going to be slow.

Could brew perhaps rebuild the completion cache when it updates itself, instead of while you try to use the completion system?

[nhooyr]

Ah looks like it was added at https://github.com/Homebrew/brew/pull/5388

[nhooyr]

@MikeMcQuaid Can you unlock https://github.com/Homebrew/brew/issues/6106 so that I can add a comment there explaining what the problem was and why I was the only one who seemed to experience it in case someone stumbles upon it?

[MikeMcQuaid]

@MikeMcQuaid If you don't leave these sorts of issues open, it's very difficult for someone looking to contribute to figure out what needs work.

Yet you seemed to find it despite it being closed. Our way of managing our issue trackers works for us: run your own projects however you wish.

Some general feedback @nhooyr: I'm not sure it's helpful for you to continue to create issues rather than PRs and your feedback on how we should run our project is definitely not helpful.

@MikeMcQuaid What're your thoughts on changing the completion scripts to instead hard code the list of commands?

If they are generated by something like e.g. brew man: this seems like a good idea. If they are hardcoded manually: this would be a step back.

Could brew perhaps rebuild the completion cache when it updates itself, instead of while you try to use the completion system?

This may be possible but I'm not sure how it'd be done. We'd review a PR.

This could perhaps be done more generically by e.g. writing $HOMEBREW_CACHE/.brew_commands or something which could then be shared with Bash (which doesn't provide completion caching).

@MikeMcQuaid Can you unlock #6106 so that I can add a comment there explaining what the problem was and why I was the only one who seemed to experience it in case someone stumbles upon it?

No, sorry.

[marlonrichert]

Could brew perhaps rebuild the completion cache when it updates itself, instead of while you try to use the completion system?

This may be possible but I'm not sure how it'd be done. We'd review a PR.

I will submit one soon.

[marlonrichert]

@MikeMcQuaid @nhooyr Here you go: #7766