Prezto: The notice about `path_helper` on OS X is incorrect

[mislav]

This:

If you have administrator privileges, you must fix an Apple-introduced problem in Mac OS X 10.5 Leopard by executing the following command, or BASH and Zsh will have the wrong PATH when executed non-interactively.

sudo chmod ugo-x /usr/libexec/path_helper

Is incorrect. Users shouldn't disable path_helper.

The problem is OS X is that it comes with /etc/zshenv that uses path_helper. However, /etc/zshenv gets sourced for every zsh shell, even non-interactive. To put a stop to that, simply:

sudo mv /etc/{zshenv,zprofile}

[sorin-ionescu]

The instruction in the README is argued from the point that path_helper is flawed for all shells. Is said premise wrong?

[mislav]

Yes. path_helper was only flawed in zsh, due to misplaced /etc/zshenv. Bash sources /etc/profile (which uses path_helper) only for login shells, not for other interactive or non-interactive shells.

There's nothing wrong with path_helper per se.

[mislav]

You might appreciate an article I wrote: Unix shell initialization

[sorin-ionescu]

My beef with path_helper is that it reorders $PATH.

Thanks for the article.

1. Mention zlogin and zlogout for Zsh.
2. ~/.MacOSX/environment.plist no longer works in Mountain Lion.

I'll make the sudo mv /etc/{zshenv,zprofile} change.

[mislav]

My beef with path_helper is that it reorders $PATH.

That's not a problem because it runs as a first thing when you start a login shell (e.g. by opening a new Terminal tab). Because it runs first, there's no PATH before it that it can mess up.

[lunaryorn]

@mislav Are there actually any adverse consequence of disabling path_helper?

[mislav]

@lunaryorn: Yes. Installers might choose to add items to /etc/paths.d, which then won't get set up if you've disabled path_helper.

[lunaryorn]

@mislav Prezto reads paths from /etc/paths.d/ and /etc/manpaths.d/, see zshenv.

[mislav]

Oh, didn't know. So you disabled it and re-implemented it? Now you can enable it and cut down on the implementation.

[sorin-ionescu]

That is indeed what I have done. My version doesn't reorder the elements in $PATH. I believe that is why I chose to disable path_helper instead of just renaming zshenv to zprofile.

[mislav]

Scenario: bash user decides to try out Prezto. They have extra entries in their /etc/paths.d which they're probably not aware of (created by installers such as Postgres, git, Vagrant, etc). They follow Prezto install instructions wherein they disable path_helper. Things continue to work because Prezto re-implements path_helper.

Later, they decide to not use Prezto anymore, or they decide to switch their login shell back to bash. Suddently, their PATH is missing entries.

An innocent looking project for zsh configuration shouldn't mess up a whole system. How about making it your philosophy to keep system functionality intact? (sudo mv /etc/{zshenv,zprofile} doesn't count because that is a proper fix for an OS X bug.)

[sorin-ionescu]

Here's my path with path_helper disabled.

/Users/sorin/Library/Ruby/Gems/1.8/bin
/Users/sorin/Library/Python/2.7/bin
/Users/sorin/Library/Perl/5.12/bin
/Users/sorin/Developer/go/bin
/Users/sorin/.tilde/bin
/Users/sorin/.tilde/sbin
/Users/sorin/.homebrew/bin
/Users/sorin/.homebrew/sbin
/usr/local/bin
/usr/local/sbin
/usr/bin
/usr/sbin
/bin
/sbin
/usr/X11/bin

Here's my path with path_helper enabled and /etc/zshenv renamed to /etc/zprofile.

/Users/sorin/Library/Ruby/Gems/1.8/bin
/Users/sorin/Library/Python/2.7/bin
/Users/sorin/Library/Perl/5.12/bin
/usr/bin
/bin
/usr/sbin
/sbin
/usr/local/bin
/usr/X11/bin
/Users/sorin/Developer/go/bin
/Users/sorin/.tilde/bin
/Users/sorin/.tilde/sbin
/Users/sorin/.homebrew/bin
/Users/sorin/.homebrew/sbin
/usr/local/sbin

Do you see the problem now?

[mislav]

Yes I see that the outcome is different. However your case doesn't prove in any way that path_helper is broken. What is happening here is that you have extra scripts in your initialization (which you haven't shown me) that were not designed to work with path_helper. Now, once you've enabled it, your setup gets messed up. I totally understand how that could happen.

Can I see the code that adds all the /Users/sorin paths? Also, do you have a ~/.zshenv or ~/.zshrc? You want the latter, to give chance for /etc/zprofile to execute first. Then in ~/.zshrc you modify PATH as you please, by reordering items and adding new ones.

[sorin-ionescu]

It's too late to modify PATH in ~/.zshrc for essential path entries that must exist for non-interactive shells. Hence why ~/.zshenv exists. There is a README in the runcoms directory that explains all the Zsh files.

Except for the Ruby, Python, and Perl paths that are added by their respective Prezto modules, the rest of the home directory path entries are set in _~/.zshenv_, which is a copy of the runcoms/zshenv file.

# Set the list of directories that Zsh searches for programs.
path=(
  $HOME/.tilde/{bin,sbin}
  $HOME/.homebrew/{bin,sbin}
  /usr/local/{bin,sbin}
  /usr/{bin,sbin}
  /{bin,sbin}
  $path
)

path_helper is not needed. Apple could have implemented /etc/paths.d properly by using shell code, just like I have, but it looks like they spent 10 minutes to write a broken C program instead.

path_helper is broken. Dancing around the issue causes nothing but frustration. The only real solution, without editing /etc shell init files, is to replace the Apple supplied path_helper with a fixed one that doesn't rearrange path entries. It's a simple C program.

[mislav]

It's too late to modify PATH in ~/.zshrc for essential path entries that must exist for non-interactive shells. Hence why ~/.zshenv exists.

Which are these non-interactive shells that need their PATH set? What starts them, and in what environment?

While I'm waiting for your answer to the above, I have an idea for the following compromise. While you don't seem to be concerned about breaking _both_ zsh and bash on the system for users that are _not_ using Prezto, I sure am concerned about that. So my following idea is a way for Prezto to disable path_helper without physically disabling it.

First, you advise people to fix the OS X bug:

sudo mv /etc/{zshenv,zprofile}

Now we've done the right thing, and this enables the hack in the following step. In Prezto's ~/.zshenv you do the following:

unsetopt GLOBAL_RCS
# I think the above syntax should work. See `man zshoptions`

This way Prezto disables loading of /etc/zprofile and prevents automatic usage of path_helper, without actually messing with path_helper.

[lunaryorn]

Editors like Vim or Emacs start these shells when executing shell commands (i.e. M-x compile or M-x grep in Emacs). Git, too, starts such a shell to start an Editor. With path_helper enabled,git commitwill always start OS X builtin/usr/bin/vim/even if there is a custom installation in/usr/local/bin` (e.g. via Homebrew).

[sorin-ionescu]

@mislav, what @lunaryorn said — there are plenty of such programs that execute scripts and commands non-interactively.

I am concerned about BASH and Zsh users who leave Prezto and forget to re-enable path_helper, are you such a user? Unfortunately, right now, path_helper is breaking my Zsh. Thus, I must scratch this itch first.

I don't consider _/etc/zshenv_ a bug. That's where the path is supposed to be set. I consider path_helper a bug.

In what versions of Zsh is that option supported? I know of the ./configure --switch that does the same. I'll test it and report back.

[sorin-ionescu]

I have created a branch for this issue. See changes at https://github.com/sorin-ionescu/prezto/tree/issue/381/.

[mislav]

Editors like Vim or Emacs start these shells when executing shell commands. Git, too, starts such a shell to start an Editor.

@lunaryorn: You're forgetting that you launch all of Vim, Emacs, and git commit from your terminal, i.e. from an interactive shell in which you've already set up PATH properly. The subprocesses will inherit that path, and whatever non-interactive shells they spawn will inherit the same PATH too.

@sorin-ionescu: /etc/zshenv (or ~/.zshenv) setting PATH is a bug because it gets executed for every zsh shell, even non-login and non-interactive. Since most non-interactive shells are executed in an environment that is already set up, it's unnecessary overhead to go through the PATH and set it up once more.

Observe what happens with stock /etc/zshenv and this ~/.zshenv:

path=(
  ~/bin
  $path
)

When I open a new terminal window, it seems OK. My ~/.zshenv and path_helper did their jobs:

echo $path
#=> /Users/mislav/bin /usr/bin /bin /usr/sbin /sbin /usr/local/bin /opt/X11/bin

However when I execute a non-interactive subshell, you can see that ~/.zshenv and path_helper ran again and went to further edit the already setup PATH:

zsh -c 'echo $path'
#=> /Users/mislav/bin /usr/bin /bin /usr/sbin /sbin /usr/local/bin /opt/X11/bin /Users/mislav/bin

path_helper _reordered_ entries to move them to the front, and ~/bin got added _again_ to the front resulting it being now present _twice_ in the PATH. If I did more advanced PATH manipulation in ~/.zshenv it would get _duplicated inside every subshell_, then every subshell inside of that, and so on.

So you see, you're adding unnecessary overhead and duplicated PATH entries to every spawning of zsh, including for instance every executable that has #!/usr/bin/env zsh for shebang.

---

You haven't yet pointed out to me what you find not to your liking about my compromise. You don't like path_helper; fine, disable loading of /etc/zprofile via GLOBAL_RCS from Prezto. Scratches your itch. Someone uninstalls Prezto or goes back to bash? Their system works like before. Seems to me like a win-win.

[sorin-ionescu]

@mislav Said programs have GUI versions. There are, of coarse, other GUI programs that have the same issue.

Environment variables, including PATH should to be set in zshenv. That's how Zsh is designed. path_helper is screwing up Zsh's design. No duplicates occur if the right array type is used. Are you a Prezto user? Based on some of your commentary, it doesn't seem like you use Prezto.

As for the compromise, it works, but I need to know how far back GLOBAL_RCS was added.

@lunaryorn, how is it working for you?

[sorin-ionescu]

Of coarse, anything that executes scripts using /bin/bash as a login shell will get a screwed up PATH.

[telemachus]

In the past, path_helper was buggy and brutally slow. That's why historically many people disabled it, I think.

• http://mjtsai.com/blog/2009/04/01/slow-opening-terminal-windows/
• http://www.kilala.nl/Sysadmin/index.php?id=934

I believe that's been fixed, and the current path_helper is a binary (and thus faster). However, like sorin-ionescu I also prefer to disable it in order to get a more standard UNIX-like path with /usr/local _before_ /bin and company.

[mislav]

GUI programs have separate ways of setting PATH. In OS X, the preferred way would probably be via setenv directives to launchd in /etc/launchd.conf. On Ubuntu, the desktop manager sources ~/.profile on login. In other Linux /etc/environment would be probably the way to go.

@telemachus: Yes I also reorder PATH in my user rc files to have /usr/local/bin before /bin. I want to give Homebrew precedence. This could be easily done globally by editing /etc/paths and reordering the lines. It's what it was meant for.

[telemachus]

@sorin-ionescu If your biggest current complaint is the re-ordered $PATH, then another compromise might be to tell people to rearrange the items in /etc/paths (as @mislav mentions above) instead of telling them to opt out of path_helper.

In the end, I don't think that there's an enormous amount of difference. That is, I don't think it's evil to recommend people disable path_helper or to recommend rearranging the path items in /etc/paths. Users who are comfortable with that level of customization can handle fixing things later if they change things or stop using a particular shell or piece of software.

The bottom line (to me) is that OSX defaults to a non-standard $PATH and many users will want to adjust that. There are different ways to fix the problem.

[sorin-ionescu]

@telemachus, if you google path_helper, you'll notice that it has caused nothing but frustration ever since it was introduced in Leopard. Some have even chosen to write their own. See https://github.com/mgprot/path_helper and https://github.com/mcavallo/path_helper.rb.

[mislav]

if you google path_helper, you'll notice that it has caused nothing but frustration ever since it was introduced in Leopard

I've searched for results in the past year and it doesn't seem like people have real issues with it. Support questions on forums/mailing lists are from people that struggle to find out how to edit their PATH and what's the right configuration file for that. The Hackernews comment about path_helper says that it used to be slow, but that's not the case since the C rewrite. As for the supposed _bugs_ it has, I haven't found any real report of someone finding a bug with it.

Here's what path_helper does, and is designed to do: It takes the current PATH and adds global entries to it, moving them to front if they already exist in PATH. Yes, it reorders entries in the PATH, because it was designed to run as the first thing in a login shell, not as the last. You're just using it wrong, and then claim frustration with the tool.

I don't think it's evil to recommend people disable path_helper

@telemachus: People blindly follow instructions, especially when copy/pasting from README files, and forget that they do so. Especially when they don't know what path_helper is (and most don't). A Prezto user might switch back to bash and wonder why their system doesn't work like before. They won't remember to chmod +x path_helper

The bottom line (to me) is that OSX defaults to a non-standard $PATH and many users will want to adjust that. There are different ways to fix the problem.

Agreed. And I am advocating the least invasive way to fix the problem here: let path_helper do its thing and then customize it further to your needs.

[fuadsaud]

Reordering entries in /etc/paths realy feels like a less intrusive way of fixing this.

[sorin-ionescu]

@mislav, I understand how it works. It's broken.

I also understand that having it enabled causes problems. You forget the path set in ~/.MacOSX/environment.plist, which is needed for GUI apps, and Terminal.app, being a GUI app, will get that environment before it launches a shell subprocesses.

path_helper reorders the PATH entries set in environment.plist, which causes GUI programs to fail when they execute their scripts via a shell subprocess. A good example is any editor that depends on Exuberant Ctags. path_helper reorders the PATH entries and /usr/bin/ctags gets executed instead of Exuberant Ctags that is installed elsewhere. The editor complains that the wrong ctags is installed.

If I don't set the PATH in environment.plist for GUI apps, then they complain that they can't find programs.

Setting PATH on Mac OS X is a mess, and path_helper doesn't help.

[pdemarti]

We are now in 2017, and running El Capitan, so this is probably not quite the right thread. But Googling 'prevent zprofile messing my path' lands me here and I can see that the comments sound relevant.

Basically, I set PATH in .zshenv, as zsh intended. The fact that /etc/zprofile is sourced after .zshenv messes up my path: it calls path_helper -s, which sneakily reorders my PATH in such a way that /usr/local/lib and other system dirs are placed before the ones in which I carefully override certain commands. For example, I use Conda to manage Python and its modules. On all my other systems, which python yields the desired $CONDA_HOME/bin/python. On this mac, it is /usr/bin/python, which is not at all what I want.

So, I'd say this path_helper thing is quite intrusive.

[pdemarti]

Anyway, for anybody who'll land here too, here is a workaround:

in .zshenv: before customizing PATH:

# MacOS workaround sneaky path_helper in /etc/zprofile
if [ -x /usr/libexec/path_helper ]; then
  eval `/usr/libexec/path_helper -s`
fi

At the end of .zshenv:

# MacOS workaround sneaky path_helper in /etc/zprofile
export GOODPATH=$PATH

Create a .zprofile (I don't usually have one):

# MacOS workaround sneaky path_helper in /etc/zprofile
export PATH=$GOODPATH

Now everything works as intended.

[mitrx]

@pdemarti, if we consider unix shell initialization, then we may come to conclusion that /etc/zprofile is not called in some cases. At least in the case when we

Executing a command remotely with ssh

In addition, it slows down the terminal session loading process.

So, I suppose the correct solution would be NOT to include

if [ -x /usr/libexec/path_helper ]; then
  eval `/usr/libexec/path_helper -s`
fi

into ~/.zshenv, but to save PATH at the END of ~/.zshenv

export PATH_SAVE=$PATH

as you, indeed, did.

Then, you just need to set PATH in ~/.zprofile, such that it would be

export PATH=$PATH_SAVE:$PATH

In this case you just prepend your path from resulting PATH from ~/.zshenv to the PATH that was mutated by /etc/zprofile.

In order to remove duplications and not just merge two paths you can use instead of this

export PATH=$PATH_SAVE:$PATH

this script:

# add unique paths from PATH to PATH_SAVE and replace PATH with PATH_SAVE
if [ -n "$PATH_SAVE" ]; then
  PATH_SAVE=:$PATH_SAVE:;
  PATH=$PATH:
  while [ -n "$PATH" ]; do
   x=${PATH%%:*}  # the first remaining entry
   case $PATH_SAVE in
     *:$x:*) ;;  # already there
     *) PATH_SAVE=$PATH_SAVE$x: ;;  # not there yet
   esac
   PATH=${PATH#*:}
  done
  PATH=${PATH_SAVE:1:-1}
  unset PATH_SAVE x
  export PATH
fi