Hi,
I cannot check dependencies or run any command- I keep getting an error message 'command not found'.
Thanks,
A
Spinal Cord Toolbox (4.3)
SCT info:
Open a new Terminal window to load environment variables, or run:
source /Users/adityavedantam/.bashrc
Installation finished successfully!
adityavedantam@aditya-vedantam spinalcordtoolbox-4.3 % sct_check_dependencies
zsh: command not found: sct_check_dependencies
@AdiVeeMiami have you started a new terminal before trying to run sct_check_dependencies (or any other command)?
If you did, it looks like this problem is related to https://github.com/neuropoly/spinalcordtoolbox/pull/2366. If that's the case, can you please verify the content of your .zshrc file?
Just to add onto what @jcohenadad has said:
Looking at the output log you've shared, it looks like SCT has tried to make its scripts available by modifying a file called .bashrc. However, the zsh shell uses .zshrc instead of .bashrc.
The configuration should have been placed in .zshrc, as @jcohenadad points out, because the SCT installer was supposed to detect that you were using zsh.
If SCT's configuration is missing, a quick fix on your end could be to move the relevant lines from .bashrc to .zshrc. On my installation, this looks something like below, but the paths will be slightly different on your end.
# ~/.bashrc: executed by bash(1) for non-login shells.
# SPINALCORDTOOLBOX (installed on 2020-07-31 15:17:14)
export PATH="/home/joshua/repos/spinalcordtoolbox/bin:$PATH"
export SCT_DIR=/home/joshua/repos/spinalcordtoolbox
export MPLBACKEND=Agg
While copying over the lines wouldn't address the larger question of why those lines weren't added in the first place, it might help get you up and running for the time being. :slightly_smiling_face:
Thanks for the feedback. I am not very familiar with the unix code, so these may be very basic questions.
@jcohenadad - where can I find the .zshrc file?
@joshuacwnewton - I checked the code lines (199-201) for the installation file and they are exactly as you mentioned above. I clicked the link on the quick fix but couldn't understand what I needed to and how to do it.
Thanks for the feedback. I am not very familiar with the unix code, so these may be very basic questions.
No worries, @AdiVeeMiami. Happy to help! There are two possible files you'll need to look at:
/Users/adityavedantam/.bashrc/Users/adityavedantam/.zshrcIf neither file is there, then you might need to enable showing hidden files in Finder. On macOS, this can be done by pressing Command + Shift + (dot).
Once you've opened these files, you'll want to look for a group of lines starting with # SPINALCORDTOOLBOX (installed on YYYY-MM-DD ...). It will likely be near the bottom of one of the two files. The goal is to have that section be placed in the .zshrc file but not the .bashrc file. :slightly_smiling_face:
Thanks-
I found the .bashrc file with the spinalcordtoolbox lines.
Could not find .zshrc even after I unhid the files in Finder.
Ok- I had to create my .zshrc file and then I moved the code and it works!
thanks for your help!
Excellent! I'm so glad to hear. :slightly_smiling_face:
If it's not too much trouble, @AdiVeeMiami, could you please run the following command in your zsh terminal and share what the output is?
echo $SHELL
I'm hoping to investigate a little more to find out the root cause, and to help keep this from happening to other users in the future. :slightly_smiling_face:
Last login: Sun Aug 2 17:57:59 on ttys002
adityavedantam@aditya-vedantam ~ % echo $SHELL
/bin/bash
adityavedantam@aditya-vedantam ~ %
Thanks much!
@jcohenadad We may need to use another method for checking which shell someone is using, as $SHELL might not be as reliable as was expected in #2366 #2500. There are some alternative methods proposed in these StackExchange questions: Link 1 and Link 2.
One possible example:
joshua@pop-os:~$ echo $SHELL
/bin/bash
joshua@pop-os:~$ echo $0
bash
joshua@pop-os:~$ zsh
joshua@pop-os ~ % echo $SHELL
/bin/bash
joshua@pop-os ~ % echo $0
zsh
EDIT: Ah, $0 will just return install_sct within the script, I think. Might be better off using something like readlink "/proc/$$/exe" but I'll do some more exploration to try to find a better solution.
It's possible not even those solutions will help here, since we start our install_sct script with a shebang:
With a little test script:
#!/bin/bash
echo $SHELL
echo $0
readlink "/proc/$$/exe"
zsh output:
joshua@pop-os ~ % ./test_script
/bin/bash
./test_script
/usr/bin/bash
Ah, it looks like we're repeating ourselves, see #2481 and #2500. It's possible #2500 didn't fix the issue after all.
@Drulex @kousu @zougloub @cfhammill do you have any idea how we could solve this problem?
Nice debugging @joshuacwnewton.
Uhhh to get the preferred shell you can do this
PW_SHELL=$(python -c 'import pwd, os; print(pwd.getpwuid(os.getuid()).pw_shell)')
or
PW_SHELL=$(grep "$USER" /etc/passwd | cut -f 7 -d ":")
But I think this "check your shell and hope" method is really fragile.
what other installers do
For comparison, brew installs itself on OS X to /usr/local/bin/ and mangles the permissions so that it can install all packages it manages under there too, which is already on $PATH; and on Linux since it can't trust that it can mangle /usr/local, it looks at $SHELL (which is funny, I wonder if they suffer the same bug?) and gives you a suggested command to add it to your ~/.bash_profile or ~/.zprofile, depending on your shell. It's also worth noting that a lot of packages will give a warning about missing LIBPATHs, but they always only give a recommendation to add to your shell, they never do it for you.
For comparison, pip install --user writes launchers to ~/.local/bin/, and like brew it's up to the user to have edited their ~/.bashrc with export PATH=~/.local/bin:$PATH.
For comparison, conda init is a command that adds scans for active shell init files and adds something like
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/miniconda3/etc/profile.d/conda.sh" ]; then
. "/opt/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
and the installer is set up to run this command (as full/path/to/conda init, since the installer knows where it installed conda to)
For comparison, npm install -g installs code to /usr/lib/node_modules and puts launchers directly in /usr/bin/ -- so it needs sudo to work. I don't think it has a npm install --user exactly like python but it does have npm link but... it apparently defaults to PREFIX=/usr, so it also needs sudo by default. So it never touches your config files either; but you could easily tell it (npm config set prefix ~/.local) to do so, which would imply you've also manually edited your ~/.bashrc (or equivalent) to match, as in the python case.
I really like python's approach here. I like that it follows the unix convention of $PREFIX directories, and I really like that when asked (--user) or just if finds it needs to it falls back to PREFIX=~/.local. I personally wish ~/.local/bin was a standard on every system, it's a really good idea (an older option some people still use is PREFIX=~/, so your local apps are /home/$USER/bin/*). I like the idea of just putting launchers somewhere into PATH and not doing surgery on people's {.bashrc,.zshrc,.zprofile,.bash_profile,.profile}, which might break things for users who actually use those files. conda does do surgery but at least their approach marks off its territory in your config file and therefore can be undone (conda init --reverse).
So what if we installed things by adding them to $PREFIX/bin? This is the more unix-standard way, and fits packaging via a distribution which does use $PREFIX to tell an install script where to install a package to. Uh, the thing that makes this _kind of weird_ is how our installer makes its own isolated install of conda, regardless of any previous install, which means defining its own $PREFIX (in ./install_sct I think, implicitly, PREFIX=$SCT_DIR/$PYTHON_DIR). But we can make symlinks into that $PREFIX from somewhere the user already has on their $PATH, and that's better than trying to figure out how to mangle their config file (that's what a nixos environment would do, and I've been thinking more and more that nixos is showing us the way).
And along the way we can introduce a user-local/system-wide distinction to the installer, the way there's npm install vs npm install -g and pip install --user vs pip install.
if [ ask "do you want to install for the current user or for everyone?" == "user" ]; then ## this isn't proper bash but hopefully it conveys the idea
: ${PREFIX:=~/.local} # set default
else
: {PREFIX:=/usr/}
fi
........
if [ "usermode" ]
ln -s $CONDA_PREFIX/bin/* $PREFIX/bin/*
else
# system-wide
sudo ln -s $CONDA_PREFIX/bin/* $PREFIX/bin/* ## also not proper bash plz forgive
# ^^ also we should take care to make these *relative* symlinks if possible, in order that they can be packaged properly by tools like arch's makepkg or whatever debian uses
fi
.......
if [[ ":${PATH}:" != *":${PREFIX}/bin:"* ]]; then # grabbed from the brew installer
warn "${PREFIX}/bin is not in your PATH. SCT will not be accessible."
warn "You can correct this with:"
warn 'export PATH="${PREFIX}/bin:${PATH}"' >> $(shell_profile)" # from a different part of the brew installer
fi
For user-local installs, it means a one-time extra manual step by the user:
echo "export PATH="${PREFIX}/bin:${PATH}"' >> ~/.zprofile # -or-
echo "export PATH="${PREFIX}/bin:${PATH}"' >> ~/.bash_profile
which we can hand-hold them with, the way brew does. And I'd argue it's a helpful step, and that lots of users probably already have it done if they're using other python tools because PREFIX=~/.local is a kinda-semi-standard that's floating out there and used by a lot of tools, even if not officially baked into any distribution that I'm aware of.
And eventually, if we get packaged in say, neurodebian, or brew, this becomes a non-issue: the convention there that the package will set PREFIX=/usr/local/ or PREFIX=/usr and install direct to $PREFIX/bin so that our tools are on $PATH for everyone.
So my proposal:
./install_sct and replace it by adding symlinks to $PREFIX/bin/./install_sct a --user flag (or maybe it should be a new third prompt: "Install for all users or just me?"?) which overrides $PREFIX with ~/.local--user, install to a global $PREFIX (a little bit tricky since different distributions define this differently, but we can cover the common cases; and allow ourselves to be overridden), which requires us to run sudo; like brew's installer, explain what sudo is being run for before it is run. ((the other alternative is to run sudo ./install_sct, and maybe that's a bit more traditional, but I think it's better to minimize how much code runs as root and to take the opportunity to explain why sudo is needed))-p PREFIX install prefix, defaults to /home/kousu/miniconda3, must not contain spaces.) (to be brainstormed)git install mode in the install script is the same as pip install -e; can it _become_ pip install -e?My immediate hot-fix proposal:
replace $SHELL in ./install_sct with PW_SHELL=$(grep "$USER" /etc/passwd | cut -f 7 -d ":") and a link back to this issue.
Just to make sure I understand correctly, @kousu, you're saying there are two ways to frame the problem here...
.bashrc, .zshrc, .bash_profile, etc.)_Depending on how we want to scope this, we could focus on the hotfix you've proposed for now, and possibly continue a larger discussion about installation methods in a separate issue? As you've pointed out, there are certainly a lot of tangled long-term implications related to how we install SCT. :)
I don't have the ability to test for macOS right now, but I'm definitely interested in checking out the PW_SHELL=$(grep "$USER" /etc/passwd | cut -f 7 -d ":") fix you've proposed. :)
Just to unravel it a little, the command uses grep to search through /etc/passwd for the line matching the current user. Each line contains "seven attributes or fields: name, password, user ID, group ID, gecos, home directory and shell. " The command then pipes that line into cut -f 7 -d ":", which then extracts the 7th token delimited by : (i.e. the default shell for the current user.)
But, just to clarify, isn't that the same information that $SHELL is supposed to be referring to? From the login(1) manual page:
The value for $HOME, $USER, $SHELL, $PATH, $LOGNAME, and $MAIL are set according to the appropriate fields in the password entry.
It was my understanding that the issue stems from some Macs having $SHELL=/bin/bash (i.e. the /etc/passwd entry) even when zsh is being used, so I have a feeling the proposed solution may still not work. But, just theorizing here -- no Mac to test (yet). :)
One other potential hotfix would be to simply edit the init files of every present shell type (see: cat /etc/shells), rather than selecting just one.
This might not be a good long-term solution for reasons @kousu has pointed out, but it would allow SCT scripts to be used in any shell available to the user at the time of installation, which would at least prevent issues like this for the time being.
isn't that the same information that
$SHELLis supposed to be referring to? From thelogin(1)manual page:
The value for $HOME, $USER, $SHELL, $PATH, $LOGNAME, and $MAIL are set according to the appropriate fields in the password entry.It was my understanding that the issue stems from some Macs having
$SHELL=/bin/basheven whenzshis being used
Oh good point. Bother. It sure /seems/ that $SHELL should work, since it's what brew uses, at least on Linux. Maybe Apple switched to zsh by changing Terminal.app instead of /etc/passwd?
This is where we get out https://github.com/foxlet/macOS-Simple-KVM.
Given that this is the first report of this issue and we have many mac users and developers, is it possible this is a problem unique to this particular user? I've had users experience strange issues on catalina with directories getting locked and software becoming uninstallable after an upgrade while having most users experience the upgrade painlessly, this may be something similar.
Good point, @cfhammill. I've just done a new install of Catalina and echo $SHELL returns /bin/zsh as expected. Perhaps I jumped the gun a bit.
I'm alright with closing this, then, as the original issue has been addressed. We can certainly return to this if it turns out to be a larger issue, of course. :)
Long-term: How could we be installing SCT so that we have more flexibility when it comes to issues like this? (i.e. could we switch to an installation method where we wouldn't have to edit shell init files to begin with?)
i think the core dev team is strongly motivated to go with pip install spinalcordtoolbox. The sooner the better, so we won't have to deal with these accumulations of problems (not the first, not the last). Related to https://github.com/neuropoly/spinalcordtoolbox/issues/1526 and (partly) https://github.com/neuropoly/spinalcordtoolbox/issues/2669.
So, maybe there's something weird about the Catalina upgrade process that messes up our installer's assumptions, but it's unclear at this time. If we have another report we'll reopen this.
Thank you for taking the time to find us on Github and work through the problem, @AdiVeeMiami. Have a great week, and if you need help figuring out the software please feel free to ask on http://forum.spinalcordmri.org/.
reopening because we had another report from @valosekj
I think you may have called it with https://github.com/neuropoly/spinalcordtoolbox/issues/2823#issuecomment-668080173 and https://github.com/neuropoly/spinalcordtoolbox/issues/2823#issuecomment-668225436, @kousu.
To quote @valosekj from Slack, emphasis mine:
I have just upgraded from Mojave to Catalina (10.15.7), so my shell has changed from bash to zsh . Then, I reinstalled SCT by ./install_sct . At the end of installation, I got:
Open a new Terminal window to load environment variables, or run:
source /Users/valosek/.bashrc
This naturally did not work and I had to copy lines regarding export of SCT variables from /Users/valosek/.bashrc to /etc/zshrc.
It is same problem as in this issue. I write this here to let you know, that it can happen to some users. Maybe we could add info about this to SCT documentation?
Because my fresh install of Catalina went fine, my hunch then is this is only an issue with people upgrading to Catalina, as @kousu said. Also, see below, emphasis mine:
Apple’s documentation doesn’t provide much insight on why it’s making the switch. It simply says Zsh will be the new default shell for “newly created user accounts,” and that Bash will remain the defualt shell for macOS Mojave and earlier. (Source)
If I had to guess, the /etc/passwd entry for the existing user account is preserved during the upgrade process, and therefore $SHELL gets set to bash on login.
Interesting is that my $SHELL variable is correct in comparison with https://github.com/neuropoly/spinalcordtoolbox/issues/2823#issuecomment-667732523:
valosek@macbook-pro:~$ echo $SHELL
/bin/zsh
valosek@macbook-pro:~$ echo $0
-zsh
If it's not too much of a bother, could you try reinstalling, @valosekj? I'm curious if the install script will detect the shell correctly now.
If it's not too much of a bother, could you try reinstalling, @valosekj? I'm curious if the install script will detect the shell correctly now.
Do you mean only re-run ./install_sct script in SCT folder? Or git clone whole repo again, git checkout and then ./install_sct - how is written here?
(I already performed reinstallation (re-run ./install_sct script) just after upgrade from Mojave to Catalina.)
Do you mean only re-run ./install_sct script in SCT folder? Or git clone whole repo again, git checkout and then ./install_sct - how is written here?
Ah, my apologies, I was unclear. You don't even need to reinstall. Just run ./install_sct in the SCT directory and share the start of the script, e.g.
*******************************
* Welcome to SCT installation *
*******************************
Checking OS type and version...
Linux pop-os 5.4.0-7642-generic #46~1598628707~20.04~040157c-Ubuntu SMP Fri Aug 28 18:02:16 UTC x86_64 GNU/Linux
Checking requirements...
OK!
SCT version ......... dev
Installation type ... in-place
Operating system .... linux (unknown)
Shell config ........ /home/joshua/.bashrc
I'm curious what Shell config says. :)
It looks that shell is recognized correctly:
*******************************
* Welcome to SCT installation *
*******************************
Checking OS type and version...
Darwin macbook-pro.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
ProductVersion: 10.15.7
Checking requirements...
OK!
SCT version ......... dev
Installation type ... in-place
Operating system .... osx (15)
Shell config ........ /Users/valosek/.zshrc
So, that makes me wonder if there's a small window of time immediately after upgrading to Catalina where the $SHELL is incorrect. And, maybe the user needs to log out/log back in for it to be set properly.
Hm, everything is different when I am using sudo:
(I had to use sudo, otherwise installation wasn't able to write to /usr/local dir)
valosek@macbook-pro:/usr/local/sct$ sudo ./install_sct
Password:
*******************************
* Welcome to SCT installation *
*******************************
Checking OS type and version...
Darwin macbook-pro.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
ProductVersion: 10.15.7
Checking requirements...
OK!
SCT version ......... dev
Installation type ... in-place
Operating system .... osx (15)
Shell config ........ /Users/valosek/.bashrc
So, that makes me wonder if there's a small window of time immediately after upgrading to Catalina where the
$SHELLis incorrect. And, maybe the user needs to log out/log back in for it to be set properly.
Yes, you are right. Just after upgrade to Catalina, shell was still set to bash and it was necessary to run command for zsh activation.
Yes, you are right. Just after upgrade to Catalina, shell was still set to
bashand it was necessary to run command forzshactivation.
Ahh, I see now. There's a warning message, right? (I've just learned about the warning message from this link)
The default interactive shell is now zsh. To update your account to zsh, please run chsh -s /bin/zsh.
If that's the case, this might be a rare enough occurrence (user installing SCT before running chsh) that we call this a wontfix on SCT's part, and instruct users on what to do if it comes up again.
Oh, my apologies, I've just seen the edit to your previous comment.
Hm, everything is different when I am using sudo:
(I had to use sudo, otherwise installation wasn't able to write to /usr/local dir)
For sudo, I imagine this is because root has a separate entry in /etc/passwd, so chsh -s /bin/zsh doesn't update that entry.
$ less /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
...
For sudo, I imagine this is because
roothas a separate entry in/etc/passwd, sochsh -s /bin/zshdoesn't update that entry.
You are right! root has still set bash as a shell:
valosek@macbook-pro:~$ cat /etc/passwd | grep root
root:*:0:0:System Administrator:/var/root:/bin/sh
Now, everything makes sense. By command mentioned in https://github.com/neuropoly/spinalcordtoolbox/issues/2823#issuecomment-700084527 I switched to zsh only for my user but not for root.
UPDATE:
I changed /bin/sh to /bin/zsh in /etc/passwd for root:
valosek@macbook-pro:/usr/local/sct$ cat /etc/passwd | grep root
root:*:0:0:System Administrator:/var/root:/bin/zsh
Restarted computer and tried sudo ./install_sct again but shell is still bash:
Shell config ........ /Users/valosek/.bashrc
UPDATE 2:
Running sudo chsh -s /bin/zsh changed shell for root. Now sudo ./install_sct shows correct shell:
Shell config ........ /Users/valosek/.zshrc
So, conclusion is that is necessary to run chsh -s /bin/zsh and also sudo chsh -s /bin/zsh if you want to install SCT with sudo. Thanks @joshuacwnewton for debugging!
Very informative indeed, thank you @joshuacwnewton . So, should we label this as wontfix and close the issue now?
UPDATE 2:
Runningsudo chsh -s /bin/zshchanged shell for root. Nowsudo ./install_sctshows correct shell:
Shell config ........ /Users/valosek/.zshrc
btw I'd wager this has unintended consequences! There's probably scripts scattered around that assume root uses bash. It's safer to do:
sudo bash ./install_sct
Most helpful comment
btw I'd wager this has unintended consequences! There's probably scripts scattered around that assume root uses bash. It's safer to do: