Sdkman-cli: 'sdk update' yields 'bash -1: substring expression < 0' error

[Dispader]

We require a conversation to take place in the appropriate Gitter Room before raising a new Github Issue. Please note that issues will be closed immediately if prior discussion did not take place. We need to do this to help manage the quality and validity of Issues raised on this project.

Please tick one:

• [ ] this is a new Feature Request, a conversation was started in the Gitter CLI Development Room.
• [x] this is a new Bug Report or Issue, a conversation was started in Gitter User Issues Room.

Please explain the Issue / Feature Request here:

While running sdk yesterday, I got the message

WARNING: SDKMAN is out-of-date and requires an update. Please run:

  $ sdk update

Attempting to issue sdk update yields what looks to be a shell error.

(~){azazel}01:33am] sdk update
bash: -1: substring expression < 0

I'm currently on MacOS High Sierra.

(~){azazel}01:34am] uname -a
Darwin azazel 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64 i386 MacBookPro12,1 Darwin
(~){azazel}01:34am] bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.
(~){azazel}01:34am] 

[Dispader]

I just learned about function tracing in bash. I'm thinking the following might help.

(~){azazel}02:06am] set -x ; set -o functrace ; sdk update 2>&1 
+ COMMAND=update
+ QUALIFIER=
+ case "$COMMAND" in
+ [[ update != \u\p\d\a\t\e ]]
+ SDKMAN_AVAILABLE=true
+ '[' -z false ']'
+ __sdkman_update_broadcast_and_service_availability
++ __sdkman_determine_broadcast_id
++ [[ false == \t\r\u\e ]]
++ [[ update == \o\f\f\l\i\n\e ]]
+++ __sdkman_secure_curl_with_timeouts https://api.sdkman.io/2/broadcast/latest/id
+++ [[ false == \t\r\u\e ]]
+++ curl --silent --location --connect-timeout 5 --max-time 4 https://api.sdkman.io/2/broadcast/latest/id
++ echo 5aba541124aa9a00051286ed
+ local broadcast_live_id=5aba541124aa9a00051286ed
+ __sdkman_set_availability 5aba541124aa9a00051286ed
+ local broadcast_id=5aba541124aa9a00051286ed
++ echo 5aba541124aa9a00051286ed
++ tr '[:upper:]' '[:lower:]'
++ grep html
+ local detect_html=
+ [[ -z 5aba541124aa9a00051286ed ]]
+ [[ -n '' ]]
+ SDKMAN_AVAILABLE=true
+ __sdkman_update_broadcast 5aba541124aa9a00051286ed
+ local broadcast_live_id broadcast_id_file broadcast_text_file broadcast_old_id
+ broadcast_live_id=5aba541124aa9a00051286ed
+ broadcast_id_file=/Users/jake/.sdkman/var/broadcast_id
+ broadcast_text_file=/Users/jake/.sdkman/var/broadcast
+ broadcast_old_id=
+ [[ -f /Users/jake/.sdkman/var/broadcast_id ]]
++ cat /Users/jake/.sdkman/var/broadcast_id
+ broadcast_old_id=5aba541124aa9a00051286ed
+ [[ -f /Users/jake/.sdkman/var/broadcast ]]
++ cat /Users/jake/.sdkman/var/broadcast
+ BROADCAST_OLD_TEXT='==== BROADCAST =================================================================
* 27/03/18: Groovy 2.4.15 released on SDKMAN! #groovylang
* 23/03/18: Kotlin 1.2.31 released on SDKMAN! #kotlin
* 21/03/18: Scala 2.12.5 released on SDKMAN! #scala
================================================================================'
+ [[ true == \t\r\u\e ]]
+ [[ 5aba541124aa9a00051286ed != \5\a\b\a\5\4\1\1\2\4\a\a\9\a\0\0\0\5\1\2\8\6\e\d ]]
+ '[' -f /Users/jake/.sdkman/etc/config ']'
+ source /Users/jake/.sdkman/etc/config
++ sdkman_auto_answer=false
++ sdkman_auto_selfupdate=true
++ sdkman_insecure_ssl=false
++ sdkman_disable_gvm_alias=false
++ sdkman_curl_connect_timeout=5
++ sdkman_curl_max_time=4
++ sdkman_beta_channel=true
++ sdkman_debug_mode=false
++ sdkman_colour_enable=true
+ [[ -z update ]]
+ CMD_FOUND=
+ CMD_TARGET=/Users/jake/.sdkman/src/sdkman-update.sh
+ [[ -f /Users/jake/.sdkman/src/sdkman-update.sh ]]
+ CMD_FOUND=/Users/jake/.sdkman/src/sdkman-update.sh
+ CMD_TARGET=/Users/jake/.sdkman/ext/sdkman-update.sh
+ [[ -f /Users/jake/.sdkman/ext/sdkman-update.sh ]]
+ [[ -z /Users/jake/.sdkman/src/sdkman-update.sh ]]
++ echo ant asciidoctorj ceylon crash gaiden glide gradle grails griffon groovy groovyserv java jbake kobalt kotlin kscript lazybones leiningen maven sbt scala spark springboot sshoogr vertx visualvm
++ grep -w ''
+ local 'sdkman_valid_candidate=ant asciidoctorj ceylon crash gaiden glide gradle grails griffon groovy groovyserv java jbake kobalt kotlin kscript lazybones leiningen maven sbt scala spark springboot sshoogr vertx visualvm'
+ [[ -n '' ]]
+ [[ update == \o\f\f\l\i\n\e ]]
++ echo update
++ tr - _
+ CONVERTED_CMD_NAME=update
+ '[' -n /Users/jake/.sdkman/src/sdkman-update.sh ']'
+ __sdk_update '' '' ''
+ local candidates_uri=https://api.sdkman.io/2/candidates/all
+ __sdkman_echo_debug 'Using candidates endpoint: https://api.sdkman.io/2/candidates/all'
+ [[ false == \t\r\u\e ]]
++ __sdkman_secure_curl_with_timeouts https://api.sdkman.io/2/candidates/all
++ [[ false == \t\r\u\e ]]
++ curl --silent --location --connect-timeout 5 --max-time 4 https://api.sdkman.io/2/candidates/all
+ local fresh_candidates_csv=ant,asciidoctorj,ceylon,crash,gaiden,glide,gradle,grails,groovy,groovyserv,java,jbake,kotlin,kscript,lazybones,leiningen,maven,sbt,scala,spark,springboot,sshoogr,vertx,visualvm
+ fresh_candidates=("")
+ local fresh_candidates
+ cached_candidates=("")
+ local cached_candidates
+ [[ false == \t\r\u\e ]]
+ OLD_IFS='     
'
+ IFS=,
+ fresh_candidates=(${fresh_candidates_csv})
+ cached_candidates=(${SDKMAN_CANDIDATES_CSV})
+ IFS='         
'
+ __sdkman_echo_debug 'Local candidates: ant,asciidoctorj,ceylon,crash,gaiden,glide,gradle,grails,griffon,groovy,groovyserv,java,jbake,kobalt,kotlin,kscript,lazybones,leiningen,maven,sbt,scala,spark,springboot,sshoogr,vertx,visualvm'
+ [[ false == \t\r\u\e ]]
+ __sdkman_echo_debug 'Fetched candidates: ant,asciidoctorj,ceylon,crash,gaiden,glide,gradle,grails,groovy,groovyserv,java,jbake,kotlin,kscript,lazybones,leiningen,maven,sbt,scala,spark,springboot,sshoogr,vertx,visualvm'
+ [[ false == \t\r\u\e ]]
++ echo ant
++ tr '[:upper:]' '[:lower:]'
++ grep html
+ local detect_html=
+ [[ -n ant,asciidoctorj,ceylon,crash,gaiden,glide,gradle,grails,groovy,groovyserv,java,jbake,kotlin,kscript,lazybones,leiningen,maven,sbt,scala,spark,springboot,sshoogr,vertx,visualvm ]]
+ [[ -z '' ]]
+ local fresh_candidates_length=176
+ local cached_candidates_length=191
+ __sdkman_echo_debug 'Fresh and cached candidate lengths: 176 191'
+ [[ false == \t\r\u\e ]]
++ echo ant asciidoctorj ceylon crash gaiden glide gradle grails groovy groovyserv java jbake kotlin kscript lazybones leiningen maven sbt scala spark springboot sshoogr vertx visualvm ant asciidoctorj ceylon crash gaiden glide gradle grails griffon groovy groovyserv java jbake kobalt kotlin kscript lazybones leiningen maven sbt scala spark springboot sshoogr vertx visualvm
++ tr ' ' '\n'
++ sort
++ uniq -u
++ tr '\n' ,
+ local diff=griffon,kobalt,
+ ((  fresh_candidates_length > cached_candidates_length  ))
+ ((  fresh_candidates_length < cached_candidates_length  ))
bash: -1: substring expression < 0

[Dispader]

Two more important notes:

I noticed the difference in candidate count, and thought I might try an sdk flush candidates. This seems to have stopped the message that I need to update, but also seems to have caused some problems for commands involving candidates.

(~/.sdkman/etc){azazel}02:16am] sdk list groovy

Stop! groovy is not a valid candidate.

And second, I am on the... beta channel, I believe it's called?. Sorry— I'd forgotten that I was.

(~/.sdkman/etc){azazel}02:16am] sdk version

SDKMAN master+293

[marc0der]

Right, so let me give a little more context. We now require a monthly sdk update for this exact reason. If candidates are added or removed, the candidate cache found at ~/.sdkman/var/candidates is updated. By doing this, the sdk flush candidates is now obsoleted. A bug was recently raised where someone ran the flush command and the candidates were no longer refreshing.

The new update feature's spec can be found in update.feature. The scenario that is failing for you is:
Scenario: A candidate has been removed

With context out of the way, it seems like the new feature is misbehaving on your machine (the tests are passing, and running on bash).

[marc0der]

Right, I just did a functrace too and a diff against your output revealed that somehow yours fails in the following lines of code where mine doesn't:

+ ((  fresh_candidates_length > cached_candidates_length  ))
+ ((  fresh_candidates_length < cached_candidates_length  ))

Can you try restoring your candidate list as follows and let me know if/when you see this happen again?

$ curl -s https://api.sdkman.io/2/candidates/all > ~/.sdkman/var/candidates

It could possibly be some bash setting that you might have in your profile.

[Dispader]

Right you are! Sorry I muddied the waters by running sdk flush candidates. A new shell that I started this morning made it clear that var/candidates was lost

-bash: /Users/jake/.sdkman/var/candidates: No such file or directory
-bash: /Users/jake/.sdkman/var/candidates: No such file or directory

I ran the curl you suggested, and it seems to have restored the file.

(~){azazel}11:35am] curl -s https://api.sdkman.io/2/candidates/all > ~/.sdkman/var/candidates
(~){azazel}11:36am] cat ~/.sdkman/var/candidates 
ant,asciidoctorj,ceylon,crash,gaiden,glide,gradle,grails,groovy,groovyserv,java,jbake,kotlin,kscript,lazybones,leiningen,maven,sbt,scala,spark,springboot,sshoogr,vertx,visualvm

It does still look like I messed up my candidate configuration, though.

(~){azazel}11:35am] sdk list groovy

Stop! groovy is not a valid candidate.

D'you think I should re-install SDKman entirely, or is having this system in a strange state helpful for debugging this as a potential issue?

[marc0der]

Great stuff, although you shouldn't need to reinstall anything. Simply opening a new shell usually does the trick. If it doesn't, it means that you are using a login shell and should simply log out and back in to your session.

[xeraa]

I ran into the same issue (using the stable channel, didn't run sdk flush candidates), but curl -s https://api.sdkman.io/2/candidates/all > ~/.sdkman/var/candidates fixed it.

[obesga-tirant]

+1 working again 👍

[marc0der]

@xeraa @obesga-tirant could you guys please let me know what version of bash you are on please? bash --version should do the trick.

[obesga-tirant]

Sure

On macos 10.11.6 elcapitan

GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
Copyright (C) 2007 Free Software Foundation, Inc.

[xeraa]

macOS 10.13.4

≻ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.

[marc0der]

Thanks both for getting back so fast! So I think I've identified the problem. It looks like this only happens on bash 3.x. bash 4.x seems unaffected. I recommend that you all update to the latest version of bash (should be able to do that with brew), but I'll also fix it by letting it degrade gracefully. When bash 3.x is detected, we will _always_ get the new candidate list.

In addition, I will also remove the flush candidates command, as this is no longer relevant.

[obesga-tirant]

Ok, for me is fixed now
I've found instructions to update bash via homebrew; because apple won't upgrade bash
https://apple.stackexchange.com/questions/193411/update-bash-to-version-4-0-on-osx

As it is fixed for me, I won't upgrade bash at this moment and try to use sdkman, I've other scripts that go fine in my mac
(I think this is apple fault, but whatever)

Could be better to make a little fix (like if bas3 then abc else xyx) perhaps ?
I don't know the internals of sdkman and have no time (sorry), so feel free to ignore my request

Thanks for yor great job on sdkman !

[marc0der]

Yes, it is very sloppy of them to ship OSX with such ancient versions of bash. Bash 4.x superseded 3.x in 2009! I highly recommend you _do_ upgrade to 4.x, even from a security perspective.

In the meanwhile I've release a fix that will perform a brute force overwrite of the candidates cache, but will also issue a warning about the outdated version of bash. In the meanwhile, could I ask you to test this for me please? It's now available on the beta channel.

Would be great if @Dispader could also give his stamp of approval and close this issue if the problem is now resolved.

[jkwuc89]

I ran into this issue as well inside a Bash 3.2.57(1)-release shell on macOS 10.13.4 inside iTerm2 3.2.57(1)-release. I then opened the built in macOS terminal and inside there, sdk update worked.

[marc0der]

My experience was somewhat different with 3.2.57(1) always failing. The moment I upgrade to 4.4 it all worked fine. I took the following steps:

1. Spin up a brand new vagrant box of yosemite.
2. vagrant ssh into that box
3. Confirm that it's bash 3.x:

    vagrant$ bash -version
    GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin14)

1. Install SDKMAN, then delete some of it's candidates under ~/.sdkman/var/candidates
2. source ~/.sdkman/bin/sdkman-init.sh
3. Do an update and watch it fail:

    vagrant$ sdk update
    -bash: -1: substring expression < 0

1. Install bash 4.4 with brew
2. source ~/.sdkman/bin/sdkman-init.sh
3. Try doing another update

    vagrant$ sdk update
    Adding new candidates(s): gaiden,glide,gradle,grails,groovy
    Please open a new terminal now...

For me this is conclusive that bash 3.x was the problem and that 4.x fixes it.

@jkwuc89 not sure what you were seeing there but I was using ssh to get into the box so no terminal emulator involved.

[Dispader]

brew install bash did the trick for me, here. Thanks @marc0der !

IMHO, we can close this issue.

[marc0der]

So I released version 5.6.3 last night that contains the workaround for bash 3.x as many OSX users will be affected by this problem. It issues a warning that you should upgrade to bash 4.x, then does a brute force update of the candidates cache regardless of whether they changed. I've also removed the flush candidates command as it no longer makes sense.