Packages: Misc. bash/shell issues, some RFEs

[okdana]

Hey there. I've been collecting some cases of erroneous or inconsistent behaviour in the new shell syntax definition. I've also included some RFEs and zsh-compatibility issues near the bottom, if you're interested. (I'd understand if zsh compatibility is not a major concern — most of the stuff i mention here is low-hanging fruit, though, for whatever that's worth.)

Edit: I've broken the zsh-specific stuff out into a separate block so it's easier to distinguish it from stuff that affects bash too/only.

#!/usr/bin/env bash

# Unescaped hash inside parameter expansion breaks file
: ${foo# #} # <- second # interpreted as comment
} # fix highlight

# Open-bracket in glob/regex character class breaks file
[[ a == [abc[]* ]] ]] # <- remove extra brackets to break file
: ${foo//[abc[]/x} ]] } # <- remove extra brackets/brace to break file

# Escaped space in glob/regex character class breaks file
if [[ ' foobar' == [\ ]foo* ]]; then ]] # <- remove extra brackets to break file
  :
fi

# `!` operator only highlighted immediately after `if`
if ! foo || ! bar; then :; fi
while ! foo; do :; done
! false && foo

# Comment not highlighted as such following redirect to fd
foo >&2 # comment

# Closing sub-shell parenthesis not highlighted as such when immediately
# following a variable assignment
(foo=bar)

# Function definitions are not highlighted as such inside command substitutions,
# and the closing `)` is interpreted as terminating the substitution
func() {
} # OK
foo=$(
  func() {
  } # broken
)

# Redirects not highlighted properly after variable assignment (legal in both
# bash and zsh)
LC_ALL=C 2> /dev/null

# Compound commands after `coproc` key word not highlighted as such (braces
# highlighted as brace expansion)
coproc { foo bar }
coproc ( foo bar )

# Unquoted array and assocation values not highlighted consistently with scalar
# values during assignment
scalar=foo
array=( foo bar )

# Association keys and punctuation not highlighted consistently during
# assignment
assoc[foo]=bar
assoc=( [foo]=bar )

# Unquoted array and association members highlighted only as 'function call'
# when assignment is performed as part of a call to `declare`, &al.
declare array=( foo bar baz )
declare assoc=( [foo]=bar )

# Variable names given to `unset` are not highlighted as such (note that the
# presence of `-f` signifies that the names belong to functions rather than
# variables, but highlighting them as variables would at least make them
# consistent with `declare`, &al.)
unset foo
unset -f foo

# Parameter expansion isn't handled consistently with `declare`, &al.
declare -a $foo
local -a $foo
typeset -a $foo
readonly -a $foo
declare -h $foo # Works if the option is unknown though

# Unknown options to `declare`, &al., are highlighted as variables (zsh and
# forwards compatibility — `-` is not a legal character for bash variables
# anyway)
declare -h foo
local -h foo
typeset -h foo
readonly -h foo

# `+`-prefixed options to `declare`, &al., highlighted as variables (zsh
# compatibility — `+` is not a legal character for bash variables anyway)
declare +x foo
local +x foo
typeset +x foo
readonly +x foo

# No way to distinguish between conditional-expression operators like `-n`,
# `-e`, and `-gt` (which are part of the shell's grammar) and generic options to
# commands (can there be some kind of group scope for brackets?)
[[ -e foo/bar ]]
[[ 2 -gt 1 ]]
foo -e -gt

# Also, option highlighting inside double brackets is not entirely accurate
[[ '-e' == -e ]] # Second -e is a string literal
[[ -e == -e ]] # zsh is even smart enough to allow this

# 'Bare' variables not highlighted as such within arithmetic constructs
(( foo > 1 ))

zsh compatibility issues:

#!/usr/bin/env zsh

# Parentheses of anonymous functions highlighted as compound-command (sub-shell)
# punctuation, and braces highlighted as brace expansion (zsh compatibility —
# empty sub-shells not legal in bash anyway)
() { : }

# Code in `always` block not highlighted correctly (zsh compatibility — not
# legal syntax in bash)
{
  : ...
} always {
  # broken now
  if :; then
    : ...
  fi
}

# 'Bare' variable with `#` expansion flag in arithmetic constructs highlighted
# as comment (zsh compatibility — not legal syntax in bash)
: $(( #foo ))
)) # fix highlight

# 'Simple' expansion flags break expansion highlighting (zsh compatibility —
# not legal syntax in bash, but seems like it'd be harmless to just match all
# punctuation characters here)
: ${#foo} # Valid in bash and zsh, highlighted OK
: ${+foo} # zsh only, broken
: ${~foo} # zsh only, broken
: ${=foo} # zsh only, broken
: ${^foo} # zsh only, broken

# Glob qualifiers break array assignments, sub-shells, command substitutions,
# &c. (zsh compatibility — not legal syntax in bash)
foo=( $dir/*.ext(-.) )
( rm -f $dir/*.ext(-.) )

# Certain characters within 'complex' expansion flags break variable
# highlighting — can characters between parentheses be skipped or ignored here?
# (zsh compatibility — not legal syntax in bash)
: ${(j:,:)array}
: ${(j<,>)array}

# Arithmetic function parentheses highlighted as grouping operators (zsh
# compatibility — not legal syntax in bash)
(( abs(1 - 5) + 3 ))

# `.` not highlighted in floats (zsh compatibility — not legal syntax in bash)
(( 1.23 ))

[FichteFoll]

A checklist for the issues in this report still outstanding would be useful, as they will hardly be solved all at once.

[rwols]

I compiled all issues into a checklist.

• [x] Unescaped hash inside parameter expansion breaks file #1372
• [x] Open-bracket in glob/regex character class breaks file #1383
• [x] Escaped space in glob/regex character class breaks file #1383
• [ ] ! operator only highlighted immediately after if
• [x] Comment not highlighted as such following redirect to fd #1369
• [x] Closing sub-shell parenthesis not highlighted as such when immediately following a variable assignment #1371
• [x] Function definitions are not highlighted as such inside command substitutions, and the closing ) is interpreted as terminating the substitution #1370
• [x] Redirects not highlighted properly after variable assignment (legal in both bash and zsh) #1369
• [ ] Compound commands after coproc key word not highlighted as such (braces highlighted as brace expansion)
• [ ] Unquoted array and assocation values not highlighted consistently with scalar values during assignment
• [ ] Association keys and punctuation not highlighted consistently during assignment
• [ ] Unquoted array and association members highlighted only as 'function call' when assignment is performed as part of a call to declare, &al.
• [ ] Variable names given to unset are not highlighted as such (note that the presence of -f signifies that the names belong to functions rather than variables, but highlighting them as variables would at least make them consistent with declare, &al.)
• [ ] Parameter expansion isn't handled consistently with declare, &al.
• [ ] Unknown options to declare, &al., are highlighted as variables (zsh and forwards compatibility — - is not a legal character for bash variables anyway)
• [ ] +-prefixed options to declare, &al., highlighted as variables (zsh compatibility — + is not a legal character for bash variables anyway)
• [ ] No way to distinguish between conditional-expression operators like -n, -e, and -gt (which are part of the shell's grammar) and generic options to commands (can there be some kind of group scope for brackets? [yes, good idea])
• [ ] Also, option highlighting inside double brackets is not entirely accurate [not sure if this is feasible]
• [ ] 'Bare' variables not highlighted as such within arithmetic constructs

Z-Shell compatibility:

• [ ] Parentheses of anonymous functions highlighted as compound-command (sub-shell) punctuation, and braces highlighted as brace expansion (zsh compatibility — empty sub-shells not legal in bash anyway)
• [ ] Code in always block not highlighted correctly (zsh compatibility — not legal syntax in bash)
• [ ] 'Bare' variable with # expansion flag in arithmetic constructs highlighted as comment (zsh compatibility — not legal syntax in bash)
• [ ] 'Simple' expansion flags break expansion highlighting (zsh compatibility — not legal syntax in bash, but seems like it'd be harmless to just match all punctuation characters here)
• [ ] Glob qualifiers break array assignments, sub-shells, command substitutions, &c. (zsh compatibility — not legal syntax in bash)
• [ ] Certain characters within 'complex' expansion flags break variable highlighting — can characters between parentheses be skipped or ignored here? (zsh compatibility — not legal syntax in bash)
• [ ] Arithmetic function parentheses highlighted as grouping operators (zsh compatibility — not legal syntax in bash)
• [ ] . not highlighted in floats (zsh compatibility — not legal syntax in bash)

[michaelblyons]

Expanding on the array bit, newlines are valid in assignment. Here's an associative array example.

declare -a owners=(
    # dogs
    [susan]=labrador
    [tim]=poodle

    # cats
    [terry]=tabby
    [lynn]=siamese
)

# oops, forgot jim.
owners+=([jim]=parakeet)

Based on experimentation with some things from an LJ post.