A summary:

@PEZ I'd like to attempt to fix this, just letting you know.
Nice! I now noticed that after a Grow Selection the next backward selection works. Maybe that is a clue.
Just an update here. I believe I have the fix for this and will submit a PR probably later today.
Right, so with your input it seems this got fixed in #507, we'll see when we have tested it some.
This has been fixed. I just verified.