let g:ale_sign_column_always = 1
I've been seeing signs stick around every now and then, and I finally know the answer.
The bug occurs when signs are placed on two places, lines are deleted, and then the signs are moved to lie on the same line by Vim before ALE next parses the signs, and expects there to be only one sign per line.
This shouldn't be too hard to add a test case for and fix.
I added a test case for this, and fixed it.
The bug will be in 1.5.x, so I'll create a patch release containing this bug fix this evening.