Argo-cd: "bind: address already in use" after switching to gRPC-Web

[glerchundi]

Describe the bug

After switching from an older version of ArgoCD to 0.12 we're encountering issues with the cli when trying to sync & wait successive applications on a releasing step. Our script looks like this:

#!/bin/bash

set -euo pipefail

# parameters
server=myserver.com

# download latest argocd cli.
curl -sSL -o /usr/local/bin/argocd https://${server}/download/argocd-linux-amd64
chmod +x /usr/local/bin/argocd

# tell argocd to sync apps.
apps=(${APPS:-})
for app in "${apps[@]}"; do
    echo "Applying changes in '${app}'..."
    argocd app sync ${app} --grpc-web --server ${server}
    argocd app wait ${app} --grpc-web --server ${server}
done

To Reproduce

Cannot say, as it fails randomly with this error:

10:34:20 Applying changes in 'edge-proxy'...
10:34:20 time="2019-04-12T10:34:20Z" level=fatal msg="Failed to establish connection to myserver.com:443: listen unix /tmp/argocd-1555065260.sock: bind: address already in use"
10:34:20 make: *** [release] Error 1
10:34:20 Makefile:33: recipe for target 'release' failed

Expected behavior

Finish successfully :)

Thanks for such a cool project btw!

[glerchundi]

I think i tracked down the issue. You are creating a listener here which is never being closed. Perhaps making a blind Close inside the goroutine here would do the trick.

Also, as the socket is timestamped making a sleep 1 between sync/wait's should work hehe :)

[alexmt]

Thank you for investigation. Working on fix

[glerchundi]

Hei, i’m ok with what you proposed as it will fix our issue but it seems
like a masking solution instead of fixing the problem in the first place.

I dont know if the unclosed socks are going to accum on the OS and i dont
really know how does the OS behave purging orphan sockets but it seems to a
good idea to close the listener as this is going to remove the underlying
socket automatically.

As per doc. on the stdlib:

The default behavior is to unlink the socket file only when package net
created it. That is, when the listener and the underlying socket file were
created by a call to Listen or ListenUnix, then by default closing the
listener will remove the socket file. but if the listener was created by a
call to FileListener to use an already existing socket file, then by
default closing the listener will not remove the socket file.

Just to let you know and thanks for being this quick!! 🙏

On Fri, 12 Apr 2019 at 23:25, Alexander Matyushentsev <
[email protected]> wrote:

Closed #1451 https://github.com/argoproj/argo-cd/issues/1451 via #1455
https://github.com/argoproj/argo-cd/pull/1455.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/argoproj/argo-cd/issues/1451#event-2273607130, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACIPlohknxmMm8VyZRTClWqBqnP06h5eks5vgPmxgaJpZM4cr10M
.

[alexmt]

Hey, thank you for checking the fix. I very appreciate it!

While debugging I've discovered that listener returned in https://github.com/argoproj/argo-cd/blob/cd25c4b3c9d0902d121c056de3e59816dec2c7ef/pkg/apiclient/grpcproxy.go#L107 is closed twice :) in useGRPCProxy method:

c.proxyServer.Stop() // apparently grpc server closes all listeners it is serving
err := c.proxyListener.Close() // listener already closed here which causes warning message #1420

So I think that listener/socket is actually closed but bind: address already in use error still might happen if two argocd command are executed in the same second. I was able to reproduce it using

argocd app list & argocd app list

Am I missing something?

[alexmt]

I manually verified that socket files are deleted, but of cours this does not prove it will be deleted in any environment. Do you think it make sense to try to delete file manually after listener is closed?

[glerchundi]

It makes perfect sense, thanks for clarifying!

Thanks for working in a such incredible product which is making our lifes
much much easier!

On Sat, 13 Apr 2019 at 00:56, Alexander Matyushentsev <
[email protected]> wrote:

I manually verified that socket files are deleted, but of cours this does
not prove it will be deleted in any environment. Do you think it make sense
to try to delete file manually after listener is closed?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/argoproj/argo-cd/issues/1451#issuecomment-482747088,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACIPllqUYWOqzZpcX1CxofhzD1mK-9MYks5vgQ8sgaJpZM4cr10M
.

[alexmt]

Thank you for the feedback!