Argo: Argo CLI does not work with load balancer (e.g. AWS ALB) that do not support HTTP/2

Created on 27 Oct 2020 · 27Comments · Source: argoproj/argo

Summary

I followed the docs to use argo CLI with the access token.

API Request

It works with API.

> curl https://$HOST/api/v1/workflow-templates/$NAMESPACE -H "Authorization: $ARGO_TOKEN"
{"metadata":{"se ...}

CLI Request

It doesn't work with CLI.

> argo template list -n $NAMESPACE -v -e -k
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true true}  0x236c500 <nil> 0x236c560}"
FATA[0000] could not find a token

> argo template list -n $NAMESPACE -v --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 false false}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:32 rpc error: code = Unavailable desc = connection closed

> argo template list -n $NAMESPACE -v -e --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true false}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:43 rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

> argo template list -n $NAMESPACE -v -e -k --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true true}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:51 rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

Diagnostics

What Kubernetes provider are you using?

EKS 1.17.9

What version of Argo Workflows are you running?

> argo version
argo: latest+5eebce9.dirty
  BuildDate: 2020-10-19T20:59:29Z
  GitCommit: 5eebce9af4409da9de536f189877542dd88692e0
  GitTreeState: dirty
  GitTag: v2.11.6
  GoVersion: go1.15.3
  Compiler: gc
  Platform: darwin/amd64

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

bug

Source

posquit0

Most helpful comment

You can use the CLI in one of three modes:

# Kubernetes API Mode (default)

Requests are sent directly to the Kubernetes API. No Argo Server is needs. Large workflows and the workflow archive are not supported.

Use when you have direct access to the Kubernetes API, and don't need large workflow or workflow archive support.

If you're using instance ID (which is very unlikely), you'll need to set it:

    ARGO_INSTANCEID=your-instance-ic

# Argo Server GRPC Mode 

Requests are sent to the Argo Server API via GRPC (using HTTP/2). Large workflows and the workflow archive are supported. Network load-balancers that do not support HTTP/2 are not supported. 

Use if you do not have access to the Kubernetes API (e.g. you're in another cluster), and you're running the Argo Server using a network load-balancer that support HTTP/2.

To enable, set ARGO_SERVER. This is in the format  "host:port" - do not prefix with "http" or "https":

    ARGO_SERVER=localhost:2746 ;

If you're have transport-layer security (TLS) enabled (i.e. you are running "argo server --secure"):

    ARGO_SECURE=true

If your server is running with self-signed certificates. Do not use in production:

    ARGO_INSECURE_SKIP_VERIFY=true 

If your server is behind an ingress with a path (you'll be running "arg server --bashhref /...) or "BASH_HREF=/... argo server""):

    ARGO_PATH=/argo

By default, the CLI uses your KUBECONFIG to determine default for ARGO_TOKEN and ARGO_NAMESPACE. You probably don't want this, so to prevent it:

    KUBECONFIG=/dev/null

You will then need to set:

    ARGO_NAMESPACE=argo 

And 

    ARGO_TOKEN='Bearer ****' ;# Should always start with "Bearer " or "Basic ". 

# Argo Server HTTP Mode

As per GRPC mode, but uses HTTP. Can be used with ALB that does not support HTTP/2. The command "argo logs --since-time=2020...." will not work (due to time-type).

Use this when your network load-balancer does not support HTTP/2.

Use the same configuration as GRPC mode, but also set:

    ARGO_HTTP=true

alexec on 2 Nov 2020

👍2

All 27 comments

why do you have this?

{{$HOST:443 true true}

looks like some envvar did not get expanded in some script?

alexec on 27 Oct 2020

why do you have this?
{{$HOST:443 true true} 
looks like some envvar did not get expanded in some script?

Ah.. I just redacted the host with $HOST in this PR. I used my company internal host.

posquit0 on 27 Oct 2020

Try setting ARGO_TOKEN instead of using --token.

alexec on 27 Oct 2020

Same error... 😢

> export ARGO_TOKEN="Bearer ..."

> curl https://${redacted}/api/v1/workflow-templates/$NAMESPACE -H "Authorization: $ARGO_TOKEN"
{"metadata":{"...,"arguments":{}}}]}

> argo template list -n $NAMESPACE -v -e -k
DEBU[0000] CLI version                                   version="{latest+5eebce9.dirty 2020-10-19T20:59:29Z 5eebce9af4409da9de536f189877542dd88692e0 v2.11.6 dirty go1.15.3 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{${redacted}:443 true true}  0x2371a20 <nil> 0x2371a80}"
I1028 02:04:50.686822   48855 loader.go:375] Config loaded from file:  /Users/posquit0/.kube/config
2020/10/28 02:04:50 rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

posquit0 on 27 Oct 2020

Do you have any proxy in between?

sarabala1979 on 27 Oct 2020

Argo Server is behined of AWS ALB load balancer. Is it the problem?

posquit0 on 28 Oct 2020

Oh, it works without ALB. Hmmm, how can I make this work with ALB?
Is there any idea?

> sudo kubefwd svc -n argo

> export ARGO_SERVER=argo-server:2746

> curl https://argo-server:2746/api/v1/workflow-templates/$NAMESPACE -H "Authorization: $ARGO_TOKEN" -k
{"metadata":{"selfLink":"....}

> argo template list -n $NAMESPACE -v -e -k
DEBU[0000] CLI version                                   version="{latest+5eebce9.dirty 2020-10-19T20:59:29Z 5eebce9af4409da9de536f189877542dd88692e0 v2.11.6 dirty go1.15.3 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{argo-server:2746 true true}  0x2371a20 <nil> 0x2371a80}"
I1028 14:58:01.603415   69242 loader.go:375] Config loaded from file:  /Users/posquit0/.kube/config
NAME
xxxx-workflow

posquit0 on 28 Oct 2020

I found that this issue is related to gRPC support of ALB. I thought argo CLI communicates via HTTP API.
Is there no option for argo CLI to use HTTP REST API ?
Like argo-cd's --grpc-web options

posquit0 on 28 Oct 2020

You need HTTP2 support for workflows. I don't think you're the first person to encounter this problem, and I believe others have fixed it. Probably can find answer in other issues.

alexec on 28 Oct 2020

OK. I believe this is a problem with AWS ALBs. They do not support HTTP/2.

alexec on 28 Oct 2020

https://github.com/argoproj/argo-cd/pull/1077

alexec on 28 Oct 2020

I've pushed argoproj/argocli:http for testing. You'll need to set ARGO_HTTP=true to use it.
I'll upload a Mac binary shortly.

alexec on 30 Oct 2020

😄1

@alexec Thank you for fast response! Where is a mount point for .kube/config in argoproj/argocli Docker image? Following error is occured when I tried with arogproj/argocli:http image.

> docker run --rm -it -e ARGO_TOKEN=$ARGO_TOKEN -e ARGO_SERVER=$ARGO_SERVER -e ARGO_HTTP=true argoproj/argocli:http template list -v -e -k
DEBU[0000] CLI version                                   version="{v2.10.0-rc1 2020-10-29T23:09:22Z 26860503d4af6e1d6dd1978c9049958f8c1f022f v2.10.0-rc1 clean go1.13.15 gc linux/amd64}"
DEBU[0000] Client options                                opts="(argoServerOpts=(url=${redacted}:443,path=,secure=true,insecureSkipVerify=true,http=true),instanceID=)"
FATA[0000] invalid configuration: no configuration has been provided

Btw, I think many commands should be executed without kubeconfig when user use the access token. What do you think about this?

Related to

https://github.com/argoproj/argo/issues/3943

posquit0 on 30 Oct 2020

You should not need KUBECONFIG. You do need ARGO_SERVER, ARGO_HTTP=true, and ARGO_TOKEN.

alexec on 30 Oct 2020

Is there another reason for FATA[0000] invalid configuration: no configuration has been provided message above?

posquit0 on 30 Oct 2020

That error occurs when trying to read KUBECONFIG. Setting ARGO_TOKEN should prevent this.

alexec on 30 Oct 2020

As you can see above, I didn't configure KUBECONFIG in the command. And I configured ARGO_SERVER, ARGO_HTTP, ARGO_TOKEN.

> docker run --rm -it \
  -e ARGO_TOKEN=$ARGO_TOKEN \
  -e ARGO_SERVER=$ARGO_SERVER \
  -e ARGO_HTTP=true \
  argoproj/argocli:http template list -v -e -k
DEBU[0000] CLI version                                   version="{v2.10.0-rc1 2020-10-29T23:09:22Z 26860503d4af6e1d6dd1978c9049958f8c1f022f v2.10.0-rc1 clean go1.13.15 gc linux/amd64}"
DEBU[0000] Client options                                opts="(argoServerOpts=(url=${redacted}:443,path=,secure=true,insecureSkipVerify=true,http=true),instanceID=)"
FATA[0000] invalid configuration: no configuration has been provided

posquit0 on 1 Nov 2020

Can you try -e KUBECONFIG=/dev/null?

alexec on 1 Nov 2020

It doesn't work 😢

> docker run --rm -it \
  -e ARGO_TOKEN=$ARGO_TOKEN \
  -e ARGO_SERVER=$ARGO_SERVER \
  -e ARGO_HTTP=true \
  -e KUBECONFIG=/dev/null \
  argoproj/argocli:http template list -v -e -k

DEBU[0000] CLI version                                   version="{v2.10.0-rc1 2020-10-29T23:09:22Z 26860503d4af6e1d6dd1978c9049958f8c1f022f v2.10.0-rc1 clean go1.13.15 gc linux/amd64}"
DEBU[0000] Client options                                opts="(argoServerOpts=(url=${redacted}:443,path=,secure=true,insecureSkipVerify=true,http=true),instanceID=)"
I1102 01:25:40.374572       1 loader.go:375] Config loaded from file:  /dev/null
FATA[0000] invalid configuration: no configuration has been provided

posquit0 on 2 Nov 2020

sorry, I forgot you must state ARGO_NAMESPACE too:
docker run --rm -it \
-e ARGO_TOKEN=ARGO_TOKEN \
-e ARGO_SERVER=ARGO_SERVER \
-e ARGO_HTTP=true -e ARGO_NAMESPACE=argo \
-e KUBECONFIG=/dev/null \
argoproj/argocli:http template list -v -e -k

alexec on 2 Nov 2020

You can use the CLI in one of three modes:

# Kubernetes API Mode (default)

Requests are sent directly to the Kubernetes API. No Argo Server is needs. Large workflows and the workflow archive are not supported.

Use when you have direct access to the Kubernetes API, and don't need large workflow or workflow archive support.

If you're using instance ID (which is very unlikely), you'll need to set it:

    ARGO_INSTANCEID=your-instance-ic

# Argo Server GRPC Mode 

Requests are sent to the Argo Server API via GRPC (using HTTP/2). Large workflows and the workflow archive are supported. Network load-balancers that do not support HTTP/2 are not supported. 

Use if you do not have access to the Kubernetes API (e.g. you're in another cluster), and you're running the Argo Server using a network load-balancer that support HTTP/2.

To enable, set ARGO_SERVER. This is in the format  "host:port" - do not prefix with "http" or "https":

    ARGO_SERVER=localhost:2746 ;

If you're have transport-layer security (TLS) enabled (i.e. you are running "argo server --secure"):

    ARGO_SECURE=true

If your server is running with self-signed certificates. Do not use in production:

    ARGO_INSECURE_SKIP_VERIFY=true 

If your server is behind an ingress with a path (you'll be running "arg server --bashhref /...) or "BASH_HREF=/... argo server""):

    ARGO_PATH=/argo

By default, the CLI uses your KUBECONFIG to determine default for ARGO_TOKEN and ARGO_NAMESPACE. You probably don't want this, so to prevent it:

    KUBECONFIG=/dev/null

You will then need to set:

    ARGO_NAMESPACE=argo 

And 

    ARGO_TOKEN='Bearer ****' ;# Should always start with "Bearer " or "Basic ". 

# Argo Server HTTP Mode

As per GRPC mode, but uses HTTP. Can be used with ALB that does not support HTTP/2. The command "argo logs --since-time=2020...." will not work (due to time-type).

Use this when your network load-balancer does not support HTTP/2.

Use the same configuration as GRPC mode, but also set:

    ARGO_HTTP=true

alexec on 2 Nov 2020

👍2

I succeed without KUBECONFIG=/dev/null.

$ docker run --rm -it \
  -e ARGO_TOKEN=$ARGO_TOKEN \
  -e ARGO_SERVER=$ARGO_SERVER \
  -e ARGO_HTTP=true \
  -e ARGO_NAMESPACE=argo \
  argoproj/argocli:http template list -v -e -k
DEBU[0000] CLI version                                   version="{v2.10.0-rc1 2020-10-29T23:09:22Z 26860503d4af6e1d6dd1978c9049958f8c1f022f v2.10.0-rc1 clean go1.13.15 gc linux/amd64}"
DEBU[0000] Client options                                opts="(argoServerOpts=(url=${redacted},path=,secure=true,insecureSkipVerify=true,http=true),instanceID=)"
DEBU[0000] curl -X GET -H 'Authorization: ******' -d '' '${redacted}/api/v1/workflow-templates/argo?'
NAME
xxxxxx-workflow

Btw, the AWS released a new feature of ALB to support HTTP2 and gRPC. 🎊

https://aws.amazon.com/about-aws/whats-new/2020/10/application-load-balancers-enable-grpc-workloads-end-to-end-http-2-support/

posquit0 on 3 Nov 2020

I saw that yesterday! Have you tried it out?

alexec on 3 Nov 2020

Yup, it works with ALB.

Target Group
- Type: IP
- Protocol: HTTPS
- Protocol Version: gRPC
- Port: 2746
ALB
- HTTPS Listener (443)

$ docker run --rm -it \
  -e ARGO_TOKEN=$ARGO_TOKEN \
  -e ARGO_SERVER=$ARGO_SERVER \
  -e ARGO_NAMESPACE=argo \
  argoproj/argocli:v2.11.5 template list -v -e
DEBU[0000] CLI version                                   version="{v2.11.5 2020-10-15T22:32:48Z 076bf89c4658adbd3b96050599f81424d1b08d6e v2.11.5 clean go1.13.4 gc linux/amd64}"
DEBU[0000] Client options                                opts="{{${redacted}:443 true false}  0x171e1b0 <nil> 0x171e200}"
NAME
xxxxxxx-workflow

posquit0 on 3 Nov 2020

excellent news!

alexec on 3 Nov 2020

😄1

I've created binaries to test:

https://github.com/argoproj/argo/releases/tag/http-rc1

alexec on 5 Nov 2020

😄1

It works! Expect a next version! :)

posquit0 on 6 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings