Argo: downloading artifact from s3 in ui, timed out waiting for condition

[haghabozorgi]

Checklist:

• [x] I've included the version.
• [x] I've included reproduction steps.
• [] I've included the workflow YAML.
• [x] I've included the logs.

What happened:
Installed the latest 2.5.0rc7 via install.yaml on eks 1.14 and added to the install.yaml the diff shown in the output below, so the archivedLogs and s3 config are enabled (workflow-controller-configmap):

336,344d322
< data:
<   config: |
<     artifactRepository:
<       archiveLogs: true
<       s3:
<         bucket: "example-argo"
<         keyPrefix: "example"
<         endpoint: "s3.amazonaws.com"
< 

to gain ui access on localhost to the ui in kubernetes
kubectl port-forward svc/argo-server 2746:2746 -n argo

run a basic hello world workflow via argo cli and the workflow completes as expected, and clicking on artifacts link in ui shows main-logs object as expected, but when you click to download the actual artfact in the ui, the browser eventually returns a "timed out waiting on condition"

What you expected to happen:
I expect clicking on the link to download the requested artifact.

How to reproduce it (as minimally and precisely as possible):
install.yaml with a s3 config similar to above and run any workflow, and then try and download the resulting main-log artifact.

Logs
argo-server log shows:

time="2020-01-31T23:02:06Z" level=info msg="S3 Load path: artifact368826374, key: example/local-script-gd5zj/local-script-gd5zj/main.log"
time="2020-01-31T23:02:06Z" level=info msg="Creating minio client s3.amazonaws.com using IAM role"
time="2020-01-31T23:02:06Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: example-argo, key: example/local-script-gd5zj/local-script-gd5zj/main.log) to artifact368826374"
time="2020-01-31T23:02:06Z" level=warning msg="Failed get file: Get https://s3.amazonaws.com/example-argo/?location=: x509: certificate signed by unknown authority"

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

@sarabala1979 before I investigate - have you seen similar before please?

[ddseapy]

generally this kind of error is because the CA that signed s3.amazonaws.com isn't being used in the request. Depending on your situation & security requirements (https://serverfault.com/questions/444186/is-it-safe-to-use-s3-over-http-from-ec2-as-opposed-to-https), this can likely be fixed by using insecure: true under your s3: configuration section.

Otherwise, it would appear that argo needs to have the CA when making the s3 request. I haven't looked at the code or docker image to know more.

[ddseapy]

It looks like the image is from scratch (https://github.com/argoproj/argo/blob/master/Dockerfile#L87-L91), which wouldn't have CA's by default. I also don't see them COPY'd in.

So unless the CA is built into the binary (which would seem odd), then I'm guessing no certs exist in the docker. You can always create a secret and mount them into the container and set AWS_CA_BUNDLE env var - requiring only yaml changes.

All this said, I'm guessing this worked previously so I'm probably missing how they got in. It's also possible they exist in the container, but are out of date.

[haghabozorgi]

@ddseapy thanks for your suggestions. I tried with insecure: true, i get http: named cookie not present in browser window, and it does not seem to log anything on the argo-server nor the workflow-controller. So I tried to remove the insecure: true line to revert back and now i get "http: named cookie not present" message in browser even after i kubectl delete -f install.yaml and reinstall, and submit a new workflow and try and download that new artifact.

I assume the secret should have the contents of /etc/ssl/certs/ca-certificates.crt? I could give that a try, but I am starting to suspect there is something else going on (as well)? Thoughts welcome.

[ddseapy]

@haghabozorgi I am hitting the same http: named cookie not present error with insecure: true on rc7. This is a bug/regression, it worked as recent as 2.4.3. @alexec let me know if you want me to create a separate ticket for this. The artifacts (both logs and other output artifacts) are in s3 and the workflow succeeds.

Yes, that's generally where ca certs are at on machines. If AWS_CA_BUNDLE points to wherever that file is within the docker then i'd suspect you wouldn't see that error. A configmap might be a bit easier than a secret actually to get it into the docker. But I agree that this would be a quick hack to get past the error. One of the argo maintainers can probably point to where that cert is supposed to come from.

[haghabozorgi]

@ddseapy is there extra config needed to get the artifact store setup in 2.4.3? I was trying to confirm if this is working in 2.4.3 but if i insert the same configmap snippet i see no artifacts list in the archive tab.

[tcolgate]

One quite way of doing this is to add the following to the argo-server deployment:

      containers:
      - volumeMounts:
        - mountPath: /etc/ssl/certs/ca-certificates.crt
          name: ssl-certs
          readOnly: true

      volumes:
      - hostPath:
          path: /etc/ssl/certs/ca-certificates.crt
          type: ""
        name: ssl-certs

[ddseapy]

@haghabozorgi I don't think the config is different. hopefully @tcolgate 's fix works for you.

[tcolgate]

The executor probably has a valid system CA as it's build on a debian image. the argocli container is more stripped down.

[alexec]

I think this is probably a bug and that the argo-server image may need to be built on the same base as the argocli. This is not straight forward, I'll own it.

[ddseapy]

@alexec are you also testing with insecure: false or should I make a separate ticket? the certificate issue wouldn't fix that.

[alexec]

I'm working of a fix that will change from scratch - what would be useful is for someone to test it for me.

[alexec]

@tcolgate thoughts on a solution

• Option 1 - update the manifests to include the example you provided?
• Option 2 - use the same debian slim image as the executor?
• Option 3 - add certs during docker build

@jessesuen - thoughts on this please?

[alexec]

https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b

[haghabozorgi]

@ddseapy when I add the volume mount and volume snippet per @tcolgate's suggestion, I still get http named cookie not present when trying to download the artifact from the ui

[alexec]

I don't have access to S3 is a way to reliably test this. I've create a PR, would you be able to use the Dockefile there to try it out?

[ddseapy]

@haghabozorgi i'm afraid I'm not sure what the cause of that issue is. I made the other ticket to track that.

[haghabozorgi]

@alexec I am building from your repo now, but not sure if i can test properly given the issue I mentioned above. I assume i will see the same http named cookie not present message?

[ddseapy]

@haghabozorgi yes. if you deploy without @tcolgate's fix, and see "http named cookie not present" instead of the original error "x509: certificate signed by unknown authority", then the PR works.

[haghabozorgi]

@alexec using the image from your docker file results in http named cookie not present message.

[alexec]

Ok, but we know that is a good fix for the certs.

[haghabozorgi]

@alexec based on @ddseapy comments it seems the http named cookie not present is a separate issue, but that seems to be the case whether insecure flag is set to true or false. The end result being the user is still not able to download artifacts from ui.

[alexec]

Ok - should we close this issue once the PR is merged?

[ddseapy]

i'ts up to ya'll. if you want to leave this open, i'll close the other one. if you want to close this one, i'll leave the other one open.

[ddseapy]

@alexec @tcolgate have you been able to download artifacts through the UI with rc7? If so, I'm curious why your setup might be different from @haghabozorgi and mine.

[alexec]

I have, but we're using MinIO over HTTP.

[ddseapy]

I use minio over http as well.

[alexec]

Can you please check you have secrets set-up?

---
apiVersion: v1
data:
  config: |
    artifactRepository:
      archiveLogs: true
      s3:
        bucket: my-bucket
        endpoint: minio:9000
        insecure: true
        accessKeySecret:
          name: my-minio-cred
          key: accesskey
        secretKeySecret:
          name: my-minio-cred
          key: secretkey
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    app: minio
  name: my-minio-cred
stringData:
  accesskey: admin
  secretkey: password
type: Opaque

[ddseapy]

as far as i can tell it's ok.

apiVersion: v1
data:
  config: |
    containerRuntimeExecutor: docker
    artifactRepository:
      archiveLogs: true
      s3:
        accessKeySecret:
          key: accesskey
          name: ddseapy-minio
        secretKeySecret:
          key: secretkey
          name: ddseapy-minio
        bucket: ds-argo-artifacts
        endpoint: ddseapy-minio
        insecure: true
        region: us-east-1
    metricsConfig:
      enabled: true
      path: /metrics
      port: 8080
    persistence:
      archive: true
      connectionPool:
        maxIdleConns: 100
        maxOpenConns: 0
      nodeStatusOffLoad: true
      postgresql:
        host: ddseapy-postgresql
        port: 5432
        database: argo
        tableName: argo_workflows
        userNameSecret:
          name: ddseapy-argo-workflow-controller
          key: postgresqlUsername
        passwordSecret:
          name: ddseapy-argo-workflow-controller
          key: postgresqlPassword
kind: ConfigMap
metadata:
  name: ddseapy-argo-workflow-controller
  namespace: ddseapy
---
apiVersion: v1
data:
  accesskey: REDACTED
  secretkey: REDACTED
kind: Secret
metadata:
  labels:
    app: minio
  name: ddseapy-minio
  namespace: ddseapy
type: Opaque

[alexec]

@ddseapy I redacted your paste as you shared un-encrypted credentials. If this is a production system, then you should immediately change your password.

[alexec]

I'm wondering where this error is coming from - can you open your browser console and share the HTTP request and response please?

[ddseapy]

no, that is simply admin password base64 encoded.

[alexec]

Oh - what auth mode are you using? Server?

[alexec]

Ok. I've reproduced this.

[alexec]

Fix implemented.

[haghabozorgi]

@alexec does your fix address downloading from s3 or is it just for the minio over http use case?

[ddseapy]

minio is an s3-compatible api. while, i don't have the ability to test s3, and it sounds like neither does @alexec it almost certainly fixes the error for s3 as well..

[haghabozorgi]

@alexec can we please re-open? I am testing install.yaml from master with same s3 snippet in my original comment on this issue and am still seeing http: named cookie not present in browser when I try and download main-logs for example. the url in the browser show http://localhost:2746/artifacts/argo/local-script-j7wg9/local-script-j7wg9/main-logs?Authorization=null

[alexec]

If bug persists - please re-open.

[ddseapy]

@haghabozorgi master install.yaml points to latest docker tag. It might be that the docker wasn't on dockerhub yet. Do you mind setting the docker tags to 2.5.0-rc8 in the yaml just to make sure.

With rc8 I am able to download logs, though all other artifacts are downloaded as .gz, when they are .tgz so I have to rename them after downloading. I'll open a separate ticket.

[alexec]

The "named cookie" error should be fixed in rc8.

[haghabozorgi]

@alexec @ddseapy thank you both for all your efforts, sincerely appreciate it. confirmed, it is working with rc8. I think we can close this.

[alexec]

Yay!