Velero: not able to fetch logs

[adityagu0910]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
when I run command velero backup logs whole-cluster-backup-04092019171429
it gives below error

[root@75 tools]# velero backup logs whole-cluster-backup-04092019171429
An error occurred: file not found

while velero describe works fine and velero backup create also works fine.
[root@75 tools]# velero backup describe whole-cluster-backup-04092019171429
Name: whole-cluster-backup-04092019171429
Namespace: velero
Labels: velero.io/storage-location=default
Annotations:

Phase: Completed

Namespaces:
Included: *
Excluded:

Resources:
Included: *
Excluded:
Cluster-scoped: auto

Label selector:

Storage Location: default

Snapshot PVs: auto

TTL: 720h0m0s

Hooks:

Backup Format Version: 1

Started: 2019-09-04 17:14:35 -0400 EDT
Completed: 2019-09-04 17:19:56 -0400 EDT

Expiration: 2019-10-04 17:14:35 -0400 EDT

Persistent Volumes:

Restic Backups (specify --details for more information):
Completed: 16

What did you expect to happen:

It should show the complete logs of that backup
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):
  velero version
  Client:
  Version: v1.1.0
  Git commit: a357f21aec6b39a8244dd23e469cc4519f1fe608
  Server:
  Version: v1.1.0
• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3+icp-ee", GitCommit:"5f4327948913b3c0bd330cae0e5bf7cf09d1f0ae", GitTreeState:"clean", BuildDate:"2018-09-21T05:53:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3+icp-ee", GitCommit:"5f4327948913b3c0bd330cae0e5bf7cf09d1f0ae", GitTreeState:"clean", BuildDate:"2018-09-21T05:53:39Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
  IBM Private Cloud
• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux Server"
  VERSION="7.5 (Maipo)"
  ID="rhel"

[skriss]

What object storage system are you using? Can you look directly in the bucket you're using, and show which files exist under backups/whole-cluster-backup-04092019171429 within it?

[adityagu0910]

I am using distributed minio
Log file is available in here
[root@17 whole-cluster-backup-04092019171429]# ll
total 0
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 velero-backup.json
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 whole-cluster-backup-04092019171429-logs.gz
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 whole-cluster-backup-04092019171429-podvolumebackups.json.gz
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 whole-cluster-backup-04092019171429-resource-list.json.gz
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 whole-cluster-backup-04092019171429.tar.gz
drwxr-xr-x. 2 minio minio 35 Sep 4 17:19 whole-cluster-backup-04092019171429-volumesnapshots.json.gz
[root@17 whole-cluster-backup-04092019171429]# pwd
/export/minio/data1/sus-icp-dev-backup/backups/whole-cluster-backup-04092019171429
[root@17 whole-cluster-backup-04092019171429]#

Interesting thing is that same minio i am using for other ICP backup and for that I am able to see logs. The only difference between two ICP is that they are running in different network and for the one which i am able to see logs is in same network as my minio.
I suspect it could be due to some network policy but not able to figure out as create backup works fine.

[skriss]

You could try manually creating a DownloadRequest by applying YAML like the following:

apiVersion: velero.io/v1
kind: DownloadRequest
metadata:
  name: test-download-request
  namespace: velero
spec:
  target:
    kind: BackupLog
    name: whole-cluster-backup-04092019171429

It should be processed shortly by Velero and get a pre-signed URL added to the status. You can then try manually accessing that URL and see what happens.

[skriss]

You might also want to check in the velero deployment logs for any errors related to DownloadRequests

[adityagu0910]

I created DownloadRequest manually as suggested and it created one URL but It seems download is not working due to proxy. I will check and update here. Thanks Steve !!

Regarding logs - do not see any specific error message in log. It just says retrieved 0 items

time="2019-09-04T21:19:56Z" level=info msg="Backing up resource" backup=velero/whole-cluster-backup-04092019171429 group=velero.io/v1 logSource="pkg/backu
p/resource_backupper.go:104" resource=downloadrequests
time="2019-09-04T21:19:56Z" level=info msg="Listing items" backup=velero/whole-cluster-backup-04092019171429 group=velero.io/v1 logSource="pkg/backup/reso
urce_backupper.go:224" namespace= resource=downloadrequests
time="2019-09-04T21:19:56Z" level=info msg="Retrieved 0 items" backup=velero/whole-cluster-backup-04092019171429 group=velero.io/v1 logSource="pkg/backup/
resource_backupper.go:238" namespace= resource=downloadrequests

[adityagu0910]

Thanks Steve !!

It worked fine after adding minio IP in no_proxy variable.
Problem was with my shell session from which I was running velero backup log command to get the logs. It was having the proxy variables set so added my minio storage IP to no_proxy env variables.