Argo: Argo doesn't delete PVC with `volumeClaimTemplates`

[yustoris]

Checklist:

• [x] I've included the version.
• [x] I've included reproduction steps.
• [x] I've included the workflow YAML.
• [ ] I've included the logs.

What happened:
When I used git artifact and volumeClaimTemplates in a workflow , Argo doesn't delete PVC and git pod even after the workflow successfully finished.

What you expected to happen:

Delete PVC and pods completely.

How to reproduce it (as minimally and precisely as possible):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
  namespace: argo
spec:
  entrypoint: git-clone-test
  podgcstrategy: upon-pod-succeeded
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-storage
  templates:
    - name: git-clone-test
      steps:
      - - name: clone-seed-data
          template: git-clone
      - - name: cat-cloned-data
          template: cat-data
    - name: git-clone
      inputs:
        artifacts:
        - name: argo-source
          path: /git-cloned
          git:
            repo: https://github.com/argoproj/argo.git
            revision: v2.6.1
      container:
        image: alpine:3.11
        command: [sh, -c]
        args: ["cp -r /git-cloned /mnt/build"]
        volumeMounts:
        - name: workdir
          mountPath: /mnt/build
        workingDir: /git-cloned
    - name: cat-data
      container:
        image: alpine:3.11
        command: [sh, -c]
        args: ["find /mnt/build"]
        volumeMounts:
        - name: workdir
          mountPath: /mnt/build
        workingDir: /mnt/build

Anything else we need to know?:

Environment:

• Argo version:

argo: v2.6.1
  BuildDate: 2020-03-04T21:25:55Z
  GitCommit: 842739d7831cc5b417c4f524ed85288408a32bbf
  GitTreeState: clean
  GitTag: v2.6.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

clientVersion:
  buildDate: "2020-02-13T18:06:54Z"
  compiler: gc
  gitCommit: 06ad960bfd03b39c8310aaf92d1e7c12ce618213
  gitTreeState: clean
  gitVersion: v1.17.3
  goVersion: go1.13.8
  major: "1"
  minor: "17"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-08-19T11:05:16Z"
  compiler: gc
  gitCommit: 96fac5cd13a5dc064f7d9f4f23030a6aeface6cc
  gitTreeState: clean
  gitVersion: v1.14.6
  goVersion: go1.12.9
  major: "1"
  minor: "14"
  platform: linux/amd64

Other debugging information (if applicable):

• kubectl result after the workflow finished

$ kubectl -n argo get all                                                                                                                                                                          [~/develop/sound_recognition/tmp/pipeline]
NAME                                       READY   STATUS      RESTARTS   AGE
pod/argo-server-7bdbdb7cfb-hkmd5           1/1     Running     0          40h
pod/input-artifact-git-h56dj-2799173413    0/2     Completed   0          8m45s
pod/input-artifact-git-h56dj-350790851     0/2     Completed   0          9m30s
pod/workflow-controller-5795456bcd-mtmvj   1/1     Running     0          40h

NAME                  TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/argo-server   NodePort   172.19.248.196   <none>        2746:32423/TCP   2d3h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-server           1/1     1            1           40h
deployment.apps/workflow-controller   1/1     1            1           40h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-server-7bdbdb7cfb           1         1         1       40h
replicaset.apps/workflow-controller-5795456bcd   1         1         1       40h
$ kubectl -n argo get pvc
NAME                               STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
input-artifact-git-h56dj-workdir   Bound    local-pv-8c257ef6   180Gi      RWO            local-storage   10m

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[yustoris]

Also happened though I don't use git artifact

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
  namespace: argo
spec:
  entrypoint: git-clone-test
  podgcstrategy: upon-pod-succeeded
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-storage
  templates:
    - name: git-clone-test
      steps:
      - - name: clone-seed-data
          template: git-clone
      - - name: cat-cloned-data
          template: cat-data
    - name: git-clone
      container:
        image: alpine:3.11
        command: [sh, -c]
        args: ["echo test > /mnt/build/test2"]
        volumeMounts:
        - name: workdir
          mountPath: /mnt/build
    - name: cat-data
      container:
        image: alpine:3.11
        command: [sh, -c]
        args: ["cat /mnt/build/test2"]
        volumeMounts:
        - name: workdir
          mountPath: /mnt/build