so the fix from #94 did not help and we're still having lingering volumes in our prod cluster
if someone could analyze prod logs and figure out what's happening, that would be lovely
Was checking out a job now which left a volume. The build failed b/c the spec was wrong - could this be the problem?
Okay, so it's not just PVs, it's also pods! We have a ton of leftover pods in our prod sandbox namespace.
We should turn on the debug logs for Kubernetes. I am afraid that a problem could be in k8s API. We have been changing installation of dependencies recently and k8s API for deletion methods is different between 8.0.0 and 9.0.0.
k8s API for deletion methods is different between 8.0.0 and 9.0.0.
pip3 show kubernetes
Version: 8.0.1
this is resolved now