Argo-cd: Application history can't be appended and new changes are never applied

[carlosjgp]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

After applying multiple changes to our Jenkins deployments we have ended up with this error message in our Argo Application

failed to record sync to history: error decoding patch

Following the code
https://github.com/argoproj/argo-cd/blob/master/controller/sync.go#L201
into...
https://github.com/argoproj/argo-cd/blob/master/controller/state.go#L516

CRDs are other K8s objects have a maximum size of 1MB
https://github.com/kubernetes/kubernetes/issues/19781#issuecomment-172553264

To Reproduce

Deploy multiple changes to a Jenkins deployment or any other large Helm release values file... could be comments!
our values.yaml is 145Kb

Expected behavior

Argo knows about this limit and only keeps as many revisions as it can or history limit is configurable so deployments can continue

Screenshots

N/A

Version

argocd: v1.3.0-rc5+24d43e4
  BuildDate: 2019-11-11T21:11:48Z
  GitCommit: 24d43e45f7a11866bc3eb5f36866f3dc4d5ed56a
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64

Logs

time="2019-12-02T16:14:42Z" level=info msg="Sync operation to 1.5.9 failed: failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})" application=jenkins dest-namespace=gringotts-jenkins dest-server="https://xxx.eks.amazonaws.com" reason=OperationCompleted type=Warning
time="2019-12-02T16:14:42Z" level=info msg="updated 'jenkins' operation (phase: Error)"
time="2019-12-02T16:14:41Z" level=info msg="Updating operation state. phase: Succeeded -> Error, message: 'successfully synced (all tasks run)' -> 'failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})'" application=jenkins syncId=339253-JbgDD
time="2019-12-02T16:06:10Z" level=info msg="Sync operation to 1.5.9 failed: failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})" application=jenkins dest-namespace=gringotts-jenkins dest-server="https://xxx.eks.amazonaws.com" reason=OperationCompleted type=Warning
time="2019-12-02T16:06:10Z" level=info msg="updated 'jenkins' operation (phase: Error)"
time="2019-12-02T16:06:10Z" level=info msg="Updating operation state. phase: Succeeded -> Error, message: 'successfully synced (all tasks run)' -> 'failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})'" application=jenkins syncId=338772-UaYaZ
time="2019-12-02T15:51:25Z" level=info msg="Sync operation to 1.5.9 failed: failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})" application=jenkins dest-namespace=gringotts-jenkins dest-server="https://xxx.eks.amazonaws.com" reason=OperationCompleted type=Warning
time="2019-12-02T15:51:25Z" level=info msg="updated 'jenkins' operation (phase: Error)"
time="2019-12-02T15:51:23Z" level=info msg="Updating operation state. phase: Succeeded -> Error, message: 'successfully synced (all tasks run)' -> 'failed to record sync to history: error decoding patch: json: Unmarshal(non-pointer map[string]interface {})'" application=jenkins syncId=337805-rotCm

[alexec]

See #2545

[alexec]

This will hard to fix as we keep history in the resource and this is limited to 1MB. Storing in separate CRDs would mitigate this.

[carlosjgp]

Hi @alexec,

thanks for pointing me to the PR, but I noticed that it was closed and there are no other tickets attached to it.

Am I right if I assume that the work continues on the branch? Would you like some help with it?
this is kind of a blocker for us that we have just migrated from Flux into ArgoCD

[alexec]

2285