Linkerd2: stabilize inject output

[siggy]

For diff and debugging purposes, it's helpful if linkerd inject does not reorder the input k8s resources.

When injecting https://run.linkerd.io/emojivoto.yml, observe the serviceAccountName get moved within the pod spec:

$ diff <(curl -s https://run.linkerd.io/emojivoto.yml) <(curl -s https://run.linkerd.io/emojivoto.yml | bin/go-run cli inject - | bin/go-run cli uninject -)

namespace "emojivoto" skipped
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
deployment "emoji" injected
service "emoji-svc" skipped
deployment "voting" injected
service "voting-svc" skipped
deployment "web" injected
service "web-svc" skipped
deployment "vote-bot" injected


namespace "emojivoto" skipped
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
deployment "emoji" uninjected
service "emoji-svc" skipped
deployment "voting" uninjected
service "voting-svc" skipped
deployment "web" uninjected
service "web-svc" skipped
deployment "vote-bot" uninjected

43d42
<       serviceAccountName: emoji
55a55
>       serviceAccountName: emoji
90d89
<       serviceAccountName: voting
102a102
>       serviceAccountName: voting
137d136
<       serviceAccountName: web
155a155
>       serviceAccountName: web

[liquidslr]

@siggy just had a small question although i didn't go in the details of this issue,but according to my knowledge the output of diff command takes the formatting of the input files, so in this case, the extra indent of serviceAccountName is coming from the yaml file I guess, Just asking out of curiosity.

[siggy]

Hi @liquidslr. Thanks for the question. The output diff is exhibiting changes to the order of the YAML elements, not the indents.

Consistent indentation between input and output of our inject/uninject commands is not currently supported (not impossible but more work).

The intent of this ticket is to stabilize the ordering of elements across inject/uninject commands, with the goal being that I can run inject on a yaml, diff the output, and easily identify what was injected. If our inject command reorders the yaml elements, it's harder to see what changed.

[liquidslr]

thanks @siggy I would like to work on this !

[liquidslr]

The output diff is exhibiting changes to the order of the YAML elements, not the indents.

sorry for the delay, I went through the deployment files https://run.linkerd.io/emojivoto.yml and couldn't notice any difference in ordering of the elements, I think I still don't understand the exact issue.

[siggy]

@liquidslr the bash session in this issue's description demonstrates that given a yaml file, injecting and then uninjecting leaves the serviceAccountName elements in a different location in the yaml. Injecting a yaml with a proxy should not re-order elements that are unrelated to the elements being injected.

Here's a similar command with stable-2.3.0:

$ diff <(curl -s https://run.linkerd.io/emojivoto.yml) <(curl -s https://run.linkerd.io/emojivoto.yml | linkerd inject --ignore-cluster --disable-identity - | linkerd uninject -)

namespace "emojivoto" skipped
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
deployment "emoji" injected
service "emoji-svc" skipped
deployment "voting" injected
service "voting-svc" skipped
deployment "web" injected
service "web-svc" skipped
deployment "vote-bot" injected


namespace "emojivoto" skipped
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
deployment "emoji" uninjected
service "emoji-svc" skipped
deployment "voting" uninjected
service "voting-svc" skipped
deployment "web" uninjected
service "web-svc" skipped
deployment "vote-bot" uninjected

43d42
<       serviceAccountName: emoji
55a55
>       serviceAccountName: emoji
90d89
<       serviceAccountName: voting
102a102
>       serviceAccountName: voting
137d136
<       serviceAccountName: web
155a155
>       serviceAccountName: web

[olix0r]

Is this still relevant now that injection is handled by the proxy-injector?

[ihcsim]

Yes, it's still relevant because the proxy injector (more specifically, the yaml library) doesn't preserve the order of the keys inside the YAML block. E.g.,

--- before-inject.yaml  2019-05-28 15:29:30.578087469 -0700
+++ after-inject.yaml   2019-05-28 15:32:20.024791349 -0700
@@ -36,11 +36,12 @@
   strategy: {}
   template:
     metadata:
+      annotations:
+        linkerd.io/inject: enabled
       creationTimestamp: null
       labels:
         app: emoji-svc
     spec:
-      serviceAccountName: emoji
       containers:
       - env:
         - name: GRPC_PORT
@@ -53,6 +54,7 @@
         resources:
           requests:
             cpu: 100m
+      serviceAccountName: emoji # the keys are now sorted alphabetically

[saurav-malani]

@siggy @ihcsim I would like to look into this, if it's still relevant and open.

[ihcsim]

@saurav-malani Go for it! Thanks!

[vaniisgh]

Hey!
I was wondering if this issue is still relevant and if so is anyone still working on it?

If there are no takers, I would like to have a go at this one :)
thanks:)

[saurav-malani]

@vaniisgh I am not working on it anymore. So, please go ahead.

[vaniisgh]

I think this issue has been solved somehow, I ran the command :

diff -c <(curl -s https://run.linkerd.io/emojivoto.yml) <(curl -s https://run.linkerd.io/emojivoto.yml | linkerd inject --disable-identity  - | linkerd uninject -) 

and get output :

namespace "emojivoto" injected
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
service "emoji-svc" skipped
service "voting-svc" skipped
service "web-svc" skipped
deployment "emoji" injected
deployment "vote-bot" injected
deployment "voting" injected
deployment "web" injected


namespace "emojivoto" uninjected
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
service "emoji-svc" skipped
service "voting-svc" skipped
service "web-svc" skipped
deployment "emoji" uninjected
deployment "vote-bot" uninjected
deployment "voting" uninjected
deployment "web" uninjected

*** /dev/fd/11  Wed May  6 11:50:52 2020
--- /dev/fd/12  Wed May  6 11:50:52 2020
***************
*** 1,6 ****
--- 1,8 ----
  apiVersion: v1
  kind: Namespace
  metadata:
+   annotations:
+     config.linkerd.io/disable-identity: "true"
    name: emojivoto
  ---
  apiVersion: v1
***************
*** 83,88 ****
--- 85,92 ----
        app: emoji-svc
    template:
      metadata:
+       annotations:
+         config.linkerd.io/disable-identity: "true"
        labels:
          app: emoji-svc
      spec:
***************
*** 120,125 ****
--- 124,131 ----
        app: vote-bot
    template:
      metadata:
+       annotations:
+         config.linkerd.io/disable-identity: "true"
        labels:
          app: vote-bot
      spec:
***************
*** 151,156 ****
--- 157,164 ----
        app: voting-svc
    template:
      metadata:
+       annotations:
+         config.linkerd.io/disable-identity: "true"
        labels:
          app: voting-svc
      spec:
***************
*** 188,193 ****
--- 196,203 ----
        app: web-svc
    template:
      metadata:
+       annotations:
+         config.linkerd.io/disable-identity: "true"
        labels:
          app: web-svc
      spec:
***************
*** 210,212 ****
--- 220,223 ----
            requests:
              cpu: 100m
        serviceAccountName: web
+ ---

and the only changes I see are the ones that are configured by the flags specified...
or am I missing something :)

[grampelberg]

@vaniisgh I believe it is because you disabled identity.

[vaniisgh]

Oh I was just trying to replicate this suggestion :

the bash session in this issue's description demonstrates that given a yaml file, injecting and then uninjecting leaves the serviceAccountName elements in a different location in the yaml. Injecting a yaml with a proxy should not re-order elements that are unrelated to the elements being injected.

Here's a similar command with stable-2.3.0:

$ diff <(curl -s https://run.linkerd.io/emojivoto.yml) <(curl -s https://run.linkerd.io/emojivoto.yml | linkerd inject --ignore-cluster --disable-identity - | linkerd uninject -)

but I ran it again right now and Its still the same

~/ diff -c <(curl -s https://run.linkerd.io/emojivoto.yml) <(curl -s https://run.linkerd.io/emojivoto.yml | linkerd inject - | linkerd uninject -)

namespace "emojivoto" injected
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
service "emoji-svc" skipped
service "voting-svc" skipped
service "web-svc" skipped
deployment "emoji" injected
deployment "vote-bot" injected
deployment "voting" injected
deployment "web" injected


namespace "emojivoto" uninjected
serviceaccount "emoji" skipped
serviceaccount "voting" skipped
serviceaccount "web" skipped
service "emoji-svc" skipped
service "voting-svc" skipped
service "web-svc" skipped
deployment "emoji" uninjected
deployment "vote-bot" uninjected
deployment "voting" uninjected
deployment "web" uninjected

*** /dev/fd/11  Wed May  6 23:33:37 2020
--- /dev/fd/12  Wed May  6 23:33:37 2020
***************
*** 210,212 ****
--- 210,213 ----
            requests:
              cpu: 100m
        serviceAccountName: web
+ ---

Does the issue persist for you?

[grampelberg]

@vaniisgh I'm getting the same output as you are (extra line and that's it). This one's taken care of! Thank you for looking into it =)