Aks-engine: User provided customFiles should take precedence over aks-engine files

[PascalVA]

We have been deploying aks-engine with pod security policies since before the feature was available. When the feature was implemented in aks-engine, we decided to use customFiles to override the psps of azure and place our own (/etc/kubernetes/addons/pod-security-policy.yaml). This has worked fine so far, but it seems it recently broke. (not sure in which version, but I assume when the option enablePodSecurityPolicy was depricated in favor of the addons)

It looks like the file we pass is completely ignored and the default file from aks-engine is placed in the azuredeploy.json file (base64 encoded and gzipped)

I assume the customFiles are loaded in the code before the files for the PSPs and are then overwriting our customFiles.

custom files by the user should have precedence over files placed by aks-engine.

Steps To Reproduce

Generate a new aks-engine cluster (no need to even deploy it) with the following settings:

• The enablePodSecurityPolicy (depricated) option is not specified
• There is no addon for pod security policy specified in
• The masterProfile contains the following for customFiles:

            "customFiles": [
                {
                    "dest": "/etc/kubernetes/addons/pod-security-policy.yaml",
                    "source": "/path/to/any/file.ext"
                }
            ],

If you then check the customData for the masters and look for the file /etc/kubernetes/addons/pod-security-policy.yaml (base64 --decode + gunzip), you will see it is the file from aksengine and not our custom file.

Expected behavior

the azuredeploy.json should include our file in the deployment.

AKS Engine version

0.51.0

Kubernetes version

1.18.1

[jackfrancis]

Hi @PascalVA, you're probably right that this functionality has regressed. There is another way to do this, though. See:

https://github.com/Azure/aks-engine/blob/master/docs/topics/clusterdefinitions.md#addons

Go down to until you reach the sentence "If you'd like to entirely customize one of the addons available..." and you'll see how to pass arbitrary data. Because a sane PodSecurityPolicy is required to properly schedule other k8s components according to how AKS Engine specs them out, you can't disable the addon. But you can include this in your api model:

"kubernetesConfig": {
    "addons": [
        {
            "name": "pod-security-policy",
            "enabled": true,
            "data": "YXBpVmVyc2lvbjoge3tpZiBJc0t1YmVybmV0ZXNWZXJzaW9uR2UgIjEuMTYuMCJ9fXBvbGljeXt7ZWxzZX19ZXh0ZW5zaW9uc3t7ZW5kfX0vdjFiZXRhMQpraW5kOiBQb2RTZWN1cml0eVBvbGljeQptZXRhZGF0YToKICBuYW1lOiBwcml2aWxlZ2VkCiAgYW5ub3RhdGlvbnM6CiAgICBzZWNjb21wLnNlY3VyaXR5LmFscGhhLmt1YmVybmV0ZXMuaW8vYWxsb3dlZFByb2ZpbGVOYW1lczogIioiCiAgbGFiZWxzOgogICAgYWRkb25tYW5hZ2VyLmt1YmVybmV0ZXMuaW8vbW9kZTogUmVjb25jaWxlCnNwZWM6CiAgcHJpdmlsZWdlZDogdHJ1ZQogIGFsbG93UHJpdmlsZWdlRXNjYWxhdGlvbjogdHJ1ZQogIGFsbG93ZWRDYXBhYmlsaXRpZXM6CiAgLSAiKiIKICB2b2x1bWVzOgogIC0gIioiCiAgaG9zdE5ldHdvcms6IHRydWUKICBob3N0UG9ydHM6CiAgLSBtaW46IDAKICAgIG1heDogNjU1MzUKICBob3N0SVBDOiB0cnVlCiAgaG9zdFBJRDogdHJ1ZQogIHJ1bkFzVXNlcjoKICAgIHJ1bGU6IFJ1bkFzQW55CiAgc2VMaW51eDoKICAgIHJ1bGU6IFJ1bkFzQW55CiAgc3VwcGxlbWVudGFsR3JvdXBzOgogICAgcnVsZTogUnVuQXNBbnkKICBmc0dyb3VwOgogICAgcnVsZTogUnVuQXNBbnkKLS0tCmFwaVZlcnNpb246IHt7aWYgSXNLdWJlcm5ldGVzVmVyc2lvbkdlICIxLjE2LjAifX1wb2xpY3l7e2Vsc2V9fWV4dGVuc2lvbnN7e2VuZH19L3YxYmV0YTEKa2luZDogUG9kU2VjdXJpdHlQb2xpY3kKbWV0YWRhdGE6CiAgbmFtZTogcmVzdHJpY3RlZAogIGFubm90YXRpb25zOgogICAgc2VjY29tcC5zZWN1cml0eS5hbHBoYS5rdWJlcm5ldGVzLmlvL2FsbG93ZWRQcm9maWxlTmFtZXM6IGRvY2tlci9kZWZhdWx0CiAgICBhcHBhcm1vci5zZWN1cml0eS5iZXRhLmt1YmVybmV0ZXMuaW8vYWxsb3dlZFByb2ZpbGVOYW1lczogcnVudGltZS9kZWZhdWx0CiAgICBzZWNjb21wLnNlY3VyaXR5LmFscGhhLmt1YmVybmV0ZXMuaW8vZGVmYXVsdFByb2ZpbGVOYW1lOiAgZG9ja2VyL2RlZmF1bHQKICAgIGFwcGFybW9yLnNlY3VyaXR5LmJldGEua3ViZXJuZXRlcy5pby9kZWZhdWx0UHJvZmlsZU5hbWU6ICBydW50aW1lL2RlZmF1bHQKICBsYWJlbHM6CiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKc3BlYzoKICBwcml2aWxlZ2VkOiBmYWxzZQogIGFsbG93UHJpdmlsZWdlRXNjYWxhdGlvbjogZmFsc2UKICByZXF1aXJlZERyb3BDYXBhYmlsaXRpZXM6CiAgICAtIEFMTAogIHZvbHVtZXM6CiAgICAtIGNvbmZpZ01hcAogICAgLSBlbXB0eURpcgogICAgLSBwcm9qZWN0ZWQKICAgIC0gc2VjcmV0CiAgICAtIGRvd253YXJkQVBJCiAgICAtIHBlcnNpc3RlbnRWb2x1bWVDbGFpbQogIGhvc3ROZXR3b3JrOiBmYWxzZQogIGhvc3RJUEM6IGZhbHNlCiAgaG9zdFBJRDogZmFsc2UKICBydW5Bc1VzZXI6CiAgICBydWxlOiBNdXN0UnVuQXNOb25Sb290CiAgc2VMaW51eDoKICAgIHJ1bGU6IFJ1bkFzQW55CiAgc3VwcGxlbWVudGFsR3JvdXBzOgogICAgcnVsZTogTXVzdFJ1bkFzCiAgICByYW5nZXM6CiAgICAgIHt7LSAvKiBGb3JiaWQgYWRkaW5nIHRoZSByb290IGdyb3VwLiAqL319CiAgICAgIC0gbWluOiAxCiAgICAgICAgbWF4OiA2NTUzNQogIGZzR3JvdXA6CiAgICBydWxlOiBNdXN0UnVuQXMKICAgIHJhbmdlczoKICAgICAge3stIC8qIEZvcmJpZCBhZGRpbmcgdGhlIHJvb3QgZ3JvdXAuICovfX0KICAgICAgLSBtaW46IDEKICAgICAgICBtYXg6IDY1NTM1CiAgcmVhZE9ubHlSb290RmlsZXN5c3RlbTogZmFsc2UKLS0tCmFwaVZlcnNpb246IHJiYWMuYXV0aG9yaXphdGlvbi5rOHMuaW8vdjEKa2luZDogQ2x1c3RlclJvbGUKbWV0YWRhdGE6CiAgbmFtZTogcHNwOnByaXZpbGVnZWQKICBsYWJlbHM6CiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKcnVsZXM6Ci0gYXBpR3JvdXBzOiBbJ2V4dGVuc2lvbnMnXQogIHJlc291cmNlczogWydwb2RzZWN1cml0eXBvbGljaWVzJ10KICB2ZXJiczogICAgIFsndXNlJ10KICByZXNvdXJjZU5hbWVzOgogIC0gcHJpdmlsZWdlZAotLS0KYXBpVmVyc2lvbjogcmJhYy5hdXRob3JpemF0aW9uLms4cy5pby92MQpraW5kOiBDbHVzdGVyUm9sZQptZXRhZGF0YToKICBuYW1lOiBwc3A6cmVzdHJpY3RlZAogIGxhYmVsczoKICAgIGFkZG9ubWFuYWdlci5rdWJlcm5ldGVzLmlvL21vZGU6IFJlY29uY2lsZQpydWxlczoKLSBhcGlHcm91cHM6IFsnZXh0ZW5zaW9ucyddCiAgcmVzb3VyY2VzOiBbJ3BvZHNlY3VyaXR5cG9saWNpZXMnXQogIHZlcmJzOiAgICAgWyd1c2UnXQogIHJlc291cmNlTmFtZXM6CiAgLSByZXN0cmljdGVkCi0tLQphcGlWZXJzaW9uOiByYmFjLmF1dGhvcml6YXRpb24uazhzLmlvL3YxCmtpbmQ6IENsdXN0ZXJSb2xlQmluZGluZwptZXRhZGF0YToKICBuYW1lOiBkZWZhdWx0OnJlc3RyaWN0ZWQKICBsYWJlbHM6CiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKcm9sZVJlZjoKICBhcGlHcm91cDogcmJhYy5hdXRob3JpemF0aW9uLms4cy5pbwogIGtpbmQ6IENsdXN0ZXJSb2xlCiAgbmFtZTogcHNwOnJlc3RyaWN0ZWQKc3ViamVjdHM6Ci0ga2luZDogR3JvdXAKICBuYW1lOiBzeXN0ZW06YXV0aGVudGljYXRlZAogIGFwaUdyb3VwOiByYmFjLmF1dGhvcml6YXRpb24uazhzLmlvCi0tLQphcGlWZXJzaW9uOiByYmFjLmF1dGhvcml6YXRpb24uazhzLmlvL3YxCmtpbmQ6IHt7aWYgSXNLdWJlcm5ldGVzVmVyc2lvbkdlICIxLjE1LjAifX1DbHVzdGVyUm9sZUJpbmRpbmd7e2Vsc2V9fVJvbGVCaW5kaW5ne3tlbmR9fQptZXRhZGF0YToKICBuYW1lOiBkZWZhdWx0OnByaXZpbGVnZWQKe3stIGlmIG5vdCAoSXNLdWJlcm5ldGVzVmVyc2lvbkdlICIxLjE1LjAiKX19CiAgbmFtZXNwYWNlOiBrdWJlLXN5c3RlbQp7e2VuZH19CiAgbGFiZWxzOgogICAgYWRkb25tYW5hZ2VyLmt1YmVybmV0ZXMuaW8vbW9kZTogUmVjb25jaWxlCnJvbGVSZWY6CiAgYXBpR3JvdXA6IHJiYWMuYXV0aG9yaXphdGlvbi5rOHMuaW8KICBraW5kOiBDbHVzdGVyUm9sZQogIG5hbWU6IHBzcDpwcml2aWxlZ2VkCnN1YmplY3RzOgotIGtpbmQ6IEdyb3VwCiAgbmFtZToge3tpZiBJc0t1YmVybmV0ZXNWZXJzaW9uR2UgIjEuMTUuMCJ9fXN5c3RlbTphdXRoZW50aWNhdGVke3tlbHNlfX1zeXN0ZW06bWFzdGVyc3t7ZW5kfX0KICBhcGlHcm91cDogcmJhYy5hdXRob3JpemF0aW9uLms4cy5pbwp7ey0gaWYgbm90IChJc0t1YmVybmV0ZXNWZXJzaW9uR2UgIjEuMTUuMCIpfX0KLSBraW5kOiBHcm91cAogIG5hbWU6IHN5c3RlbTpzZXJ2aWNlYWNjb3VudHM6a3ViZS1zeXN0ZW0KICBhcGlHcm91cDogcmJhYy5hdXRob3JpemF0aW9uLms4cy5pbwp7e2VuZH19Ci0ga2luZDogR3JvdXAKICBuYW1lOiBzeXN0ZW06bm9kZXMKICBhcGlHcm91cDogcmJhYy5hdXRob3JpemF0aW9uLms4cy5pbwojRU9GCg=="
        }
    ]
}

Note that the above base64-encoded string is a result of this statement in the git root:

$ base64 < parts/k8s/addons/pod-security-policy.yaml

So don't copy that string exactly! Base64-encode your known-working, custom PodSecurityPolicy spec, and use that as the value of the "data" property.

Let me know if you have any questions.

[jackfrancis]

FYI I've reclassified this issue, as I don't think the customFiles interface was ever meant to take precedence over the various "built-in" files paved by the AKS Engine-generated ARM template.

[PascalVA]

Thanks for the quick reply as always @jackfrancis !

I can live with this workaround (or rather better way to provision the psp files), thanks!
I do disagree on the customFiles part. Even though it may not have been the intention, I think letting the user have the final say about what goes on the system, gives aks-engine a lot more flexibility. Though granted, it could make it easier for people just starting out to make mistakes.

In any case with the workaround, for now, it does not really matter to me if it is implemented or not.

[jackfrancis]

I'm happy to leave this issue open in the backlog, I'll add the feature label.

[jackfrancis]

cc @rbitia

[jackfrancis]

@PascalVA realistically we won't be doing this work anytime soon. Feel free to re-open, especially if you feel like you have a handle on a solution and want to open a PR. :)

[PascalVA]

@PascalVA realistically we won't be doing this work anytime soon. Feel free to re-open, especially if you feel like you have a handle on a solution and want to open a PR. :)

That makes sense to me, in any case, passing the base64 for the pod security policies in the add-ons was enough of a workaround for me.