Ambassador: Ambassador fails to start: regex_max_size is not respected

Created on 18 Mar 2020 · 2Comments · Source: datawire/ambassador

Describe the bug
Ambassador 1.2.2 does not seem to respect the regex_max_size configuration parameter.
This results in the ambassador pod failing start when a large regex is present in in a mapping configuration.

To Reproduce
Steps to reproduce the behavior:

Apply the following configuration that I have used to reproduce the issue.

apiVersion: v1
kind: Namespace
metadata:
  name: jvv
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: authservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: AuthService
    plural: authservices
    singular: authservice
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: consulresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: ConsulResolver
    plural: consulresolvers
    singular: consulresolver
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: filterpolicies.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: FilterPolicy
    plural: filterpolicies
    shortNames:
    - fp
    singular: filterpolicy
  scope: Namespaced
  version: v1beta2
  versions:
  - name: v1beta2
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: filters.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: Filter
    plural: filters
    shortNames:
    - fil
    singular: filter
  scope: Namespaced
  version: v1beta2
  versions:
  - name: v1beta2
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: hosts.getambassador.io
spec:
  additionalPrinterColumns:
  - JSONPath: .spec.hostname
    name: Hostname
    type: string
  - JSONPath: .status.state
    name: State
    type: string
  - JSONPath: .status.phaseCompleted
    name: Phase Completed
    type: string
  - JSONPath: .status.phasePending
    name: Phase Pending
    type: string
  - JSONPath: .metadata.creationTimestamp
    name: Age
    type: date
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: Host
    plural: hosts
    singular: host
  scope: Namespaced
  subresources:
    status: {}
  version: v2
  versions:
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: kubernetesendpointresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: KubernetesEndpointResolver
    plural: kubernetesendpointresolvers
    singular: kubernetesendpointresolver
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: kubernetesserviceresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: KubernetesServiceResolver
    plural: kubernetesserviceresolvers
    singular: kubernetesserviceresolver
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: logservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: LogService
    plural: logservices
    singular: logservice
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: mappings.getambassador.io
spec:
  additionalPrinterColumns:
  - JSONPath: .spec.prefix
    name: Prefix
    type: string
  - JSONPath: .spec.service
    name: Service
    type: string
  - JSONPath: .status.state
    name: State
    type: string
  - JSONPath: .status.reason
    name: Reason
    type: string
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: Mapping
    plural: mappings
    singular: mapping
  scope: Namespaced
  subresources:
    status: {}
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: modules.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: Module
    plural: modules
    singular: module
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: ratelimits.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: RateLimit
    plural: ratelimits
    shortNames:
    - rl
    singular: ratelimit
  scope: Namespaced
  version: v1beta1
  versions:
  - name: v1beta1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: ratelimitservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: RateLimitService
    plural: ratelimitservices
    singular: ratelimitservice
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: tcpmappings.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: TCPMapping
    plural: tcpmappings
    singular: tcpmapping
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: tlscontexts.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: TLSContext
    plural: tlscontexts
    singular: tlscontext
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    helm.sh/hook: crd-install
  labels:
    app.kubernetes.io/name: ambassador
    product: aes
  name: tracingservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: TracingService
    plural: tracingservices
    singular: tracingservice
  scope: Namespaced
  version: v1
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv
  namespace: jvv
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador
  namespace: jvv
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - services
  - secrets
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - getambassador.io
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
  - update
  - patch
  - create
  - delete
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.internal.knative.dev
  resources:
  - clusteringresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador-crds
rules:
- apiGroups:
  - apiextensions.k8s.io
  resourceNames:
  - authservices.getambassador.io
  - mappings.getambassador.io
  - modules.getambassador.io
  - ratelimitservices.getambassador.io
  - tcpmappings.getambassador.io
  - tlscontexts.getambassador.io
  - tracingservices.getambassador.io
  - kubernetesendpointresolvers.getambassador.io
  - kubernetesserviceresolvers.getambassador.io
  - consulresolvers.getambassador.io
  - filters.getambassador.io
  - filterpolicies.getambassador.io
  - ratelimits.getambassador.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - list
  - watch
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador
  namespace: jvv
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jvv-ambassador
subjects:
- kind: ServiceAccount
  name: jvv
  namespace: jvv
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador-crds
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jvv-ambassador-crds
subjects:
- kind: ServiceAccount
  name: jvv
  namespace: jvv
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: ambassador-service
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador
  namespace: jvv
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    nodePort: 30005
    port: 80
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/name: ambassador
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
    service: ambassador-admin
  name: jvv-ambassador-admin
  namespace: jvv
spec:
  ports:
  - name: ambassador-admin
    port: 8877
    protocol: TCP
    targetPort: admin
  selector:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/name: ambassador
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  myservice
      prefix: /
      service: myservice.default:3000
      host: (ab|abcdefg)\.qwerty\.com|abcdefg\.poiuyt\.com|asdfghj\.poiuyt\.com|xyz\.asdfghj\.ghjkl\.com|zxcvbn\.poiuyt\.com|www\.zxcvbncvbnm\.com|zxcvbncvbnm\.com|dfghjkl\.poiuyt\.com|xyz\.dfghjkl\.uioplkjhg\.com|.*\.cvbnm\.poiuyt\.com|xyz-ertyui\.poiuyt\.com|xcvbgffds\.poiuyt\.com|myoiuytrfg\.poiuyt\.com|myoiuytrfg\.oiuytrfgth\.com|wszxzaq\.poiuyt\.com|ruiekc\.imnju\.com
      host_regex: true
      ambassador_id: [ "jvv" ]
  name: myservice
  namespace: jvv
spec:
  ports:
  - name: myserviceport
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: myservice
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador
  namespace: jvv
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: jvv
      app.kubernetes.io/name: ambassador
  strategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
        prometheus.io/path: /metrics
        prometheus.io/port: "8877"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/instance: jvv
        app.kubernetes.io/name: ambassador
        app.kubernetes.io/part-of: jvv
    spec:
      containers:
      - env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: AMBASSADOR_DRAIN_TIME
          value: "300"
        - name: AMBASSADOR_ID
          value: jvv
        - name: AMBASSADOR_SHUTDOWN_TIME
          value: "315"
        - name: AMBASSADOR_SINGLENAMESPACE
          value: "false"
        image: quay.io/datawire/ambassador:1.2.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ambassador/v0/check_alive
            port: admin
          initialDelaySeconds: 30
          periodSeconds: 3
        name: ambassador
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8877
          name: admin
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ambassador/v0/check_ready
            port: admin
          initialDelaySeconds: 30
          periodSeconds: 3
        resources:
          limits:
            memory: 800Mi
          requests:
            cpu: 400m
            memory: 400Mi
        volumeMounts:
        - mountPath: /tmp/ambassador-pod-info
          name: ambassador-pod-info
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: false
      imagePullSecrets: []
      restartPolicy: Always
      securityContext:
        runAsUser: 8888
      serviceAccountName: jvv
      volumes:
      - downwardAPI:
          items:
          - fieldRef:
              fieldPath: metadata.labels
            path: labels
        name: ambassador-pod-info
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  labels:
    app.kubernetes.io/instance: jvv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ambassador
    app.kubernetes.io/part-of: jvv
    helm.sh/chart: ambassador-6.2.1
  name: jvv-ambassador
  namespace: jvv
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ambassador
      app.kubernetes.io/part-of: jvv
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: jvv-ambassador
  namespace: jvv
spec:
  acmeProvider:
    authority: none
  ambassador_id: jvv
  hostname: '*'
  requestPolicy:
    insecure:
      action: Route
---
apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: jvv-ambassador
  namespace: jvv
spec:
  config:
    ambassador_id: jvv
    enable_grpc_web: true
    envoy_log_type: json
    gzip:
      window_bits: 14
    regex_max_size: 400

Run kubectl describe --namespace jvv module jvv-ambassador and note that the ambassador module has config regex_max_size: 400
Check the logs of the ambassador pods kubectl logs --namespace=jvv -l app.kubernetes.io/instance=jvv,app.kubernetes.io/name=ambassador,app.kubernetes.io/part-of=jvv
It should state something along the lines of:

[2020-03-18 08:52:27.826][166][critical][main] [source/server/config_validation/server.cc:59] error initializing configuration '/ambassador/snapshots/econf-tmp.json': regex '(ab|abcdefg)\.qwerty\.com|abcdefg\.poiuyt\.com|asdfghj\.poiuyt\.com|xyz\.asdfghj\.ghjkl\.com|zxcvbn\.poiuyt\.com|www\.zxcvbncvbnm\.com|zxcvbncvbnm\.com|dfghjkl\.poiuyt\.com|xyz\.dfghjkl\.uioplkjhg\.com|.*\.cvbnm\.poiuyt\.com|xyz-ertyui\.poiuyt\.com|xcvbgffds\.poiuyt\.com|myoiuytrfg\.poiuyt\.com|myoiuytrfg\.oiuytrfgth\.com|wszxzaq\.poiuyt\.com|ruiekc\.imnju\.com' RE2 program size of 316 > max program size of 200. Increase configured max program size if necessary.

Aborting update...

Observe that the ambassador pods never enters the running state.

Expected behavior
I expected Ambasador to load the mapping and accept its RE2 size since the regex_max_size parameter is set to 400.
Furthermore, I expected Ambassador to discard the mapping it cannot load and load other mappings I have and startup properly.

Versions (please complete the following information):

Ambassador: 1.2.2 OSS edition
Kubernetes environment: Amazon EKS v1.14.9-eks-502bfb
Yaml created using the datawire Helm chart version 6.2.1

Additional context
We have multiple mappings using the deprecated APIversion ambassador/v0. We are upgrading our ambassador from version 0.86 to 1.2.2 but are blocked by this issue.

stale

Source

van-vliet

👍8

Most helpful comment

The issue still persists, and prevents from writing long regex in pattern property under regex_rewrite.
I have a use case in which I need to explicitly capture UUID4 on a long URL, although currently there is a workaround using [0-9a-fA-F-]+ I prefer to be strict and use [a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89aAbB][a-f0-9]{3}-[a-f0-9]{12}.

eddy-curv on 20 Oct 2020

👍3

All 2 comments

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.