Charts: How to setup Spinnaker with Ingress?

[pluttrell]

I have installed stable/spinnaker on a k8s cluster at AWS into the namespace spinnaker.

At the end of the chart run, I see these notes:

NOTES:
1. You will need to create 2 port forwarding tunnels in order to access the Spinnaker UI:
  export GATE_POD=$(kubectl get pods --namespace spinnaker -l "component=gate,app=wrinkled-porcupine-spinn" -o jsonpath="{.items[0].metadata.name}")
  export DECK_POD=$(kubectl get pods --namespace spinnaker -l "component=deck,app=wrinkled-porcupine-spinn" -o jsonpath="{.items[0].metadata.name}")
  kubectl port-forward --namespace spinnaker $GATE_POD 8084 &
  kubectl port-forward --namespace spinnaker $DECK_POD 9000 &

2. Visit the Spinnaker UI by opening your browser to: http://127.0.0.1:9000

I need to share access to this install to other team members, whom we are not going to expose direct kubectl access to. As I understand it, the recommended way to expose Spinnaker external is through an Ingress.

I have installed an Nginx Ingress Controller via this kops addon.

I haven't been able to find any examples of how to construct an Ingress resource for Spinnaker, however after reading these docs, I've created this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: spinnaker-deck
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: spinnaker.{mydomain.com}
    http:
      paths:
      - backend:
          serviceName: wrinkled-porcupine-spinn-deck
          servicePort: 9000
      - backend:
          serviceName: wrinkled-porcupine-spinn-gate
          servicePort: 8084

I've also setup our DNS server to point spinnaker.{mydomain.com} to the ELB address connected to the Ingress Controller.

When I navigate to spinnaker.{mydomain.com}, the page appears to load without error. I'm assuming a brand new install shouldn't show much.

When I go to Actions -> New Application, I get a popup with a large spinning graphic indefinably. I've waited as long as 10 minutes.

How do you solve this?

I've reviewed the logs for the following pods: -gate, -deck, ingress-nginx and ingress-backend, but none contain anything insightful other then possibly this, but it references using the default, which I assume is correct:

2017-02-17T08:50:21.702299336Z I0217 08:50:21.702237       1 event.go:216] Event(api.ObjectReference{Kind:"Ingress", Namespace:"spinnaker", Name:"spinnaker-deck", UID:"819d70e1-f4b3-11e6-9be9-0a2ca1dda54a", APIVersion:"extensions", ResourceVersion:"201574", FieldPath:""}): type: 'Warning' reason: 'MAPPING' Ingress rule 'spinnaker/spinnaker-deck' contains no path definition. Assuming /

How do you properly setup Spinnaker with Ingress?

[pluttrell]

@viglesiasce do you happen to know how to solve this?

[lachie83]

@pluttrell thanks for raising this. It would be incredibly valuable to get this working as implement as part of the chart

[lachie83]

Conversation for spinnaker kubernetes channel

@lwander I'm not sure what you meant by "reverse proxy the API through gate". And I don't have direct access to settings.js as I've installed Spinnaker via the helm chart. The chart appears to have installed properly but Spinnaker is of no use to our team if in order to access the UI, you have to run `kubectl proxy` or `kubectl port-forward`, so I'm trying to figure out how to setup an Ingress for it. Attempt 1 above doesn't work. Checkout the screenshots of Spinnaker partially working here: https://github.com/kubernetes/charts/issues/678

How to setup Spinnaker with Ingress? · Issue #678 · kubernetes/charts · GitHub
charts - Curated applications for Kubernetes

[11:17]  
In looking at https://github.com/spinnaker/deck/blob/master/settings.js#L4, I see that it would also take the API_HOST env variable, but there's no way to set that through the chart.
github.com
deck/settings.js at master · spinnaker/deck · GitHub
deck - Management UI for Spinnaker

lwander [11:18 AM]
i'm not very familiar with helm, but can't you edit this file prior to deployment? https://github.com/kubernetes/charts/blob/master/stable/spinnaker/config/settings.js

sorry, I meant "reverse proxy the API through deck" to elaborate, deck is the component serving the UI via apache2. I'm not too familiar with how the helm chart works, but the end result of what you need is an apache2 entry that looks like this: https://github.com/spinnaker/spinnaker/blob/206e93e5abbde7dbc792e9c20cefd23024ffd110/etc/apache2/sites-available/spinnaker.conf, and the changes i pointed out above made to the settings.js file in the helm chart
github.com
charts/settings.js at master · kubernetes/charts · GitHub
charts - Curated applications for Kubernetes
github.com
spinnaker/spinnaker.conf at 206e93e5abbde7dbc792e9c20cefd23024ffd110 · spinnaker/spinnaker · GitHub
spinnaker - Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence.

pluttrell [11:22 AM]
I don't believe that I can edit specific files prior to deployment with helm, but I will check with that team. There are predefined values, that you can change, but there's nothing related.

[lachie83]

The problem is almost certainly the fact that the service returned for gate is on localhost. See this.

https://github.com/spinnaker/spinnaker/blob/206e93e5abbde7dbc792e9c20cefd23024ffd110/etc/apache2/sites-available/spinnaker.conf

[lachie83]

I've played around with this a little bit and was unable to get a working solution without running the kubectl port-forward for gate locally. Any ideas if there is a document around for running spinnaker behind a reverse proxy?

[lachie83]

This document may be helpful -- http://blog.armory.io/exposing-spinnaker-sub-services-through-a-reverse-proxy/

[pluttrell]

The armory.io instructions look like they'd work if you're deploying Spinnaker on dedicated EC2 instances, where the IP and/or hostname won't change. Not sure how easily they can be adapted to k8s deployments where many of the assumptions deploying on dedicated EC2 instances can't be relied upon.

Hopefully, the Spinnaker chart can be enhanced soon to support an Ingress, so these internal details are hidden from users of the chart. The port-forwards that it requires only stay up for 1-2 minutes for me and aren't a feasible way to use Spinnaker in a team based environment.

[joshuatalb]

@pluttrell I bumped into this issue earlier. Here's what I did:

1) Created a values.yaml file with:

deck:
  host: spinnaker.ops.hostname.com
  port: 443
  protocol: https

gate:
  allowedOriginPattern: https?://(?:spinnaker\.ops\.hostname\.com|localhost(?::[1-9]\d*)?/?$

2) Create an ingress in that namespace. Here's my example using kube-lego.

[abrass]

I setup the helm installation with an AWS Loadbalancer as entry point like this.

I delete the service my-app-spinnaker-deck and created an own one. "my-app" must be replaced by whatever name you gave to the spinnaker helm release:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: my-app-deck
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: deck
  - name: ninethousand
    port: 9000
    protocol: TCP
    targetPort: deck
  selector:
    app: my-app-spinnaker
    component: deck
  sessionAffinity: None
  type: LoadBalancer

I changed the allowedOriginPattern in the values.yaml, to avoid Allow-Origin-Exceptions in Browser to:

deck:
  host: spinnaker.example.com
  port: 9000
  protocol: http

gate:
  allowedOriginsPattern: '^https?://(?:localhost|spinnaker.example\.com)(?::[1-9]\d*)?/?$'

This only works (for now) with spinnaker.example.com:9000... did not get it work with spinnaker.example.com yet

[seanknox]

Hi @pluttrell is this still an issue?

[cristim]

@seanknox I've just set up Spinnaker and I have the same problem, I run a private VPC and I couldn't yet figure out how to configure an Ingress for it, so basically it's running but I can't connect to it.

[tomarv2]

I faced the same problem. I changed service for Deck from ClusterIP to NodePort, gave a a fixed port number and updated the gate.yaml to use the same and everyone in the team was able to connect using that NodePort IP.

[djsd123]

Hi,
For anyone wanting to get this working with CoreOS's ALB controller, just grab the default values.yaml from the stable branch of this chart and modify the deck and serviceType config like so:

# Change this if youd like to expose Spinnaker outside the cluster
deck:
  host: spinnaker.example.co.uk
  port: 9000
  protocol: http
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "my-alb-class"
      alb.ingress.kubernetes.io/port: "443"
      alb.ingress.kubernetes.io/subnets: subnet-XXXXXXXX,subnet-XXXXXXXX,subnet-XXXXXXX
      alb.ingress.kubernetes.io/security-groups: sg-XXXXXXXX
      alb.ingress.kubernetes.io/scheme: internet-facing
      external-dns.alpha.kubernetes.io/hostname: spinnaker.example.co.uk.
      alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:eu-west-1:XXXXXXXXXXXX:certificate/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

gate:
  allowedOriginsPattern: '^https?://(?:localhost|127.0.0.1|[^/]+\.example\.com)(?::[1-9]\d*)?/?$'

# Bucket to use when storing config data in S3 compatible storage.  DO NOT USE "spinnaker"
storageBucket: SomeBucketName

# Change service type for UI service
serviceType: NodePort

Should be enough to get you going. Good Luck!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[jsarkar]

@pluttrell...interested to know how you resolved the issue? I am having same issue which you faced and look ing for some solution