Conductor: Creating Conductor Deployments and Services For Kubernetes

[mysteva]

I am currently evaluating Conductor as an orchestration framework, and am attempting to get all four containers (elasticsearch, dynomite, conductor-ui and conductor-server) to spin up successfully in Kubernetes. Is there a set of yaml files that exist that would provide the same functionality as the docker-compose.yaml file?

I've tried creating my own yaml files to apply to kubernetes, and they all seem to be running, but I can't get them communicating with each other (for instance, the ui clearly shows that there's no underlying data source). Any help would be greatly appreciated. Thanks!

[james-deee]

It sounds like it's not the yaml files you need help with then? It sounds like you don't have the correct configuration settings in your config.properties that got built into your conductor server image?

[mysteva]

@demichej It's possible. Maybe a better way to frame the question: is there a best practice that should be used for getting conductor, in its entirety, up and running in the cloud (whether it be AWS or Kubernetes)? Am I going about it the wrong way, and docker-compose is really the correct way to do it?

[james-deee]

I am not aware of a best practice @mysteva. We have decided to use Kubernetes to deploy our Conductor services + our workers.

We created our own yaml files for the Conductor services and we build it up with docker-compose and deploy manually.

I'm not sure if others would find it useful having some kind of "template" or something for their yaml files. Seems like a reasonable idea, but I dont think anything like that exists right now.

[mysteva]

@demichej Thanks for your response. Going forward, I do think that would be very valuable to include. Is there a way to make that request officially?

[james-deee]

You could take a stab at putting a PR together :)

I'm not sure of the best way to label this as an enhancement though. That's another option. I dont think i have the permissions to do that though.

[mysteva]

Ok, thanks. I don't suppose you'd by chance have your generic yaml files that you used, would you?

[apanicker-nflx]

@demichej Thanks for the suggestion. I have labeled this discussion as an enhancement. Please consider sharing your yaml files and Kubernetes deployment guide to benefit the community. This would be a great addition to the project.

[mysteva]

Hi, just wanted to check in to see if there were any sort of "stock" yaml files that we can use for testing via Kubernetes. Thanks!

[james-deee]

Hi @mysteva I'm still here :) I'm sorry I haven't had time to do that, but I will try and get this in a PR this week. Note though, we're using straight Redis (no dynomite), so I'm not sure if that will 100% solve your use case or not.

[eins]

Hello folks, any news about this ticket, conductor and kubernetes support ?

[apanicker-nflx]

@demichej Checking in to see if you would be able to pick this back up soon? Thanks.

[james-deee]

@apanicker-nflx Yeah it is on our list still, but has been pushed further down :(

Also, do y'all have plans to push a Docker image (maybe to the Netflix OSS Hub)? We are able to override all configuration from environment variables, so if an image was made available in the public domain, it would be pretty nice/easy to use.

Thanks.

[apanicker-nflx]

Thanks for the update, we are currently chalking out plans for Conductor 3.0 and publishing docker images as part of new releases is one of the items on the roadmap.