Conductor: Cluster setup and redis queue shards

[kelnos]

I'm on AWS, and I've set up a 3-node dynomite cluster across 3 availability zones (so, 9 nodes total). To start playing around with conductor, I booted a single node with conductor on it, and configured it to speak to the dynomite cluster using a custom HostSupplier implementation that talks to our internal service discovery system, as well as the dyno-supplied HttpEndpointBasedTokenMapSupplier.

I loaded up the sample 'kitchensink' workflow to test things out, but found that I couldn't get it to start; task_1 would be in the SCHEDULED state, and polling for it with the API would just return 204 no content.

After digging into the redis queues implementation, I found that when it pushes data onto a queue, it round-robins each push across available shards (of which I have three, since my dynomite/redis cluster is booted on 3 AZs), but when it _pops_ data from the queue, it only tries to pop from the current (conductor) node's shard. After looking at redis directly, I found that the task_1 item queue was tagged with a different shard, so there's no way for conductor to ever poll for that task using this setup.

I ended up booting two more conductor nodes, so that I'd have a node in each AZ. To poll for a new task, however, I have to issue an API call to each node until I find the one that will successfully dequeue it. Ultimately, I want to put all conductor nodes behind a load balancer, with a single hostname in front of it. Then it'll be a game of whack-a-mole to repeatedly request from the poll URL until the load balancer happens to round robin to the "correct" node.

All of this seems awkward to me -- is this intended behavior, or have I set up my cluster in an unexpected way?

[apanicker-nflx]

@kelnos Your findings are correct, this is currently a limitation enforced by dyno-queues which requires conductor to be deployed across 3 AZs. Having said that however, dyno-queues provide high availability and high-durability guarantees in such a configuration, so the tradeoffs seem quite small in comparison.

[naveenchlsn]

Hi @apanicker-nflx. As per my understanding, Dynomite only provides high availability but not durability.

Can we refill the conductor task queues in case of multi-az failure?

[kelnos]

Great, thanks for the clarification!

[msf]

I'm having what seems to be a similar issue.

however, in my case I only have 1 single host per RACK (AZ).
I've configured dynomite w/ DC_SAFE_QUORUM, so all instances host the same data and provide redundancy to each other.

I have a single conductor instance and all workers connecting to that instance (for now).

must I have basically a "triplicate" deployment, where all AZ have workers, conductor instances and dynomite instances, and where within each AZ, workers should coomunicate to their respective conductor instance?

what about the UI server? do I also need to run 3 instances of it?

[apanicker-nflx]

@naveenchlsn Dynomite stores the replicas of the queue shards in other AZs even if the tasks are served from one shard. When an instance in an AZ is rebooted, it refills its task queues from these replica shards. @smukil Please confirm.

@msf As stated in the reply above, you would need a conductor instance in each AZ for the server to be able to poll tasks from shard of the queue (which is in a separate AZ). The workers, however, need not be deployed in all AZs.

[kelnos]

I realized a possible bad failure scenario here: what if there's an AZ outage? The queue data will still be available on dynomite in the unaffected AZs, but there will be no functioning Conductor nodes dequeuing from the dead AZ's queue shard. What is the recommended way to recover from this scenario, without having to wait for AWS to restore the AZ?

[kimabd]

@kelnos @msf could you please share your working configs of dynomite and conductor

[kelnos]

@kimabd sure. dynomite:

dyn_o_mite:
  auto_eject_hosts: true
  rack: us-east-1e
  distribution: vnode
  dyn_listen: 0.0.0.0:8101
  stats_listen: 127.0.0.1:22222
  dyn_seed_provider: florida_provider
  gos_interval: 10000
  hash: murmur
  listen: 0.0.0.0:8102
  preconnect: true
  server_retry_timeout: 30000
  timeout: 5000
  tokens: '2815085496'
  secure_server_option: none
  datacenter: us-east-1
  read_consistency: DC_SAFE_QUORUM
  write_consistency: DC_SAFE_QUORUM
  mbuf_size: 16384
  max_msgs: 500000
  pem_key_file: /mnt/dynomite/conf/dynomite.pem
  dyn_seeds:
  - [redacted]:8101:us-east-1d:us-east-1:1383429731
  - [redacted]:8101:us-east-1c:us-east-1:1383429731
  - [redacted]:8101:us-east-1e:us-east-1:1383429731
  - [redacted]:8101:us-east-1c:us-east-1:2815085496
  - [redacted]:8101:us-east-1d:us-east-1:2815085496
  - [redacted]:8101:us-east-1e:us-east-1:4246741261
  - [redacted]:8101:us-east-1d:us-east-1:4246741261
  - [redacted]:8101:us-east-1c:us-east-1:4246741261
  servers:
  - 127.0.0.1:6379:1
  data_store: 0

conductor:

APP_ID=conductor
EC2_REGION=us-east-1
EC2_AVAILABILITY.ZONE=us-east-1d
STACK=dev
environment=dev
db=dynomite
workflow.dynomite.cluster=dynomite
workflow.dynomite.cluster.name=dynomite
workflow.dynomite.connection.maxConnsPerHost=16
workflow.namespace.prefix=conductor
workflow.namespace.queue.prefix=conductor.queues
queues.dynomite.threads=32
queues.dynomite.nonQuorum.port=6379
workflow.elasticsearch.url=localhost:19300
workflow.elasticsearch.index.name=conductor

[kimabd]

@kelnos Thanks for fast reply , do you mention all redis hosts for workflow.dynomite.cluster.hosts option?

[kelnos]

@kimabd no, I've written a HostSupplier that uses our internal service discovery mechanism to provide the list of dynomite hosts. We're also using dynomite-manager, so I use HttpEndpointTokenMapSupplier in Conductor as well.