Prefect: Add RQ backend

[grozail]

This library is such a nice solution for my needs except for flow run backend.
Are there any thoughts to add rq as a backend to run flows?
Thanks in advance for your answers!

[cicdw]

Hi @grozail - happy to hear Prefect fits your needs! In terms of a backend / orchestration layer, that is actually the purpose of Prefect Cloud, which we are actively rolling out to early partners right now. Cloud ships with a database backend as well as a UI. I'd be happy to connect with you if Cloud access is something you might need for your work - just send us an e-mail using the link and reference this convo.

[cicdw]

My apologies, I read this too quickly --> I believe there _is_ room to implement an RQ executor in Prefect Core so I'm reopening this issue for that!

[grozail]

@cicdw Thanks for your reply.
Maybe I will try to implement it and submit a pr.

[cicdw]

That would be awesome @grozail! Let me know if you need any guidance. I recommend looking at the DaskExecutor as an example to work off of.

[grozail]

I have done naive implementation of RQ-based executor.
Now I'm writing tests for it.
Should I submit PR with a WIP status?
Also I have questions about DaskExecutor implementation, can I ask them here so that I could implement better rq-executor?
Moreover, should I write a PIN?

[cicdw]

Hi @grozail - opening a WIP PR would be fantastic! Feel free to ask any / all questions either here or in the PR, and I'd be happy to work with you on polishing the RQ executor.

You don't need to write a PIN for a new executor - we usually reserve PINs for fundamentally new ideas that require "architectural" decisions to be implemented. While a new executor is a very high value add, it most likely won't require new large scale changes to the Prefect codebase.

[grozail]

I have opened a PR.
The question about DaskExecutor is about its map behavior. According to test in test_executors.py, if I understood right, it maps a function over zipped args instead of mapping over each arg in *args.
What motivation stands behind this implementation?