How about using SignalR for spreading out the Jobs to external workers ?
I mean, you then have just 1 connection to the database that fetches the jobs, and sends them out to all the connected workers.
So not every worker has to penetrate the database that much, and you don't need a new connection each worker.
What's your major concern about workers fetching jobs directly from the database? I don't think the difference between 1 and 20 connections is significant, and if you are concerned with performance or have thousands of workers, you could always go for the redis backend, which should be blazing fast for any number of connections.
Unless there's a more convincing reason I fail to see, I'd say it's a major effort for questionable gains.
I've tried to use just one connection for all the workers, this resulted in a performance degradation, especially when multiple processes are used. I like the idea of independent workers, with no centralized management much more.
SignalR would be a very heavy dependency for Hangfire (and for every other library or framework), IMO it should be installed only to target applications, because it is very complex by itself. Own broadcasting implementation is a very complex thing too.
Without a major performance increase, it is totally useless.
Most helpful comment
What's your major concern about workers fetching jobs directly from the database? I don't think the difference between 1 and 20 connections is significant, and if you are concerned with performance or have thousands of workers, you could always go for the redis backend, which should be blazing fast for any number of connections.
Unless there's a more convincing reason I fail to see, I'd say it's a major effort for questionable gains.