Identityserver4: How to run Identity on multiple servers under a load balancer?

[ryancosans]

We currently have Identity server running on two web servers under a load balancer. We have a single client application and single resource API, both running in .net core, running on two web servers under a load balancer as well.

We have an issue when authenticating with the client application it returns a 400 error. We think it is because the request is hopping between the two identity servers and is getting confused. The client application and resource API authenticate fine when Identity is running on a single server under the load balancer with the exact same settings.

An unhandled exception occurred while processing the request.

HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
MoveNext

AggregateException: Unhandled remote failure. (Response status code does not indicate success: 400 (Bad Request).)
MoveNext

We have double checked return URLs and other client settings on all our applications and it appears to be fine hence it working with a single instance of our Identity server.

We have a log output from a basic authentication request between the our client application and Identity server below.

Concatenated logs from both servers

[brockallen]

And you've read this? https://identityserver.github.io/Documentation/docsv2/advanced/deployment.html

[ryancosans]

Yeah we've read the docs and many other blog posts etc and are still a little confused. In the documentation it covers certs and we are currently using the default test certs (idsvr3test.pfx) on both servers due to being unsure of where we are meant to keep them. Also we aren't sure if we should have a single shared cert or multiple machine specific certs (or if this just our choice and it makes no difference).

We are currently loading the cert (.pfx) directly from the file system, not from a cert store.

var cert = new X509Certificate2(Path.Combine(_environment.ContentRootPath, "idsvr3test.pfx"), "idsrv3test");

[brockallen]

You need an SSL cert for HTTPS and that's used by IIS and that needs to be from a trusted CA. You need another cert for signing and that's used by IdentityServer and that can be a self-signed cert. You should not use the SSL cert for signing -- you want 2 distinct certs. Both are typically stored in the windows certificate store.

[ryancosans]

We got SSL setup no problem. The part we are confused about is the .pfx file we load in using the line of code above which as far as we're aware doesn't have anything to do with SSL.

Our question is what if anything do we have to do with that second cert (signing cert) to get it working on production or more specifically on multiple servers?

Did you manage to whiz through the log attached above. We have the following, does this mean that there is an issue with the signing cert setup?

...
2016-09-23 15:09:47.443 +01:00 [Error] Authorization code cannot be found in the store: d2912c5e46194cd8a22cbafb3423e286
...

[brockallen]

which as far as we're aware doesn't have anything to do with SSL

You're right, it doesn't.

Our question is what if anything do we have to do with that second cert (signing cert) to get it working on production or more specifically on multiple servers

Just share it across all the servers. Or if you're using the cert store, export it from one, and import it to the other (and ensure the private key is included).

[ryancosans]

When you say share what exactly do you mean? The idsvr3test.pfx currently resides on all the Identity servers.

Also if possible we want to avoid using the windows cert store, are there any downsides to not using the cert store?

[leastprivilege]

The cert store is the most secure location on a Windows machine to store key material.

[masduo]

Ok, but is there a way to avoid using cert store (windows, azure keyvalut, etc.) or it is a must have? and if it is avoidable what are the steps to get it working on multiple instances of identityserver?

[leastprivilege]

It is not a must have. Just a recommendation for situations where you have access to it. In Azure you can e.g. deploy the cert via the portal.

Any other mechanism of securely storing the cert is fine as well.

[ryancosans]

Wait why did this get closed my original question hasn't been answered.

I have a situation where I have "shared" copied the .pfx file to all the servers where Identity is running so each one has access to it (not sure if this what you meant by share since you didn't specify) and it doesn't work (not sure if it's the cert at this point).

I was hoping you would point me in any other direction than, or elaborate on, the documentation that I have read but maybe not entirely understood.

[leastprivilege]

Can't see any cert related problems in your logs. Rather there seems to be an issue with auhorization codes. How do you store them?

[ryancosans]

Thank you. That question pointed us in the right direction and we now have it running on multiple servers and successfully authenticating 👍

The way we were storing the operational data particularly the authorization codes... we weren't. So we looked at the Identity Server Entity Framework implementation and learned what we needed to implement from that, used the JSON serialization stuff to avoid storing a big ol' object and the rest from there was easy SQL.

[brockallen]

So we can close this now?

[ryancosans]

Yeah 👍

[masduo]

Thanks guys for your help!

[oakdemirci]

Hi all,

And you've read this? https://identityserver.github.io/Documentation/docsv2/advanced/deployment.html

it's an old post but
@brockallen is3 documentation ? Will we consider is3 documentation
or is4 version, or both?
http://docs.identityserver.io/en/release/topics/deployment.html
About the second; how can I determine the required Forwarded Headers ? By checking LB brand?
regards.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.