Identityserver4.admin: Docker deployment login loop

[cculver]

I've got an issue where I'm trying to deploy the Admin UI and the STS server. The process works fine locally, but once deployed to production through docker and put up behind a reverse proxy providing TLS offloading, the process gets stuck in a loop. I've been all over the internet trying to figure this out, have modified my nginx configuration extensively to try and get this working, have incorporated ForwardedHeaders (including the reverse proxy IP, both v4 and v6) and have spent a good bit of time investigating myself. I think I may have enough information for you to quickly identify the problem, because I believe it lies in Identity Server somewhere or with the configuration I've specified.

Most of the settings when running locally are defaults. In the configuration file running in production I replaced all instances of localhost: with the appropriate FQDN. They both operate off of the same database, so to get the redirection working properly, I also created a client for the public-facing Admin UI. The settings reflect the local one but with the appropriate FQDN and specifying https instead of http like it does for the default seed data. The client secrets are identical, as are the other elements that might matter. I can look in each table in the database and find identical values, only replaced with the right FQDN. When logging in, my logs indicate that the Admin UI was unable to locate the AspNetCore.Correlation.oidc cookie.

I took a look at the traffic that happens locally and through nginx with both Fiddler and Wireshark. They both report drastically different behavior between the two sites right after logging in. Localhost, which again works perfectly, submits a POST to AdminUILocal/signin-oidc with the following cookies:
.AspNetcore.Antiforgery.<...>
.AspNetCore.Correlation.oidc.<...>
.AspNetCore.Identity.Application
.AspNetCore.OpenIdConnect.Nonce

I'm not sure how, because no response prior sets these cookies, but I'm just going to accept it as is for now because it works. On the other hand, the public facing version submits a POST to AdminUIPublic/signin-oidc with no cookies whatsoever. Upon seeing this, we get redirected back to STS to /connect/authorize. Then it just goes in a loop between those two.

Does this indicate anything to you?

[cculver]

Some additional info from tonight's debug run. When running on localhost and comparing verbose log messages against production's, I see three log messages that are missing when running in production. These messages are:

1) Augmenting SignInContext
RequestPath: /connect/authorize/callback
SourceContext: IdentityServer4.Hosting.IdentityServerAuthenticationService

2) Performing protect operation to key {3a209bdd-eeac-42c0-b8a0-f8c26c952728} with purposes ('STS.Identity', 'Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationMiddleware', 'Identity.Application', 'v2').
RequestPath: /connect/authorize/callback
SourceContext: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector

3) AuthenticationScheme: Identity.Application signed in.
RequestPath: /connect/authorize/callback
SourceContext: Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler

These log messages happen prior to the postback to my admin server's /signin-oidc url. I have a certificate being loaded up that I self-signed and the STS server did not complain. It seems to match behavior I'm seeing in the previous post in that it appears that in production, it is not able to encrypt the information contained in the cookie for whatever reason.

Does this help?

Do these mean anything?

[skoruba]

@xmichaelx - any idea with this issue? I think you have similar docker setup. Thanks!

[cculver]

Just a note from debugging today. I attempted to get my local configuration as close to what would be running in docker as possible, getting rid of anything IIS related and making sure the .net framework versions were exactly equal. I didn't see any change in behavior, but I did notice that the three log messages I mentioned were missing are there now. I'm not sure anything I did had any reason to change this, so maybe I missed them before. So, back to basics: the required cookies aren't being passed along with the request and that's causing a correlation failure. Does this have anything to do with the DataProtection API in linux? I'm not doing anything there but have read that in server farms that's necessary, but for now my application seems to be generating its own keys and is the only one there. I'm lost.

[cculver]

One final note, if I didn't mention before. I haven't changed anything at all from the defaults with respect to CORS. Could this have an effect?

[skoruba]

BTW: Which configuration for Docker do you use? From some PR in this repo or yours? I have currently trouble with docker in general, therefore I can not provide useful advice for you. Maybe someone will help us with this issue. 😊

[cculver]

I'm using docker-compose, which as I'm sure you know sits on top of docker and uses a yaml file to specify options. The script is as simple as possible and in each of the two services, the build context refers to a dockerfile that simply contains the dockerfile. In each docker file the only command is FROM /. I build the docker images locally and push them out to a container registry to be pulled down from this server.

version: '3'

services:
  identityadmin:
    build:
      context: AdminServer/
    restart: always
    ports:
      - "5002:80"
    networks:
      - identityadmin
    depends_on:
      - identityserver

  identityserver:
    build:
      context: LoginServer/
    restart: always
    ports:
      - "5001:80"
    networks:
      - identityserver

networks:
  identityadmin:
    driver: bridge
  identityserver:
    driver: bridge

My nginx reverse proxy forwards requests to the appropriate ports along with X-Forwarded-For headers. I've confirmed that STS and admin server are both able to replace the appropriate scheme and original ip address for the requests.

[skoruba]

How did you solve this issue? 😊

[cculver]

Somewhat embarassingly given the amount of time I've put into this: more careful inspection of the headers being returned upon the first 302 redirect from the admin server. Before I explain, I want to say that I appreciate now everything this project is doing. I've been through the code up and down and through debugging it I now have a good understanding of how much time goes into making something like this, so I certainly appreciate your and your team's efforts. I hope not much time was spent helping me with this. It ended up being my nginx server - the system time did not match the kestrel server's time and the correlation and nonce cookies were actually being set, but ignored by the browser! nginx was putting a datetime into the response which rendered the cookies invalid based on their own expiration date.

Also, because Identity sends large headers/cookies, I ended up having to add the following lines to my nginx configuration files:

nginx.conf:

http{
...
proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;
large_client_header_buffers 4 16k;
...
}

default.conf:

location /{
    ...
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
    ...
}

I'm just happy to be done with this.

[skoruba]

Thank you very much for your feedback, I will put this information into some docs - I don’t use linux and nginx, that is currently out of my scope of knowledge. 😊 I am able to test everything on windows only, so I happy for your info. 👍🏼 If you will have any information for deploy on linux, please share it with us. 😎