Identityserver4: Stateless or not stateless?

[LindaLawton]

I just tried to spin up two instances of the identity sever. (yes load balancing is setup) The documentation says that Identity server 4 is stateless so this should work.

An unhandled exception occurred while processing the request.
CryptographicException: The key {ec55dd66-7caf-4423-9dd6-74768e80675d} was not found in the key ring.
Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, bool allowOperationsOnRevokedKeys, out UnprotectStatus status)

InvalidOperationException: The antiforgery token could not be decrypted.
Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(string serializedToken)

This makes me think the token was created on one instance and trying to be decrypted on the other.

[scottbrady91]

It's stateless but signing and encryption keys need to be shared :)

What you're seeing here is the DataProtection API using a different key and being unable to decrypt the anti-forgery cookie issued by a different instance.

[LindaLawton]

Ok let me see if i understand this. A new key is created for the identity server what when it starts? This key is used to verify the anti-forgery token. and create it or is it the client that's creating it?

So because I am running it with two instances I have two keys and if the user hits the server that didn't create its key it pukes?

Which would then mean i need to convince them to use the same key?

As you can see i haven't dug around in this part of IDS before. Would be nice to understand whats happening while i fix it.

[scottbrady91]

Pretty much. This isn't IdentityServer here but the ASP.NET Core data protection API: https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/

[leastprivilege]

well - RTFM ;)

https://identityserver4.readthedocs.io/en/release/topics/deployment.html

[LindaLawton]

@leastprivilege I actually have read about 90% of your documentation. However as you know security is a very extensive subject. Your documentation is written for people who are already experts of security not for beginners. Here copy this code and use it. Not this code does x and you should do x because of Y and if you have any issues please check stack overflow with this tag.

I would rather learn why something is done then just copy your code. Maybe someday i can become as wise as you guys :)

Solution

Now so that i can help someone else in the future.

Adding the following code to ConfigureServices will cause the key to be saved to a directory on my machine

services.AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(settingsSetup.Settings.PersistKeysDirectory));

I now have a key-f438262b-377e-4f12-a07b-f4af3737f8f3 sitting in the directory all fat and happy.

_I am still not sure how often these keys are created and if i need to set up a cron job to delete them every night._

The docker then needs to be configured so that all the instances of the identity server read from the same local directory. @scottbrady91 thanks for pointing me in the right direction that was a big help.

[brockallen]

I am still not sure how often these keys are created and if i need to set up a cron job to delete them every night.

They are re-gen'd each 90 days, but old keys are still needed. Check the MSFT docs on it (which are even more extensive than ours): https://docs.microsoft.com/en-us/aspnet/core/security/data-protection/

[LindaLawton]

I have moved on to this now key encryption

Ids is not as stateless as one would have thought apparently.

Current error.

idsrv was not authenticated. Failure message: Unprotect ticket failed

Followed by

CryptographicException: Unable to retrieve the decryption key.

[brockallen]

Ids is not as stateless as one would have thought apparently.

Normally stateless in web apps means the web server does not maintain in-memory state across requests to function properly. Would you call using a database stateless or not? Same goes for ASP.NET Core cookie authentication requiring state (keys) on the server.

[chrisowhite]

@LindaLawton IIRC, when supplying a thumbprint for resolving a cert on Windows in DPAPI, the cert is expected to be in the user store (not local machine) for cross platform compatibility between nix and Windows. We ran into this on net core 1.x and realized at the time that encrypting keys at rest using a cert was not supported out of the box on platforms other than windows. Given that we chose to just restrict file system access where the data protection keys are stored.

[leastprivilege]

Given you store your key container in a "secure" place in the back-end - the value of key encryption vs the overhead is questionable IMHO.

[LindaLawton]

@chrisowhite Thanks for the verification on this. That's exactly the conclusion I came up with yesterday. This is running in a docker swarm so its nix based which means our cert is not in the store its actually a file. Did you find any solution in running more then one instance of Identity server on a nix based OS? Its still saying "Failure message: Unprotect ticket failed" From what i understand this means that the keys must be encrypted. I was actually wondering if i could just put it in the DB that must be considered secure.

@leastprivilege right now I have a file share that all the docker nodes have access to. The keys are in this directory. All the instances of the identity server have access to the same folder in the file share they can read it and appear to be loading. But its appears that these keys because they are in a server environment need to be encrypted somehow. I am not getting an error when i run locally on my windows machine. Persisting keys when hosting in a Docker container

@brockallen I apologize for my Ids is not as stateless as one would have thought apparently. comment if it seamed harsh. When management reads something in the documentation that should work out of the box and then it doesn't. It can be frustrating for people like me who have to work out the logistics of why it doesn't work and how to make it work. And of course it should only have taken five minutes ...

And in the even this isnt an IDs issue and more of a Docker configuration issue i have posted this over on Stack Overflow Unprotect ticket failed

[brockallen]

Sure, but my point was that you would have had the same issue for any ASP.NET [Core] app. The issue of synchronizing keys has always been a task that one needs to consider for deployment.

[chrisowhite]

@LindaLawton we currently run on Windows and our keys are stored on a network share. If you're looking to store in a DB, you can reference the DPAPI source (or I think I saw a blog post with a Redis example at or point) for guidance but you're kind of on your own to implement that. If the apps are running on different physical/virtual hosts, inside a container or not, the keys need to be stored in a shared location.

[LindaLawton]

@chrisowhite I am trying to avoid putting them in the Db I would rather just have it in shared storage. I have gotten a bit further with it to day. Still not working though.

[heavenwing]

I also host IdentityServer in a cluster with front load balance,
I have added this code:

            var redis = ConnectionMultiplexer.Connect(redisHost);
            services.AddDataProtection()
                .PersistKeysToRedis(redis, "DataProtection-Keys");

I still got error:

idsrv was not authenticated. Failure message: Unprotect ticket failed

[leastprivilege]

Any update on the issue? closing for now - feel free to re-open if it needs further discussion.

[LindaLawton]

Ask for an update and close at the same time. No update still not working.

Can't run multiple instance in docker.

[leastprivilege]

Well - we can offer you commercial support. That seems to be the most sensible way to go forward.

Write us an email.

[LindaLawton]

That's not an option for me sorry.

I am sure i can figure this out when I have the time its currently my 20% time project. I am sure it will make a great blog post / tutorial / sample project when i am done as it appears i am not the only one having issues.

[leastprivilege]

That's not an option for me sorry.

May I ask why?

[LindaLawton]

This is my job I am in charge of the identity server project where I work, I was hired for my background in OAuth, Identity servers and security. I cant pay you to do my job that would kind of be defeating the purpose and I wouldn't learn anything.

Running multiple instances isn't something that must be supported right now its something we will need down the road. That's why i have taken it on as my 20% project once we get it running i will make sure to document it to help others.

I must say you guys have done a great job with this project. I have only been working with Identity server 4 since October its amazing how configurable you guys have made it.

[leastprivilege]

OK. no problem.

Closing this issue now - create a new more specific issue when the time has come. Also let us know when you documented it so we can incorporate it in our docs.

[LindaLawton]

Solution:

New settings

  "Redis": {
    "Enabled": true,
    "HostPort": "redis:6379",
    "ApplicationName": "Identityserver-v2-Production-Key"
  },

Add data protection in ConfigureServices.

if (Configuration["Redis:Enabled"] == bool.TrueString)
            {
                _logger.LogInformation(LoggingEvents.StartupConfiguringServices, "Redis Enabled.");

                services.ConfigureApplicationCookie(opts =>
                    opts.SessionStore = new RedisCacheTicketStore(new RedisCacheOptions()
                    {
                        Configuration = Configuration["Redis:HostPort"]

                    })
                );
                var redis = ConnectionMultiplexer.Connect( Configuration["Redis:HostPort"]);
                services.AddDataProtection()
                    .SetApplicationName(Configuration["Redis:ApplicationName"])
                    .PersistKeysToRedis( redis, "DataProtection-Keys");
            }

[inpicksys]

@LindaLawton Thank you!!! You save my life!

[tuongntk]

@LindaLawton Thanks so much. You're saving a lot of lives. Tks for your contribution and have a nice day

[TomCJones]

since I was the program manager long ago, I thought I would try to explain what DPAPI was designed to do and not to do. https://tcwiki.azurewebsites.net/index.php?title=Shared_Data_Protection#Context

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.