Server: Self hosted bitwarden login always fails the first time

[Pulsar]

Setup

on global.override.env
globalSettings__sqlServer__connectionString= ...Connect Timeout=480;...

Server load

Description

Bitwarden throws an unhandled exception when trying to log-in. Also a notification saying an unknown error has occurred. It seems that failure rate can be changed by running or stopping services on the server.

Logs

bitwarden-identity         | crit: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity         |       Unhandled exception: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
bitwarden-identity         |       The statement has been terminated.
bitwarden-identity         | System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
bitwarden-identity         | The statement has been terminated.
bitwarden-identity         |  ---> System.ComponentModel.Win32Exception (258): Unknown error 258
bitwarden-identity         |    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
bitwarden-identity         |    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.RunParser(BulkCopySimpleResultSet bulkCopyHandler)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinuedOnSuccess(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         | --- End of stack trace from previous location where exception was thrown ---
bitwarden-identity         |    at Bit.Core.Repositories.SqlServer.EventRepository.CreateManyAsync(IList`1 entities)
bitwarden-identity         |    at Bit.Core.Services.RepositoryEventWriteService.CreateManyAsync(IList`1 e)
bitwarden-identity         |    at Bit.Core.Services.EventService.LogUserEventAsync(Guid userId, EventType type, Nullable`1 date)
bitwarden-identity         |    at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.BuildSuccessResultAsync(User user, ResourceOwnerPasswordValidationContext context, Device device, Boolean sendRememberToken)
bitwarden-identity         |    at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
bitwarden-identity         |    at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
bitwarden-identity         |    at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
bitwarden-identity         |    at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
bitwarden-identity         | ClientConnectionId:97263c8c-9de9-4920-9039-7ac733f1834d
bitwarden-identity         | Error Number:-2,State:0,Class:11
bitwarden-identity         | fail: Microsoft.AspNetCore.Server.Kestrel[13]
bitwarden-identity         |       Connection id "0HM05FFJB15KH", Request id "0HM05FFJB15KH:00000001": An unhandled exception was thrown by the application.
bitwarden-identity         | System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
bitwarden-identity         | The statement has been terminated.
bitwarden-identity         |  ---> System.ComponentModel.Win32Exception (258): Unknown error 258
bitwarden-identity         |    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
bitwarden-identity         |    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
bitwarden-identity         |    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.RunParser(BulkCopySimpleResultSet bulkCopyHandler)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinuedOnSuccess(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         |    at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
bitwarden-identity         | --- End of stack trace from previous location where exception was thrown ---
bitwarden-identity         |    at Bit.Core.Repositories.SqlServer.EventRepository.CreateManyAsync(IList`1 entities)
bitwarden-identity         |    at Bit.Core.Services.RepositoryEventWriteService.CreateManyAsync(IList`1 e)
bitwarden-identity         |    at Bit.Core.Services.EventService.LogUserEventAsync(Guid userId, EventType type, Nullable`1 date)
bitwarden-identity         |    at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.BuildSuccessResultAsync(User user, ResourceOwnerPasswordValidationContext context, Device device, Boolean sendRememberToken)
bitwarden-identity         |    at Bit.Core.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
bitwarden-identity         |    at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
bitwarden-identity         |    at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
bitwarden-identity         |    at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
bitwarden-identity         |    at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
bitwarden-identity         |    at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
bitwarden-identity         |    at IdentityServer4.Hosting.MutualTlsTokenEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
bitwarden-identity         |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
bitwarden-identity         |    at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
bitwarden-identity         |    at Bit.Core.Utilities.CurrentContextMiddleware.Invoke(HttpContext httpContext, CurrentContext currentContext, GlobalSettings globalSettings)
bitwarden-identity         |    at Bit.Core.Utilities.ServiceCollectionExtensions.<>c__DisplayClass10_0.<<UseDefaultMiddleware>b__1>d.MoveNext()
bitwarden-identity         | --- End of stack trace from previous location where exception was thrown ---
bitwarden-identity         |    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
bitwarden-identity         | ClientConnectionId:97263c8c-9de9-4920-9039-7ac733f1834d
bitwarden-identity         | Error Number:-2,State:0,Class:11

[kspearrin]

Looks like your server may be overloaded. RAM seems maxed out. Can you increase?

[Pulsar]

I can't increase RAM without buying/renting new Hardware. Is there a way to increase timeout expect in globalSettings__sqlServer__connectionString?

[kspearrin]

I am not familiar with a way to change timeouts off-hand.

[cscharf]

You can not increase a command timeout via the connection string for SQL Server, that is set per command (vs. per connection) and .NET maintains a connection pool to run multiple commands for efficiency. It would require extensive changes to Bitwarden's DAL in order to facilitate a setting that would allow longer command execution and to make that configurable (we would never want that in our SaaS hosting cloud environment, for example).

How many organizations is this user a member of attempting the login? I would also take a look at that, while it shouldn't be a limitation, that call which is timing out may be hitting contention on your SQL Server instance via a table lock, there are simply too many records to insert into the dbo.Events table upon login, but the login is working, it's the event recording that's failing (it will record 1 record for the user + 1 record for each org the user is a member of in a Sql Bulk command).

[Pulsar]

@cscharf , Thank you for your detailed reply.
I understand, that means It would be a great amount of work to change that.

It would be great if you consider a refactoring (other services under same conditions don't cause any problems), and as a motivation the following:
Currently Bitwarden is being considered of being promoted as a solution for Companies from 50- 300.000 Employees (no promise). Bitwarden Inc. will get the Licenses from the self-hosted services. As the customers want to host their services, we can't afford unavailable service, as RAM upgrade (on a VM), or a second instance could take over a week (sadly). Although their problem, the end result is that the service won't be adopted.

Anyway, I'll check for a workaround, if any, I will post it here :)

[cscharf]

We do have on our roadmap supporting multiple RDBMS outside of MSSQL which is loosely scheduled for development this year. That would include PostgreSQL and mySQL -> https://github.com/bitwarden/server/issues/453