Efcore: System.InvalidOperationException: 'Nullable object must have a value.'

[NetTecture]

with 2.1.1 (which incientally is the last published version, no trace of 2.1.2 on nuget).

The following is the LAMBDA executed:

``C# .Call System.Linq.Queryable.Select( .Constant<Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable1[Data.Core.OBTActionLog]>(Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable1[Data.Core.OBTActionLog]), '(.Lambda #Lambda1<System.Func2[Data.Core.OBTActionLog,Api.Odata.ActionLog]>))

.Lambda #Lambda1 .New Api.Odata.ActionLog(){
Created = $dtoOBTActionLog.DCreateDate,
Element = (Api.Odata.ActionLogElements)$dtoOBTActionLog.IElement,
Equipment = .If ($dtoOBTActionLog.OBTInstallEquipment == null) {
null
} .Else {
.New Api.Odata.Equipment(){
Id = ($dtoOBTActionLog.OBTInstallEquipment).PkInstallEquipmentID,
Identity = .Call (($dtoOBTActionLog.OBTInstallEquipment).GIdentity).ToString(),
Comment = ($dtoOBTActionLog.OBTInstallEquipment).SComment,

        }
    },
    Id = $dtoOBTActionLog.PkActionLogId,
    IsAnynomous = $dtoOBTActionLog.BIsAnonymous,
}

}

This is a relatively easy projection. I have removed the actual filling of a number of properties, but the gist is here. Equipment is working perfectly when asked for and projected to it's own DTO.

There is a null check in place, so no, there is no reason for it to fail like this.

The corresponding stack trace is:

at lambda_method(Closure , QueryContext , TransparentIdentifier2 ) at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.ProjectionShaper.TypedProjectionShaper3.Shape(QueryContext queryContext, ValueBuffer& valueBuffer)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func3 operation, Func3 verifySucceeded) at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable1.Enumerator.MoveNext()
at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor1.EnumeratorExceptionInterceptor.MoveNext() at System.Collections.Generic.List1.AddEnumerable(IEnumerable1 enumerable) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Api.Odata.Web.Code.ODataEntityController2.Query(ODataQueryOptions1 options) in C:\Work\OBB\Source.BackendApi.Odata.Web\Code\ODataEntityController.cs:line 127
at Api.Odata.Web.Code.ODataEntityController2.Get(ODataQueryOptions1 options) in C:\Work\OBB\Source.BackendApi.Odata.Web\Code\ODataEntityController.cs:line 105
at lambda_method(Closure , Object , Object[] )
at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
```

This should work without any issues. Any null condition (in this case that OBTActionLog.OBTInstallEquipment is null - is already checked

Sidenote: it is possible that this is a "broken relationship" - i.e. there isa value in the foreign key field, but no corresponding entry exists in the database. While this NORMALLY indicates missing relational integrity, this is not a bug in this place. Actually the foreign eky relationships are in place (for code generation) but disbled. The ActionLog table in this case is supposed to survive when referenced objects get deleted and the last entry in it will be "deleted" (i.e. you can see an entry then for the not more existing equipment 33344 and it says in a field that this item was deleted). While this is disputable from a db point of view, it is not from an ORM view and should defeinitely be supported. No problems with EF6.

[NetTecture]

If anyone can point me where to put a unit test for that in the 2.1 tree (i.e. which class should have this unit test) I am more than willing to give this a try on my own.

[NetTecture]

@ajcvickers I have some more information to that.

On a :1 relationship (a -> b) where a has a foreign key value where the other object (b) is not existing (which comes from not there as in foreign key relationsips not enabled, but can also come from.... the object being filtered out in a global filter..... because while the user can see a, the user is not allowed to see b)

You can not ask for a when the b "property" is in the object being queries. Which means you can NEVER ask for the object, you ALWAYS have to put in a projection, which does not include the property pointing to b. Which is a ridiculous amount of work because I need to make DTO for every single combination of possibly visible objects.

Because EfCore.... will throw this exception when it does not find b and has a foreign key value in a that points to an object existing.

This is brutal. Takes the whole global filter mechanism and throws it into the garbage, because everytime I filter out an object I may run into a little issue that objects the USER can see are not materializable. In fact I got that now through half an API because I filter out some objects (b) for security reasons and... whow. Red unit tests on half the objects because some B are now hidden for non admins.

Stack trace:

   at lambda_method(Closure , QueryContext , TransparentIdentifier`2 )
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.ProjectionShaper.TypedProjectionShaper`3.Shape(QueryContext queryContext, ValueBuffer& valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
... my code from here

[smitpatel]

The issue indicate that a null value from database was read which was being assigned to non-null property. That is all information possible out of just stack trace. In the absence of posting any CLR types involved in the query, it is difficult to figure out where is nullability mismatch is happening.

[smitpatel]

EF Team Triage: This issue is lacking enough information for us to be able to effectively triage it. In particular, it is missing the following information requested in the new issue template. Can you please provide this information?

Steps to reproduce

Ideally include a complete code listing that we can run to reproduce the issue.
Alternatively, you can provide a project/solution that we can run.

BTW we're not just doing this to be mean :smile:... we get a lot traffic on this project and it takes time to attempt to reproduce an issue based on fragments of information. In addition, our attempt is often unsuccessful as the exact conditions required to hit the issue are often not explicitly included in the code provided. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we ask that folks give us a self-contained way to reproduce an issue.

For a guide on submitting good bug reports, read Painless Bug Tracking.

BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

[NetTecture]

@smitpatel Please. Read. The. Error. Description. Damn.

I say exact steps to reproduce. One can take the horse to the water - but you can not make it drink. I can take you to the bug, but I can not make you read the error. Shame. For. Developers.

Yes, there isa null value read from the database. And it is because - oh, wait:

Because EfCore.... will throw this exception when it does not find b and has a foreign key value in a
that points to an object existing.

Yes. RIGHT THERE. It blindly assumes that because a.foreignkeypropertyforb has a value, b actually exists. Which may not be the case for various reasons - at the end, though, an ORM is not responsible for envorcing referential integrity on a database when not told so. In this case, this is a logging table that can point to objects deleted.

II gladly make a full test case. €90 per hour because my customer does NOT pay for this and I have better things to do with my personal time than writing test cases for one of the largest IT companies in the world for free. Personally I prefer dumping the POS that EFCore still is and going back to EF because I have WAY too many problems. Queries that work when I pull data into memory (and return a fully materialized List and fail when they run against the database are so brutal that I basically have a switch that determines whether or not. I had to f* write my own lambda walker to extract all necessary includes and now when I ever find ONE include, I assume the query blows and materialize in full. SHAME. But the only way to get any result out of Microsoft EF Core with Microsoft OData on top.

[ajcvickers]

@NetTecture Please keep conversations civil and respectful as described in the code of conduct.

I am genuinely sorry that EF Core is not working out for you. We have taken on board issues with the way querying works and are taking concrete steps to improve this, as I outlined in a previous response. We would be happy to hear any constructive ideas that you have on how we can do better. However, we will not engage in conversations that are not civil and respectful.

[ajcvickers]

We discussed this in triage, and based on information above it sounds like the data does not comply with the model definition. Specifically, that a non-nullable property exists for which the database is returning null values. Such cases will cause an exception to be raised since there is no way to populate a non-nullable property with a null value. Therefore, we are closing this as by-design. As stated above, if this is not the case, then we will need a runnable project/solution or complete code listing that demonstrates the behavior you are seeing.

[klaussj]

I'm having the same issue, and it's a real problem for me.

I made a gist to reproduce. Can you re-open please ?

[ajcvickers]

@klaussj Filed a new issue: #13517

[NetTecture]

Whow. Never would have thought to look at the lower level filters for that. Ouch.

[johnkwaters]

I have a similar issue. I have a trip object that points to a device object. The device has been soft deleted, but not the trip. I want to retrieve the trip with its device using ProjectTo. However, I get this error. The soft delete filter is in global query filters. The only two solutions I can see are:

1. Disable global query filters on this call (since I want to "see" deleted data). But I do want my multi tenant global queries still in place... so that's not an option
2. Read the parent object first without the reference, then read the referenced object without the global query filter, then piece it back together.
   Neither is a great option. Any other ideas?

[seekingtheoptimal]

While not sure if there is an issue for this open somewhere, in 2.2.3, the steps to reproduce are:

• have a nullable nav property
• in a query, when you try to check if it exists, like MyEntity.SomeNullableNavProp != null
  Then it throws this exception.
  If I add the actual Id field to my model and use that as a null check, like MyEntity.SomeNullableNavPropId.HasValue, then there is no random InvalidOperationException thrown by EF trying to instantiate the nav property even though it does not exist.
  Regardless whether it is by design or not, I agree with those who say that this is an intuitive way to check if a nav property is null or not.