Efcore: Unbound navigations for migration purposes have regressed

[NinoFloris]

Unbound navigations for migration purposes have regressed

This scenario apparently used to be supported.
The usecase here is when you want migrations to track your table "as is", including any foreign keys you may not use in the app (yet).
See this 1.0 issue https://github.com/dotnet/efcore/issues/2140

Steps to reproduce

Today, trying this:

// b: EntityTypeBuilder<Bar>
b.HasOne(typeof(Foo), "Foo").WithMany();

Throws during dotnet ef migrations add:

The navigation property 'Foo' cannot be added to the entity type 'Bar' because there is no corresponding CLR property on the underlying type and navigations properties cannot be added to shadow state.
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.HasNavigations(Nullable`1 navigationToPrincipal, Nullable`1 navigationToDependent, EntityType principalEntityType, EntityType dependentEntityType, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.HasNavigations(Nullable`1 navigationToPrincipal, Nullable`1 navigationToDependent, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalRelationshipBuilder.HasNavigation(String name, Boolean pointsToPrincipal, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.HasRelationship(EntityType targetEntityType, Nullable`1 navigationToTarget, Nullable`1 inverseNavigation, Boolean setTargetAsPrincipal, ConfigurationSource configurationSource, Nullable`1 required)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.HasRelationship(EntityType targetEntityType, String navigationName, ConfigurationSource configurationSource, Boolean setTargetAsPrincipal)
   at Microsoft.EntityFrameworkCore.Metadata.Builders.EntityTypeBuilder`1.HasOne[TRelatedEntity](String navigationName)
   at AppDbContext
   at Microsoft.EntityFrameworkCore.ModelBuilder.Entity[TEntity](Action`1 buildAction)
   at AppDbContext.OnModelCreating(ModelBuilder builder)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.GetModel(DbContext context, IConventionSetBuilder conventionSetBuilder)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_3(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
   at Microsoft.EntityFrameworkCore.DbContext.Microsoft.EntityFrameworkCore.Infrastructure.IInfrastructure<System.IServiceProvider>.get_Instance()
   at Microsoft.EntityFrameworkCore.Infrastructure.Internal.InfrastructureExtensions.GetService[TService](IInfrastructure`1 accessor)
   at Microsoft.EntityFrameworkCore.Infrastructure.AccessorExtensions.GetService[TService](IInfrastructure`1 accessor)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(Func`1 factory)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(String contextType)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)

A workaround when the foreign key is named like the type is to do:

// b: EntityTypeBuilder
b.HasOne(typeof(Foo)).WithMany();

However this breaks down when there are multiple foreign keys to the same table (with different column names) as EF will just number them sequentially. (foo_id, foo_id1, foo_id2 etc)

_I might be able to fix that naming afterwards by digging deep into the metadata but it would be very helpful if the original syntax would just work again. Though if you have a snippet that would do the trick that would be very helpful in the meantime._

Further technical details

EF Core version: 3.1.3
Database provider: EF PG

[smitpatel]

The scenario you describe was not supported with the issue you linked either, unbound navigations are only supported on entity types which are in shadow state (i.e. without any backing CLR type).
In your case entity type builder is backed by Bar CLR type.

[roji]

FYI I suggested (offline) to simply try a relationship without a navigation property, and in addition making the foreign key a shadow property. @NinoFloris let us know if that works for you.

[NinoFloris]

@smitpatel I wasn't able to let migrations handle shadow types, dotnet ef threw an exception in the direction of "a valid model Bar cannot be a shadow type"

[smitpatel]

Unbounded property from migration context only works inside the model created by ModelSnapshot, which is used in diffing.

[NinoFloris]

@roji thanks that absolutely works well enough, silly of me to miss it, I have even used this api a few years back in another app...

@smitpatel can you elaborate? I'm not sure what the difference is.
Let me rephrase, can I configure shadow types in OnModelCreating that are valid for migrations? (or do I always need to create a CLR type for them)

[smitpatel]

EF Core does not support shadow entity types. See https://github.com/dotnet/efcore/issues/749
Since model snapshot tracks model from previous iteration, user types from which could have changed or deleted, it cannot use actual CLR types. Hence model snapshot explicitly use shadow entities. #748
It works out fine since model snapshot is never used in runtime hence those types are actually not needed.

2140 allowed shadow navigations on shadow entity types only. Support for shadow navigation on non-shadow entity type is tracked by #3864

tl;dr - As of now, you cannot use a shadow type or a shadow navigation in OnModelCreating.

Can you elaborate that what you mean by "for migration purposes"? OnModelCreating is used to build the model for all purposes. There is no model building which user can do which is specific for migration.

[NinoFloris]

There is no model building which user can do which is specific for migration.

@smitpatel exactly, this is what I thought. This is the crux of it, if the feature does exist, how can I opt into building shadow entities, if I purely want them specified for migration purposes.

I don't want or need them to exist at runtime, I just need them to be managed by EF Core's migration tooling.

The reason is simple, a legacy codebase we're 'strangling' currently manages migrations, we want to cut this over to EF Core.

However we need to retain some of these legacy tables and foreign keys as the legacy app is still actively querying that database.

[smitpatel]

I don't want or need them to exist at runtime, I just need them to be managed by EF Core's migration tooling.

Can you elaborate how do you want them to be managed? It looks to me that in your scenario, you are mapping it to an existing database. If you are not using EF Core to create the database, then migration do not touch any additional objects you have in the database. So I expect even without mapping those legacy tables in EF model, they will remain as is in the database after using migrations.

[roji]

@NinoFloris if I understand correctly you want to use EF Core to manage some tables which exist in your legacy database, with including the corresponding CLR types in your code model.

Aside from @smitpatel's suggestion to use migrations to manage only some of the tables in the existing database, out of curiosity, is there a specific reason you can't just add CLR types for these tables without actually using them (e.g. under some namespace nobody cares about)?

[NinoFloris]

@smitpatel with our running databases this works. However we do spin up new databases per tenant from time to time, these tables still need to be there in that case too. Legacy is an active participant in the functionality of the product, sadly it cannot be omitted.

@roji no reason, it's how I did it at this moment (empty private classes in the DbContext), my question was one out of curiosity as well ;)

[roji]

OK. Note that we do have property bag entities (#2282, see also #9914), which are related to shadow entities.

Is there anything outstanding we want to track here?

[NinoFloris]

That would allow someone to successfully specify type Dictionary<string, object> for all unused tables right?

If so, that would work really well.

[roji]

Yep, that would work.