Note: this is a copy of a very old EF specification and reflects thinking from several years ago. A lot of things aren't valid anymore.
We define EF Core lifecycle hooks as the general feature that enables an application or library to sign up to be invoked or notified whenever certain interesting conditions or actions occur as part of the lifecycle of entities, properties, associations, queries, context instances, and other elements in the Entity Framework stack.
For example:
DbConnection
is opened (to use features such as SQL Server App Role)We want to enable customers to write business logic that triggers in the different stages of the lifecycle of these objects, following well factored coding patterns. We also want framework writers to be able to use these hooks to extend EF Core in useful ways.
In previous versions of Entity Framework we already exposed a few lifecycle hooks. For instance, we had the AssociationChanged
and ObjectStateManagerChanged
events since the first version, and the ObjectMaterialized
event was added in EF4. Up until EF6.x many of the existing hooks are not exposed in the DbContext API. In EF6 we also added several low level extensibility points in Interception that can be used too as lifecycle hooks.
There is a continuum of capabilities related and overlapping with lifecycle hooks, e.g.:
Framework/tool writers that need to extend the behavior of Entity Framework to support new application scenarios or to integrate it with other products or frameworks can write their own frameworks on top of EF that customize the behavior, collect data about the execution of specific actions, etc. Someone can write a full-fledged profiler taking advantage of the hooks.
Should work well and be consistent with DbContext design principles
There are not only different interesting conditions or actions an application may need to listen to, but also different kinds of mechanisms to implement hooks that present distinctive characteristics along the following dimensions:
Events are the most common hook pattern that almost every API in .NET uses. Events are messages sent by sender object to one or more receiver objects through a multicast delegate that acts as a dispatcher. Among the characteristics of events, they support runtime subscribe/unsubscribe, multiple listeners, and are relatively easy to use. Events can be slower than other hook mechanisms, but they have the advantage of being very discoverable (they are usually public members on the sender object) and familiar to customers. Events also provide a standard way to model cancellable actions with CancelEventArgs.
Virtual methods require the application code to declare a derived type and override the method. Virtual methods provide better performance than events but are slower than regular method invocation. Virtual methods are easy to discover and provide a very nice model for overriding/customizing default behavior and chaining with subsequent derivate types. Visual Studio provides a nice Intellisense experience for virtual methods: when you write the overrides keyword in C#, Intellisense provides the list of all the virtual methods available.
A more efficient alternative to events, regular (non-multicast) delegates can also be used as a hook. Users can normally provide some implementation of a predefined delegate signature (usually a Func Partial methods were introduced in .NET 3.5 as a means to extend generated code in separate partial classes. Partial methods are void methods that are both defined and invoked in the right places in generated code. Users can choose to provide the implementation of partial methods in a separate partial class, and the compiler will resolve the partial method to the implementation provide by the user. When the implementation of a partial method is not provided, all calls to the method and its definition are removed by the compiler. Since partial methods are either turned into regular methods or removed, the mechanism is extremely efficient. Partial methods are discoverable because Visual Studio provides a nice Intellisense experience for them. Similar to virtual methods, once you write the keyword partial inside the partial class, Visual Studio editor will list all the partial method definitions that havenât been implemented. The concept of magic methods is that the user can write a method that follows a particular naming convention and signature, and a framework component will make sure the method will get invoked automatically at runtime. Usually an expression is compiled at runtime to produce a delegate that be used to invoke the method multiple times very efficiently. There is some runtime overhead in compiling the expression, but this is paid only once. Magic methods can be instance or static method. A common practice for magic methods, when code generation is involved, is to provide the declaration of the magic method as a partial method. Although no actual calls are generated, the partial method that gets an implementation will be compiled into the assembly so that it can be invoked at runtime. The only reason for this is the Intellisense experience you get. Similar to magic methods, an instance or static method with the right signature can decorated with a special attribute that specifies a runtime role for this method. The runtime examines types for the presence of these attributes and registers the method to be executed on the occurrence of certain conditions. This approach consists on defining a class that implements a custom interface provided by the framework and at a later point register an instance of this class as a listener for particular events. Usually, the interface defines one or more methods with a very specific purpose, but the listener class can be a composite of multiple interfaces. Discoverability can be improved in this programing model with a base listener interfaces and by placing all the related interfaces under the same namespace. In many situations a mechanism is required to handle a stream of asynchronous events coming from the same source. IObservable DbContext provide a very good place for us to focus when defining extensibility hooks for anything that has to do with the functions that they encompass: The one potential issue with this approach is that the hook configuration would become part of the model and therefore it would not be possible to change it once the context object has been instantiated. This might be an acceptable limitation however, since any necessary changes in behavior can be coded into the listener class itself. Making the configuration of hooks immutable also has the advantage of allowing for compiling the invocations to listeners into efficient delegates. We are trying to find a design that has the following characteristics: The following is an incomplete list that presents various hooks we could consider adding. Intentionally the list does not try to be specific on each hook about certain details: Name | Pri | Location | Cancel or override | Description & sample scenario Name | Description & sample scenarioPartial methods
Magic methods
Magic methods can be used to override default behaviors but it is necessary to expose a public method that implements the default behavior so that the user has the option to invoke this method from within the magic method if he doesnât want to completely override it.
LINQ to SQL uses this mechanism pervasively. Attributed methods
WCF Data Services uses this pattern pervasively.Listener interfaces
IObservable
Context hooks vs. entity and property hooks
But for hooks that are specific to the lifecycle of entity or complex types, properties, etc., other options exist:mb.Entity<Product>().Notify(new EntityLoadedListener());
mb.Entity<Customer>().Property(c => c.Orders).Notify(new PropertyLoadedListener());
EF requirements for a hooking mechanims
This requires further analysis, but it seems that it would be reasonable to support a mix of hook mechanisms (but not too many) to optimize for different scenarios.
The current thinking is that we would use a single generic class to represent a hook. The sender should just need to call a method or instantiate a class and call a method, nothing much more complex than the typical âOn[EventName]â pattern, and the hook class would make sure all the hook handlers are invoked.
We are also considering building a convention system for wiring up hook handlers in the entity types.
When designing this we should take advantage of any opportunity to reuse some hook mechanism as building blocks for others. For instance:
âLifecycle hooks list
-- | -- | -- | -- | --
QueryExecuting | 0 | DbContext | Yes | Query interception, custom query caching.
QueryExecuted | 3 | DbContext | No | When Execute happened, before the reader is read. Tracing?
QueryCompleted | 3 | DbContext | No | After DbDataReader is closed. Tracing?
EntityStateChanged | 0 | DbContext | No | Signals all state changes
EntityStateChanging | 3 | DbContext | ? | Undo changes or change proposed values before they are set?
ConnectionProvisioning | 2 | DbContext | Yes | Execute additional code to make sure the connection is alive, or do logging
ConnectionReleasing | 2 | DbContext | Yes | Cleanup something done during Ensure / StartUsingConnection
ConnectionOpened | 2 | Â | No | More likely for tracing. Since SqlClient has fixed invalid connection pools, then this is lower priority
ConnectionOpening | 1 | DbContext | Yes | Slightly simpler to use than Ensure/Start, would not require user to check current state. Could also be used for tracing.
ConnectionClosed | 1 | Â | No | Â
ConnectionClosing | 1 | DbContext | Yes | Slightly simpler to use than Release/Stop, would not require user to check the initial state. Could also be used for tracing.
OnModelCreating | 0 | DbContext | Yes | Tweak model before it is cached.
OnModelCreated | 1 | DbContext | Yes | Signal that the model is done and execute some custom code, possibly related to caching logic. . Issue: do we need this for ObjectContext? Issue: if the user is going to implement his own caching, we should have an abstract class or interface for that.
ModelCacheLookup | 2 | DbContext | Yes | Implement your own caching logic. Tracing?
ModelCacheHit | 2 | DbContext | Yes | Execute additional code when the model is found in the cache. Tracing?
EntityLoading | 1 | DbContext, DbEntityEntry | No | After object instance is created but before its properties are initialized. Can be used to reset a flag that will be set in newly created instances but shouldnât be set during initialization, i.e. for validation.
EntityLoaded | 0 | DbContext, DbEntityEntry | No | Can be used to setup anything after an object has been materialized, i.e. event handlers, flags, etc.
CollectionLoading | 1 | DbContext, DbEntityEntry DbCollectiohnEntry | No | Can be used to setup anything on a collection after it is created but before it is populated. Issue: Could be used to provide your own collection?
CollectionLoading | 1 | Context, Entity or Collection | No | Can be used to setup anything on a collection after it has been created and populated, i.e. listeners for its changed event.
ObjectTypeResolving | 1 | Context | Yes | Could be used to specify a different type than the original one, i.e. to implement your own proxy mechanism. It should be per type but could return a Func
Virtual OnSavingChanges | Medium | DbContext | No | Can be used to re-implement SaveChanges but still invoke the existing SavingChanges event
SavedChanges | Low | Context | No | Could be used to execute cleanup code after SaveChanges. For instance, to call AcceptChanges on each STE change tracker. It is lower priority because virtual SaveChanges covers most scenarios.
EntityStateChanging | Low | Context, Entity | Yes | For an entity instance or type in particular we could avoid putting in the modified state. So even if the properties are read-write, the context ignores changes to this entity. Could be also used to suspend fixup on an entity that is being detached.
EntityStateChanged | High | Context, Entity | No | Executes logic after an entity has been put in a certain state. Can be used to setup property values, restore state after the changing event.
PropertyChanging | Low | Context, Entity | Yes | Any time a property is about to be changed by the framework or any party, if notification or interception is enabled by the entity type. Should make original and new value available. Should also work for navigation, scalar and complex types properties. Tracing?
PropertyChanged | High | Context, Entity | No | Any time a change in a property value change is detected.
PropertyLoading | High | Context, Entity, Collection | Yes | Intercepts, overrides de loading of a property. Could be used to support loading of properties using stored procedures.
PropertyLoaded | Medium | Context, Entity, Collection | No | Tracing?
Writetable IsLoaded | High | Context, Entity | Yes | Allows cancelling the loading of a property.
CollectionChanging | Medium | Context | Yes | Any time a collection is about to be changed by the framework or any party, if interception is enabled
CollectionChanged | Medium | Context, Entity | No | Any time a change to a collection has been detected.
AssociationChanging | Low | Context, RelatedEnd | Yes | Can be used to prevent an association from being changed, or to execute business logic when the association is about to change.
AssociationChanged | Medium | Context, RelatedEnd | No | Can be used to execute additional logic after an association is changed, i.e. user can explicitly cascade relationships removals into dependent removals, workaround current databinding shortcomings.
RowValidate , RowValidateAdded, RowValidateModified, RowValidateDeleted | Medium | Context | Yes | Storage level version of ObjectValidate. Tracing?
SavingChanges event | 0 | DbContext | ? | Currently only available on ObjectContext. Should make trigger OnSavingChanges method protected.Existing hooks
-- | --
virtual Dispose | This can be used to do additional cleanup, i.e. on entity instances.
virtual SaveChanges | Can be used to execute additional logic before, after or instead of saving changes.Some open issues:
I started prototyping some ideas as a part of my investigation of setting SQLite pragma options when a connection opens. It seems this feature would be generally useful in solving several other feature requests. Can we considering moving this off backlog?
I think we need @divega in the office to design this one, the all up feature is pretty large and wide-reaching. We also have some higher priority things to work on first. If you are specifically looking at connection pre-amble then I think we can treat that as a smaller feature that we work on now.
Will this make it into the next release?
@rowanmiller In terms of an EF Core ObjectMaterialized style hook, can you give me some pointers as to where I can tackle this as of now?
Permission validation and filtering are other scenarios that could be attained with lifecycle hooks (see #6440).
Will this also take care of the following scenario?
I'd like to add an entity to a DbContext (to be inserted or updated etc), but also subscribe to be notified once the entity has been saved, so i can handle taking its newly updated values (i.e the database generated values that get poplated after a SaveChanges() - like it's ID etc) and do something with them.
var dbContext = GetExistingDbContextInstance();
var newItem = new SomeEntity();
dbContext.SomeEntities.Add(newItem);
// I am quite far down in an object graph, and SaveChanges() will eventually be called
// later on by something higher up, co-ordinating this transaction. I'd like to be notified
// here though once SaveChanges() has been called and this entity has been persisted, so I can grab the
// the updated entity values.
// dbContext.OnceSaved(newItem, ()=>{ // ooh someEntity.Id is now populated! })
Something along the lines of:
dbContext.OnceSaved(someEntity, ()=>{ // ooh someEntity.Id is now populated! })
Where the callback would be invoked after dbContext.SaveChanges() is called.
@dazinator I'm currently working on an extension project for EFCore (https://github.com/Antaris/EntityFrameworkCoreExtensions) which will allow you to do what you want, eventually.
I have a number of working hooks including change tracking, value materialization, querying and storage. It's still a WIP though.
@rowanmiller What's the expected release date on this? It's been open for almost 2-1/2 years
@sidshetye it's planned to be in our 2.0 release (at least the first set of hooks). No specific dates on that yet, but it's the release we are starting work on now.
In the mean time, using EF Core 1.1, what would be the recommended method for executing a SQL statement straight after a DbConnection is opened?
We are using RLS for implementing multitenancy and I'd like to know if there's any method for setting the TenantId until EF Core 2.0 is released.
@tpanthier You could try registering for an event on the DbConnection itself.
Punting this because we are out of time. /cc @divega
With this issue being open for 2.5 years and the various other issues that have been closed due to referencing this issue, I've lost track of whether two specific scenarios we care about are covered by other cases and are or are not supported in 2.0.
As part of our multi-tenancy solution, we currently subscribe to DbContext.Database.Connection.StateChange. Whenever the state changes to Open, we issue a SQL command to set the the current tenant id in session_context.
In EF 6, dates loaded from the DB have a kind of Unspecified. In order to coerce these into UTC, we're hooking into ((IObjectContextAdapter)DbContext).ObjectContext.ObjectMaterialized and calling DateTime.SpecifyKind for specific fields that we care about being in UTC.
Can you confirm if either or both of these scenarios will be possible with EFCore 2.0?
@michaelaird Unfortunately, there is no change on either of those issues for 2.0. There is a high chance that they will make it in for 2.1.
Got it. Thanks for the update.
Off-topic but slightly related: My observation of the 2.0 milestone across all the .net core projects is that it represents a solid "step up" in stability and general breadth of feature support. With that stability in mind, it would be great if there were more frequent pre-release packages available between 2.0 and 2.1. In particular it would help us get to try these features that just barely didn't make the 2.0 release....
@michaelaird I hear you. On the EF team we don't really control when the releases happen or how many pre-releases there are, but we do push for them to be frequent as much as we can.
Guys - do you genuinely think it's acceptable performance when you've had 2.5 years to track, plan and execute - and yet it consistently misses releases? This is open since Sept of 2014. Very disappointing. Especially knowing you guys are clever :(
hey @sidshetye , I don't know if you ship commercial software for a living, but I do. And I aim to get things into releases all the time that get re-prioritized by the business and other things that are more important get pulled in ahead. It drives me crazy, as I'm sure it does to the EF team, when I don't have enough hours in the day to deliver all the features that I know my users want.
Let's support the EF team in what they are delivering and not beat them up too much for what is missing the cut.
@michaelaird I agree, however negative feedback is as important as positive feedback and both kinds are better than no feedback at all. If you aren't getting any feedback it usually means no one cares! What you are witnessing is someone who cares enough about EF to express their dissapointment over an issue and I think it's perfectly valid and even useful for the EF team to hear that. Any attempt to silence such a thing would not be constructive imho. I am also certain that the EF team, being highly professional, are perfectly capable of handling both positive and negative feedback on issues without needing any one of us to jump in for them.
@michaelaird Yes, I ship commercially. Including security and voice for 60% of all mobile phones - globally. Code that defined "carrier grade". But making this about me is a distraction - the observation stands even if I personally never shipped a single line of code.
Since we're quickly off topic now, I'll get back with the following suggestions to the EF team:
I hope my/anyone's feedback doesn't distract from the teams achievements. At the end of the day, we're all trying to make the world a better place through better software. Just don't keep us waiting ;) !
Thanks,
Sid
This turned out longer than I expected, so I'll get back to work. Cheers.
As we don't have hooks yet, I would like to ask about DbContext.Database.Connection.StateChange
event. Is it safe to subscribe on it without un-subscribe? What is DbConnection
CLR object lifetime if EFCore is used in ASP.NET app? As I see, IRelationalConnection
is registered as Scoped
in DI, and DbConnection
's lifetime is the same. I could assume if the subscriber also is registered as Scoped
, we could relax and don't worry about memory leaking... or not?
@michaelaird, @ajcvickers could you help?
@ilya-chumakov When EF creates the DbConnection for you, then it's lifetime is the same as the DbContext--it will be disposed when the DbContext is disposed. You can also create your own DbConnection and pass it to UseSqlServer (or equivalent) in which case you can make the lifetime longer.
So, if your subscriber has the same lifetime as your DbContext and EF is creating the DbConnection, then the subscriber, context, and connection should all have the same lifetime.
@rowanmiller commented that some hooks are coming in 2.0, and in https://github.com/aspnet/EntityFramework/wiki/Roadmap there is a one-sentence description, but I haven't been able to find anything else on it. Is there more documentation or pointers out there?
I am interested in hooks for entity state transition. To this end I am using the following approach:
var localViewListener = dbContext.GetService<Microsoft.EntityFrameworkCore.ChangeTracking.Internal.ILocalViewListener>();
localViewListener.RegisterView(StateManagerChangedHandler);
However the callback for entity state transition does not provide all the information one might desire. Specifically, for example, if a transition from Modified to Unmodified occurs then we do not know whether this is due to AcceptAllChanges or due to some form of Reject Changes, or possibly due to some other reason. For that, you need to add a flag in your calling code. You need this information to know, for example, in your entity view model whether the current value has changed or the original value has changed. So when you get around to the hooks for entity state transitions, I would ask that you include some indication of whether AcceptAllChanges is in progress or RejectAllChanges is in progress (and I'm hoping you will add RejectAllChanges !)
@sjb-sjb if you look in the roadmap you will see that lifecycle-hooks is in the list of "Features originally considered but for which we have made no progress and are essentially postponed".
I will let @ajcvickers comment on whether there is any reasonable way to workaround current limitations and get notified of those changes by using internal APIs but without having to wrap the whole set of change tracking services, but that seems unlikely.
I can't think of a workaround off the top of my head.
Are there any, or going to be any, workaround/solutions for DbInterception? Based on this thread and the roadmap it seems that it's not going to be available for awhile. We are starting some new projects and want to use .Net Core / EF Core, but I'm not sure how to translate our existing DbInterception process.
We are using interception to inspect the sql query and replace [SECURITY_VIEW] with an actual view name. In our current system, each user has their own security view. Views could be added/removed each day, so we cannot create a static context.
In our latest code, which uses EF6, we tested dynamically compiling the model vs interception. Once we started load testing, we found interception to be the winner.
We could go back to writing sql stored procedures, but I'd prefer not to.
Any thoughts/ideas would be greatly appreciated.
@kdcarlisle42 You could investigate using the DiagnosticSource
events documented in https://github.com/aspnet/EntityFramework/blob/dev/src/EFCore.Relational/Diagnostics/RelationalEventId.cs
These events give you access to the DbCommand, etc, before it is executed, and it may be possible for code to manipulate these objects to do what you need. However, it's not as flexible as DbInterception in EF6. I would be interested in hearing about your experience if you go this route.
(Note that these events have changed significantly between 1.1 and 2.0, so if you write something for 1.1, then it will need to be modified to work with 2.0.)
I don't know what the limitations are that were referred to by @divega and @ajcvickers.
Could one not simply add an event like this to DbContext?
public event TypedEventHandler<IEntityStateWatcher, EntityStateTransitionEventArgs> EntityStateTransition;
public DbContext( ... )
{
...
var localViewListener = this.GetService<Microsoft.EntityFrameworkCore.ChangeTracking.Internal.ILocalViewListener>();
localViewListener.RegisterView(StateManagerChangedHandler);
...
}
private void StateManagerChangedHandler(Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry entry, EntityState previousState)
{
var entityEntry = entry.ToEntityEntry();
this.EntityStateTransition?.Invoke(this, new EntityStateTransitionEventArgs(entityEntry, previousState));
}
I've been following this issue for a couple of months, I need to perform a soft-delete feature in my Application, I know how to "override" actual deletes with "update the delete flag" but not a clue how to filter the "soft-deleted" records once a query is performed, this would be easy with hooks, but in this case I can't imagine how to workaround this.
@olman21 Take a look at Global Query Filters: https://blogs.msdn.microsoft.com/dotnet/2017/05/12/announcing-ef-core-2-0-preview-1/
Consider what can be done to observe transactions commit/rollback--see #9429
Can this task (if it ever gets prioritized) support passing some sort of timings in with the hooks?
Right now our Asp.net core with EF core running on Azure is having periods of non-reproducible high 75, 90, 95, and 99 percentile response times. We want to hook into DB operations so when our special request logging is returned it returns the # of queries and average db request/response time for each so we can see if this inconsistency is db related or not.
Right now it seems possible to do this for SaveAsync
and FindAsync
calls on the DbContext by overriding those calls and putting a stopwatch around them, but I don't see an easy way to get the timings of any Linq based queries, without manually adding code around every single EF core code block.
Still looking for ObjectMaterialized and SavingChanges equivalent hooks. With the passage of time (and .NET standard 2.0 out), do we have a cleared release date? Pretty please?
@sidshetye In case you haven't noticed this issue has been recently assigned to the 2.1 release as a "stretch priority". The exact hooks we will prioritize hasn't been decided yet, but your input will be considered.
@KallDrexx it sounds like you should be able to use the logging & diagnostics functionality in EF Core for this. Have you had a chance to try that?
cc @ajcvickers
@divega Maybe I am missing something but I can't find any documentation that shows how this can be effectively used, at least for my use case.
The first issue is I don't see much in the logging facilities that would allow me to output query times (including network times) so I am not totally sure how to grab that.
The second issue is that in production we whitelist what logs are produced to keep costs down and to keep logs concise and easy to search through (i.e. one log record per request with all relevant data in a json record). This makes it extremely easy to utilize tools such as Elastic Search to find complete details about a total request instead of making hard assumptions and having to coalesce 20 different log entries to get a complete view of a single request.
From what I can currently see, utilizing the current logging infrastructure means creating a custom ILogger
with a lot of non-trivial hoop jumping to get query data to flow out of our custom Logger into the request context, so at the end of the request the data can be coalesced into our simple request.
@KallDrexx if logging doesn't meet your requirements, EF Core also publishes diagnostics events using DiagnosticSource.
There is some information about how to consume this information programmatically at https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md#consuming-data-with-diagnosticlistener. It also possible to consume this diagnostics events through ETW on Windows.
Thank you very much, I had not known that there was a diagnostic system
built into .net core, so I will investigate that route. Really appreciate
the response.
On Mon, Aug 28, 2017 at 1:31 PM, Diego Vega notifications@github.com
wrote:
@KallDrexx https://github.com/kalldrexx if logging doesn't meet your
requirements, EF Core also publishes diagnostics events using
DiagnosticSource.There is some information about how to consume this information
programmatically at https://github.com/dotnet/
corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/
DiagnosticSourceUsersGuide.md#consuming-data-with-diagnosticlistener. It
also possible to consume this diagnostics events through ETW
https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md#consuming-diagnosticsource-data-with-eventlisteners-and-etw
on Windows.â
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/626#issuecomment-325420694,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIggYAUt_9e-jdhU8-9EajopCLRy0nWks5scvlugaJpZM4Cesel
.
@divega thanks for the update, missed that tag update. Though it's not very comforting that this is as low of a priority as 'stretch' / "if we have bonus time left".
To us this is a deal breaker for EF core (and therefore .NET core). Hope .NET core and EF core get more investments. It ultimately boosts Azure deployments - at least for us.
@sidshetye Although not ideal, a couple of possible workarounds for your needs are a custom Entity Materializer Source and another library EntityFramework.Triggers.
@ajcvickers Add hooks for customizing a model and the model cache key using with a context instance
For a concrete application of SavingChanged and ObjectMaterialized in EF6, see my article about hybrid storage on CodeProject.
Inspired by the Core way of doing things, I defined an extension method "UseHybridStorage()" to DbContext that registers to the ObjectMaterialized and SavingChanges events. Hence no subclassing is needed.
It would also be very usefull to have access to the Context from within those lifecycle hooks.
Would what I did in the article be possible with EF Core 2.1 too ? (See the OnObjectMaterialized and OnSavingChanges methods just above the "Entity State Handling Issue" paragraph.)
I think it would also be beneficial to have some form of hook to notify if a LINQ expression was not able to be translated. I am aware that this gets logged, but it would be helpful to have a programmatic way to escalate it from a simple WARN log to some other form of notification. This, of course, makes the most sense during development but it is not always obvious to the developers.
The primary situation I'm thinking of is one where a library (for json:api or graphql as examples) sits on top of entity framework and is responsible for interacting with EF. This would enable the library to surface these kinds of problems in a more obvious way to the developers and provide framework-specific solutions.
@janhartmann It is already possible to:
```C#
public class MyListener : IObserver
{
public void OnCompleted()
{
}
public void OnError(Exception error)
{
}
public void OnNext(DiagnosticListener listener)
{
if (listener.Name == DbLoggerCategory.Name)
{
listener.Subscribe(this);
}
}
public void OnNext(KeyValuePair<string, object> value)
{
if (value.Key == RelationalEventId.QueryClientEvaluationWarning.Name)
{
var payload = (QueryModelClientEvalEventData)value.Value;
Console.WriteLine("Client eval happening: " + payload);
}
}
}
Then somewhere in app startup:
```C#
DiagnosticListener.AllListeners.Subscribe(new MyListener());
This would be nice to have in docs
Thanks Erik. Filed https://github.com/aspnet/EntityFramework.Docs/issues/583
@ajcvickers @divega : is there a firm release date we can go by? I have multiple customers asking us to expand Crypteron to support EF Core. We've been deferring them to EF6 for years now but I'm wondering if 2017 will be special
I would like to renew my vote for state transitions to become part of the public API (see Jul 13 above).
@sidshetye Unfortunately we don't have any dates that we can share yet.
@ajcvickers That's disappointing. It makes roadmap planning impossible for your customers (us and in-turn our customers). We certainly appreciate the work by the team and I know it's comforting to not state a release date (that you're now held accountable to!). But may I challenge the team to step outside that comfort zone? There has got to be some plan and some target date ... doesn't help to keep the community in the dark.
Again, love the work - just wish there was more transparency to facilitate business.
@sidshetye Thanks for the feedback--I will certainly pass it on to those who make the calls on the release dates.
For anyone following this issue. in order to help us prioritize work can you let us know which hooks are important to you and what you intend to do with the hooks?
@ajcvickers
During a save operation of a business object, my parent / root business object typically calls SaveChanges() but only after passing the DbContext
instance to all its child business objects in the graph, so that they can add / update / delete entities on the DbContext in order to take part in the transaction. Once the entire business object graph has worked with the DbContext instance, the root business object will call SaveChanges()
. At that point, the database can generate values for all of those persisted entities. I would like for my Child business objects in the graph to be able to register their interest in entities that they have added / updated, so that after SaveChanges is called, they can get notified, and get access the new state of the entity now that it has any database generated values (such as primary keys etc) in order to update their own state to match.
A very useful use case of Lifecycle hooks is to implement Revisions (I've used this in several applications with EF6)
How it works: when a user edits an entity IE: Post, during save the EntityState is changed from Modified to Added and the RevisionNr is increased. This way as simple workflow, Load, Edit, Commit is used (that does not require copying object values and adding them as new) to create a Revisions based system that always inserts new revisions (instead of updating the original entity).
A second use case is to compute complex properties eagerly. Like decode a binary blob and update a specific graph (In cases where this cannot be done lazily), IE: loading complex scheduling rules and updating a scheduling graph (this will be done eagerly)
A third use case is Detailed Change Auditing (usually the more enterprisey the project, the, more this is required)
Our use case is to implement global query caching. Basically we'd need a hook to capture a read query just before its executed. If the key does not exist in the cache the query will be executed on the database and the result will be cached.
The other hook we'd need is when a transaction is committed (or rolled back), so we can invalidate the proper items in the cache. Overriding SaveChanges() will probably also work instead of this hook.
See also: https://github.com/aspnet/EntityFrameworkCore/issues/5858#issuecomment-306441806
In EF6 we used https://github.com/moozzyk/EFCache and we implemented our own Redis provider for this.
We have some "StatusService", where clients can subscribe (via .NET events or WebSocket) so they get notified when something changes. This includes some hysteresis / rate limiting of notifications.
Part of the changes are changes to (a subset of) our database entities, so they can react accordingly (e. G. update their UI, or schedule synchronization of some data, etc...).
So an "this database entity has changed" event is what we'd need.
(Currently, we do this via overwriting SaveChanges[Async]()
Methods which triggers events on an injected singleton service, and subscribers use the dbcontext ChangeTracker to check which entities they're interested.)
We have several uses for them:
1) In the OnSaving event (which we trigger from an overloaded SaveChanges) we update simple tracking information (UpdatedBy, UpdatedWhen,...).
2) A more advanced version of the tracking logs old and new values of changed properties in new entities.
3) Some objects have complex graphs of subobjects associated which we prefer to store as a JSON string in a single database field. We then use both OnObjectMaterialized and OnSaving to deserialize/serialize the JSON string and populate the entity. I have described this solution in detail on https://www.codeproject.com/Articles/1210354/Hybrid-Storage-relationalplusdocument-with-Entity
4) Though there should probably be a better solution for this one, but we capture the OnSaving event to throw an exception when an entity(type) is considered read-only and has a Changed state.
@codetuner For your use case 3 (JSON): https://github.com/aspnet/EntityFrameworkCore/issues/4021 may help there, and I could also envision a different implementation (with conversion logic in the Entity) which works without overriding those methods or LifeCycle hooks.
We have 2 cases
1 is really our primary concern, 2 is a nice to have.
Per my posts of Jul 13 and Dec 4, I am using hooks on the DbContext to detect entity state transitions. The ask is to include ILocalViewListener or similar in the public API.
I use the hooks primarily for two purposes
(a) to inform an observable list of entities of a change in entity state, so that the list can be maintained appropriately (e.g. entities added or removed from the selected set of entities). The observable list is being used as a view model, more or less like a customized LocalView but not attached to a DbContext. It is a free-standing list of entities that is being accessed by the end user. Individual units of work are sent in from various parts of the program and the list provides an integrated view of the resulting entities.
(b) to inform the view model of an individual entity that there has been a change of entity state. This is being used in an entity view model system along the lines of the one in Template 10 but including entity state information. When a property is changed in an entity, a unit of work is opened and the view model reflects a modified status for the property; when SaveChanges is used to complete the unit of work, the entity state transition hook is used to inform the view model that the property is no longer dirty (i.e. the value has been updated). This is very convenient because the callback provides exact information about the specific entity that was changed (or whose changes were rolled back), and there is no tight coupling between the part of the program that submitted the SaveChanges and the part of the program that is using the entity view model.
Broadly speaking, hooking into the entity state change provides a way of driving view model updates and maintaining a single consistent internal state (a) without having to explicitly tie together the receiving view and view model with the UI or other program code that was used to generate and submit the entity changes and (b) without having to reproduce the logic that EF uses to track state and to commit or rollback changes.
These uses are somewhat similar to the objective that @markusschaber cited ("changes to (a subset of) our database entities, so they can react accordingly e.g. update their UI, or schedule synchronization of some data") and to the use that @codetuner described ("logs old and new values of changed properties in new entities") although we are updating UI instead of logging.
Note: in this application, concurrency contention is low so it is OK for a copy of the data to stay around in the app for a while.
We need these/equivalents from ObjectContext
...
ObjectMaterialized
SavingChanges
And preferably another hook _after_ SavingChanges
completes (maybe SavedChanges
?) Hope to see these soon!
@michaelaird I think a general option to specify whether Timestamp columns are UTC or local would be a better alternative here, what do you think? (Maybe this should be discussed somewhere else?)
I think they should deprecate DateTime and âblessâ NodaTime (the way they
did with Json.net). But that is definitely a discussion for somewhere else.
That said we jump through hoops to translate nodatime types to datetime for
storage so a mechanism to define those translations at the dbcontext would
be hugely useful.
On Fri, Jan 19, 2018 at 3:02 AM Markus Schaber notifications@github.com
wrote:
@michaelaird https://github.com/michaelaird I think a general option to
specify whether Timestamp columns are UTC or local would be a better
alternative here, what do you think? (Maybe this should be discussed
somewhere else?)â
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/EntityFrameworkCore/issues/626#issuecomment-358893510,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANX6YLjFey2wq0V4r-xY9rCQwM30d9qks5tMEwegaJpZM4Cesel
.
@michaelaird you can workaround point 2 with something like:
public static class DateTimeMapper
{
public static DateTime SetUtc(DateTime value)
{
return DateTime.SpecifyKind(value, DateTimeKind.Utc);
}
}
public class MyMaterializerSource : EntityMaterializerSource
{
private static readonly MethodInfo SetUtcMethod = typeof(DateTimeMapper).GetTypeInfo().GetMethod(nameof(DateTimeMapper.SetUtc));
public override Expression CreateReadValueExpression(Expression valueBuffer, Type type, int index, IProperty property = null)
{
if (type == typeof(DateTime))
{
return Expression.Call(
SetUtcMethod,
base.CreateReadValueExpression(valueBuffer, type, index, property)
);
}
return base.CreateReadValueExpression(valueBuffer, type, index, property);
}
}
And when setting your DbContextOptions
, on your DbContextOptionsBuilder
:
options.UseSqlServer(/* ... */);
options.ReplaceService<IEntityMaterializerSource, MyMaterializerSource>();
I agree it could be more user friendly with a hook like ObjectMaterialized
, but it works.
For point 1, you can pass your own connection if you don't want the DbContext
to create it for you and handle StateChange
on it (for example, with UseSqlServer
, using this overload), for example:
var myConnection = new SqlConnection(connectionString);
myConnection.StateChange += (s,e) => { /* */ }
/* then, in your options builder */
options.UseSqlServer(myConnection);
@sidshetye just out of curiosity, what would you need on a SavingChanges
hook that can't be done by overriding SaveChanges
/SaveChangesAsync
?
You could implement that easily, with something like:
public class MyDbContext : DbContext
{
public event EventHandler SavingChanges;
public event EventHandler SavedChanges;
public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
SavingChanges?.Invoke(this, EventArgs.Empty);
var returnValue = base.SaveChanges(acceptAllChangesOnSuccess);
SavedChanges?.Invoke(this, EventArgs.Empty);
return returnValue;
}
public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = default(CancellationToken))
{
SavingChanges?.Invoke(this, EventArgs.Empty);
var returnValue = await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
SavedChanges?.Invoke(this, EventArgs.Empty);
return returnValue;
}
}
Maybe there's something I'm missing?
@javiercampos
I'm thinking a lifecycle hook inside EF could potentially allow access to the Update pipeline allowing you to modify the resulted SQL, or modify the way store generated values are read back, making this feature much more powerful and making more scenarios possible than overriding SaveChanges. For example instead of single Update statement, you could send multiple statements interleaved with other updates. You cannot do this by overriding SaveChanges.
@popcatalin81 oh yes, I definitely agree as many "advanced scenarios" lifecycle hooks as possible should be implemented (and in a user-friendly way if possible), that's why I'm following this thread. Was just curious about @sidshetye's request (he was asking for the equivalent to ObjectContext.SavingChanges
, and a possible SavedChanges
event which are simple enough to implement yourself, not to overload the busy team :smile: ).
@javiercampos , the challenge with just overriding SaveChanges and SaveChangesAsync is that you then have to keep track of the entities that were changed. While it is true that you can copy this information from ChangeTracker before the save and then iterate through it again after the save, it seems more natural to me to have the system notify you of the changes in each entity. In addition this approach will not notify you of changes that are not due to SaveChanges, e.g. due to a property assignment, a revert changes (state = Unmodified) change, etc. Again you can build this yourself but, given that the ChangeTracker is already figuring all this out, it makes more sense to simply have the system notify you when there is a change.
About 'overloading the busy team', I agree they are busy (!!) and doing a fantastic job. On the other hand, this is a hook that is already there (in the ILocalViewListener service) that just needs to be included in the public API.
@sjb-sjb again, yes, but that's not what ObjectContext.SavingChanges
did, and that's what I was replying to (@sidshetye specifically asked for the equivalent to ObjectContext.SavingChanges
)
@javiercampos our data encryption platform (see http://www.crypteron.com) supports C# and within C#, we also support Entity Framework. We need SavedChanges
to make it user friendly for our common Entity Framework customers. Because we can then directly plug into the hooks EF provides (plug and play) rather than us telling them "... and don't forget this snippet of custom code" - which busy developers always seem to forget! Specifically, we need this hook to cryptographically verify data integrity after saving to storage (SQL in EF's case). For contrast, the Java Persistence API (JPA) has such hooks, so our guidance there is very simple and I'd like to keep my C# customers equally happy! I raised this since adding a hook is simpler when already adding other hooks and seems the implementation could be straight forward.
About ObjectContext
: We're not married to this specific interface but it's what we've used for our EF6 integrations, so we're going by that. I checked with my team and we're using the following API's. A same/improved API would be desired
((ObjectContext)sender).ObjectStateManager.GetObjectStateEntries(/* Added or Modified */)
((ObjectContext)sender).ObjectStateManager.ChangeObjectState(/* obj to modified or unchanged */)
In addition we're using these APIs
DbContext.Configuration.AutoDetectChangesEnabled
DbContext.Entry(..)
DbEntityEntry.OriginalValues
DbEntityEntry.CurrentValues
@sidshetye I see, that makes sense... in the meantime, one possible workaround (which involves work from the customer though) would be making a base DbContext from which they must inherit to use your framework. Not ideal on your specific case, yes, but better than having your customers override SaveChanges
themselves
Thanks for all the feedback!
Notes for triage:
Decision from triage: we will attempt to get state change events in for 2.1 as a stretch goal. Moving other events out for now.
Just sharing a quick and dirty implementation to get the type of notification I needed for grabbing ID values after a save:
// DbContext declared as partial class so implementation not lost if rescaffolding from db.
public partial class FooDbContext
{
private readonly List<Action> _notifySavedList = new List<Action>();
protected void NotifySavedChanges()
{
foreach (var item in _notifySavedList)
{
item();
}
_notifySavedList.Clear();
}
public void OnAfterSaveChanges(Action p)
{
_notifySavedList.Add(p);
}
public override int SaveChanges()
{
var result = base.SaveChanges();
NotifySavedChanges();
return result;
}
public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
var result = base.SaveChanges(acceptAllChangesOnSuccess);
NotifySavedChanges();
return result;
}
public override async Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = default(CancellationToken))
{
var result = await base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
NotifySavedChanges();
return result;
}
public override async Task<int> SaveChangesAsync(CancellationToken cancellationToken = default(CancellationToken))
{
var result = await base.SaveChangesAsync(cancellationToken);
NotifySavedChanges();
return result;
}
public override void Dispose()
{
_notifySavedList.Clear();
base.Dispose();
}
}
Use case looks something like this:
public class FooChildBusinessObject
{
public int Id { get; private set; }
public string Name { get; set; }
public bool IsNew { get; set; } // Whatever logic you use to track inserts vs updates.
public void Update(DbContext dbContext)
{
if(IsNew)
{
var fooEntity = new Foo();
fooEntity.Name = Name;
dbContext.Foo.Add(fooEntity);
dbContext.OnAfterSaveChanges(() =>
{
// grab the database generated ID..
Id = fooEntity.Id;
});
}
else
{
var fooEntity = dbContext.Foo.Single((f)=>f.Id == Id);
fooEntity.Name = Name;
}
}
}
public class RootBusinessObject
{
public FooChildBusinessObject Foo { get; set; }
public void Save()
{
// This is some parent / root object saving children in its object graph.
// It calls SaveChanges() after all objects in the graph have added their changes to the dbcontext.
using (var dbContext = GetDbContext())
{
Foo.Update(dbContext);
dbContext .SaveChanges();
// Foo.Id should now be populated!
}
}
}
@ajcvickers Is there documentation for committed public API for entity state changes? EntityFrameworkCore/src/EFCore/ChangeTracking/ChangeTracker.cs
seems to be evolving on which APIs are internal and which ones are public. We're investigating if we can make any progress till SavingChanges
and ObjectMaterialized
(and hopefully SavedChanges
) are implemented ... hopefully not too far away!
@sidshetye Docs are not done yet--tracking issue is https://github.com/aspnet/EntityFramework.Docs/issues/661. The public surface as it is now is what we plan to ship for 2.1, unless we get significant feedback from the prereleases that cause us to re-think.
Thanks for this!
A few comments based on looking at ChangeTracker.cs and also based on the experience I've had with the earlier, nonpublic version (LocalViewListener).
It might be worth cautioning in the documentation that during the state change callback, one should be careful not to re-enter into another state change on the same entity (for example by assigning the state or by calling SaveChanges). I had a case where due to some very indirect interactions I was reverting the state from Added --> Detached during an Add, i.e. during a Detached --> Added state change callback. The result was that after the revert, subsequent continued processing of the Detached --> Added callback looked like the transition was from Detatched --> Detached. In the new interface I believe the effect would be that NewState would still be Added but this would no longer be equal to the new State of the EntityEntry, which would be Detached -- certainly a confusing state of affairs. Incidentally it would be useful to document that NewState is equal to the State of the entry at the time the state transition event is raised (assuming I understand it correctly).
I am not clear on the reason for the Tracked event. Is this not redundant with entity state transitions from the Detached state to one of the other states?
I am using these callback to, essentially, construct a view model which includes information about the entity states and entity values. As part of this I frequently would like to test whether or not there are any attached entries with certain states. For example, to enable UI for saving changes, one should test among other things whether or not there are any entities in Added, Modified, or Deleted states. This can be easily accomplished using the ChangeTracker.Entries().Where( ee => ee.State == X). However I am not very comfortable with that approach because it does not seem efficient to loop over all the attached entities simply to find the few that have a specific state. Consequently I ended up keeping my own tables for each of the states that I am interested in. I would prefer that there be a more efficient way.
@sjb-sjb Thanks for the comments--I referenced them from the docs issue.
With regard to the Tracked event, that was the result of a long discussion on whether or not something being tracked for the first time is just a state change from Detached to something else, or whether it is a distinct event. Specifically, are there common scenarios where they would be used independently. My initial thought was that it was all just one state change event, but the ultimate outcome of the discussion was a team decision to create two different events. So you will never get state changed events from Detached to something else--those will instead be Tracked events.
Can you file a new issue for your last point? I think this is useful information for EF to provide. However, keep in mind that when normal non-notifying entity types are used, then it requires DetectChanges to run to determine if there are any entities in a given state, and this always involves scanning all entities, so this may never be super-performant unless notifying entities are being used.
Is there any docs on "notifying entities"? This is the first time I read about them...
@markusschaber Docs are being tracked as https://github.com/aspnet/EntityFramework.Docs/issues/531
In a nutshell, have your entities implement INotifyPropertyChanging and INotifyPropertyChanged, and then use ObservableCollection for collection navigation properties. There are a quite a few places in the tests that you could look at.
@ajcvickers thanks for the comments.
I think that the FromQuery flag in the tracked event is useful. It looks like this will let you know whether the tracking is due to materialization or not. I could imagine this being extended to indicate whether, for example, tracking was caused by (a) an explicit assignment to the EntityEntry.State or by being the head entity in an Add, Update, etc, or (b) was due to the entity being reachable from a head entity that was tracked due to Add, Update, etc.
It is a little bit odd that we have a NewState in the entity transition event but do not have it in the Tracked event. Presumably the state of a newly tracked entity could be either Unchanged or Added, for example. If the user prevents entity state changes during the transition event and during the tracked event, then there is no need for NewState since it will remain equal to the EntityEntry.State. On the other hand since this locking is not built into EF there is always the possibility that the state could change. In conclusion it would make sense to me to add NewState to the tracked event. Caveat: I'm doing some guessing here since I haven't tried the new interface.
I will create the issue as you requested. As relates to the efficiency of state testing, as a matter of fact I am using notifying entities. One reason in fact for using notifying entities is to avoid the inefficiency of DetectChanges. Another reason, however, is the same as the reason for using EF state callbacks, namely to build a view-model system. If you are going to implement a view-model system then you have to notify property changes anyway, so you may as well leverage these notifications for use by EntityFramework tracking. Concerning the state change, I attach the entity to a context the first time a property is assigned and then the state callbacks are used to manage the entity's presence and sorting in observable collections supporting the interfaces required by ListView (this is a view-model activity and, I might say, not a very simple one -- especially if the collection is paged).
I'll tell you an overall comment. I realize that EF is focused on data access, that is it's raison d'etre. At the same time I don't think MS should ignore the fact that data access operates within UI programs (and within web services, etc.) As a consumer of MS products I think MS has done a fantastic job on the data access part, but it is really surprising to me that little has been done on integrating this into a coherent program-building paradigm. Specifically I feel MS should put effort into building a really professional view-model layer that fits on top of EF. Many people are out there cooking up their own, it is very duplicative and I imagine in most cases leading to not nearly as robust a solution.
Are you planning to have an event after entities are created and the context is available to be used again?
After I load an entity from the DB, I want to initalize it, which would look something like this:
ctx.ChangeTracker.Tracked += (sender, e) => {
((BOBase)e.Entry.Entity).Init();
};
The problem is, if Init()
attempts to execute a DB query, it will throw
System.InvalidOperationException: 'A second operation started on this context before a previous
operation completed. Any instance members are not guaranteed to be thread safe.'
because the Tracked
event is invoked while the context is still busy with an operation.
If a query starts tracking multiple entities, I'd want an interception point after the whole query is completed (so, the context can be used again), not after each individual entity is loaded.
@Wain123 We discussed such an event back in the EF4 days, but it is very problematic to implement because it would have to fire only after each entire query has been enumerated, which is something that is controlled by the caller. So at that time we decided that it was better for the application to define and control such events. That way the application can know when it has finished executing whatever queries it needs, even if that is several, and then fire and event to other parts of the application that should respond.
@ajcvickers Ok, that's inconvenient, maybe there's another approach - is there a way to not get the InvalidOperationException when executing a DB query inside the ChangeTracker.Tracked event? The context becomes available very soon after the Tracked event (it's available inside the loop where I'm enumerating the query), so could this Tracked event be sent slightly later to allow me to Init each entity right as it becomes tracked?
Did any of this got through into 2.1? If so, any docs available?
I'm particulary interested on an ObjectMaterialized
replacement... I see you have now the Tracked
event on the ChangeTracker
, but I'm afraid that will not do if you make a query with AsNoTracking()
.
Is there any "official" hook (apart from EntityMaterializerSource
which, although it's what we are using, is internal and has actually changed from 2.0 to 2.1) to get when entities are created?
@javiercampos The only new hooks added in 2.1 were for the Tracked and StateChanged events. There is no ObjectMaterialized yet--the main reason being that the code and patterns in this area are changing with the introduction of constructor binding and factory support. However, how all the API hooks up and when to use the event over a binding, if ever, is till to be determined. See #10789
The InternalEntityEntry
received via IEntityStateListener
's StateChanged
doesn't seem to contain the values set by the database itself such as those which are set by .ValueGeneratedOnAddOrUpdate()
.
Should they? (No I learned)
Update - This was a fundamental misunderstanding on my part. Entities are not updated _from_ the Db after SaveChanges[Async]()
call unless they are specific properties such as Primary Keys or RowVersion(I don't know the full extent but certainly Primary Keys). Therefore properties which are not the above and are .ValueGeneratedOnAddOrUpdate()
cannot be expected to be current.
@jdevmike - I worked around this using the hand rolled solution I posted above. If I should be replacing this with a new EF mechanism please let me know.
@ajcvickers when can we expect ObjectMaterialized
and SavingChanges
equivalents to be released? Waiting but hoping it's not a long wait đ€
Hi @divega
What about intercepting connection like we used to do on EF6 using IDbConnectionInterceptor.
Interception is not for logging only!
My asp core app requires executing session context
EXEC sp_set_session_context ...
on the server each time database connection is opened.
My only workaround now is to hook on the event Database.GetDbConnection().StateChange
. which is my last option.
Thanks for help
@ajcvickers Following up on the April 7th comment; our proof-of-concept with EF Core 2.1 confirms that ChangeTracker.Tracked
and ChangeTracker.StateChanged
events in a DbContext
are insufficient for us to get the job done on EF Core 2.1. Coming from EF 6.0, we did have this hunch. But can now confirm that we are indeed blocked from supporting EF Core 2.x till SavingChanges
and ObjectMaterialized
(and hopefully SavedChanges
) make the release. Would appreciate you sharing your roadmap on this so we can adjust ours accordingly.
@SidShetye re Tracked and StateChanged, are they insufficient because they don't trigger when non-tracked objects are materialized or because you need to be able to perform operations that can potentially use the context to execute queries while processing the event?
@divega for us they are insufficient because there are no event hooks at the moment of entity materialization from the db (like ObjectMaterialized
) as well as at the moment just prior to the entity hitting the network (like SavingChanges
). A 3rd event hook that fired after an entity (e.g. SavedChanges
) has been written to the network would also be extremely helpful.
@SidShetye concerning the 3rd event hook, given that before the SaveChanges the state will be Modified, Added or Deleted and afterward it will be Unchanged or Detached, can you not use the StateChanged event?
@sjb-sjb
Just to play devils advocate, given you can detach and attach entities, and change the state of entities yourself, a state changing event is not necessarily any guarantee that the entity was actually written to the network
right?
@sjb-sjb : Our reason is the same as @dazinator i.e. the state transitions
(Modified | Added | Deleted ) => (Unchanged | Detached)
could happen for certain permutations of user-level usage patterns even when the entity doesn't actually hit the network.
Yeah, I agree if you let the user do anything then state transitions won't correspond to writes. I could see it happening with a library intended to complement EF for example.
In my case I just don't let them do anything weird, and if they do then they pretty much just get what they deserve đ.
@Wain123
When you materialize a query with many entities you get a bunch of ChangeTracker.Tracked events, one per each entity in the query result.
With this code you can call Init() when the object graph is completely materialized:
(You still can't use the object context to make further queries, but you can eager load everything you need in the same graph and then Initialize it on due time. Another option would be to use directly a DbCommand or another instance of the context for these queries).
private void ChangeTracker_Tracked(object sender, EntityTrackedEventArgs e)
{
if (!e.FromQuery)
return;
var entity = e.Entry.Entity;
if (entity is IInitializable initializable)
_initializableEntities.Add(initializable); //keep until materialization completes and then call Init()
if (_dbConnection.State == ConnectionState.Closed) //last entity in graph
{
e.Entry.State = EntryState.Detached;
e.Entry.State = EntryState.Unchanged; //force change tracker to fix all entry navigations
_initializableEntities.ForEach(x => x.Init());
_initializableEntities.Clear();
}
}
Just a comment based on some recent experience and reviewing the various comments above.
Many of the comments / issues raised are connected to the fact that Tracked / StateChanged events occur during materialization or during SaveChanges. For example we see a number of comments seeking initialization hooks, which would be addressed if we knew that state transitions occurred after completion of queries or completion of saves (writes).
What I recently realized is that the processing of state changes during the process of materialization or during the process of SaveChanges can lead to inconsistent application state because the various entities in the application (for example a set of entities currently attached to a context and being saved) will be at different stages of being processed by EF. The example I ran into was a SaveChanges of a graph of entities in the context. Some of the entities had been saved and their store-generated ID's set, while other dependent entities had foreign keys (and navigation properties) onto these saved entities but the foreign keys had not yet been updated to match the store-generated id's. That is, the foreign key was a negative number (EF-generated placeholder) while the referenced entity pointed to by the navigation property had a positive id (store-generated). Clearly a bad situation.
Thinking about this, I wonder if I need to put the Tracked / StateChanged events into a producer/consumer queue and pull them off asynchronously for processing after the materialization or SaveChanges is complete. The asynchrony could also help to avoid accidental re-entrancy while processing the state changes. This leads back to the question of having a hook for when materialization or SaveChanges is complete.
If people feel it would be useful to have these events be processed asynchronously after the materialization or save is complete then maybe it could be considered for inclusion as an option in EF.
On the other hand it is also true that sometime you do want the synchronous invocation. For example this is useful if you are trying to track state information that is synchronized with EF state information.
@divega and @ajcvickers : it's been several months since an update so wanted to ask if you have clarity on ObjectMaterialized
and SavingChanges
hook release dates?
@SidShetye Unfortunately, the schedule for 3.0 is pretty full, and this is not planned for that release--as indicated by being in the "Backlog" milestone. That being said, it's possible something small will get slipped in depending on how things go.
For those that are blocked by A second operation started on this context before a previous
operation completed. Any instance members are not guaranteed to be thread safe
, we worked around this by delaying our custom ObjectMaterialized
-like event in a few different places.
High-level steps are:
1) For queries that can return multiple results (e.g. Where
, Select
, ...):
IEnumerable<T>
with a custom Enumerator
MoveNext
recursively2) For queries that can only return one results (e.g. FirstOrDefault
, Single
, ...):
IQueryable<T>
Execute
recursively3) For lazy loading:
ILazyLoader.Load(...)
NOTE: You also need to keep track of whether the entity has already had the event raised somewhere, otherwise you could end up raising the event multiple times.
Happy to share an implementation somewhere if people are interested.
@ajcvickers So, let's say someone wanted to rewrite queries (to implement complex filtering not supported by the built in filters), where would you suggest they insert themselves into the process?
There seems to be ample opportunity to do this (for example, the Expression passed to the QueryCompiler looks so tempting, and I'd like to beat the query cache with a baseball bat), but lots of warnings not to, and a corresponding lack of documentation on the internals.
A little guidance would be appreciated. I've been investigating and prototyping rewriting queries for the past month and have come up agonizingly close to a practical solution.
@WillSullivan We don't have any great general documentation here. It's something I would like to improve, but unfortunately resources are limited, especially considering we only have engineering resources to write docs. If you have specific questions, then please file a new issue with those questions and we'll try to answer.
@ajcvickers uh, "So, let's say someone wanted to rewrite queries (to implement complex filtering not supported by the built in filters), where would you suggest they insert themselves into the process?"
@WillSullivan - If you want to do before any kind of query execution starts, then override IQueryCompiler.
If you want to change only when query is being compiled then override IDatabase.CompileQuery* methods.
I need to run certain operations on the database when it got created. Is there a way to get a notification when it's creation completed (i.e. after calling EnsureCreated
created the DB)?
Is this for testing? EnsureCreated should not be used in production.
Thanks for your notice @ErikEJ, my question is also about context.Database.Migrate()
. Is there a way to intercept those calls and perform stuff afterwards?
@weitzhandler - How about writing custom SQL in migration files? After database is created, the first line executed on server would be first line in up method of very first migration file
@smitpatel @ErikEJ
I've opened a new issue.
Hi folks - any news on when this will make the release? Been requested since 2014, now 2019.
2019 - 2014 = 5 :(
It's sad that EF still doesn't support this when Hibernate has had this capability for many years now.
If I want to add basic auditing and Guid generation I have to hand roll it myself each time.
This should be something that's possible within EF.
`
public class Audit {
@Column(name = "created_on")
private LocalDateTime createdOn;
@Column(name = "created_by")
private String createdBy;
@Column(name = "updated_on")
private LocalDateTime updatedOn;
@Column(name = "updated_by")
private String updatedBy;
@PrePersist
public void prePersist() {
createdOn = LocalDateTime.now();
createdBy = LoggedUser.get();
}
@PreUpdate
public void preUpdate() {
updatedOn = LocalDateTime.now();
updatedBy = LoggedUser.get();
}
//Getters and setters omitted for brevity
}
`
@Igneous01 You can already achieve this in EF Core - see Generate tracking columns for more details.
SaveChanges seems like a great place to be able to intercept/extend a context (auditing, caching events) and now with DbContextPooling overriding those methods to call services is less manageable as they can't be passed in to the constructor. A BeforeSaveChanges(Async), AfterSaveChanges(Async) and ErrorSavingChanges(Async) event would be awesome.
@jakenuts A couple of points:
I would support the events mentioned by @jakenuts. I implemented very similar events in a wrapper class as a way to update entity view models.
A very handy function to complement SaveChanges would be RevertChanges, with somewhat similar events associated with it. One typically wants to revert changes when a user cancellation occurs.
sjb
@sjb-sjb Not saying those aren't useful events. They are still on the table.
Hi Arthur and Diego - can we raise the priority of the lifecycle hooks? The community demand has been very strong (30+ participants, 50+ thumbs, over 5+ years) but this item has been waiting (waiting, waiting ...) in the queue to be punted across every release. I understand the team has a challenging schedule but perhaps a swap/re-prioritization against another issue/feature would help get this long-waiting item out of the queue? Thanks
CC: @ajcvickers @divega
EDIT: More precisely, the worry is this isn't on the 3.0.0 milestone or on any plan in general.
@SidShetye Unfortunately, at this point there isn't anything that can be bumped to give this a higher priority.
Hell, lifecycle hooks have been a community want since EF v4 in 2010.
@ajcvickers That is disappointing - and hard to believe. Why? Because looking at the 474 issues open in 3.0.0 milestone, there is only ONE. solitary. single. issue with more customer demand than this issue. Every one of the remaining 473 issues has less customer expressed demand or importance (!).
I mean your customers are literally telling you "this issue is more desired than the other 99.78% of the issues". And yet the product road-map simply tosses this aside? Even with noisy measurements, the balance is overwhelmingly off.
I believe product development should be rationally driven by the customer and markets. But this issue gives the impression that it's highly whimsical. Like: "Yeah, you customers can go take a hike, we're going to build whatever we want." On top of it, this is also nothing new (since you said "at this point") - we've been at it since 2014!
We strive to work closely but this is a disappointing project dynamic.
CC: @divega
@SidShetye Customer feedback is very important to us. But there are several other sources of information we use to decide what goes into each release. By the way, we made an attempt to describe the process at https://docs.microsoft.com/ef/core/what-is-new/roadmap#release-planning-process.
Besides that, this isn't really a regular issue. It is more of an "umbrella" issue that represents several individual features. Each lifecycle hook clearly should have its own priority, but because we haven't broken it down into individual issues (something I would like to do, but I haven't found the time for), we get lots of votes here.
We actually implemented some of the features that used to be tracked here (state change events) in EF Core 2.1.
You are welcome to comment here about what specific lifecycle hooks would be most valuable for you and why.
You can offer to help by contributing some of them.
You are also welcome to complain about the fact that your favorite features aren't implemented and all the customers that are tracking this issue will see it. But let me set the expectation that it is very late in the 3.0 release, and given the number of work items we have in progress and the resources we have available, it is very unlikely that we will be adding any more enhancements to the list of this release.
@divega @ajcvickers - believe it or not, we're huge fans of the EF team and respect the work coming out right from EFs early days. So much so, that at the unique intersection of ORMs and Data Security where our Crypteron products, services and IP reside, we've heavily invested and have always treated EF (till EF 6.x) as our 'golden target' that drive the designs for other ORMs we support like Java's JPA/Hibernate. We're also promoting enterprise migration to the cloud/Azure by reducing data security concerns on those workloads - these are common customers between your team and us. In fact we're an Azure Security Partner working on an even closer commercial integration. The synergies are great and we're EF proponents in general.
Mapping to some of the guiding questions on your roadmap planning process:
In terms of what lifecycle hooks are desired, why etc., we discussed that here last year. I hear you about challenges with time on 3.0.0. You guys know best about internal resource allocation, but perhaps 3.0.1 :) ?
Thanks
sid(at)crypteron.com
I very much appreciate the EF teams work and am hesitant to step into what is best described as a passionate back and forth. I have one request however.
Objectively, I would assert if this umbrella issue has the 2nd most customer demand, but can't be treated as a single feature request. Its worthy of the time to break into individual issues to track demand more accurately.
@SidShetye thanks for the pointer to https://github.com/aspnet/EntityFrameworkCore/issues/626#issuecomment-359082350. To give you some perspective, there were ~90 comments before that, and ~70 comments after that in this issue. That is one of the reasons it has become extremely hard to distill the comments in this issue into actionable work items that we can prioritize. I would be very grateful if you go ahead and create a new issue for the specific hooks you need. Please, try to be very specific about what you need (unfortunately, after re-reading that comment a couple of times, I am still not sure), and feel free to link to it from here.
@AFDevMike
I would assert if this umbrella issue has the 2nd most customer demand
I wonder how you are counting. In https://github.com/aspnet/EntityFrameworkCore/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-desc this appears to be in the 14th position. But as I said, the main problem is that it is impossible hard to distill what it really means in terms of actionable work items for the individual lifecycle hooks and their relative priorities.
@divega I was counting based upon @SidShetye's assertion here: https://github.com/aspnet/EntityFrameworkCore/issues/626#issuecomment-496730616
Because looking at the 474 issues open in 3.0.0 milestone, there is only ONE. solitary. single. issue with more customer demand than this issue. Every one of the remaining 473 issues has less customer expressed demand or importance (!).
And since it wasn't refuted in your direct response, I assumed it to approximate the reality. My error, apologies for noise.
@divega
I opened a new issue that is related to life-cycle hooks. In my scenario, I need to have a post-OnConfiguration
hook, so I can call Database.Migrate
. See #15846.
See #15066 for IDbCommandInterceptor
(for command interception), for which there is now a PR: #16113
For folks tracking this directly or coming from search (SEO!) - the actual work items are in #15910 and #15911. Hoping to see this released in EF Core 3.1 đ€ !
We need interception to rewrite the SQL before execution because there is no support in EF for FULLTEXT functions nor XML XPath query syntax. We dynamically build an IQueryable based on an ad hoc query builder interface in our application, and that turns our XML data typed model properties into column LIKE '%value%' statements in the SQL.
Interceptors were the ONLY work around we could come up with to parse the SQL, replacing the LIKE '%value%' with CONTAINS(fulltextcolumn, 'value') and xmlcolumn.query('xpath value') = 1
Where's the common work around described in this opened issue for manipulating the SQL before execution?
I'd personally like to see full XML and FULLTEXT syntax support. Is there a issue tracked for those? Can the community code and submit these changes?
@vasont For full-text search see #10488, #11481, #11486. For the low-level hook to modify SQL, database interceptors is the way to do this.
@vasont For full-text search see #10488, #11481, #11486. For the low-level hook to modify SQL, database interceptors is the way to do this.
@ajcvickers Thanks so much for the references. A project this big can get unruly trying to find the right info with different search terms.
Awesome news on the FreeText search functions. It would be more awesome to see similar for the XML query functions. :)
@vasont I'm happy I'm not alone in finding information discovery challenging with large projects like this!
@ajcvickers Which is why I (and I'm sure many others) greatly appreciate this repos openness to question issues. As opposed to offloading those to a gitter or chat where the information is rapidly lost to time. It makes it possible to find conversations and solutions to niche issues that would otherwise be neigh-impossible to locate or discover, and gives way for some niche discussions to turn into features.
Hope this can be added soon, this is good for some cloud services that require to keep track of what records have been changed in DB.
For example, we use Algolia for searching, algolia needs to index records or I should say keep track of all the records. so that it can provide the latest results. In this case, it would be really nice that we could notify algolia whenever there is a model event happens. Such like created, updated, saved, deleted
Is this still not created in EF Core?
@nefcanto Take a look at interception of database operations in New features in Entity Framework Core 3.0, but lifecycle hooks didn't make it into 3.0 release.
For those who came here via SEO: because I needed this functionality myself, I created my own package with lifecycle hooks.
You can download it via NuGet. Full docs are available on the Github page of the project.
I was just wondering how I might detect changes to a relationship collection to update the owner and it seems like this is the same issue. You've built a great system for loading and persisting data but between those two actions there is "change" and it is only addressed in terms of the first two. When I modify a property, add an item to a collection I want an easy method for hooking into that process and reacting. We've probably all written some complicated method of using change tracking to do tiny little tasks like setting an UpdatedDate and that just promotes brittle redundant code across all of our projects. Prioritize change hooks, please.
@jakenuts you can use notification tracking entities with a collection that implements INotifyPropertyChanged/Changing, see https://blog.oneunicorn.com/2016/11/16/notification-entities-in-ef-core-1-1 You can also use a collection type that implements INotifyCollectionChanged.
In this approach, when you assign to a principal entity reference and both entities are attached to the same context, EF will immediately add the dependent entity to the parent collection resulting in a CollectionChanged from the collection.
Watch out for the fact that this only happens when both entities are attached to the same context. In my work I attach the entities if they are not already attached.
You can also expect to get PropertyChanged notifications for database-assigned primary keys when they are set during a SaveChanges.
We also need to propagate changes to several systems.
I looked at @JValck solution, found it too fine-grained for our purposes.
At the end I implemented simple publish/subscribe by overriding DbContext.SaveChangesAsync
method.
It would be great to have such mechanism in the library itself, because IDbCommandInterceptor
API is too low-level
Propagating changes in a transaction safe way is not easily possible - the service might crash in the wrong moment, leaving an inconsistency between the propagated changes and the real database contents.
We're thinking about using PostgreSQL LISTEN
and NOTIFY
mechanism for this problem.
Propagating changes in a transaction safe way is not easily possible - the service might crash in the wrong moment, leaving an inconsistency between the propagated changes and the real database contents.
We're thinking about using PostgreSQL
LISTEN
andNOTIFY
mechanism for this problem.
Iâd suggest youâd be better going with with one:
Propagating changes in a transaction safe way is not easily possible - the service might crash in the wrong moment, leaving an inconsistency between the propagated changes and the real database contents.
We're thinking about using PostgreSQLLISTEN
andNOTIFY
mechanism for this problem.Iâd suggest youâd be better going with with one:
- publish with later polling, or
- DTC, or
- a mechanism that can mimic DTC like the NServiceBus Outbox (https://docs.particular.net/nservicebus/outbox/)
DTC is not supported in .NET Core https://github.com/dotnet/runtime/issues/715
@optiks LISTEN and NOTIFY in PostgreSQL are integrated with the database transactions, so what's the reason you advise against them?
Publish with later polling puts a high load if you have lots of data, and only small amounts change. Also, you increase latency by the polling interval.
We'll have a closer look at NServiceBus Outbox. But at first glance, it just looks like reinventing a transactional messaging system in the database by storing message in tables, so why not just use the one provided by the database?
Most helpful comment
hey @sidshetye , I don't know if you ship commercial software for a living, but I do. And I aim to get things into releases all the time that get re-prioritized by the business and other things that are more important get pulled in ahead. It drives me crazy, as I'm sure it does to the EF team, when I don't have enough hours in the day to deliver all the features that I know my users want.
Let's support the EF team in what they are delivering and not beat them up too much for what is missing the cut.