This is my issue for tracking performance and memory optimization across Avalonia.
Most of my findings come from checking ControlCatalog running on x64 Windows using JetBrains dotMemory.
IDeferredRendererLock.TryLock allocates per-frame (https://github.com/AvaloniaUI/Avalonia/pull/2918)
Scene gets cloned completely if any element gets invalidated.
Scene.HitTest allocates crazy amounts of iterator blocks
IVisualNode.Children cause enumerator boxing due to access via IReadOnlyList.
RenderLoop and DeferredRenderer allocate new lambdas and lambda closure per-frame (Fixed in https://github.com/AvaloniaUI/Avalonia/pull/2960)
RoutedEvent<T>.AddClassHandler<T> and AvaloniaProperty<T>.AddClassHandler<T> make it very easy (or almost force) to write allocation heavy code. Result below is from moving mouse for few seconds.
InputExtensions.GetInputElementsAt cause method group allocation per-call. (Fixed in https://github.com/AvaloniaUI/Avalonia/pull/2927).RoutedEvent is using Delegate.DynamicInvoke which is slow (relies fully on reflection) and allocates a lot of garbage. (Fixed in https://github.com/AvaloniaUI/Avalonia/pull/2931 and https://github.com/AvaloniaUI/Avalonia/pull/2943)Logger API causes each binding and new value to allocate object arrays and box values.

ItemsPresenterBase using ItemVirtualizerNone will cause controls to materialize twice. First materialize will be caused by PanelCreated and second by ItemsChanged. https://github.com/AvaloniaUI/Avalonia/blob/3b38aea2b9a5ddc4a15cadff7ba74c658c4ba2af/src/Avalonia.Controls/Presenters/ItemsPresenterBase.cs#L223-L225AvaloniaObject.Bind allocate strings for descriptions that most likely will be never seen (in dev tools).
StyledProperty default value is always boxing value types. (https://github.com/AvaloniaUI/Avalonia/pull/2935)
PriorityBindingEntry is allocating two delegates per instance to subscribe to the binding. Can be avoided completely by using IObserver interface. (https://github.com/AvaloniaUI/Avalonia/pull/2931) 
@ahopper @kekekeks Wrote down my initial findings, I will add more when I find more time. I am already working on certain issues (marked as working on a fix), and I will open PRs slowly as I verify each fix.
Wow!
Another thing that might be worth investigating: LightweightObservableBase always creates a List<IObserver<T>> even if it only has a single observer.
We currently have SingleSubscriberObservableBase but it might be worth merging that with LightweightObservableBase and only creating the list when there are >1 subscribers.
@grokys Hopefully I will get there eventually, right now we still have an order of magnitude worse allocation patterns and I generally pick "easy" things right now that give decent return of investment.
Most helpful comment
@grokys Hopefully I will get there eventually, right now we still have an order of magnitude worse allocation patterns and I generally pick "easy" things right now that give decent return of investment.