React-native-windows: Support lottie-react-native for RNW vNext

[namrog84]

https://github.com/react-native-community/lottie-react-native

Used by 62k repos and about 12k stars.

Lottie component for React Native (iOS and Android)

Lottie is a mobile library for Android and iOS that parses Adobe After Effects animations exported as JSON with bodymovin and renders them natively on mobile!

[chrisglein]

This would be a great thing to do after we have our view managers story up and running (after M3).

[chrisglein]

Details on WinUI's support of Lottie:
https://docs.microsoft.com/en-us/uwp/toolkits/winui/release-notes/winui-2.1
https://docs.microsoft.com/en-us/windows/communitytoolkit/animations/lottie
https://docs.microsoft.com/en-us/uwp/api/microsoft.ui.xaml.controls.animatedvisualplayer?view=winui-2.2

[namrog84]

Everything in there seems to be for C# .NET and I don't really see a C++/WinRT way of using that.

[chrisglein]

Accidental close

[chrisglein]

Partner is currently unblocked with their existing lottie implementation. We will need to wait for some improvements from Composition before lottie support can be added without either codegen or the overhead of .NET. Moving to M5 as we no longer need this to exist M4.

[chrisglein]

A think to keep in mind that the Lottie-Windows team noted: Lottie files are generally just vector graphics, but may also include images, and those images may be a uri. Lottie-Windows won't handle the policy of that, and will need some sort of "image asset delegate" to call to the lottie-react-native module and have it resolve the uri. This is functionality that doesn't exist in Lottie-Windows, but they are anticipating needing, and and expecting us to drive requirements... when we get to that point.

[chrisglein]

Some history on this item:
Lottie support on windows exists and is exposed to XAML through AnimatedVisualPlayer (available with WinUI2, which we bring in for RNW). So the pieces are in place to hook this up today.

There is one hitch: Lottie-Windows was built out for the most optimal path for the Windows, where Shell developers use codegen to load their assets instead of just handing over the file at runtime. That's a great option that doesn't align with the React Native module's mechanism for exposing your Lottie assets. There is an option available from Lottie-Windows that works more like the Android/iOS asset handling, but it internally uses managed code to handle the assets. This will cause an otherwise native app to bring in the CLR, which is a big barrier to teams like Xbox that are looking to keep things lean. For that reason, when the Xbox team used Lottie in their RNW app they used the codegen. This is unblocking, but requires a Platform fork in the code from what could be otherwise be cross platform Android/iOS logic.

We could today contribute a Windows solution to the Lottie community module with the caveat that it'll hit the managed codepath for assets. There will be that performance asterisk for native apps, but they'll be unblocked. Long term the Lottie-Windows team will be converting that asset loading code so that this performance asterisk can go away. In the meantime some consumers may opt to use the codegen to avoid that problem, doing some Platform.select forking in their code.

[jonthysell]

Hi @chrisglein, @stmoy,

Here’s my analysis of what lottie-react-native needs in order to add a new implementation, and what LottieWindows does (and doesn’t) provide.

It’s not a large module, but I’m concerned that LottieWindows (as it currently exists) does not provide key functionality necessary to move forward. See "Problems" at the end.

Requirements

• Native Modules: 0
• View Managers: 1
  
  
  
  • LottieAnimationViewManager
  
  

Name | Type | Description | lottie-android | LottieWindows + WinUI
-- | -- | -- | -- | --
LottieAnimationView | View | The actual UI element to put in to the tree. | LottieAnimationView | AnimatedVisualPlayer
VERSION | Constant | Lottie version? | 1 | 1
play(int startFrame, int endFrame) | View Command | Play the animation from startFrame to endFrame (reverse if endFrame < startFrame, use (-1, -1) to play all). | setMinAndMaxFrame(), reverseAnimationSpeed() | ???
reset() | View Command | Stop the animation and set the progress to 0. | cancelAnimation(), setProgress() | Stop(), SetProgress()
pause() | View Command | Pause the animation. | pauseAnimation() | Pause()
resume() | View Command | Resume the animation. | resumeAnimation() | Resume()
animationFinish(bool isCancelled) | View Event | Event fired whenever the animation completes, with a boolean specifying if the animation was canceled. | onAnimationEnd(), onAnimationCancel(), | ???
sourceName | View Property | Specify a lottie json file to be loaded. | setAnimation() | Source, SetSourceAsync()
sourceJson | View Property | Specify a lottie json string to be loaded. | setAnimationFromJson() | ???
resizeMode | View Property | Specify the resize/scale mode: CENTER, CENTER_INSIDE, CENTER_CROP | setScaleType() | Stretch
renderMode | View Property | Specify the render move: AUTOMATIC, HARDWARE, SOFTWARE | setRenderMode() | ???
progress | View Property | Set the progress (in percentage?) | setProgress() | SetProgress()
speed | View Property | Set the playback speed. | setSpeed() | PlaybackRate
loop | View Property | Set how many times to repeat playback. | setRepeatCount() | (Might be possible to implement locally if there were playback events)
imageAssetsFolder | View Property | Set the root path for use when setting sourceName. | setImageAssetsFolder() | (Implement with local property to inject path when loading image)
enableMergePathsAndroid ForKitKatAndAbove | View Property | ? | setEnableMergePaths ForKitKatAndAbove() | N/A?
colorFilters | View Property | Apply color filters to the animation. | SimpleColorFilter(), KeyPath(), LottieValueCallback(), addValueCallback() | ???

Problems

• The LottieWindows API surface is in "progress" percentages, while everywhere else uses actual frames.
  
  
  
  • There is no way to convert between the two, let alone accurately.
  
  
• There are no exposed playback events fired we can hook into, specifically when an animation finishes.
• LottieWindows cannot load lottie JSON from a string, only from file paths.
• LottieWindows provides no method for implementing color filters.
• The latest LottieWindows 6.1 requires WinUI 2.4, RNW is only on MUX 2.3 and we must be in sync.
  
  
  
  • LottieWindows 6.0 uses WinUI 2.3, so we would have to use that or otherwise bump RNW's WinUI
  
  

[simeoncran]

Some of these can be easily fixed in a new version of Lottie-Windows. Some can be most easily solved by bypassing AnimatedVisualPlayer (i.e. write your own equivalent based on its source, but with different semantics). Some are not possible because the rendering is done out of process.

1. Progress vs frames. We can add a LottieMetadata property to LottieVisualSource that would provide FPS so you can translate between frame number and progress. A contributor is rumored to be working on this already.
2. Event when animation finishes. This is already handled by AnimatedVisualPlayer. The PlayAsync method completes when the play finishes, so you can just hook up to the completed event of it.
3. Loading JSON from a string. LottieVisualSource can get a new SetJsonSource(string) method.
4. Color filters. Lottie-Windows does have a system for modifying colors at runtime, but it is designed for codegen, not for LottieVisualSource. We could fairly easily expose something that made every palette color modifiable from LottieVisualSource, but it would be a different way of doing things so it is unlikely to be usable with the same API as the other platforms.
5. Dependency on WinUI2.4. I don't think the 2.4 version is necessary - we should be able to bump LottieVisualSource back down to 2.3, however there may be a bug fix in AnimatedVisualPlayer that would make 2.4 preferable. Or, use your own implementation based on AnimatedVisualPlayer source.
6. ImageAssetsFolder. This is equivalent to our "Image asset delegate" feature that has not been implemented only because we have been waiting to get requirements from RNW. :-) We can do that pretty quickly.
7. Loop. The PlayAsync method allows endless looping already. It doesn't support repeat count. However you can do a set number of repeats via "await PlayAsync" in a for loop.
8. Play a segment. Already handled by AnimatedVisuaPlayer. There is a subtle difference in how play direction is handled. In AnimatedVisualPlayer playing is always forward from "from" to "to". If from>to, the animation will play around the end (i.e. from from to 1.0 then from 0.0 to to). To play in reverse you set the playback speed negative. If you used your own implementation of AnimatedVisualPlayer then you get decide these semantics.

[jonthysell]

I've started the implementation here: https://github.com/jonthysell/lottie-react-native/tree/windows/src/windows

I haven't tested it yet, but it builds.

@simeoncran:

1. I've implemented a static framerate for now, but any frame to progress percentage conversion will be lossy. I think people will really need frame-accurate playback.
2. Implemented my own event.
3. I've stubbed this out for now.
4. I've stubbed this out for now.
5. I think we should bump our dependency on WinUI to 2.4, no reason for us to be on an older version.
6. I've implemented this locally for now.
7. This was a misreading on my part - yes lottie-android exposes a setRepeatCount(int), but lottie-react-native only exposes a boolean and sets it to 0 or infinity. So we're good here.
8. I think I have a decent recreation of the logic, but I'll have to test android and windows side-by-side to verify the behavior matches up. But again, I really think we need frame-accurate APIs.

[jonthysell]

In trying to create the windows sample app for lottie-react-native I've run into new blockers:

lottie-react-native targets RN 62, which should be great for us.

However, C++/WinRT apps can't consume C# libraries without a bunch of work-around code in the app project. We used this workaround in RNW 62 for our own apps, but didn't expose it to external developers until RNW 63.

So in order for lottie-react-native to be in C#, we need to backport https://github.com/microsoft/react-native-windows/pull/5851 to 62.

So the options are port that fix back to 62, and keep lottie-react-native in C# or rewrite the module in C++/WinRT.

The former means keeping the pretty small and straight-forward C# implementation I've just made, but getting a fix backported that might not get approved.

The later means writing an uglier C++/WinRT implementation, but also better sets us up for consuming a C++/WinRT lottie library in the future.

[jonthysell]

In the meantime, here's the PR to backport the C# fix: https://github.com/microsoft/react-native-windows/pull/6231

[simeoncran]

@jonthysell I don't understand what you mean by "frame accurate APIs". Frames and progress should be interchangeable as long as you know the right factor to multiply by. It might help if you can be more explicite about what API you would like, which class it should go on.

[jonthysell]

Currently working on an all C# stack until #6231 is released, so I can test the module functionality. I've finally got their example app up and running, but it uses sourceJson, which we don't support yet.

Part of getting the app to work is to fix a module called react-native-safe-modules that they depend on. PR here to add windows support to that module: https://github.com/emilioicai/react-native-safe-modules/pull/1

[jonthysell]

@jonthysell I don't understand what you mean by "frame accurate APIs". Frames and progress should be interchangeable as long as you know the right factor to multiply by. It might help if you can be more explicite about what API you would like, which class it should go on.

The developer passes in integer start frame and end frame. I can convert those into progress percentages given a total number of frames, which I'm currently doing by multiplying the duration in seconds by a hard-coded framerate.

But with rounding, there's no guarantee that the exact specified frames will be played back.

[jonthysell]

Using some weird workarounds (like taking the json they loaded from disk, and resaving it back to disk, to be reloaded by LottieWindows, and doing too much on the UI thread), but this is their sample app: 😃

[jonthysell]

PR started here: https://github.com/lottie-react-native/lottie-react-native/pull/692

[simeoncran]

The developer passes in integer start frame and end frame. I _can_ convert those into progress percentages given a total number of frames, which I'm currently doing by multiplying the duration in seconds by a hard-coded framerate.

But with rounding, there's no guarantee that the exact specified frames will be played back.

I don't think it should matter. Frames are floating point numbers so they're already subject to rounding elsewhere in the system. Do you have any repros where it's definitely displaying the wrong thing so that I can investigate?

[simeoncran]

Using some weird workarounds (like taking the json they loaded from disk, and resaving it back to disk, to be reloaded by LottieWindows, and doing too much on the UI thread), but this is _their_ sample app: 😃

You should be able to pass a stream to LottieVisualSource. That API was hidden due to a missing attribute, but that has been fixed and will be available in V7.0 (preview expected this week).
Is the problem with too much work on the UI thread something in LottieVisualSource or is it something you can fix? LottieVisualSource does most of it's loading on a separate thread. Maybe more needs to moved off.

[jonthysell]

Using some weird workarounds (like taking the json they loaded from disk, and resaving it back to disk, to be reloaded by LottieWindows, and doing too much on the UI thread), but this is _their_ sample app: 😃

You should be able to pass a stream to LottieVisualSource. That API was hidden due to a missing attribute, but that has been fixed and will be available in V7.0 (preview expected this week).
Is the problem with too much work on the UI thread something in LottieVisualSource or is it something you can fix? LottieVisualSource does most of it's loading on a separate thread. Maybe more needs to moved off.

Looking at the 7.0.0-preview4 now. The UI threading issues has nothing to do with LottieWindows and were just how I wrote my prototype - explicitly dispatching everything on the UI thread to avoid thread mismatch issues and just prove I could get their example app up and running. I have since then gone through and redone the threading properly.

[aschultz]

@jonthysell I took a stab at a C++/WinRT implementation.
https://github.com/lottie-react-native/lottie-react-native/pull/714

[jonthysell]

@aschultz I'm glad to see another implementation, I'll take a look at it.

I think, unfortunately, there's no solution that meets everyone's needs:

| Requirement | C# | C++/WinRT |
|-|-|-|
| Dynamically load assets at runtime | ✔ | ❌ |
| Codepush / bundle-only updates | ✔ | ❌ |
| Non-managed code | ❌ | ✔ |

Until there's a full C++ implementation of lottie, with loading json at runtime, I think we'll need both solutions, and let app developers pick which version works best for them. For some customers, requiring them to publish a native app update to change assets is a deal-breaker. For other customers, introducing managed code (especially having to switch to a managed app) is a deal-breaker.

[aschultz]

@jonthysell I was thinking that the C++/WinRT version might work for both scenarios. It defers the loading of the IAnimatedVisualSource back to the app (via ILottieSourceProvider), so devs can choose whether to use codegen or LottieWindows to fulfill the requests.

interface ILottieSourceProvider
{
    Microsoft.UI.Xaml.Controls.IAnimatedVisualSource GetSourceFromName(String name);
    Microsoft.UI.Xaml.Controls.IAnimatedVisualSource GetSourceFromJson(String json);
}

A C++ app could elect to only implement GetSourceFromName for now, and then add support for JSON once a better dynamic loader is available to them.

This also allows use of both modes in the same app -- codegen for built-in animations, JSON for remotely fetched content -- based on whether the React code passes in a string or JSON for a source.