Mobile-ffmpeg: Does this work with Unity

[nsmith1024]

Description
Description of what the issue is about.

Does anybody know how to use this mobile-ffmpeg library with Unity from C# to work for both Android and IOS?

Expected behavior
What you expected to happen.

Current behavior
What happened.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
Post logs here or paste them to Ghostbin and insert the link here.

Environment

• Platform: [e.g. Android/IOS]
• Architecture: [arm-v7a, arm-v7a-neon, arm64-v8a, x86, x86_64, armv7, armv7s, arm64, i386, x86_64]
• Version (if applicable) [e.g. v1.2]
• Source branch (if applicable) [e.g. master, dev-v2.x]
• Xcode version (if applicable) [e.g. 7.3.1, 9.0.1]
• Cocoapods version (if applicable) [e.g. 1.2.1]
• Android Studio version (if applicable) [e.g. 3.1]
• Android NDK version (if applicable) [e.g. 16b, 17c]

Other
Add any other context about the problem here.

[tanersener]

I really don't know, I'm not familiar with Unity.

[nsmith1024]

Ok so here is Unity-3D is a video game engine, millions of people use it, if you can make this FFMPEG work with it, you will be a hero!

One developer was selling a FFMPEG Unity binding, but he stopped developing it, so now there is a big gaping hole to fill there if you could do it.

[cyliax]

I'm trying to build a wrapper for Unity right now. It works with Android so far, but not perfect. I need to add Async calls and a progress and then.... iOS. But what I can say so far: The encoding of my example case is 2x faster than using the FFmpeg Unity Bind Asset in the Asset Store with an old FFmpeg version and without support.

This plugin here is well written and the AAR file contains 64-bit libraries, which is necessary to upload apps to the Play Store. Hopefully I will be able to build on this great work. I'll let you know if my wrapper works completely with Android.

PS: I feel with @tanersener on all problems related to FFmpeg: it is really worth installing FFmpeg on a Linux system (like WSL on Windows) to test the commands before complaining about mobile-ffmpeg. This is just a friendly advice for everyone, because I feel that the developers of the old FFmpeg Unity Bind Asset are tired of explaining the FFmpeg commands, and so they have stopped their support. I don't want to see this again :-)

[tanersener]

Unfortunately, I don't think I'll have enough time to create a new binding for Unity. I hope @cyliax can complete his/her work and Unity developers can use it.

Meanwhile, I created a Using MobileFFmpeg in Unity guide to explain how mobile-ffmpeg can be imported into Unity. I explained all the steps for Android but I don't have a step by step guide for iOS / tvOS yet. Integration on iOS / tvOS is complicated and requires Unity Cloud Build which is not used by all developers. I'll update that section when a more general solution is available.

Additionally, I must remind that my guide is written for developers. It includes the basic integration and does not include advanced features.

[nsmith1024]

Dude, listen to me good... YOU THA MAN!!!

[nsmith1024]

Can this be used to play a video to a Unity texture, the built in Unity VideoPlayer is no good.
Can it play mp4 or live streams like HLS RTMP etc?
Even if it cant do those things, it still great if it can do anything at all in unity

[tanersener]

No, unfortunately you can't play a file or a stream.

[mantraspace]

Several monthes ago I got this great library to work with Unity for iOS platform (in a frequency generator app of mine, i had to allow users to convert recorded audio to various audio formats, as well as to create an mp4 video from an image / animated gif and an audio file), i will try to publish a little tutorial as soon as possible.

@tanersener, thanks for your library, and for explaining how to import ffmpeg-mobile into Unity for Android. For Android platform, i have been using another similar library dedicated to Android (Bravobit), but due to the requirement to make Android apps compatible with 64-bit architectures in addition to 32-bit ones, i am very interested into following your tutorial. I tried a few weeks ago, but for some reason i couldn't make your library work with Unity for Android.

[tanersener]

@mantraspace Great, let me know if you need assistance.

[cyliax]

It's me again and I like to share my progress with you. I started to call sync androidjavaobject calls like @tanersener explained in his example and a lil bit more. But I wasn't happy that it wasn't async and I haven't had a progress during encoding, so this ist what I did:

package de.kids_interactive.mobileffmpeg;

import android.os.AsyncTask;
import android.util.Log;

import com.arthenica.mobileffmpeg.Config;
import com.arthenica.mobileffmpeg.FFmpeg;
import com.arthenica.mobileffmpeg.Statistics;
import com.arthenica.mobileffmpeg.StatisticsCallback;
import com.arthenica.mobileffmpeg.util.AsyncExecuteTask;
import com.arthenica.mobileffmpeg.util.ExecuteCallback;

import com.unity3d.player.UnityPlayer;

public class FFmpegHelper
{
public static void executeAsync(final String arguments, final String receiver, final String onProgressCallback, final String onCompleteCallback)
{
Config.resetStatistics();
Config.enableStatisticsCallback(new StatisticsCallback()
{
@Override public void apply(final Statistics statistics)
{
int time = statistics.getTime();
Log.d("Unity", String.format("FFmpeg progress: %d", time));
UnityPlayer.UnitySendMessage(receiver, onProgressCallback, Integer.toString(time));
}
});

    final ExecuteCallback executeCallback = new ExecuteCallback()
    {
        @Override public void apply(final int returnCode, final String commandOutput)
        {
            Log.d("Unity", String.format("FFmpeg completed with returnCode: %d", returnCode));
            // this ### is just to split my string afterwards
            UnityPlayer.UnitySendMessage(receiver, onCompleteCallback, Integer.toString(returnCode) + "###" + commandOutput);

            Config.enableStatisticsCallback(null);
        }
    };

    final AsyncExecuteTask asyncCommandTask = new AsyncExecuteTask(executeCallback);
    asyncCommandTask.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR, arguments.split(" "));
}

public static void cancel()
{
    FFmpeg.cancel();
}

}

(sorry for faulty formatting)

As you see, I moved some logic from the C# classes to my own Java helper, because it's much easier to access object and methods in Java itself and just call "exectuteAsync" from Unity. There has to be a gameobject with the receiver name in your Unity scene root with public methods for progress and complete callback. UnitySendMessage can only send strings and passing multiple results is a lil bit nasty. CommandOutput can also be very long and I skipped to pass as a text to TextMeshPro, because those textfields are limited.

At the end I managed it to run complex FFmpeg commands with filter_complex, cropping, merging, adding an overlay and so on, which is great. But that's not the end, I'll let you know when it comes to iOS and mayby another Unity mobile-ffmpeg bridge.

PS: I'm not a Java dude, so there might be some way to improve my code.

[nsmith1024]

When you get it all working put it on the Unity Asset store, im sure people need it badly!

On Monday, September 9, 2019, 3:17:35 AM EDT, Carsten Cyliax <[email protected]> wrote:

It's me again and I like to share my progress with you. I started to call sync androidjavaobject calls like @tanersener explained in his example and a lil bit more. But I wasn't happy that it wasn't async and I haven't had a progress during encoding, so this ist what I did:

package de.kids_interactive.mobileffmpeg;

import android.os.AsyncTask;
import android.util.Log;

import com.arthenica.mobileffmpeg.Config;
import com.arthenica.mobileffmpeg.FFmpeg;
import com.arthenica.mobileffmpeg.Statistics;
import com.arthenica.mobileffmpeg.StatisticsCallback;
import com.arthenica.mobileffmpeg.util.AsyncExecuteTask;
import com.arthenica.mobileffmpeg.util.ExecuteCallback;

import com.unity3d.player.UnityPlayer;

public class FFmpegHelper
{
public static void executeAsync(final String arguments, final String receiver, final String onProgressCallback, final String onCompleteCallback)
{
Config.resetStatistics();
Config.enableStatisticsCallback(new StatisticsCallback()
{
@override public void apply(final Statistics statistics)
{
int time = statistics.getTime();
Log.d("Unity", String.format("FFmpeg progress: %d", time));
UnityPlayer.UnitySendMessage(receiver, onProgressCallback, Integer.toString(time));
}
});
final ExecuteCallback executeCallback = new ExecuteCallback()
{
@Override public void apply(final int returnCode, final String commandOutput)
{
Log.d("Unity", String.format("FFmpeg completed with returnCode: %d", returnCode));
// this ### is just to split my string afterwards
UnityPlayer.UnitySendMessage(receiver, onCompleteCallback, Integer.toString(returnCode) + "###" + commandOutput);

        Config.enableStatisticsCallback(null);
    }
};

final AsyncExecuteTask asyncCommandTask = new AsyncExecuteTask(executeCallback);
asyncCommandTask.executeOnExecutor(AsyncTask.THREAD_POOL_EXECUTOR, arguments.split(" "));

}

public static void cancel()
{
FFmpeg.cancel();
}

}

(sorry for faulty formatting)

As you see, I moved some logic from the C# classes to my own Java helper, because it's much easier to access object and methods in Java itself and just call "exectuteAsync" from Unity. There has to be a gameobject with the receiver name in your Unity scene root with public methods for progress and complete callback. UnitySendMessage can only send strings and passing multiple results is a lil bit nasty. CommandOutput can also be very long and I skipped to pass as a text to TextMeshPro, because those textfields are limited.

At the end I managed it to run complex FFmpeg commands with filter_complex, cropping, merging, adding an overlay and so on, which is great. But that's not the end, I'll let you know when it comes to iOS and mayby another Unity mobile-ffmpeg bridge.

PS: I'm not a Java dude, so there might be some way to improve my code.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[cyliax]

I managed it to have a working Unity iOS plugin with async calls and progress callback as well. @tanersener is there a way to contact you, to discuss how we or I could release an Unity asset based on your mobile-ffmpeg lib.

[nsmith1024]

One thing to note before releasing it (to maybe Unity asset store or any other place) ... you have to make sure the total size of the mobile-ffmpeg build is as small as possible, because Android play store only allows apk files of 100M or less to be placed on the play store. So if mobile-ffmpeg is 300M then nobody can use it in a real project because play store wont accept it because its way over the 100M maximum. Im running into that problem now because my build is 250M (without ffmpeg)

[tanersener]

@cyliax there is a mobile-ffmpeg channel on gitter which I don't promote too much. You can send me a private message on that channel.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.