Javacpp-presets: Crash while writing frame with org.bytedeco:ffmpeg-platform:4.2.1-1.5.2

[mel4tonin4]

Hello.

I've just started to use bytedeco's ffmpeg Java wrappers in my project to perform a basic transcoding of the audio stream of a file. I modeled my code after this tutorial.

I get a crash at [libavformat.so.58+0xa44d7] avio_write+0x17 and I have a core dump but it's not helpful, because I don't have the debugging symbols for that library.

Do you have any suggestions on how I could debug this issue?

Below, I'll add the Kotlin source code of the offending function and the output message, but here is a gist with line numbers. The last Kotlin line executed is line 138 in AudioDecompression.kt.

This is the source code:

import java.io.File

import mu.KotlinLogging

import org.bytedeco.ffmpeg.avcodec.AVCodecContext
import org.bytedeco.ffmpeg.avformat.AVFormatContext
import org.bytedeco.ffmpeg.avformat.AVStream
import org.bytedeco.ffmpeg.avutil.AVDictionary
import org.bytedeco.ffmpeg.avutil.AVRational
import org.bytedeco.ffmpeg.global.avcodec.*
import org.bytedeco.ffmpeg.global.avformat
import org.bytedeco.ffmpeg.global.avformat.*
import org.bytedeco.ffmpeg.global.avutil.*

private val logger = KotlinLogging.logger {}

fun decompressAudioFile(source : File, destination : File) {
    val inputFormatContext = openInputFormatContext(source)
    val inputStream = inputFormatContext.firstAudioStream() ?: throw Exception("Could not find audio stream")

    logger.info { "Decompressing stream ${inputStream.index}" }

    inputFormatContext.dumpStreamInfo(inputStream.index, source.path)

    val inputCodecContext = inputStream.createCodecContext()

    val codec = inputCodecContext.findDecoder() ?: throw Exception("Codec for file $source is not supported")

    logger.info { "Input codec '${codec.name}' found" }

    inputCodecContext.openCodec(codec)

    logger.info { "Input codec '${codec.name}' open " }

    val channelCount = inputCodecContext.channels()
    val sampleRate = inputCodecContext.sample_rate()

    val timeBase = AVRational().num(1).den(sampleRate)

    val outputFormat = avformat.av_guess_format("s16le", null, null)
            ?: throw Exception("Could not create output format")

    logger.info { "Output audio format '${outputFormat.name}' found" }

    val outputFormatContext  = createOutputFormatContext(outputFormat, destination)

    logger.info { "Output audio format context created" }

    val outputStream = avformat_new_stream(outputFormatContext, null)?.apply {
        time_base(timeBase)
    } ?: throw Exception("Could not create output stream")

    logger.info { "Output audio stream created" }

    val outputCodec = avcodec_find_encoder(outputFormat.audio_codec())
            ?: throw Exception("Output audio codec '${outputFormat.audio_codec()}' not found")

    logger.info { "Output audio codec found" }

    val outputCodecContext = avcodec_alloc_context3(outputCodec).apply {
        channels(channelCount)
        channel_layout(av_get_default_channel_layout(channelCount))
        sample_rate(sampleRate)
        sample_fmt(outputCodec.sample_fmts()[0])
        //bit_rate()
        time_base(timeBase)
    }

    logger.info { "Output audio codec context created and set up" }

    if (avcodec_open2(outputCodecContext, outputCodec, null as AVDictionary?) < 0) {
        throw Exception("Could not open ouput codec context")
    }

    logger.info { "Output audio codec context open" }

    if (outputFormatContext.oformat().flags() or AVFMT_NOFILE == 0) {
        if (avio_open(outputFormatContext.pb(), destination.path, AVIO_FLAG_WRITE) < 0) {
            throw Exception("Could not open output file \"$destination\"")
        }
    }

    logger.info { "Output file \"$destination\" open" }

    if (avformat_write_header(outputFormatContext, null as AVDictionary?) < 0) {
        throw Exception("Could not write output file header")
    }

    logger.info { "Output file header written" }

    runTranscodingLoop(
        inputFormatContext, inputCodecContext, inputStream,
        outputFormatContext, outputCodecContext, outputStream
    )

    if (av_write_trailer(outputFormatContext) < 0) {
        throw Exception("Could not write output file trailer")
    }

    logger.info { "Output file trailer written" }

    avformat_close_input(inputFormatContext)

    logger.info { "Input file closed" }

    if (outputFormatContext.oformat().flags() or AVFMT_NOFILE == 0) {
        avio_close(outputFormatContext.pb())
    }

    logger.info { "Output file closed" }

    avcodec_close(inputCodecContext)
    avcodec_free_context(inputCodecContext)

    avcodec_close(outputCodecContext)
    avcodec_free_context(outputCodecContext)
}

private fun runTranscodingLoop(
    inputFormatContext : AVFormatContext, inputCodecContext : AVCodecContext, inputStream : AVStream,
    outputFormatContext : AVFormatContext, outputCodecContext : AVCodecContext, outputStream: AVStream
) {
    inputFormatContext.forEachPacket { inputPacket ->
        if (inputPacket.stream_index() == inputStream.index) {
            logger.info { "Decoding packet: PTS=${inputPacket.pts()}, DTS=${inputPacket.dts()}" }

            try {
                inputCodecContext.forEachDecodedFrame(inputPacket) { frame ->
                    logger.info { "Processing frame: ${frame.best_effort_timestamp()} (${frame.nb_samples()} samples)" }

                    try {
                        outputCodecContext.forEachEncodedPacket(frame) { outputPacket ->
                            outputPacket.stream_index(outputStream.index)
                            av_packet_rescale_ts(outputPacket, inputStream.time_base(), outputStream.time_base())
                            logger.info { "Output packet: PTS=${outputPacket.pts()}, DTS=${outputPacket.dts()}" }
                            if (av_write_frame(outputFormatContext, outputPacket) < 0) {
                                logger.error { "Can't write packet to output file" }
                            }
                        }
                    } catch (e : Exception) {
                        logger.error { "Error encoding frame: ${e.message}" }
                    }
                }
            } catch (e : Exception) {
                logger.error { "Error decoding packet: ${e.message}" }
            }
        } else {
            logger.info { "Skipping packet" }
        }
    }
}

The code above uses the following extension methods:

import java.io.File
import java.nio.IntBuffer

import org.bytedeco.ffmpeg.avcodec.AVCodec
import org.bytedeco.ffmpeg.avcodec.AVCodecContext
import org.bytedeco.ffmpeg.avcodec.AVPacket
import org.bytedeco.ffmpeg.avformat.AVFormatContext
import org.bytedeco.ffmpeg.avformat.AVOutputFormat
import org.bytedeco.ffmpeg.avformat.AVStream
import org.bytedeco.ffmpeg.avutil.AVDictionary
import org.bytedeco.ffmpeg.avutil.AVDictionaryEntry
import org.bytedeco.ffmpeg.avutil.AVFrame
import org.bytedeco.ffmpeg.global.avcodec
import org.bytedeco.ffmpeg.global.avformat
import org.bytedeco.ffmpeg.global.avutil
import org.bytedeco.ffmpeg.presets.avutil.AVERROR_EAGAIN
import org.bytedeco.javacpp.Pointer
import org.bytedeco.javacpp.PointerPointer

class Exception(message : String) : kotlin.Exception(message)

fun enumerateAllCodecs() = sequence {
    val p = Pointer()
    var codec : AVCodec? = avcodec.av_codec_iterate(p)
    while (codec != null) {
        yield(codec!!)
        codec = avcodec.av_codec_iterate(p)
    }
}

fun openInputFormatContext(file : File, findStreamInfo : Boolean = true) =
        AVFormatContext(null).also {
            if (avformat.avformat_open_input(it, file.path, null, null) < 0) {
                throw Exception("Could not open file $file")
            }
        }.apply {
            if (findStreamInfo) {
                findStreamInfo()
            }
        }

fun createOutputFormatContext(outputFormat : AVOutputFormat, file : File) =
        AVFormatContext().also {
            if (avformat.avformat_alloc_output_context2(it, outputFormat, outputFormat.name, file.path) < 0) {
                throw Exception("Could not create output context")
            }
        }

fun AVFormatContext.enumerateStreams() = (0 until nb_streams()).asSequence().mapNotNull { streams(it) }
fun AVFormatContext.findStreamInfo() {
    if (avformat.avformat_find_stream_info(this, null as AVDictionary?) < 0) {
        throw Exception("Count not find stream information")
    }
}

fun AVFormatContext.firstAudioStream() : AVStream? =
    enumerateStreams().firstOrNull { it.codecpar().codec_type() == avutil.AVMEDIA_TYPE_AUDIO }

fun AVStream.createCodecContext() : AVCodecContext =
    avcodec.avcodec_alloc_context3(null).also {
        avcodec.avcodec_parameters_to_context(it, codecpar())
    }

fun AVCodecContext.findDecoder() : AVCodec? =
    avcodec.avcodec_find_decoder(codec_id())

fun AVCodecContext.computeAudioBufferSize() =
    avutil.av_samples_get_buffer_size(null as IntBuffer?, channels(), frame_size(), sample_fmt(), 0)

val AVCodec.name get() : String = name().string
val AVOutputFormat.name get() : String = name().string

fun AVCodecContext.openCodec(codec : AVCodec) {
    if (avcodec.avcodec_open2(this, codec, null as PointerPointer<*>?) < 0) {
        throw Exception("Can not open codec '${codec.name}'")
    }
}

fun AVDictionary.enumerateEntries() : Sequence<AVDictionaryEntry> = let { dictionary ->
    sequence {
        var entry : AVDictionaryEntry? = null
        while (true) {
            entry = avutil.av_dict_get(dictionary, "", entry,  avutil.AV_DICT_IGNORE_SUFFIX)
            if (entry == null) {
                break
            }
            yield(entry!!)
        }
    }
}

fun AVFormatContext.dumpStreamInfo(streamIndex: Int, path : String, isOutput : Boolean = false) =
    avformat.av_dump_format(this, streamIndex, path, if (isOutput) 1 else 0)

val AVStream.index get() : Int = index()

inline fun AVFormatContext.forEachPacket(block: (AVPacket) -> Unit) {
    val packet = AVPacket()

    while (avformat.av_read_frame(this, packet) >= 0) {
        block(packet)
        avcodec.av_packet_unref(packet)
    }
}

inline fun AVCodecContext.forEachFrame(block : (AVFrame) -> Unit) {
    val frame = AVFrame()
    while (true) {
        val response = avcodec.avcodec_receive_frame(this, frame)
        if (response == AVERROR_EAGAIN() || response == avutil.AVERROR_EOF) {
            break
        } else if (response < 0) {
            throw Exception("Error while receiving frame from decoder")
            //av_make_error_string()
        }
        block(frame)
        avutil.av_frame_unref(frame)
    }
}

inline fun AVCodecContext.forEachDecodedFrame(packet : AVPacket, block : (AVFrame) -> Unit) {
    if (avcodec.avcodec_send_packet(this, packet) < 0) {
        throw Exception("Could not forward packet to decoder")
    } else {
        forEachFrame(block)
    }
}

inline fun AVCodecContext.forEachEncodedPacket(frame : AVFrame, block : (AVPacket) -> Unit ) {
    if (avcodec.avcodec_send_frame(this, frame) < 0) {
        throw Exception("Could not forward frame to encoder")
    }
    forEachPacket(block)
}

inline fun AVCodecContext.forEachPacket(block : (AVPacket) -> Unit ) {
    val packet = AVPacket()
    while (true) {
        val response = avcodec.avcodec_receive_packet(this, packet)
        if (response == org.bytedeco.ffmpeg.presets.avutil.AVERROR_EAGAIN() || response == avutil.AVERROR_EOF) {
            break
        } else if (response < 0) {
            throw Exception("Error while receiving frame from encoder")
            //av_make_error_string()
        }
        block(packet)
        avcodec.av_packet_unref(packet)
    }
}

and, finally, this is the output of a run of the application:

14:50:39.318 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Decompressing stream 0
Input #0, matroska,webm, from 'downloads/QoIw0Q5oEyI.webm':
  Metadata:
    encoder         : google/video-file
  Duration: 00:05:49.22, start: -0.007000, bitrate: 119 kb/s
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
14:50:39.336 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Input codec 'opus' found
14:50:39.337 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Input codec 'opus' open 
14:50:39.349 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio format 's16le' found
14:50:39.349 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio format context created
14:50:39.349 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio stream created
14:50:39.350 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio codec found
14:50:39.350 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio codec context created and set up
14:50:39.350 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output audio codec context open
14:50:39.350 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output file "downloads/QoIw0Q5oEyI.raw" open
14:50:39.351 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output file header written
14:50:39.356 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Decoding packet: PTS=-7, DTS=-7
[opus @ 0x7fa2ed1aaa40] Could not update timestamps for skipped samples.
14:50:39.360 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Processing frame: -7 (648 samples)
14:50:39.360 [main] INFO com.aeyeonmovement.ffmpeg.AudioFileDecompression - Output packet: PTS=-336, DTS=-336
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa23186c4d7, pid=102959, tid=102960
#
# JRE version: OpenJDK Runtime Environment (11.0.5+10) (build 11.0.5+10-post-Ubuntu-0ubuntu1.1)
# Java VM: OpenJDK 64-Bit Server VM (11.0.5+10-post-Ubuntu-0ubuntu1.1, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libavformat.so.58+0xa44d7]  avio_write+0x17
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /home/maurizio/choreoplayer/service/core.102959)
#
# An error report file with more information is saved as:
# /home/maurizio/choreoplayer/service/hs_err_pid102959.log
#
# If you would like to submit a bug report, please visit:
#   Unknown
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
/home/maurizio/choreoplayer/service/bin/cli: line 28: 102959 Aborted                 (core dumped) java $DEBUGGING_ARGS $ASSERTIONS_ARGS -classpath "$JARS_IN_LIBS" -Djava.library.path=$NATIVE_LIBRARY_PATH $MAIN_CLASS "$@"

Is it possible to have a "debug" version of the "org.bytedeco:ffmpeg-platform:4.2.1-1.5.2" jar? Don't people ever run into this kind of issues while using the javacpp presets?

Please let me know if I can improve this issue. Sorry for the wall of code.

Thanks for reading.

[saudet]

Is it possible to have a "debug" version of the "org.bytedeco:ffmpeg-platform:4.2.1-1.5.2" jar? Don't people even run into this kind of issues while using the javacpp presets?

Yes, it's possible to build binaries in debug mode. Edit the cppbuild.sh file with the flags you need and follow the instructions here:
https://github.com/bytedeco/javacpp-presets/blob/master/README.md#build-instructions
It's not too hard, but typical users don't know C/C++ and wouldn't be able to use the debug information, while the ones who are able to like you can easily build from source.

BTW, FFmpegFrameGrabber and FFmpegFrameRecorder already implement that kind of functionality, so we can use them as reference for clues:
https://github.com/bytedeco/javacv

[mel4tonin4]

Thanks, Samuel. I'll look into FFmpegFrameRecorder, first, to check if I can spot what I'm doing wrong which causes the crash. If that fails, I'll build the debugging jar of the preset.

[mel4tonin4]

The code of FFmpegFrameRecorder is not the best place to learn how to use _libavcodec_ to perform transcoding: part of the API calls are deprecated, part are redundant because they set up properties that are set up in the same way by the library as consequence of previous calls and part are redundant because they are always overridden by the code few lines down the road.

So, most of (if not all) the calls that are missing in my code have no effect.

Some don't make sense, like setting the time base of the (deprecated) codec context associated with the output stream. I practiced voodoo programming and set it, but it doesn't affect my problem. That codec() is not actually a _codec_ but a _codec context_, a different one from the one we created. What's its role?

It augmented my confusion.

I'm not criticizing you: I read that class is based on old code from the _ffmpeg_ codebase. I just wanted to warn other people who may want to learn how to use the _ffmpeg_ from it.

In my case, specifically, this example also uses JVM callbacks to perform I/O, possibly bypassing the call to the _avio_ function that is crashing in my code.

I'd try to use JVM callbacks, too, but here's another problem: some formats (having the AVFMT_NOFILE flag set) don't require or allow you to provide an _avio_ context, because they do it all by themselves. That is my case, as I'm trying to encode to a raw signed 16-bit little endian file.

I'll try to build the libraries with debugging symbols, I guess.

[mel4tonin4]

I decided that, for the time being, I'm going to run _libavcodec_ via the command line interface, out of process.
Besides the specific problem of this crash, I realized that I don't like the libav* APIs and their documentation, so I'll stay away from them as long as possible.

This is not a judgment on _javacpp_. I'm sure I'll use it again in future.

Thank you very much for _javacpp_ and for trying to help me!

[saudet]

If the ffmpeg program does what you need, yes, that's the easiest thing to use, even with JavaCPP:
http://bytedeco.org/javacpp-presets/ffmpeg/apidocs/org/bytedeco/ffmpeg/ffmpeg.html