Javacv: Play Audio of Movie with FFmpegFrameGrabber

[zavataafnan]

I open a video and get its frames with FFmpegFrameGrabber. I can get and show the frames in SurfaceView in Android and successfully. Unfortunately, I am not able to play sound correctly. It seems the output contains many noises in the background. I have searched all over the internet about the ways that I can do. But I am not successful.

val SC16 = 0x7FFF.toDouble() + 0.4999999999999999
var full_path = Dirs[1].toString()+ "/video/first_video.mkv"
var grabber:FFmpegFrameGrabber
var exampleFile: File = FileUtils.getFile(full_path)
var exampleStream = FileUtils.openInputStream(exampleFile)

var inputStream: FileInputStream? = null
var bufinputStream: BufferedInputStream? = null

inputStream = FileInputStream(exampleFile)
bufinputStream = BufferedInputStream(inputStream)
grabber = FFmpegFrameGrabber(bufinputStream)
grabber.setPixelFormat(avutil.AV_PIX_FMT_RGBA)
grabber.start()

grabber.setVideoFrameNumber(0)
grabber.setAudioFrameNumber(0)

//I should do that to output in FloatBuffer! I think there is a bug here
grabber.sampleFormat = AV_SAMPLE_FMT_FLT

var frame1: Frame = grabber.grabSamples()

            var mBufferSize = AudioTrack.getMinBufferSize(
                    frame1.sampleRate, AudioFormat.CHANNEL_OUT_STEREO,
                    AudioFormat.ENCODING_PCM_16BIT
                )

                val myTrack = AudioTrack(
                    AudioAttributes.Builder()
                        .setUsage(AudioAttributes.USAGE_MEDIA)
                        .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                        .build(),
                    AudioFormat.Builder().setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                        .setSampleRate(frame1.sampleRate)
                        .setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
                        .build(),
                    mBufferSize,
                    AudioTrack.MODE_STREAM,
                    AudioManager.AUDIO_SESSION_ID_GENERATE
                )

            for(i in 0..100000) {
                var frame1: Frame = grabber.grabSamples()
                var shrBuf = frame1.samples[0] as FloatBuffer
                shrBuf.rewind()
                val outBuffer = ByteBuffer.allocate(shrBuf.capacity() * 4)

                for(index in 0 until shrBuf.capacity()){
                    val value:Short = (shrBuf.get(index) * SC16).toShort()
                    outBuffer.putShort(value)
                }


                myTrack.write(outBuffer.array(), 0,outBuffer.capacity())
                myTrack.play()
                outBuffer.clear()
            }

            myTrack.release()

Any help is apprietiated...

[saudet]

Duplicate of #110 ?

[zavataafnan]

Yes, the basic idea is from that issue. But this is for android. There were some libraries that are not used in the Android SDK. I also read the frames with Matlab. There are some differences in the values of pitches. I want to know why the output of grabberSamples is not the type of the input sound? For example, my input sound is a type of AV_SAMPLE_FMT_FLTP. why the output is type of AV_SAMPLE_FMT_U8? I should set the grabber.sampleFormat = AV_SAMPLE_FMT_FLT. Is it correct?

[saudet]

There's actually another SampleMode property for that, for example, as demonstrated here:
https://github.com/bytedeco/javacv/blob/master/platform/src/test/java/org/bytedeco/javacv/FrameGrabberTest.java#L91

[zavataafnan]

Yes, I have used it. But the output is not planar in the case of input as planar! I want to know it is correct that the output format should be the same as input format!!? Why the initialize output is base on ByteBuffer and non planar.

[saudet]

Most codecs support only 1 or 2 formats. To avoid any conversions, you can also use SampleMode.RAW.

[zavataafnan]

Thank you, man. without any changes in output audio format. It plays perfectly.

   ```

var frame1: Frame = grabber.grabSamples()

    if(frame1.audioChannels != 0) {
        var shrBuf = frame1.samples[0] as ShortBuffer

        val outBuffer = ShortBuffer.allocate(shrBuf.capacity())

        for (index in 0 until shrBuf.capacity()) {
            outBuffer.put(shrBuf.get(index))
        }

        myTrack.write(outBuffer.array(), 0, outBuffer.capacity())

        //myTrack.write(outBuffer.array(), 0,outBuffer.capacity())
        //myTrack.write(outBuffer, outBuffer.capacity(), 1)
        myTrack.play()
        outBuffer.clear()
    }

```