Javacv: FFmpegFrameGrabber can't grab audio frame from InputStream

[yzw607]

Hi guys,
My javacv version is 1.5.3, I'm trying to receive InputStream and push to rtmp. Actually the input data I receive every time is a complete frame data(video or audio frame). Now the video frames work fine and I can see video through rtmp server, but audio frames seem missing so can't hear any sound. Below is the related code:

`public class PushTask implements Runnable {

    private volatile boolean stop = false;

    private String taskName;

    private String rtmpAddr;

    private ChannelHandlerContext context;

    private FFmpegFrameGrabber grabber;

    private FFmpegFrameRecorder recorder;

    private PipedOutputStream pos;

    private PipedInputStream pis;

    private boolean running;

    public PushTask(String taskName, String rtmpAddr, ChannelHandlerContext context)
            throws IOException {
        this.taskName = taskName;
        this.rtmpAddr = rtmpAddr;
        this.context = context;
        pos = new PipedOutputStream();
        pis = new PipedInputStream();
        pis.connect(pos);
    }

    @Override
    public void run() {
        log.info("create frame push task : {}", taskName);
        try {
            grabber = new FFmpegFrameGrabber(pis, 0);
            grabber.setFormat("h264");
            FFmpegLogCallback.set();
            log.info("start to initialize grabber!!");
            grabber.start();
            log.info("initialize frame push grabber over! :{}", taskName);
            int width = grabber.getImageWidth();
            int height = grabber.getImageHeight();

            // video setting
            int videoCodecid = grabber.getVideoCodec();
            double frameRate = grabber.getVideoFrameRate();
            double bitRate = grabber.getVideoBitrate();
            // audio setting
            // below three params are required：audioChannels > 0 && audioBitrate > 0 && sampleRate > 0
            int audioCodecid = grabber.getAudioCodec();
            int audioChannels = grabber.getAudioChannels();
            double audioBitrate = grabber.getAudioBitrate();
            int sampleRate = grabber.getSampleRate();
            log.info("===============grabber videoCodecid is: {},  frameRate is : {}, bitRate is {}, " +
                    " audioCodecid is: {}, audioChannels is: {}, audioBitrate is: {}, width is: {}, height is : {}, sampleRate is {} ==============",
                    videoCodecid, frameRate, bitRate, audioCodecid, audioChannels, audioBitrate, width, height, sampleRate);

            this.rtmpAddr = "rtmp://127.0.0.1:23895/yzw/616";
            recorder = new FFmpegFrameRecorder(rtmpAddr, width, height);`

I tried to pick one video/audio frame data(raw byte[]) and test on my local pc, video frame is ok, but audio frame will block in FFmpegFrameGrabber.avformat_find_stream_info, and throw below exception finally:

`Warning: Invalid return value 0 for stream protocol

Warning: Invalid return value 0 for stream protocol

Error: [h264 @ 000000002d431540] missing picture in access unit with size 34080

Error: [AVBSFContext @ 000000002adebcc0] No start code is found.

org.bytedeco.javacv.FrameGrabber$Exception: avformat_find_stream_info() error -1094995529: Could not find stream information.
    at org.bytedeco.javacv.FFmpegFrameGrabber.startUnsafe(FFmpegFrameGrabber.java:870) ~[javacv-1.5.3.jar:1.5.3]
    at org.bytedeco.javacv.FFmpegFrameGrabber.start(FFmpegFrameGrabber.java:790) ~[javacv-1.5.3.jar:1.5.3]
    at org.bytedeco.javacv.FFmpegFrameGrabber.start(FFmpegFrameGrabber.java:785) ~[javacv-1.5.3.jar:1.5.3]`

As I mentioned above, the data I received every time is already a complete video or audio frame, actually for audio frame I don't need to grab and can push to rtmp directly with some additional setting like sampleRate and audioChannels.
But I still want to know if my above implementing is feasible for both video and audio frames. It'll be very convenient if I can get audio setting from audio frame directly like video frame because our application may support different type of devices with different audio settings(like sampleRate, audioChannels).
If above implementing should work, then what's the problem in my code? It's about format? or some other settings?
Thanks for help!

[saudet]

Try to skip over avformat_find_stream_info().

[saudet]

If that doesn't work, try to set maximumSize to something greater than 0.

[yzw607]

If that doesn't work, try to set maximumSize to something greater than 0.

Thanks for you response, saudet! But seems don't work for my case. Actually I tested a local mp4 file, the video and audio setting can be grabbed easily, I guess for audio inputstream, the audio setting info may be stored differently.
So I already changed my implementing: handle audio inputstream separately, but seems still have some audio codec issues: the source audio codec may be adpcm/g711-a/g711-u, the client player hope the codec is aac, so i need to convert the audio codec, is there some api in javacv to do audio codec converting? Thanks!

[saudet]

Thanks for you response, saudet! But seems don't work for my case. Actually I tested a local mp4 file, the video and audio setting can be grabbed easily, I guess for audio inputstream, the audio setting info may be stored differently.

MP4 doesn't work too well with streaming, that's well known.

So I already changed my implementing: handle audio inputstream separately, but seems still have some audio codec issues: the source audio codec may be adpcm/g711-a/g711-u, the client player hope the codec is aac, so i need to convert the audio codec, is there some api in javacv to do audio codec converting? Thanks!

Yes, that's what FFmpegFrameRecorder is for.

[yzw607]

FFmpegFrameRecorder

Thanks for you response, saudet! But seems don't work for my case. Actually I tested a local mp4 file, the video and audio setting can be grabbed easily, I guess for audio inputstream, the audio setting info may be stored differently.

MP4 doesn't work too well with streaming, that's well known.

So I already changed my implementing: handle audio inputstream separately, but seems still have some audio codec issues: the source audio codec may be adpcm/g711-a/g711-u, the client player hope the codec is aac, so i need to convert the audio codec, is there some api in javacv to do audio codec converting? Thanks!

Yes, that's what FFmpegFrameRecorder is for.

Thanks for your quick response!
I already set audiocodec like below:
recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);

now I use FFmpegFrameRecorder to push my audio info directly:

    public boolean recordSamples(int sampleRate, int audioChannels, Buffer ... samples) throws Exception {
        if (audio_st == null) {

my question is: the audio codec in parameter 'Buffer samples' should be a specific one? As I mentioned above, my received audio codec may be adpcm/g711-a/g711-u, I should convert them to a standard one before push? or I can put any audio codec here and push audio info directly?

[saudet]

my question is: the audio codec in parameter 'Buffer samples' should be a specific one? As I mentioned above, my received audio codec may be adpcm/g711-a/g711-u, I should convert them to a standard one before push? or I can put any audio codec here and push audio info directly?

It needs to be PCM. If you have something else, you can decode it with FFmpegFrameGrabber first.

[yzw607]

my question is: the audio codec in parameter 'Buffer samples' should be a specific one? As I mentioned above, my received audio codec may be adpcm/g711-a/g711-u, I should convert them to a standard one before push? or I can put any audio codec here and push audio info directly?

It needs to be PCM. If you have something else, you can decode it with FFmpegFrameGrabber first.

Ok, thank you, I'll try!

[yzw607]

I tried to decode the audio to pcm, then push to rtmp, the sound can be heard now, thanks for your help @saudet !