Javacv: Decoding JPEG bytes into a Mat?

[MC-2016]

I have a ByteBuffer that contains the data of a JPEG file (the actual source of data is not a file, so I cannot use _imread_). I need decode these bytes into a _Mat_ that I can display, transform, store in a file and so on.

I am talking about something like this:

channel.read(byteBuffer);    // byteBuffer has JPEG data now
Mat encoded = byteBufferToMat(byteBuffer);    // Magic happens here
Mat decoded = opencv_imgcodecs.imdecode(encoded, opencv_imgcodecs.CV_LOAD_IMAGE_COLOR);
canvasFrame.showImage(decoded);

The question is: how do I code the _byteBufferToMat()_ method? I've tried googling it up and I can't find any useful examples. I am using OpenCV 3.0 or 3.1. Most examples tell me to use _Mat.put(int,int,byte[])_ and I don't think it exists anymore.

(NB: I would prefer a solution that doesn't require copying the contents of the byte buffer, but I am not very picky at the moment.)

[saudet]

I'm guessing imdecode(new Mat(new BytePointer(encoded))) should do the trick!

[MC-2016]

I tried that and got this...

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000007fef0dc911b, pid=3864, tid=4992
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [opencv_imgcodecs300.dll+0x911b]
#

That's _imdecode()_ blowing up:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.bytedeco.javacpp.opencv_imgcodecs.imdecode(Lorg/bytedeco/javacpp/opencv_core$Mat;I)Lorg/bytedeco/javacpp/opencv_core$Mat;+0
...

I tried the other flavor of _imdecode()_ and got the same result:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.bytedeco.javacpp.opencv_imgcodecs.imdecode(Lorg/bytedeco/javacpp/opencv_core$Mat;ILorg/bytedeco/javacpp/opencv_core$Mat;)Lorg/bytedeco/javacpp/opencv_core$Mat;+0
...

[saudet]

Right, I forgot an overload for the signed flag. This should work better:

imdecode(new Mat(new BytePointer(encoded), false))

[MC-2016]

Yep! That one works. Thank you.

By the way, does this constructor copy the contents of the byte buffer? It takes a pretty long time to run (around 300 microseconds on my desktop).

[saudet]

It shouldn't, not as far as I know. If you see it do otherwise, it's probably a bug in OpenCV. :)

[saudet]

Unless byteBuffer is not a direct buffer. Then of course BytePointer has no choice but to copy the Java array to native memory.

[MC-2016]

Ah-ha, that was it. 300 microseconds with heap buffers, ~20 microseconds with direct buffers.

Makes sense.

[bashimao]

Hi, this thread helped me understanding of the javacv wrapper a lot. However, I have the vice-versa problem. I want to encode an image into memory. imencode looks like it is the right function for this purpose. But I do not know how much memory to allocate or, if I just allocate a ridiculously large buffer, how many bytes contain meaningful data. imencode just returns a bool.

[saudet]

@bashimao According to the docs "Output buffer resized to fit the compressed image", so just pass in an empty BytePointer, and check limit() on return.

[bashimao]

Ok, that sounds reasonable. But won't that create a memory leak? I mean: Does the pointer know that it has to call the destructor on the implied std::vector?

[saudet]

@bashimao Yeah, that's the idea, some explanations:
http://bytedeco.org/news/2015/10/25/moving-to-java-se-7/
http://bytedeco.org/news/2016/05/15/deeper-in-deep-learning/

[bashimao]

Thank you for the links. I got it working and it looks as if there is no memory leak. I was not aware that JavaCPP can encapsulate complex type that use internal reallocation like std::vec. Just used it for cuDNN so far. And there I have to provide custom allocation and deallocation code.

If somebody else is interested in this. Here is some code...

... to encode into an array:

override def encode(encoding: String)
  : Array[Byte] = {
    using(new BytePointer())(ptr => {
      val success = imencode(s".$encoding", mat, ptr)
      assume(success)
      BufferEx.toArray(ptr.asByteBuffer())
    })
  }

... to grab the raw pixels (important: This works even if the image was cropped!):

override def toIntArray
  : Array[Int] = {
    val w = mat.cols
    val h = mat.rows
    val c = mat.channels
    val rowStride = w * c

    val result  = new Array[Int](rowStride * h)
    val indexer = mat.createIndexer[UByteIndexer]()
    var off = 0
    var y   = 0
    while (y < h) {
      indexer.get(y, result, off, rowStride)
      off += rowStride
      y   += 1
    }
    result
  }