Javacv: RealSenseFrameGrabber connection on D435i

[stephnicolas]

I'm using javacv-platform-1.5.1-bin with Java8 on windows 10

When I try to open RealSense D435 Video camera with this simple source code:
RealSenseFrameGrabber rsgrabber = RealSenseFrameGrabber.createDefault(0);
rsgrabber.tryLoad();
rsgrabber.enableDepthStream();
rsgrabber.start();

I have this message:
RealSense devices found: 0
Exception in thread "main" org.bytedeco.javacv.FrameGrabber$Exception: FATAL error: Realsense camera: 1 not connected/found
at org.bytedeco.javacv.RealSenseFrameGrabber.start(RealSenseFrameGrabber.java:239)
at WebcamAndMicrophoneCapture.main(WebcamAndMicrophoneCapture.java:64)

When I try to connect to the camera number one, I have the same result.
The realSense video works fine with realsenseviewer

[saudet]

It looks like a model that needs librealsense2:
https://github.com/IntelRealSense/librealsense/releases
If that's the case, we'll need to create a RealSense2FrameGrabber using this:
https://github.com/bytedeco/javacpp-presets/tree/master/librealsense2
@poqudrof Is it something that you could patch up together?

[stephnicolas]

There's also some useful methods that are not available in the RealSenseFrameGrabber:
• rs2.config.enable_stream
• rs2.video_stream_profile.get_intrinsics
• rs2.rs2_deproject_pixel_to_point
• rs2.pipeline.wait_for_frames
• rs2.depth_frame.get_distance
• rs2.align

[stephnicolas]

I can help maybe but I need help to manage code generation/packaging
I'm using IntelliJ

[saudet]

You shouldn't need to do anything special for IntelliJ IDEA. Just open this repository as a Maven project and it should work fine.

[poqudrof]

Hello,
My guess would be that the library is outdated. The last time I checked there were many updates in librealsense.
The SDK 2.0 is for D400 and SR300 series and the presets were for 1.12.4 for earlier dev kits. Unfortunately I stopped using Realsense devices in favor of Orbbec devices so I cannot update this easily. Moreover I did not do any javacpp code for a long time.

You mentionned the rs2 methods that probably stands for realsense 2.0: 

There's also some useful methods that are not available in the RealSenseFrameGrabber:
• rs2.config.enable_stream
• rs2.video_stream_profile.get_intrinsics
• rs2.rs2_deproject_pixel_to_point
• rs2.pipeline.wait_for_frames
• rs2.depth_frame.get_distance
• rs2.align

You could consider another wrapper that then wraps with JavaCV:

• OpenCV (should be a good solution)
• OpenNI2 (if the versions matches which I doubt)

You could also try out the java wrapper.

Cheers,
Jeremy

[cansik]

@poqudrof As developer of the realsense2 Java wrapper, I have to say that it is currently not recommended to use the Java wrapper. It was more a proof of concept to provide the different streams of the camera. Frame-alignment, pointcloud generation and the post-processing stack are not implemented in that version.

I started working on a new implementation of the wrapper, based on JavaCPP which supports all of it, but it is still under development. Thanks to @saudet for creating the librealsense2 C-binding wrapper. The Java wrapper orientates itself on the API structure of the C# wrapper.

Maybe I find a bit of time to look into developing a Framegrabber for the Realsense2, but I am currently quite busy with other stuff.

It would help me if someone could give me a hint on how to include the rs2-c api wrapper into JavaCV? Because it is currently not available on the maven repository.

[saudet]

@cansik I've already included a dependency on librealsense2 actually in commit https://github.com/bytedeco/javacv/commit/9db7b9f50dd6e11cb2a5c30c77b5eb01ac600b5e, so it's good to go! You'll need to use the snapshots to build it though: http://bytedeco.org/builds/ Then we just need to create a RealSense2FrameGrabber.java, based on RealSenseFrameGrabber.java I guess, and that should be pretty much it. Let me know if you encounter any problems with that. Thanks for your help!!

[cansik]

I've created a very basic class to access the depth frame of RealSense2 cameras (RealSense2FrameGrabber.java). I could not test the code because I don't have a rs2 at home, but I will tomorrow. And of course, my aim is to support all kind of streams and then create a PR.

[saudet]

Thanks! Please open a PR when you can.

[cansik]

Ok, it works (was a bit tricky to setup snapshots with gradle. Maybe an example project would help).

Now there are some implementation questions: For the realsense2 it would make sense to grab all the frames at once from the camera, and then access them later, until you want a new frame. This is because you need the different frames together, to create alignment, post processing and so on.

I could not find a comment about the FrameGrabber.trigger() method, but it sounds like this should trigger the camera to read a new image, right?
Is it ok if I implement the reading of the frames in there and then just return the images through grab() and some none API methods like grabIR(), grabDepth() & grabAligned()?

The API has to be used like this:

start();
while(running) {
    trigger();
    grabDepth();
    grabWhatever();
}
stop();

I think each FrameGrabber implementation interprets the API a bit different and its not so intuitive to have an order of methods which applies to some implementations but not to others. How do you think about?

[saudet]

Yes, Gradle doesn't support snapshots well:
http://bytedeco.org/builds/
A sample project wouldn't help. That's something that needs to be fixed in Gradle.

All FrameGrabber work like you mention above. Which ones do you believe do not?

We can use trigger() to flush all captured frames, and that would sync all the different frame types. No need to buffer anything manually though, if that's your question? The driver does this for us, so please don't try to do the same thing in Java.

[cansik]

Ok, so as far as I understand the RealSenseFrameGrabber class, it does wait for frames in trigger and then wait again in the grab method. This is fine if only grab() is called, otherwise you wait twice for new frames to appear, or am I missing a part there?

Regarding "buffering", there is a difference between RS1 and RS2 on how to wait for new frames. The first api handled the frame-pointer storing and allowed to access the different frame data through methods on the device. Now in rs2 we have to store the pointer to the frameset ourself (IntelRealSense/librealsense/wiki/API-How-To). So it would make sense to just have one method calling wait_for_frames, instead of two as in the rs1-framegrabber. The resulting frameset-pointer would then be stored inside our java class to be accessed by later calls like rs2_extract_frame.

Maybe I am understanding something totally wrong and sorry for the discussion, but it seems counterintuitive to call wait_for_frames in trigger() and grab(). Or to call it in every function like grabIR() and grabDepth(). In my opinion it should just be called once (in trigger()) and only if you want to have a new coherent frameset.

But maybe it's better to create a draft of the FrameGrabber and then just discuss the actual code. Maybe I find some time today in the evening.

[saudet]

I'm not aware of all the internals of librealsense, but if we just have to store pointers for some reason, as long as we don't copy buffers around, that sounds fine.

In trigger(), if we can't tell the camera to get a frame "now", it should as least flush all the past buffered frames. That doesn't look like what wait_for_frames() does so that's probably not correct. Don't worry about it, but if you know how to fix it, please do!

[saudet]

@stephnicolas Thanks to @cansik we now have a RealSense2FrameGrabber! Please give it a try with JavaCV 1.5.2-SNAPSHOT and let us know if you encounter any issues with that one:
http://bytedeco.org/builds/

@poqudrof BTW, do Orbbec cameras already have a usable Java API? Are you still working on projector-camera systems? In any case, please do open a pull request for an AstraFrameGrabber, if you create one! Thanks

[stephnicolas]

Thank you @cansik for the effort, thanks also to you @saudet, I will try it today

[cansik]

While using the FrameGrabber in an own project, I found some minor bugs (multi-stream reading) and added some more functionality like disabling the IR emitter and adding the Timestamp to the frame object.

I will add a new PR when finished with implementing the features / bug fixes.

[saudet]

JavaCV 1.5.2 has been released with RealSense2FrameGrabber! Thanks again @cansik for the contribution and please keep sending pull requests for any additional fixes or enhancements.

[stephnicolas]

Thank you for your help, the grabber is working fine, I have to manage several realsense cameras, how should I proceed ? is it possible to select a camera ? I've maybe to open a new ticket
Thanks

[saudet]

We can specify different device numbers, that should work fine: https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/RealSense2FrameGrabber.java#L55

[stephnicolas]

I have two detected cameras:
Device: Intel RealSense D435I 05.11.01.100 846112070870 Locked: true
Device: Intel RealSense D435I 05.11.06.250 843112070838 Locked: true
but there's no effect when I change the index in :
final RealSense2FrameGrabber rs2 = new RealSense2FrameGrabber(1);

[stephnicolas]

When I try to enable two stream types:
rs2.enableColorStream(640, 480, 30);
rs2.enableDepthStream(640, 480, 30);
the grabColor() method failed after a trigger
Process finished with exit code -1073740940 (0xC0000374)
in private rs2_frame findFrameByStreamType(rs2_frame frameset, int streamType, int index)

[cansik]

Yes, I've already updated and fixed some things in the findFrameByStreamType method, but just a simple color and depth stream should work. At the moment I am busy with finishing a project, but after that I can have a look at it.

Could you try out the wrapper of my fork, just copy it into your project and change the package name or the classname:

https://github.com/cansik/javacv/blob/master/src/main/java/org/bytedeco/javacv/RealSense2FrameGrabber.java

[stephnicolas]

Sorry but I've some trouble when I try to use the source code version such as:
Error:(32, 1) java: package org.bytedeco.flycapture.PGRFlyCapture does not exist
with intellij

[saudet]

@stephnicolas That file is supposed to be ignored. You can remove it.

[cansik]

@stephnicolas Ok, the error should be fixed now. I have tested it with the following code and it's working:

rs2.enableColorStream(640, 480, 30);
rs2.enableDepthStream(640, 480, 30);

// ...

Frame frame = rs2.grab();

There are some special cases with this wrapper, because it's very limited. Maybe I am going to write a short readme for it to cover these special cases.

Update: Readme written
https://github.com/bytedeco/javacv/wiki/RealSense2FrameGrabber

[KennyBarraud]

Hi all,

I'm currently testing the updates @cansik 's done and it's working great.
The next step of our test would be to add some filters on the depth frame.
I saw that the realsense2 library has integrated the filters, but they're not available in the RealSense2FrameGrabber.

Is there a solution to add filters using javacv ?

[saudet]

@KennyBarraud The native API is fully accessible from Java:
https://github.com/bytedeco/javacpp-presets/tree/master/librealsense2
It should not be too hard to extend RealSense2FrameGrabber with the features you need.

[KennyBarraud]

Thanks.
I have a question about this API.
To use filters, I need to create a rs2_processing_block instance and then use the rs2_set_option to set the valid filter parameters. However the API takes a rs2_options in parameter and the rs2_processing_block class doesn't seem to extend this last.

In the API provided by InterRealSense that I found here, the ProcessingBlock class extends the Options class.

So is there a clean way to set the options of the filter ?

[saudet]

Are you looking for the C++ API? That's on the to-do list, see https://github.com/bytedeco/javacpp-presets/issues/808. Contributions are welcome! But I don't think there are any limitations with the C API...

[KennyBarraud]

No I'm using Java for now.
Here is how I create and use the filter.

   rs2_frame_queue holeQueue = rs2_create_frame_queue(1, error);
    rs2_processing_block holeFilter = rs2_create_hole_filling_filter_block(error);
    rs2_set_option(holeFilter, RS2_OPTION_FILTER_OPTION, 5f, error);

...

        rs2_start_processing_queue(holeFilter, holeQueue, error);
        rs2_process_frame(holeFilter, frame, error);
        rs2_frame holes_frame = rs2_wait_for_frame(holeQueue, 5000, error);

The problem I have is from the line rs2_set_option(holeFilter, RS2_OPTION_FILTER_OPTION, 5f, error); where I get an error because the rs2_processing_block doesn't extend the rs2_options class.

[saudet]

Do you have a working code example in C?

[cansik]

@KennyBarraud I was working a bit more on porting the realsense C# library to Java, which uses the same C-warpper I used for the RealSense2FrameGrabber: cansik/librealsense-java

There I have implemented filters and options and so on. The ProcessingBlockTest show how to use them and even how to set options (colorizer option).

Please be aware, that a rs2_sensor and a rs2_options pointer is the same in C. So it is possible to just create a new pointer at the same address and use a sensor as an options pointer. I hope this is enough help to guide you.

rs2_sensor sensor = // from somewhere;
rs2_options options = new rs2_options(sensor);

Adding the processing pipeline to the library would be great, but atm I am working more on the realsense-processing port. Would be great if you could contribute something to the FrameGrabber for rs2.

Another solution would be to switch completely to my java lib and rewrite the wrapper with it (would be simple). But for that, we would need to add the java lib to the javacv dependencies. @saudet I am not sure if you are happy to add more dependencies?

[KennyBarraud]

@saudet I saw the C code in this page.

The question is that I don't know if the realsense library used in javacv allows us to do the same thing. As far as I know, it's seems unlikely with the actual version.

@cansik Atm I was using javacv RealSense2FrameGrabber and extending it so I can use some filters, but if I'm not misunderstanding the library you use is another library from the one used by javacv. Am I correct to assume that I don't need javacv to use your library ?

[cansik]

@KennyBarraud Maybe this answers of all your questions:

1. Yes, it is possible to use post-processing filter (as in your link) with the C library @saudet has ported for us.
2. Yes, RealSense2FrameGrabber is based on that C library port.
3. No, post-processing is not yet implemented into RealSense2FrameGrabber
4. My library librealsense-java is also based on the C library port.
5. Yes, it is possible to use the C library port, as well as librealsense-java without javacv.

Here a diagram to show the dependency hierarchy:

librealsense2 C library (by Intel)
├── librealsense2 javacpp port (by @saudet)
│   ├── RealSenseFrameGrabber2 (by @cansik, for javacv) 
│   ├── librealsense-java (by @cansik, same API structure as the C# API of rs2) 
│       └── realsense-processing (by @cansik, for processing) 

[KennyBarraud]

@cansik It's getting clearer.
I've read some of the classes you created in your librealsense-java and indeed you're using librealsense2 javacpp portage from the C library. I thought it was something else.

Thank you for your explanations !

[KennyBarraud]

Hello again !
I have another problem, and I need some explanations.
I get the next exception :
Exception in thread "main" org.bytedeco.javacv.FrameGrabber$Exception: rs_error was raised when calling rs2_wait_for_frame(queue:0000000040F41140): Frame did not arrive in time!

And there is some of my code called in my while loop which is based on the RealSenseFrameGrabber2.

frameset = rs2_pipeline_wait_for_frames(pipeline, RS2_DEFAULT_TIMEOUT, error);
checkError(error);
rs2_frame frame = findFrameByStreamType(frameset, 1, 0);
if (frame == null)
    return null;
rs2_process_frame(holeFilter, frame, error);
checkError(error);
rs2_frame holesFrame = rs2_wait_for_frame(holeQueue, 5000, error);
checkError(error);

Have I done something wrong ? Do I need to start a thread in which I will call the rs2_wait_for_frame
function ?

[KennyBarraud]

Hi again !
I've noticed that I forgot to call the rs2_frame_add_ref function. But I've got another error :
Exception in thread "main" org.bytedeco.javacv.FrameGrabber$Exception: rs_error was raised when calling rs2_pipeline_wait_for_frames(pipe:0000000040F9BCA0): Frame didn't arrived within 15000

I've written my own RealSense2FrameGrabber "MyRealSense2FrameGrabber" for the tests and to be sure the errors came from my filter :

    public void createHoleFillingFilter() throws Exception {
        holeQueue = rs2_create_frame_queue(1, error);
        checkError(error);
        holeFilter = rs2_create_hole_filling_filter_block(error);
        checkError(error);
        rs2_start_processing_queue(holeFilter, holeQueue, error);
        checkError(error);
    }

    public void setHoleFillingFilterOption(float factor) throws Exception {
        rs2_sensor sensor = new rs2_sensor(holeFilter);
        rs2_options holeOption = new rs2_options(sensor);
        rs2_set_option(holeOption, RS2_OPTION_HOLES_FILL, factor, error);
        checkError(error);
    }

    private Frame grabCVFrame(int streamType, int streamIndex, int iplDepth, int iplChannels) throws Exception {
        Frame outputFrame;

        // get frame of type if available
        rs2_frame frame = findFrameByStreamType(this.frameset, streamType, streamIndex);
        if (frame == null)
            return null;

        Pointer frameData;
        Size size;

        if(streamType == RS2_STREAM_DEPTH && holeFilter != null) {
            rs2_frame_add_ref(frame, error);
            checkError(error);
            rs2_process_frame(holeFilter, frame, error);
            checkError(error);
            rs2_frame holesFrame = rs2_wait_for_frame(holeQueue, 5000, error);
            checkError(error);
            // get frame data
            if(holesFrame != null) {
                frameData = getFrameData(holesFrame);
                size = getFrameSize(holesFrame);
            } else {
                return null;
            }
        } else {
            // get frame data
            frameData = getFrameData(frame);
            size = getFrameSize(frame);
        }

        // create cv frame
        IplImage image = IplImage.createHeader(size.width(), size.height(), iplDepth, iplChannels);
        cvSetData(image, frameData, size.width() * iplChannels * iplDepth / 8);
        outputFrame = converter.convert(image);

        // add timestamp
        double timestamp = getFrameTimeStamp(frame);
        outputFrame.timestamp = Math.round(timestamp);

        // cleanup
        rs2_release_frame(frame);

        return outputFrame;
    }

And I'm also using the RealSense2DepthMeasuring demo available in the samples.
I can't see what I am doing wrong since I'm doing approximately what @cansik in it's librealsense-java. What should I do ?

[KennyBarraud]

Ok so I forgot to release the modified frame. That was my error.