Javacpp-presets: GPU engine in DNNL disabled by default

[blueberry]

javacpp ships DNNL with the GPU engine disabled. This is the default build in upstream DNNL:

GPU Engine
Intel Processor Graphics and Xe architecture-based Graphics are supported by the oneDNN GPU engine. The GPU engine is > disabled in the default build configuration. The following additional requirements apply when GPU engine is enabled:

OpenCL* runtime library (OpenCL version 1.2 or later)
OpenCL driver (with kernel language support for OpenCL C 2.0 or later) with Intel subgroups extension support

What is required to enable it in javacpp (by default)? Does the absence of javacpp preset for OpenCL block it, or it it not required, but you simply chose to follow the DNNL default?

[saudet]

No idea, OpenCV builds and runs fine with OpenCL enabled. Would you mind inquiring upstream?

[blueberry]

@saudet Thanks! I've opened the issue upstream. https://github.com/oneapi-src/oneDNN/issues/823

[saudet]

It looks like DNNL doesn't link with libOpenCL.so.1, etc dynamically, so we'll need to create a -gpu extension for this.
Probably the easiest one to use as a template for this are the presets for ONNX Runtime:
https://github.com/bytedeco/javacpp-presets/tree/master/onnxruntime
If you could try to patch this up together and send a pull request, it would help!

[blueberry]

Yes, I'll investigate this. If I understand correctly, that extension would give the user a choice to either use javacpp's dnnl default, or the one where both cpu and gpu are enabled (*-gpu extension)? That would be great, since both the "safe and stable" and "all features on" are available.

[saudet]

Yes, that's correct, and it can fall back automatically on the CPU-only binary when the GPU one fails to load.

BTW, it seems to support only Intel GPUs on Linux and Windows (Mac isn't supported), so I'm not sure if it's worth the trouble.

[blueberry]

If I understood correctly, based on the CPU use cases, the idea is not that these engines are exclusive, but that, similarly to OpenCL, you query the DNNL library for available engines, select the one you like (there can be even more than one CPU engine), and use it. So, the same binary should be able to load both CPU and GPU. But I might be wrong (but then the related DNNL API doesn't make much sense).

As GPU part is disabled by the default build, the query for engines will only return the CPU engine. As for whether it's worth the effort, I thought that all underlying specifics are handled by DNNL, so from the user's point of view, including GPU in the build would enable more options, which shouldn't hurt. Most Intel's GPUs would not be competitive even with CPUs, but, on the other hand, Intel seems to be stepping up their game, so increasingly better options might be available soon.

Anyway, this is not a priority, but would be nice to have at least for experimentation...

[blueberry]

Here's what the DNNL folks say:

When you build oneDNN with OpenCL runtime the library will be dependent on OpenCL Installable Client Driver (libOpenCL.so):

$ ldd libdnnl.so.1
linux-vdso.so.1 (0x00007ffe3a532000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fedfe459000)
libOpenCL.so.1 => /lib/x86_64-linux-gnu/libOpenCL.so.1 (0x00007fedfe24e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fedfe248000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fedfe067000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fedfdf18000)
libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fedfded6000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fedfdeb9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fedfdcc7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fee00732000)
So if ICD is not available in the system the application will fail. It does not matter whether Intel OpenCL Runtime is actually available or not, what matters is the loader. I believe ICD is available on most systems out of the box. Other than that CPU engine should work as expected.

In our case, can javacpp's DNNL ship with libopencl.so included? (I guess that would be it, but it's better to check before I try messing up with the build. In that case, do I have to configure that manually, or javacpp will figure it out and include it automatically?)

[saudet]

If I understood correctly, based on the CPU use cases, the idea is not that these engines are exclusive, but that, similarly to OpenCL, you query the DNNL library for available engines, select the one you like (there can be even more than one CPU engine), and use it. So, the same binary should be able to load both CPU and GPU. But I might be wrong (but then the related DNNL API doesn't make much sense).

The API doesn't look that easy to use to me. You're probably looking more for something like ONNX Runtime.

Anyway, this is not a priority, but would be nice to have at least for experimentation...

Sure, we probably just need to add the -DDNNL_GPU_RUNTIME=OCL build flag to experiment. Please give it a try!

In our case, can javacpp's DNNL ship with libopencl.so included? (I guess that would be it, but it's better to check before I try messing up with the build. In that case, do I have to configure that manually, or javacpp will figure it out and include it automatically?)

Like I said, if we create an -gpu extension, it can fall back on the CPU-only binary, so we don't need to try to bundle OpenCL. (Unless you do also want to create presets for OpenCL, and then that would make sense, yes.)

[blueberry]

The API doesn't look that easy to use to me. You're probably looking more for something like ONNX Runtime.

I already use DNNL and, though the API could be much more straightforward, I use it through my wrappers that simplify the use a lot, so I'm fine with it :)

Anyway, this is not a priority, but would be nice to have at least for experimentation...

Sure, we probably just need to add the -DDNNL_GPU_RUNTIME=OCL build flag to experiment. Please give it a try!

Will do!

[blueberry]

This is just a note that I haven't forgotten this. I was just overwhelmed with other pressing work. I plan to do this within the next two weeks.

[saudet]

I've enabled the OpenCL backend by default for Linux in the latest commit. Maybe we can reasonably assume that almost all users will have libOpenCL.so.1 somewhere on their systems. We'll see if anyone complains about this with the snapshots, and if not, maybe we'll be able to release it like that.

[blueberry]

Hi @saudet

Thank you!
I've took time today to work on this, and the first thing that I tried is, of course, to try the snapshots. Good thing is that the things that worked before still work. The bad part is that it seems that I can't try the new functionality because my Intel platform on i7-4790k does not support its meager integrated GPU. I have 3 OpenCL platforms on this computer (Intel, AMD, and Nvidia), and I expected that Intel's expose its GPU, but it doesn't.

I have two newer computers, but neither of them has an integrated GPU, since they are a combination of powerful Inetl CPU and a recent Nvidia GPU.

Related to the libOpenCL issue. Is this going to work as before if a user doesn't have libOpenCL? I've moved that file temporarily just to try this out, and it doesn't make any difference. The old code still works, and the library doesn't even try to load libOpenCL, although I've tried to list GPU engines in the appropriate DNNL api.

In that regard, it seems that the latest changes might be optimal:

1. for existing users, everything works well even if OpenCL is not present
2. if there is OpenCL and the appropriate hardware, the library should pick this up.

However, I haven't been able to test the 2nd point...

[saudet]

OpenCL doesn't need to be present, but a library named libOpenCL.so.1 still needs to be loaded from somewhere. You probably have a couple of those lying around on your system, and that's why it's still loading without error.

[blueberry]

Hmmm, I thought so, but then when I remove that libOpenCL.so, clinfo does break and complains that there's no libopencl available...

[saudet]

clinfo might be looking for libOpenCL.so, I'm not sure, but that's not the file that matters here.
The one that matters is libOpenCL.so.1, with a .1 after .so.

[blueberry]

I was writing from the top of my head. This is the exact file I removed: /usr/lib/libOpenCL.so.1.0.0

[blueberry]

... and both libOpenCL.so and libOpenCL.so.1 are soft links to the libOpenCL.so.1.0.0...

[saudet]

What does ldd $(which clinfo) return?

[blueberry]

linux-vdso.so.1 (0x00007fff1c345000)
    libOpenCL.so.1 => /usr/lib/libOpenCL.so.1 (0x00007f563a6bf000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f563a6b9000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f563a4f0000)
    librt.so.1 => /usr/lib/librt.so.1 (0x00007f563a4e5000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f563a39f000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f563a37d000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f563a924000)

[saudet]

And if you remove /usr/lib/libOpenCL.so.1, what's the output?

[blueberry]

clinfo: error while loading shared libraries: libOpenCL.so.1: cannot open shared object file: No such file or directory

[blueberry]

At the same time, my code using javacpp dnnl platform "1.6.2-1.5.5-SNAPSHOT" is loading without problems.

[blueberry]

I mean, that's not a bug, that's a wonderful feature! We just have to make sure there's not something else at play here. Because I don't think that the majority of users have opencl installed....

[saudet]

Try again with 1.6.3-1.5.5-SNAPSHOT to make sure it's not using some old binaries from somewhere...

[blueberry]

Still working without libOpenCL.so.1.

[saudet]

Nope, we need libOpenCL.so.1: https://groups.google.com/g/javacpp-project/c/3s29mosfyTI/m/3NimBckyAgAJ

[blueberry]

Hmmm. That seems quite logical. I'm puzzled how it works on my machine when I remove libOpenCL.so (and .1 etc.)...

[saudet]

I've added presets for OpenCL, and made DNNL depend on them, so it can use the OpenCL libraries from them. They don't need to be installed on the system anymore. OpenCL is now enabled for both Linux and Windows builds. Let me know if you're having any issues with that though! Thanks

[blueberry]

Thanks! I'll check it out. Additionally, it might be a good opportunity to try to support the new OpenCL bindings in my libraries. I see that you support OpenCL 3.0. Are they compatible with existing OpenCL 2.0? I remember that when they announced OpenCL 3.0 they mentioned that it's based on 1.2, but 2.0 is supported for existing implementations, but I was confused what that actually meant...

[saudet]

Right, OpenCL 3.0 is basically OpenCL 1.2 with everything from 2.0 as optional extensions. That way, they are making NVIDIA GPUs compliant with the latest version of OpenCL :)

[blueberry]

Thanks. I meant: would I be able to port existing OpenCL 2.0 code to bytedeco's OpenCL 3.0? Are the functions that I'd expect in OpenCL 2.0 available somewhere in bytedeco's bindings? (I'm not sure what being an extension means regarding a Java JNI binding).

[saudet]

Everything from OpenCL 2.x is available, yes, it will just error out if you try to call those functions on a GPU from NVIDIA:
https://github.com/bytedeco/javacpp-presets/blob/master/opencl/src/gen/java/org/bytedeco/opencl/global/OpenCL.java

[blueberry]

That's great news. I'll definitely check this out and see how to integrate it in my libraries. Will cry for help if I stumble on a problem, I hope you wouldn't mind :)

[saudet]

OpenCL-enabled builds of DNNL 2.1.1 have now been released with version 1.5.5. Enjoy!