Turicreate: Add runtime message to show which GPUs are being used

[Walter0807]

I'm running the object detection task.

MacOS Version 10.14 Beta (18A293u)

MacBook Pro (Retina, 13-inch, Early 2015)
Processor 2.7 GHz Intel Core i5 Memory 8 GB 1867 MHz DDR3
Startup Disk Macintosh HD
Graphics Intel Iris Graphics 6100 1536 MB

[srikris]

Are you using Turi-create 5.0 beta?

[Walter0807]

@srikris I've installed turicreate==5.0b1. Should I do something other than that?

[znation]

@gustavla Is OD supposed to be GPU accelerated on an Intel Iris Graphics 6100? (Perhaps this was tested, and didn't give a significant speedup so it was disabled?)

[gustavla]

@Walter0807 Fast GPU acceleration on Mac is only available if it has an AMD GPU (or on older Macs with NVIDIA GPU), but unfortunately not if you only have Intel integrated graphics. Technically, the Metal Performance Shader backend runs on it, but it is not powerful enough to improve over CPU training speeds (so we automatically use that instead).

Let's think about how we can make this clear in either documentation or verbose printing.

[znation]

Retagging as docs so we can update the documentation to answer this question.

[mdenty]

While performing the training, turicreate should display in the log which device is used and eventually the reason of this choice (no mxnet-cu detected, no suitable AMD card found, not implemented yet...) if the training is done on the CPU.

[b-straub]

How could you actually test if the build in GPU doesn't speed up anything? When forcing metal usage for object dectection by overwriting "HasHighPowerMetalDevice" I get the following internal metal error:

Using 'image' as feature column
Using 'annotations' as annotations column
Using GPU to create model (Intel Iris Pro Graphics)
+--------------+--------------+--------------+
| Iteration | Loss | Elapsed Time |
+--------------+--------------+--------------+
/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MetalPerformanceShaders-114.8/MPSCore/Utility/MPSLibrary.mm:210: failed assertion `MPSKernel MTLComputePipelineStateCache unable to load function cnnConv_Update_32x32.
Compiler encountered an internal error: (null)
'
Abort trap: 6

[srikris]

@mdenty Great suggestion. That's something we should incorporate.

[gustavla]

@b-straub You are right, we do not know this for certain on all platforms. On the ones we have tested, it was slower, but we would love to hear if this is not consistently true, in which case we could revisit the decision mechanism.

I have only seen this be slower though and it has never produced an error like the one you are showing. However, there are two tests inside the HasHighPowerMetalDevice functino: 1. Does the device support Metal? 2. Is it a high-power device? So, if you bypass this check and it turns out that the device wasn't just low-power, but it also did not support Metal, failure is expected.

Can you double-check that the device does support Metal? In that's the case, I'll make sure the right bug report is filed to look into this. Thanks for the investigation!

[b-straub]

@gustavla Yes the Intel Iris Pro Graphics is supporting Metal just fine. Anyway, I commented the second part && (!dev.isLowPower) means the device initialization remained untouched.

[b-straub]

Should be tagged with "object detection" instead of "image classifier"

[znation]

Thanks @b-straub, I'll update the tag.

[gustavla]

I have forward the Intel Iris Pro Graphics error message to the Metal Performance Shaders developers. As for this error message on the Turi Create side, we don't consider this compute device to be part of what we officially support and the error message does not occur without modifications, so in that regard I think this issue is closed. @b-straub Thanks for the bug report and we do appreciate the scrutiny!

However, we also decided to be more explicit about the compute device decision, so I will leave this issue open until that has been addressed.

[afranklin]

Todo: Add runtime message to show which (if any) GPUs are being used and why.

[znation]

Dupe of #137.

[cianiandreadev]

@gustavla can you please confirm that still integrated graphics are not used? Referring to TC version 6.0.