Onnxruntime: Switch between providers

[fumihwh]

Is there any way to run op in specific provider? or does priority of provider exist?

[snnn]

Which API you are using? C/C++/Python?

[fumihwh]

@snnn Python.

[linkerzhang]

@fumihwh ONNXRUNTIME provides APIs to set execution provider in C/C++. Python does not have. Should be added. Will keep this open to track this task.

[faxu]

@fumihwh : can you share some more details on your usage scenario? The python packages that are released are only built for single execution providers right now.

[jinfagang]

Does specific providers available now in onnxruntime python?

Otherwise we have to installed 2 different environments in order to switch between CPU and GPU as backend.

[pranavsharma]

This commit attempts to fix this.

[jinfagang]

@pranavsharma this PR is good, will it merge?

[pranavsharma]

This was merged 4 days back.

[leimao]

Is there any documentation on how to switch between CPU and GPU runtime?

[ShaiBet]

Is there a support from python to use GPU runtime at all?

[vladbph]

@fumihwh ONNXRUNTIME provides APIs to set execution provider in C/C++. Python does not have. Should be added. Will keep this open to track this task.

The issue is closed - does this mean API already supports switching between providers CPU and GPU?. Does python API support it?

[bminixhofer]

Yes, the Python API supports it:

https://github.com/microsoft/onnxruntime/blob/78a29aebbcbd0c3b6dab734f221e0f3bf1e24c97/onnxruntime/python/session.py#L49-L86

I just tried it. Before installing onnxruntime-gpu:

In [1]: session.get_providers()                                                                                                                      
Out[1]: ['CPUExecutionProvider']

afterwards:

In [1]: session.get_providers()                                                                                                                      
Out[1]: ['CUDAExecutionProvider', 'CPUExecutionProvider']

"CUDAExecutionProvider" being before CPU in the array means that onnxruntime will execute nodes with CUDA if capable and fallback to CPU if not. You can change the priority with session.set_providers.

[vladbph]

Great, thank you!

[vladbph]

session.get_providers()   

Well, the thing is the fallback to CPU must(?) happen IF GPU is busy or not available. But it is not.. It throws an exception.
See the link: https://colab.research.google.com/drive/11ISt8zaGOKaLWCdsRcapcM9Cme109Zwq?usp=sharing

[bminixhofer]

Good point, that should probably be a separate issue.