Tfjs: Add the ability to specify gpu_options

[vmarkovtsev]

TensorFlow.js version

0.12.6

Browser version

Node.js 10.9

Describe the problem or feature request

I am using tfjs-node-gpu and it allocates the whole memory of my GPU (the default TF behavior).
I request to have an ability to specify the backend options, particularly gpu_options.allow_growth.

[johnnyvf24]

I have also run into this limitation and would love a solution (even if it involves messing with the bindings).

[rthadur]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

[vmarkovtsev]

@rthadur What?

[rthadur]

@vmarkovtsev feel free to reopen if this is still an issue .

[vmarkovtsev]

@rthadur I cannot reopen. If a maintainer closes an issue, its original author does not have the permission. It has always worked this way for all repos on GitHub.

Anyway, I really like how you deal with the issues :+1:

[iacovlev-pavel]

+1, having access to gpu_options would be great.

[pavelkukov]

+1 for gpu_options, gpu_memory_fraction will be very useful when training smaller models

[single9]

+1 for gpu_options, gpu_memory_fraction

[cjue]

There appears to be a workaround for setting allow_growth through an environment variable. This works for my tfjs-node-gpu project:

#Limit GPU memory usage with an environment variable
export TF_FORCE_GPU_ALLOW_GROWTH=true

[3lk0k0]

There appears to be a workaround for setting allow_growth through an environment variable. This works for my tfjs-node-gpu project:

#Limit GPU memory usage with an environment variable
export TF_FORCE_GPU_ALLOW_GROWTH=true

where need to write these command ?

[Infinitay]

There appears to be a workaround for setting allow_growth through an environment variable. This works for my tfjs-node-gpu project:

#Limit GPU memory usage with an environment variable
export TF_FORCE_GPU_ALLOW_GROWTH=true

where need to write these command ?

This is how I did it:

1. Install the package dotenv via npm i dotenv
2. Create a new file .env, make sure you include the ., within the root of your project's workspace
3. Add TF_FORCE_GPU_ALLOW_GROWTH=true to the file and save it
4. Add require('dotenv').config(); in your code, somewhere where it can be executed _first_, and before requiring tfjs of course

[Infinitay]

I tried adding TF_FORCE_GPU_ALLOW_GROWTH=true as an environment variable, and it seemed to have worked briefly, but upon trying to run my program once more, the error started appearing again.

cc. https://github.com/tensorflow/tfjs/issues/1469#issuecomment-586091169

[emrebarutcu]

I still get the error "Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED" after I set "TF_FORCE_GPU_ALLOW_GROWTH=true". And still, there is no option to specify gpu_options.