Keras: Transfer of a new Python optimizer to Keras in R

[faltinl]

Important progress has been made in developing adaptive optimizers which are able to overcome the hurdle of oscillating between local optima, causing non-convergence problems and poor generalization performance. The challenge is beautifully illustrated by pictures devoted to a collection of optimizers for Pytorch, designed to overcome these problems.
I am particularly interested in the optimizer adamod which has been investigated thoroughly elsewhere. It is easy to install and import this Python optimizer into the virtual environment established by Keras for R applying the usual simple process to transfer Python functions into this environment. In Python power shell,

conda activate r-reticulate
pip install adamod

installs adamod 0.0.3 into the virtual environment r-reticulate. An then, in Keras for R, with

library reticulate
optimizer_adamod <- import('adamod')

the import into R is readily done. The usual compile() function of Keras, however, does not recognize the newly defined optimizer_adamod() and its parameters. And this raises the question: How can I transform this optimizer into a form compatible with Keras for R?
Any hints would be highly appreciated!

[henry090]

Hi. I think for now you could experiment with optimizers from TF Addons.
https://github.com/henry090/tfaddons

[faltinl]

Hi,-
thanks a lot, @henry090, for this hint to TF-Addons, which is really good news for me and other R fans. There are many quite interesting opportunities for further work with Keras for R & TF. Hopefully AdaMod can soon be found there in an R-version, too!
Cheers,
Leo

[faltinl]

Hi,-
I have installed package tfaddons successfully from Github, including the required updates of all dependencies (keras 2.3.0.0, , tensorflow 2.2.0, reticulate 1.16, etc.). The package shows up in my RStudio package list as it should do. However, library(tfaddons) produces an error message as follows:

Error: package or namespace load failed for ‘tfaddons’:
 .onLoad in loadNamespace() for 'tfaddons' failed, details:
  call: py_module_import(module, convert = convert)
  Error: ModuleNotFoundError: No module named 'tensorflow_addons'`

I suspect I have to install tfaddons::install_tfaddons(), beforehand - but how do I do that? After attaching library(reticulate), the input tfaddons::install_tfaddons() produced an error message of almost identical content as the one shown above.

Please advise how I should proceed. Tnx in advance!

[henry090]

The best solution is the following from fresh Session:

reticulate::py_config()
tensorflow::install_tensorflow(version = '2.2', extra_packages = 'tensorflow-addons')

[faltinl]

Thanks for Support. I got the following returns:

reticulate::py_config()
python:         C:/Miniconda3/envs/r-reticulate/python.exe
libpython:      C:/Miniconda3/envs/r-reticulate/python36.dll
pythonhome:     C:/Miniconda3/envs/r-reticulate
version:        3.6.10 |Anaconda, Inc.| (default, May  7 2020, 19:46:08) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Miniconda3/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.18.4
tensorflow:     C:\MINICO~1\envs\R-RETI~1\lib\site-packages\tensorflow\__init__.p
python versions found: 
 C:/Miniconda3/envs/r-reticulate/python.exe
 C:/Miniconda3/python.exe

Then, installation started quite smoothly...

tensorflow::install_tensorflow(version = '2.2', extra_packages = 'tensorflow-addons')
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
# All requested packages already installed.
Collecting tensorflow==2.2
  Downloading tensorflow-2.2.0-cp36-cp36m-win_amd64.whl (459.1 MB)
Collecting tensorflow-addons
  Downloading tensorflow_addons-0.10.0-cp36-cp36m-win_amd64.whl (890 kB)
Requirement already satisfied
...

...but towards the end some problems appeared:

...
Requirement already satisfied, skipping upgrade: oauthlib>=3.0.0 in c:\minico~1\envs\r-reti~1\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2) (3.1.0)
ERROR: tensorflow-gpu 2.0.0 has requirement gast==0.2.2, but you'll have gast 0.3.3 which is incompatible.
ERROR: tensorflow-gpu 2.0.0 has requirement tensorboard<2.1.0,>=2.0.0, but you'll have tensorboard 2.2.2 which is incompatible.
ERROR: tensorflow-gpu 2.0.0 has requirement tensorflow-estimator<2.1.0,>=2.0.0, but you'll have tensorflow-estimator 2.2.0 which is incompatible.
Installing collected packages: gast, astunparse, tensorboard-plugin-wit, tensorboard, tensorflow-estimator, tensorflow, typeguard, tensorflow-addons
  Attempting uninstall: gast
    Found existing installation: gast 0.2.2
    Uninstalling gast-0.2.2:
      Successfully uninstalled gast-0.2.2
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.0.2
    Uninstalling tensorboard-2.0.2:
      Successfully uninstalled tensorboard-2.0.2
  WARNING: The script tensorboard.exe is installed in 'C:\MINICO~1\envs\R-RETI~1\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.0.1
    Uninstalling tensorflow-estimator-2.0.1:
      Successfully uninstalled tensorflow-estimator-2.0.1
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.0.0
    Uninstalling tensorflow-2.0.0:
      Successfully uninstalled tensorflow-2.0.0
  WARNING: The scripts estimator_ckpt_converter.exe, saved_model_cli.exe, tensorboard.exe, tf_upgrade_v2.exe, tflite_convert.exe, toco.exe and toco_from_protos.exe are installed in 'C:\MINICO~1\envs\R-RETI~1\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Zugriff verweigert: 'c:\\minico~1\\envs\\r-reti~1\\lib\\site-packages\\tensorflow_core\\~ite\\experimental\\microfrontend\\python\\ops\\_audio_microfrontend_op.so'
Consider using the `--user` option or check the permissions.
Error: Error installing package(s): "tensorflow==2.2", "tensorflow-addons"  

Any ideas what I could do? Namely - the worst of all is that the GPU part of Tensorflow seems to be damaged somehow, since when trying to run one of my previously functioning network programs I get a long error log ending in

...
2020-06-02 19:08:21.405164: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-06-02 19:08:21.487822: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-06-02 19:08:23.242942: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-06-02 19:08:23.244791: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
 Error in py_call_impl(callable, dots$args, dots$keywords) : 
  UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[node model/conv3d/Conv3D (defined at \keras\engine\training.py:88) ]] [Op:__inference_predict_function_1415]

Function call stack:
predict_function 

OMG.

[henry090]

Sometimes, the installation of python modules can cause an issue for me as well. In this case, I remove the environment (folder) from the path and then run reticulate::py_config().
Then I start installing python modules, again. So, you could delete here:
C:/Miniconda3/envs/r-reticulate/python.exe the environment r-reticulate. Then reticulate::py_config().
For GPU try this from fresh session:

tensorflow::install_tensorflow(version = '2.2.0-gpu', extra_packages = 'tensorflow-addons')

[faltinl]

After having removed the environment r-reticulate I find

reticulate::py_config()
python:         C:/Miniconda3/python.exe
libpython:      C:/Miniconda3/python37.dll
pythonhome:     C:/Miniconda3
version:        3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Miniconda3/Lib/site-packages/numpy
numpy_version:  1.18.4

and the remaining environment Miniconda3

conda_list()
        name                     python
1 Miniconda3 C:\\Miniconda3\\python.exe

Do I install tensorflow into this environment? I rather thought to install it via RStudio in R, but when I tried that, I got

tensorflow::install_tensorflow(version = '2.2.0-gpu', extra_packages = 'tensorflow-addons')
Error: .onLoad in loadNamespace() for 'tensorflow' failed, details:
  Call: py_module_import(module, convert = convert)
  Error: ModuleNotFoundError: No module named 'tensorflow'

which is not unexpected, since the existing package tensorflow has been removed when I deleted the environment r-reticulate. (I am sorry, but I have no idea, how I managed to install TF2.0 the first time; Miniconda3 was somehow installed together with TF2.0 or so - I'm completely lost at the moment).

[henry090]

Yes, you have to create an environment and then start installing. With reticulate you can install :

reticulate::py_install(c('tensorflow-gpu==2.2','tensorflow-addons'), pip = TRUE)

[faltinl]

Sorry to molest you again and again...
I did the following:

> reticulate::py_config()
python:         C:/Miniconda3/python.exe
libpython:      C:/Miniconda3/python37.dll
pythonhome:     C:/Miniconda3
version:        3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Miniconda3/Lib/site-packages/numpy
numpy_version:  1.18.4
> library(reticulate)
Warning:
Package ‘reticulate’ has been created under R Version 3.6.3 **Note: I have v3.6.2** 
> conda_list()
        name                     python
1 Miniconda3 C:\\Miniconda3\\python.exe
> use_condaenv("Miniconda3")  
> use_python("C:\\Miniconda3\\python.exe")
> reticulate::py_install(c('tensorflow-gpu==2.2','tensorflow-addons'), pip = TRUE)
**Error: Installing Python packages into a virtualenv is not supported on Windows**

What I have also tried was

> install.packages("tensorflow")
try URL 'https://cran.rstudio.com/bin/windows/contrib/3.6/tensorflow_2.2.0.zip'
Content type 'application/zip' length 145138 bytes (141 KB)
downloaded 141 KB
package ‘tensorflow’ successfully unpacked and MD5 sums checked
The downloaded binary packages are in
    C:\Users\lf\AppData\Local\Temp\Rtmp6963xb\downloaded_packages
> library(tensorflow) 
**Error: package or namespace load failed for ‘tensorflow’**:
 .onLoad in loadNamespace() für 'tensorflow' failed, details:
  Call: py_module_import(module, convert = convert)
  **Error: ModuleNotFoundError: No module named 'tensorflow'**

This was the process I applied when I installed TF2.0 in February 2020 the first time. It should have been followed by install_tensorflow(version = "gpu") etc. which then worked without any problem. Today I don't even succeed to attach the package. I don't understand the present problems at all.

[henry090]

Does it work? https://rstudio.github.io/reticulate/articles/python_packages.html

library(reticulate)

# create a new environment 
virtualenv_create("r-reticulate")

# install TF
virtualenv_install("r-reticulate", "tensorflow-gpu==2.2")

[faltinl]

Thank you so much. In the meantime I did more or less that, except that for compatibility with Windows I wrote

conda_create("r-reticulate")       
use_condaenv("r-reticulate")   
conda_install("r-reticulate",c('tensorflow-gpu==2.2','tensorflow-addons'), pip = TRUE)

After quite a lot of problems I finally managed to get my keras & tensorflow installation working again, including GPU. (What a relief...)

Regarding tfaddons, I got the package to show up in my RStudio package list again and with

> reticulate::py_install(c('tensorflow-addons'), pip = TRUE)
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
# All requested packages already installed.
Collecting tensorflow-addons
  Using cached tensorflow_addons-0.10.0-cp36-cp36m-win_amd64.whl (890 kB)
Collecting typeguard>=2.7
  Using cached typeguard-2.8.0-py3-none-any.whl (15 kB)
Installing collected packages: typeguard, tensorflow-addons
Successfully installed tensorflow-addons-0.10.0 typeguard-2.8.0

I finally got the message

> library(tfaddons)
2020-06-03 21:31:14.916468: I tensorflow/stream_executor/
platform/default/dso_loader.cc:44] 
Successfully opened dynamic library cudart64_101.dll

I guess this means that tfaddons is ready to use. I think I'll first try one of the new optimizers to find out whether it's really working or not...

By the way - is tfaddons only applicable to keras sequential models or to keras functional APIs as well? I am using the latter more frequently, that is.

[henry090]

That's great news! Yes, they should work with Functional API, as well.

[faltinl]

Oh yeah, good to know.

Just one more question regarding the installation process: Towards the end of the tensorflow installation there are lots of confirmations regarding successfuly installed packages. Among them, however, are also occasional warnings, e.g. like this one:

WARNING: The script tensorboard.exe is installed in 'C:\MINICO~1\envs\R-RETI~1\Scripts' 
which is not on PATH.
Consider adding this directory to PATH , if you prefer to suppress this warning, 
use --no-warn-script-location.

I know, this is not directly related to our main topic tfaddons, but since you obviously are so experienced with these practical questions: how would I "add this directory to PATH"? Do I need that? Do I just move the folder of that particular script? And if so, where - and which one exactly is the PATH in that case? Would be nice I you could provide some hints.

In any case, I am very grateful for your support - I really appreciate it. All the best to you and the project tfaddons!

[henry090]

Please, follow the steps here: https://helpdeskgeek.com/windows-10/add-windows-path-environment-variable/
In your case, you have to add this dir C:\MINICO~1\envs\R-RETI~1\Scripts to path. (Control Panel + advanced system settings + environment variable + new + C:\MINICO~1\envs\R-RETI~1\Scripts + OK + end. Then you can restart OS and try, again. (to check warning)

[faltinl]

Ok, I did as advised and it's working - thank you again!