Addons: Make the tests faster

[gabrieldemarmiesse]

Describe the bug

Currently, the tests take around 600s. But only a few tests take the majority of this time. The short term solution is to make those specific tests shorter, the long term solution is to put a specific solution in place to monitor the changes in test time and have automatic report of new long tests/tests which became much longer after a change.

Here are the current number from the linux build in the master branch:

//tensorflow_addons/activations:activations_test                         PASSED in 2.6s
//tensorflow_addons/activations:gelu_test                                PASSED in 2.4s
//tensorflow_addons/activations:hardshrink_test                          PASSED in 3.9s
//tensorflow_addons/activations:lisht_test                               PASSED in 2.6s
//tensorflow_addons/activations:mish_test                                PASSED in 2.7s
//tensorflow_addons/activations:rrelu_test                               PASSED in 2.2s
//tensorflow_addons/activations:softshrink_test                          PASSED in 4.1s
//tensorflow_addons/activations:sparsemax_test                           PASSED in 11.2s
//tensorflow_addons/activations:tanhshrink_test                          PASSED in 2.4s
//tensorflow_addons/image:connected_components_test                      PASSED in 5.3s
//tensorflow_addons/image:dense_image_warp_test                          PASSED in 40.9s
//tensorflow_addons/image:distance_transform_ops_test                    PASSED in 2.8s
//tensorflow_addons/image:distort_image_ops_test                         PASSED in 4.4s
//tensorflow_addons/image:filters_test                                   PASSED in 56.8s
//tensorflow_addons/image:interpolate_spline_test                        PASSED in 6.8s
//tensorflow_addons/image:resampler_ops_test                             PASSED in 9.7s
//tensorflow_addons/image:sparse_image_warp_test                         PASSED in 14.2s
//tensorflow_addons/image:transform_ops_test                             PASSED in 66.1s
//tensorflow_addons/image:translate_ops_test                             PASSED in 5.5s
//tensorflow_addons/image:utils_test                                     PASSED in 4.2s
//tensorflow_addons/layers:gelu_test                                     PASSED in 4.2s
//tensorflow_addons/layers:layers_wrappers_test                          PASSED in 37.3s
//tensorflow_addons/layers:maxout_test                                   PASSED in 6.0s
//tensorflow_addons/layers:normalizations_test                           PASSED in 24.0s
//tensorflow_addons/layers:optical_flow_test                             PASSED in 27.0s
//tensorflow_addons/layers:polynomial_test                               PASSED in 2.4s
//tensorflow_addons/layers:sparsemax_test                                PASSED in 4.7s
//tensorflow_addons/layers:tlu_test                                      PASSED in 8.3s
//tensorflow_addons/losses:contrastive_test                              PASSED in 2.9s
//tensorflow_addons/losses:focal_loss_test                               PASSED in 3.1s
//tensorflow_addons/losses:giou_loss_test                                PASSED in 4.6s
//tensorflow_addons/losses:lifted_test                                   PASSED in 3.5s
//tensorflow_addons/losses:metric_test                                   PASSED in 3.0s
//tensorflow_addons/losses:npairs_test                                   PASSED in 3.1s
//tensorflow_addons/losses:quantiles_test                                PASSED in 2.3s
//tensorflow_addons/losses:sparsemax_loss_test                           PASSED in 7.3s
//tensorflow_addons/losses:triplet_test                                  PASSED in 5.7s
//tensorflow_addons/metrics:cohens_kappa_test                            PASSED in 16.2s
//tensorflow_addons/metrics:f_test                                       PASSED in 11.6s
//tensorflow_addons/metrics:hamming_test                                 PASSED in 2.3s
//tensorflow_addons/metrics:matthews_correlation_coefficient_test        PASSED in 4.4s
//tensorflow_addons/metrics:multilabel_confusion_matrix_test             PASSED in 2.2s
//tensorflow_addons/metrics:r_square_test                                PASSED in 3.3s
//tensorflow_addons/optimizers:conditional_gradient_test                 PASSED in 6.1s
//tensorflow_addons/optimizers:cyclical_learning_rate_test               PASSED in 2.7s
//tensorflow_addons/optimizers:lamb_test                                 PASSED in 10.5s
//tensorflow_addons/optimizers:lazy_adam_test                            PASSED in 5.7s
//tensorflow_addons/optimizers:lookahead_test                            PASSED in 40.1s
//tensorflow_addons/optimizers:moving_average_test                       PASSED in 42.3s
//tensorflow_addons/optimizers:novograd_test                             PASSED in 81.8s
//tensorflow_addons/optimizers:rectified_adam_test                       PASSED in 124.2s
//tensorflow_addons/optimizers:stochastic_weight_averaging_test          PASSED in 82.2s
//tensorflow_addons/optimizers:weight_decay_optimizers_test              PASSED in 8.1s
//tensorflow_addons/optimizers:yogi_test                                 PASSED in 13.0s
//tensorflow_addons/rnn:cell_test                                        PASSED in 8.1s
//tensorflow_addons/seq2seq:attention_wrapper_test                       PASSED in 74.2s
//tensorflow_addons/seq2seq:basic_decoder_test                           PASSED in 3.1s
//tensorflow_addons/seq2seq:beam_search_decoder_test                     PASSED in 13.9s
//tensorflow_addons/seq2seq:beam_search_ops_test                         PASSED in 2.5s
//tensorflow_addons/seq2seq:decoder_test                                 PASSED in 2.8s
//tensorflow_addons/seq2seq:loss_test                                    PASSED in 5.1s
//tensorflow_addons/text:crf_test                                        PASSED in 23.3s
//tensorflow_addons/text:parse_time_op_test                              PASSED in 3.1s
//tensorflow_addons/text:skip_gram_ops_test                              PASSED in 2.8s

As you can see some tests are quite long. If we add more and more feature with the time and we don't monitor tests duration, we'll end up with a build which takes hours to run.

In short:

• In the short term, make the tests faster, help is welcome, it's easy to do. Each test often have one task to do, everything done outside the scope of this task can be removed.

• Long term: We need a reporting on what changed in the tests duration (likely a bot saying on the pull request if there are changes in term of running time).

[Squadrick]

Most of the offenders tend to be from optimizers since we run a training loop to check for convergence. The fix in #1161 is fairly straightforward to extend to other optimizer tests.

[Squadrick]

List of the slower tests (>20s):

• [x] //tensorflow_addons/seq2seq:attention_wrapper_test: 74.2s
• [x] //tensorflow_addons/optimizers:rectified_adam_test: 124.2s
• [x] //tensorflow_addons/optimizers:stochastic_weight_averaging_test: 82.2s
• [x] //tensorflow_addons/optimizers:novograd_test: 81.8s
• [x] //tensorflow_addons/optimizers:lookahead_test: 40.1s
• [x] //tensorflow_addons/optimizers:moving_average_test: 42.3s
• [x] //tensorflow_addons/image:transform_ops_test: 66.1s
• [x] //tensorflow_addons/image:dense_image_warp_test: 40.9s
• [x] //tensorflow_addons/layers:layers_wrappers_test: 37.3s
• [ ] //tensorflow_addons/layers:normalizations_test: 24.0s
• [ ] //tensorflow_addons/layers:optical_flow_test: 27.0s
• [ ] //tensorflow_addons/text:crf_test: 23.3s
• [x] //tensorflow_addons/image:filters_test: 56.8s

[Squadrick]

filters_test can't be made faster, it runs two massive test suites where each function takes ~1s to run, but it stacks up due to the sheer number of them. Making them faster would require some effort.

[autoih]

I replace divide op with constant for filters_test, it improves 15s.

[seanpmorgan]

Can this be closed as the highest test times have been reduced? For the long term fix of reporting high test times in PRs perhaps we create a new issue?

[gabrieldemarmiesse]

Yes, let me open another issue concerning the monitoring of test times before closing this one. I'll do that after the pytest PR is merged.

[gabrieldemarmiesse]

A small note, all our tests are decorated with tf.function (because of the run_all_in_graph_and_eager_mode) and tracing is very expensive in our test suite.

By making the graphs of only the functions being tested we can save a lot. I believe that #1288 will make our tests much faster .I did some early speed tests and some of our most time consuming tests are 10x faster in eager mode, which proves that we need to trace only what's needed.

[gabrieldemarmiesse]

Closing as our tests are already very fast and #1288 will make them even faster. Moving to #1316 for the long term monitoring.