Recommenders: [BUG] DKN produces F1 score=0

[miguelgfierro]

Description

In some of the smoke tests (not always), DNK produces F1=0. These are the logs:

tests/smoke/test_deeprec_model.py ..                                     [ 28%]
tests/smoke/test_notebooks_gpu.py ....F                                  [100%]

=================================== FAILURES ===================================
______________________________ test_notebook_dkn _______________________________

notebooks = {'als_deep_dive': '/data/home/recocat/cicd/18/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/re...aseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/18/s/notebooks/01_prepare_data/data_split.ipynb', ...}

    @pytest.mark.smoke
    @pytest.mark.gpu
    @pytest.mark.deeprec
    def test_notebook_dkn(notebooks):
        notebook_path = notebooks["dkn_quickstart"]
        pm.execute_notebook(
            notebook_path,
            OUTPUT_NOTEBOOK,
            kernel_name=KERNEL_NAME,
            parameters=dict(epoch=1),
        )
        results = pm.read_notebook(OUTPUT_NOTEBOOK).dataframe.set_index("name")["value"]

        assert results["res"]["auc"] == pytest.approx(0.4707, TOL2)
        assert results["res"]["acc"] == pytest.approx(0.5725, TOL2)
>       assert results["res"]["f1"] == pytest.approx(0.7281, TOL2)
E       assert 0.0 == 0.7281 ± 3.6e-01
E        +  where 0.7281 ± 3.6e-01 = <function approx at 0x7fd3472e5620>(0.7281, 0.5)
E        +    where <function approx at 0x7fd3472e5620> = pytest.approx

Examples of failed runs: this and this

In which platform does it happen?

Azure Data Science Virtual Machine.

How do we replicate the issue?

pytest tests/smoke -m "smoke and not spark and gpu"

[miguelgfierro]

Super weird error @yueguoguo @gramhagen @Leavingseason @yexing99 @anargyri

[yueguoguo]

Investigating...

[miguelgfierro]

I commented the line that was causing trouble, so the tests pass: https://github.com/Microsoft/Recommenders/blob/staging/tests/smoke/test_notebooks_gpu.py#L126

When we start working on this, we need to have it in mind

[miguelgfierro]

FYI @almudenasanz, we have seen this weird error in DKN. We didn't have time to work on it. I just wanted you to be aware in case you also find it