Datasets: [GSoC] Investigate current kokoro failure.

[Conchylicultor]

Our builds are failing at head: https://source.cloud.google.com/results/invocations/08727437-e079-4920-bd0d-4929781edcef/targets/tensorflow_datasets%2Fgh_testing%2Fcontinuous/log

____________________ DocumentDatasetsTest.test_with_config _____________________
...
>     if builder_cls.BUILDER_CONFIGS:
E     AttributeError: type object 'EmptyDatasetBuilder' has no attribute 'BUILDER_CONFIGS'

tensorflow_datasets/core/registered.py:456: AttributeError

As far as understand:

• (1) Executing document_datasets_test.py execute core/registered_test.py, probably during imports.
• (2) When registered_test is executed, the EmptyDatasetBuilder is registered
• (3) document_datasets_test crash because EmptyDatasetBuilder is not a registered dataset.

The issue is (1). It's not clear why executing import tensorflow_datasets as tfds execute the core/registered_test.py even though the file isn't imported anywhere. This seems like a bug. Is this an issue with pytype used to run the tests ?

[vijayphoenix]

As you have described, the minimal command to reproduce this bug seems to be

pytest ./tensorflow_datasets/scripts/document_datasets_test.py ./tensorflow_datasets/core/registered_test.py

[Conchylicultor]

@vijayphoenix Thanks for reproducing the issue. It is possible that pytest import all tests files before executing the tests, leaking registration across tests.
I'll try to work on a fix.

[vijayphoenix]

Yes, I could see EmptyDatasetBuilder as a part of list_full_names in document_dataset_tests.py.
This is not desirable in our case.

[Eshan-Agarwal]

Is it still open for investigate or fixed by #1959 :)

[vijayphoenix]

Fixed as the tests are passing.
(See Kokoro tests results of that PR)

[Eshan-Agarwal]

Yes :)