Dvc: DVC metrics show doesn't find metrics files

[e3bo]

Bug Report

After marking some outputs as metrics using the -M option to dvc run, dvc metrics show tells me there are no metrics files.

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.4.0 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-4.15.0-111-generic-x86_64-with-Ubuntu-18.04-bionic
Supports: http, https, s3, ssh
Cache types: hardlink, symlink
Repo: dvc, git

as well as

$dvc version
DVC version: 1.4.0 (pip)
---------------------------------
Platform: Python 3.7.7 on Linux-4.19.132-1.pvops.qubes.x86_64-x86_64-with-fedora-30-Thirty
Supports: http, https, s3, ssh
Workspace directory: ext4 on /dev/xvdb
Repo: dvc, git

Additional Information (if any):
Steps to reproduce:

$ git clone --depth 1 --branch metrics-show-bug https://github.com/e3bo/random-walks.git
$ cd random-walks
$ dvc metrics show --verbose
2020-08-07 14:29:47,439 DEBUG: fetched: [(3,)]                                  
2020-08-07 14:29:47,459 DEBUG: fetched: [(0,)]
2020-08-07 14:29:47,466 ERROR: failed to show metrics - no metric files in this repository. Use `-m/-M` options for `dvc run` to mark stage outputs as  metrics.
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/venv/lib/python3.7/site-packages/dvc/command/metrics.py", line 49, in run
    recursive=self.args.recursive,
  File "/home/user/venv/lib/python3.7/site-packages/dvc/repo/metrics/__init__.py", line 8, in show
    return show(self.repo, *args, **kwargs)
  File "/home/user/venv/lib/python3.7/site-packages/dvc/repo/__init__.py", line 34, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/home/user/venv/lib/python3.7/site-packages/dvc/repo/metrics/show.py", line 105, in show
    raise NoMetricsError()
dvc.exceptions.NoMetricsError: no metric files in this repository. Use `-m/-M` options for `dvc run` to mark stage outputs as  metrics.
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-08-07 14:29:47,468 DEBUG: Analytics is enabled.
2020-08-07 14:29:47,522 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmps0gegceh']'
2020-08-07 14:29:47,524 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmps0gegceh']

The dvc.yaml has the paths to the metrics files, which are present in the repository. It was generated using a script similar to forecast.sh. I asked about this problem on discord chat and ivan suggested I open a bug report here.

[pared]

Ok so metrics in this particular repo are structured as list:
{"wall":[67.889]}
And this is the reason why don't get any results, as we are expecting singular values.

I have to admit that this is unfortunate, as the error message is unclear what actually happened.
The main problem lies in this line
AFAIK it is done that way to be able to show as many metrics as possible and skip the ones wrongly structured.

I think we should print some warnings that the unknown metric type is present and we are skipping it.
@shcheklein @e3bo What do you think?

[shcheklein]

Thanks, @pared for this research! It's helpful.

I guess for the metrics show we should be able to support arrays as well as scalar types. I don't see any reasons against it (except may be some code simplification?).

Error message is indeed misleading and hint is wrong in this situation. It can be improved to at least include both - mention of -m/-M and supported metrics formats.

I would not print warnings though (they all look very ugly in DVC, and we have one per every wrong metric that will be disastrous for us I think). We can print debug messages though, so that we can see what's happening in the -v mode.

[e3bo]

Thanks, @pared and @shcheklein ! I think the mention of supported metrics formats in the message would have helped me self-correct. I have no immediate need for arrays.

[jorgeorpinel]

Should run check format before allowing metrics/plots files to be marked as such? Avoiding having to check later (i.e. by mettrics/plots commands).

UPDATE: I guess you can always edit a metrics/plots file manually thus those commands need to check anyway...