Dvc: Tabular view for multiple metrics

[grighi]

dvc metrics show -a currently gives

workspace:
        eval.json:
                AUC: 0.66729
                error: 0.16982
                TP: 516
master:
        eval.json:
                AUC: 0.65115
                error: 0.17304
                TP: 528
increase_bow:
        eval.json:
                AUC: 0.66524
                error: 0.17074
                TP: 521

which is much less useful to me than this

AUC error TP workspace: 0.66729 ..... master: 0.65115 ... increase_: 0.66524 ....

Ruslan requested that I make this feature request and noted: metrics.py#L14, so we need to just orginize it into rows and pass to the table() helper, as we do it in diff metrics.py#L104

[efiop]

Discord context: https://discordapp.com/channels/485586884165107732/485596304961962003/759036937813753887

[indhupriya]

@efiop @grighi Can I work on this issue?

[efiop]

@indhupriya Sure! Thank you so much for looking into it! :pray:

[nischaymehta09]

The flattened dictionary method can be used. Please let me know if the repo is open.

[indhupriya]

Hi @efiop @grighi
Is this format and headers okay?

Branch        Path       AUC      error    TP
workspace     eval.json  0.66729  0.16982  516
master        eval.json  0.65115  0.17304  528
increase_bow  eval.json  0.66524  0.17074  521

Im calling the branch as Branch and filename as Path and the metrics' headers are derived from the json.

[indhupriya]

Also is it safe to assume the number of metrics in all files will be equal? I mean if eval.json in workspace contains AUC, error and TP, can eval.json in master contain just AUC and TO and miss error?

[pmrowla]

@indhupriya the number of metrics in a given file may vary across different commits and branches. Using your example, the output table should contain something like - or None to denote that error does not exist in the master branch/commit

[indhupriya]

@efiop @pmrowla I am having troubles running the tests on my environment. I followed the instructions here: https://dvc.org/doc/user-guide/contributing/core#running-tests. But the tests fail on my environment even before my changes. Could you please help me? Am I missing something? This is one of the failed tests:

[gw0] darwin -- Python 3.8.5 /Users/Indhu/dvc/.env/bin/python3

tmp_dir = PosixTmpDir('/private/var/folders/hk/nc0y26jx1wx143mhdy02hqnh0000gn/T/pytest-of-Indhu/pytest-31/popen-gw0/test_no_commits0')

    def test_no_commits(tmp_dir):
        from dvc.repo import Repo
        from dvc.scm.git import Git
        from tests.dir_helpers import git_init

        git_init(".")
>       assert Git().no_commits

/Users/Indhu/dvc/tests/func/metrics/test_diff.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/Indhu/dvc/dvc/scm/git.py:471: in no_commits
    return not self.list_all_commits()
/Users/Indhu/dvc/dvc/scm/git.py:304: in list_all_commits
    return [c.hexsha for c in self.repo.iter_commits("--all")]
/Users/Indhu/dvc/dvc/scm/git.py:304: in <listcomp>
    return [c.hexsha for c in self.repo.iter_commits("--all")]
/Users/Indhu/dvc/.env/lib/python3.8/site-packages/git/objects/commit.py:277: in _iter_from_process_or_stream
    finalize_process(proc_or_stream)
/Users/Indhu/dvc/.env/lib/python3.8/site-packages/git/util.py:329: in finalize_process
    proc.wait(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <git.cmd.Git.AutoInterrupt object at 0x11f0abf70>, stderr = b''

    def wait(self, stderr=b''):  # TODO: Bad choice to mimic `proc.wait()` but with different args.
        """Wait for the process and return its status code.

        :param stderr: Previously read value of stderr, in case stderr is already closed.
        :warn: may deadlock if output or error pipes are used and not handled separately.
        :raise GitCommandError: if the return status is not 0"""
        if stderr is None:
            stderr = b''
        stderr = force_bytes(data=stderr, encoding='utf-8')

        status = self.proc.wait()

        def read_all_from_possibly_closed_stream(stream):
            try:
                return stderr + force_bytes(stream.read())
            except ValueError:
                return stderr or b''

        if status != 0:
            errstr = read_all_from_possibly_closed_stream(self.proc.stderr)
            log.debug('AutoInterrupt wait stderr: %r' % (errstr,))
>           raise GitCommandError(self.args, status, errstr)
E           git.exc.GitCommandError: Cmd('git') failed due to: exit code(129)
E             cmdline: git rev-list --all --
E             stderr: 'usage: git rev-list [OPTION] <commit-id>... [ -- paths... ]
E             limiting output:
E               --max-count=<n>
E               --max-age=<epoch>
E               --min-age=<epoch>
E               --sparse
E               --no-merges
E               --min-parents=<n>
E               --no-min-parents
E               --max-parents=<n>
E               --no-max-parents
E               --remove-empty
E               --all
E               --branches
E               --tags
E               --remotes
E               --stdin
E               --quiet
E             ordering output:
E               --topo-order
E               --date-order
E               --reverse
E             formatting output:
E               --parents
E               --children
E               --objects | --objects-edge
E               --unpacked
E               --header | --pretty
E               --abbrev=<n> | --no-abbrev
E               --abbrev-commit
E               --left-right
E               --count
E             special purpose:
E               --bisect
E               --bisect-vars
E               --bisect-all
E           '

/Users/Indhu/dvc/.env/lib/python3.8/site-packages/git/cmd.py:408: GitCommandError

Please let me know if you need anything else.

[pmrowla]

@indhupriya it looks like git-python is having issues running git itself, are you able to run git rev-list --all -- (in any git repo)?

[indhupriya]

@pmrowla Yeah, I can run git rev-list --all -- from command line

[efiop]

@indhupriya That looks pretty weird, can't put my finger on anything yet. If that is the only test failure that you run into (or even if it is not), please feel free to create a WIP PR (don't forget to add WIP to the PR title), so that our CI could run the tests too and we will see if that issue is reproducible there as well.

[efiop]

@indhupriya Btw, that last error log looks a bit off, are you sure you've copied the whole thing?

[indhupriya]

@efiop The failing tests are now clear to me. I'll work on those. In the meantime, it'd be very helpful if you could review the code and suggest any changes.