Conan: [OUTPUT] Provide a "plumbing" interface for stable cli integration with json output

[sztomi]

Git has a distinction between "porcelain" and "plumbing" commands. The latter is meant to provide a stable, machine-consumable output, rather than something that is human-readable.

We often find ourselves using the output of "porcelain"-style commands from conan and awkwardly parsing it to get rid of human-targeted text, such as status messages about authentication etc. Even "machine-looking" formats like the output of build_order has to be parsed due to the lack of quotes.

I'm proposing that conan adopts a stable json interface for all of its commands in the following format:

conan json <cmd> <args>

For example, the plumbing equivalent of the command conan search "*" -r remote -q os=Windows would be called as conan json search "*" -r remote -q os=Windows. Its output should always be valid json without any status messages. The format of the output should be kept reasonably backwards-compatible. Maybe an extra parameter could be added in the style of REST APIs to allow calling certain versions of queries.

I don't know if it makes sense to extend this to all commands, but search definitely needs something like this.

[tru]

Yeah this would solve https://github.com/conan-io/conan/issues/1864 as well.

[memsharded]

I partly agree with this, in fact we have started advanced in this direction:

• with the internal conan_api
• some command (info --build_order I think) already accept the --json argument

However I think the command conan json <cmd> ... is semantically confusing, and that the type of output is an argument of the command, and not the opposite. The analogy with git plumbing and porcelain is not very accurate in this case IMHO, because the commands are exactly the same, and perform exactly the same, just provide a different output. So I would suggest continue improving the api with a more structured output, maybe extending --json to commands that need it.

[sztomi]

Yeah, I'm not too strongly opinionated about the exact syntax, only that there should be a stable and machine consumable interface (possibly with json).

[vadixidav]

Based on #2566, it would be nice to add a --json flag (as suggested above for commands which need it) to conan info so it can get back the package it was invoked on and all packages in the list, specifically expanding the package ref into a struct with name and version for the invoked package and name, version, user, and channel for each package in the list. Right now <name>/<version>@PROJECT is what is printed for the package ref of the package conan info is invoked on. Also, sometimes the conanfile.py prints arbitrary things to stdout using self.output.info, which might increase the difficulty of parsing.

[sztomi]

I recently came across the --json flag for build_order. It is great, but has a serious flaw: it is still possible that the output is not only json. For example, in one run I got b"boost/1.59.0-20@plex/stable: Not found in local cache, looking in remotes...\nboost/1.59.0-20@plex/stable: Trying with 'toolchain'...\nboost/1.59.0-20@plex/stable: Trying with 'stable'.... etc

This is a message for humans. Tooling that calls conan does not need status messages like this and in fact, this makes the --json parameter much harder to use. The point of this GHI would be to add machine-parseable output that is always parsable.

[vadixidav]

@sztomi I think such status messages should go to stderr instead, if they aren't already. I mentioned in my above post how there is an issue with self.output.info where it also prints to the output and breaks some machine parsing of the output as well. I don't think cmd on Windows has a stdout/stderr though, so it might be sufficient to suppress everything in cmd when --json is specified aside from the actual JSON output. In my case, I could spin up powershell to get the output messages for logging and use the output stream for JSON, since cmd wouldn't let me. cmd also has a stderr and piping by default from what I can tell, so that should work fine too.

[lasote]

We are going to add experimental support for the install/create commands generating a json file with install information (will be released in 1.2 today). I keep this one opened until we complete the support to other commands and stabilize it.

[lasote]

For 1.3 we propose to improve this with the upload information. Very similar to the json for the install:

{"error":  True/False
 "uploaded": [
    {"recipe": {"id": "lib/1.0@conan/stable", "remote_url": "http://...", "remote_name": "conan-center", "time": "2018-09-15T15:53"}, "packages": [{"id": "92E3498F2734", "remote_url": "http://...", "remote_name": "conan-center", "time": "2018-09-15T15:54"},  {"id": "1EE3498F2734", "remote_url": "http://...", "remote_name": "conan-center"}]},
    {"recipe": None, "packages": [{"id": "92E3498F2734", "remote_url": "http://...", "remote_name": "conan-center", "time": "2018-09-15T15:54"},  {"id": "1EE3498F2734", "remote_url": "http://...", "remote_name": "conan-center", "time": "2018-09-15T15:54"}]},
]}

[lasote]

For 1.4 the improved output will be, at least, the search.

[lasote]

1.3 (Will be released today) manages the upload output.

[lasote]

1.4 (Will be released soon) manages the search output. Look at https://github.com/conan-io/conan/pull/2902

[danimtb]

user output for 1.5 release listing users per remote #3025

[lasote]

Look at: https://github.com/conan-io/conan/issues/3053

[lasote]

I think the next natural step would be to improve the conan info command output. But currently the command is quite a mess and we have to start deprecating current options (--json and --build-order and --build probably) and add new options (--json2 for example) that will replace the current --json at Conan 2.0.

Better than commit to implementing something in 1.6 I would aim to propose here what could be the new interface of the command and the json output.

To start discussing, some questions:

• In 1.5 we have introduced a replacement for --build called --dry_build. It should replace --build in Conan 2.0. The current option returned by --build could be included in the outputted json following the specified policy.

• About the --build_order: Without --json should be kept as is? With --jsonStudy if it should be shown together with the --build policy to know not only the order but if the package should be rebuilt according to the package_id computation and the available packages? Related to #555

{...., 
"build_order": [{"OpenSSL/1.0.2m": {"missing": True, ... }, 
                {"zlib/1.2.11": {"missing": False...}]

• About #3053. Should all the conanfiles be class-parsed to show the class attributes?

• About the --json2, study a proposal that fits all the needs, should the output keep always the same structure with more or fewer fields? Start analyzing the different "modes" of the info command before defining the json outputted.

• Think about #2490