Dhall-haskell: json-to-dhall

[antislava]

Implemented a simple json-to-dhall tool: https://github.com/dhall-lang/dhall-haskell/compare/master...antislava:antislava/json-to-dhall.
Haddock documentation following the structure of dhall-to-json is a bit rough but reasonably complete in Main.hs (cabal new-haddock exe:json-to-dhall works in dhall-json's nix-shell).

The conversion function is relatively straightforward but I had to put some thought into what the default behavior should be (in particular with regards to unions and records).

So far tested it only on simple cases (listed in the documentation). For a more "real-life" example, check out bower.dhall.

Related discussions:

• Discussion: transpile dhall to other languages in both directions? · Issue #346 · dhall-lang/dhall-lang
• Discussion: Support for importing from JSON · Issue #121 · dhall-lang/dhall-lang
• Interest in json-to-dhall and yaml-to-dhall? · Issue #23 · dhall-lang/dhall-json

[Profpatsch]

Does dhall-to-json(json-to-dhall(x)) == x hold?

[antislava]

@Profpatsch json-to-dhall(t, dhall-to-json(x : t)) == x, where t is the Dhall type (schema) which is the first argument to json-to-dhall, the second being the actual JSON data, should hold in most cases. Proper proof/tests will be the next step, but it does already hold for the provided bower.dhall/bower.json example (up to the key re-ordering, obviously) and in basic cases in the Main.hs documentation section:

json-to-dhall Bool <<< 'true' | dhall-to-json
json-to-dhall Bool <<< 'false' | dhall-to-json
json-to-dhall Integer <<< 2 | dhall-to-json
json-to-dhall Natural <<< 2 | dhall-to-json
json-to-dhall Double <<< -2.345 | dhall-to-json
json-to-dhall Text <<< '"foo bar"' | dhall-to-json
json-to-dhall 'List Integer' <<< '[1, 2, 3]' | dhall-to-json
json-to-dhall '{foo : List Integer}' <<< '{"foo": [1, 2, 3]}' | dhall-to-json
json-to-dhall '{foo : List Integer}' <<< '{"foo": [1, 2, 3], "bar" : "asdf"}' | dhall-to-json
json-to-dhall '{ a : Integer, b : Text }' <<< '[{"key":"a", "value":1}, {"key":"b", "value":"asdf"}]' | dhall-to-json
json-to-dhall 'List { mapKey : Text, mapValue : Text }' <<< '{"foo": "bar"}' | dhall-to-json
json-to-dhall "Optional Integer" <<< '1' | dhall-to-json
json-to-dhall '{ a : Integer, b : Optional Text }' <<< '{ "a": 1 }' | dhall-to-json
json-to-dhall 'List < Left : Text | Right : Integer >' <<< '[1, "bar"]' | dhall-to-json
json-to-dhall '{foo : < Left : Text | Right : Integer >}' <<< '{ "foo": "bar" }' | dhall-to-json
json-to-dhall '{foo : < Left : Text | Middle : Text | Right : Integer >}' <<< '{ "foo": "bar"}' | dhall-to-json

json-to-dhall(x) without Dhall schema argument is generally not possible, obviously. One could have a separate type-guessing tool (JSON → Dhall Type) facilitating the schema authoring, but in many cases of practical importance when you basically need a tool for importing JSON data of a standard format over and over again (see spacchetti/spago for an example), manually preparing a Dhall schema for each format is a relatively minor one-time effort.

[Profpatsch]

As long as the round-trip works it’s fine I guess.
I see, you have to give it a type separately, otherwise it couldn’t distinguish e.g. Double, Natural or Integer.

[antislava]

Indeed! Also, you generally cannot derive Optional and Union from the JSON data (not from a _single_ instantiation of a schema at least).

There are also situations where you would prefer to interpret/import a JSON object as a key-value association list rather than a Dhall record (for example, dependencies field in my bower.dhall).

So json-to-dhall(t, dhall-to-json(x : t)) ~ x is probably as close as we can get in terms of reversibility (without too many options/flags, constraints, and assumptions), but it is actually fine: t is basically our import _configuration_ ;)

[Gabriel439]

This is great! I think the main thing we need is to upstream this into the standard, mainly to sort out corner cases (like how to handle unions, if at all), to document the expected behavior, and to give other implementations input into how JSON support evolves.

[antislava]

@Gabriel439 I agree, the JSON interface needs to be standardized. The proper JSON importing functionality could increase Dhall usefulness and adoption significantly. Currently, in json-to-json, json-to-nix, etc... workflows, where the JSON source is provided by some external API (e.g. the GitHub API source) I need to resort to a pipe of jq (or maybe nix eval), probably some shell interpolation, and then jq or yq again, and adding Dhall in the middle currently doesn't improve the workflow enough to justify it.
Using Haskell script for importing and defining the Dhall type is straightforward but involves quite a bit of boilerplate and will (inevitably) lack consistent behavior and error reporting (and, more importantly, falling to Haskell kind of makes Dhall itself redundant, unless Dhall is not the script's target?). A standard Dhall.FromJSON module (with the conversion function, conversion parameters and the exception data type) would also greatly improve the Haskell approach.

Back to your suggestion, where should this upstream to more exactly (I am not yet very familiar with the project hierarchy). Is there a specific file or folder in dhall-lang/dhall-lang or dhall-lang/dhall-haskell?
So far, I just tried to follow the dhall-to-json's code as much as possible (but for now keeping all the logic in Main.hs - it will obviously have to be moved to something like src/Dhall/FromJSON.hs, if simultaneously renaming current Dhall.JSON to Dhall.ToJSON for consistency, and perhaps factoring out common parts to a separate shared module, named Dhall.JSON?). Is dhall-to-json already _upstreamed_ in this sense so that I could piggyback on that as well?

[antislava]

@Gabriel439 Regarding the corner cases, I added the following options/flags for importing Unions:

1. --unions-first (default) selects the first successful match (error of none matched)
2. --unions-strict results in 'undecidable union' error if there are more than one successful matches (all the alternatives being printed out in the error message)
3. --unions-none forbids unions altogether (probably redundant since the user is responsible for the import schema anyway?)

With regards to the Records I added a --records-strict switch requiring all the keys in JSON object to be handled, reporting the error otherwise. The default behavior is to allow the unhanded keys (which I think is a more common situation as one normally needs a subset of an external API adding new fields incrementally if needed).

JSON key-value lists are imported as Records, while Dhall mapKey-mapValue lists match JSON objects (see bower.dhall for an example) by default. One can opt out of these defaults using --no-keyval-arrays and --no-keyval-maps switches, respectively.

Are there other important corner cases to consider? What did I miss?

[joneshf]

I think the main thing we need is to upstream this into the standard, mainly to sort out corner cases (like how to handle unions, if at all), to document the expected behavior, and to give other implementations input into how JSON support evolves.

Are you suggesting we standardize before merging this PR (and releasing the program) or are you saying we standardize as an auxiliary thing to do? I ask because as @antislava hinted to:

Is dhall-to-json already _upstreamed_ in this sense so that I could piggyback on that as well?

There's no standardization for going the other way. And, printing seems way more important than parsing.

I'm all for standardization of JSON decoding/encoding. I don't think it's that big a burden to add, either. This isn't me fighting against standardizing. It's just a little unclear why we need it now before adding a program to a separate repo.

[f-f]

I agree with @joneshf that standardization should be an independent process from just merging this - we can always standardize the other way around (i.e. from here to the standard), as it happened with the main implementation.

I'll also add that I think implementing a JSON interface should not be compulsory for a given implementation (the usual reason is to lighten the implementation-bootstrapping process, as this feels a bit orthogonal)

[Gabriel439]

@antislava: I originally envisioned first upstreaming this into the standard, but I've changed my mind and think it's worth merging this even before we standardize it. If you open the pull request I'll accept it

[antislava]

Done! #884

Note that I was getting some unrelated Dhall test failures after merging the latest master into my branch, so I had to disable tests in dhall-json's nix-shell temporarily (in nix/shared.nix - outside this commit obviously).

ps @Gabriel439 I keep seeing minor errors in Hydra builds (most recently, related to Semigroups/Monoids), which I don't see locally, using the "official" shell.nix and the cabal file. Please, advise how to align the local setup with that of the Hydra server, so that I could realistically iron out those. (Maybe, some advice on this should be included in the README, unless I missed it somehow?)

pps ah, I see it was ghc-7.10.3 build that was causing problems. I'd obviously need to build under import ./nix/shared.nix { compiler = "ghc7103"; }; locally to take care of those errors properly, but currently running out of space on my machine ;) Will try this during the weekend, disabling ghc < 8 for now in .cabal to see that everything else checks.

[singpolyma]

For interest's sake: there are also experimental json-to-dhall and yaml-to-dhall tools shipped with dhall-ruby

[antislava]

@singpolyma Thank you for the info! Could you please link to some list of simple examples of how it works in your repository? (Do you also require some sort of a schema as an input argument?)

[singpolyma]

@singpolyma Thank you for the info! Could you please link to some list of simple examples of how it works? (Do you also provide any sort of a schema as an input argument?)

No schema. You run it like:

    bundle exec json-to-dhall < path/to.json | dhall decode

(it emits binary encoded dhall, so if you want souce use dhall decode command).

Basic types (number, string) do what you expect. Homogenous array does what you expect. Almost-homogenous array with nulls generates list of optionals. Heterogenous array generates list of union. Map becomes record. Map keys that are null are omitted from the output.

[Gabriel439]

I'll mark this resolved now that #884 is merged