Absinthe: [Proposal] Provider scalar type for supporting raw json

[secretworry]

Hello,

The JSON type as a leaf type is not defined in the spec, but it would be useful when you can specify a field as a row JSON. People of graphql-js have discussed this on their PR #172.

[benwilson512]

I have a couple issues with this. First and foremost, I'm generally against people using a raw JSON type. It's an end run around the type system, and usually you don't need to resort to that. Secondly, it's trivial to add your own. Lastly, it would require us including a JSON codec dependency, and right now we don't have any dependencies at all.

I do think this is worth covering in the guides, but I don't think it belongs in the built in scalars list.

@bruce what are your thoughts?

[secretworry]

Actually, It's not trivial to add one. I got a bunch of errors when I did so, and that's why I'm trying to ask for a built-in support.
I tried to define a JSON scalar like this:

  scalar :json do
    serialize &(&1)
    parse parse_with([Absinthe.Blueprint.Input.Boolean,
                      Absinthe.Blueprint.Input.Float,
                      Absinthe.Blueprint.Input.Integer,
                      Absinthe.Blueprint.Input.List,
                      Absinthe.Blueprint.Input.String], &parse_json/1)
  end

and the schema:

object :foo, name: "Foo" do
    field :bar, :json
end
mutation do
    field :create_foo do
        arg :bar, not_null(:json)
        resolve fn %{bar: bar} -> {:ok, %{bar: bar}} end
    end
end

and when I was tring to pass an json-typed variable to the system, I got a type error on the bar field.
The absinthe tried to validate the fields of the JSON I passed in and those fields definitely got no definition. So, in the module Absinthe.Phase.Document.Arguments.FlagInvalid, those fields were flagged invalid with :extra.

FYI,
I just want to use JSON to represent an i18n field like this %{locale => message} , and there are an undetermined number of locales we are trying to support, so the types could not be predetermined.

[bruce]

Hi, @secretworry. Thank you for your proposal!

When @benwilson512 says "trivial," he's talking about the common JSON feature people have asked for from us -- deserializing from a JSON string, serializing back to a JSON string, eg:

scalar :json do
  parse fn input ->
    case Poison.decode(input.value) do
      {:ok, result} -> result
      _ -> :error
    end
  end

  serialize &Poison.encode!/1
end

What you're asking for is, of course, completely different.

Regarding the PR you mention:

• It didn't end with JSON being added to graphql-js, but in a proposal being made to the Relay client framework, and a statement that seems to indicate it's unlikely to be added, as the current "deserializing/serializing" approach works well.
• My opinion on the issue, generally, is the same as @taion's; I don't think adding an "exit hatch" to the type system is usually a good idea, and deserializing from a string and serializing back to a string seems like the best way to handle this when you need it -- as an opaque type.
• If serializing/deserializing on the _client_ is a problem, I think that's best addressed by the client framework you're using, if any.

All that said, considering you're talking about adding a type to Absinthe itself:

We're not inclined to add types that aren't part of the GraphQL specification as built-ins, at least until the specification supports some type of namespacing. (Otherwise we'd have added :time long ago.)

We are _very_ interested in other people releasing packages with custom scalars, phases, resolution plugins, type-building macros, etc -- even if those people have _completely_ different opinions about what constitutes "best practices."

To that end, for this, if it's something that's important to you -- you're free to swap out a later phase in the pipeline (or add your own) that either prevents the invalid flag from being added or removes the flag.

We can certainly help you understand how to tackle that (our Slack channel is a particularly good way to reach us), and we'd be open to general-purpose changes that make Absinthe more flexible for people that want to modify it to suit their needs.

Extensibility is something we want to foster with our users, but I hope that you understand that we're trying to add to Absinthe's core pretty conservatively until clear patterns emerge.

[taion]

BTW – my disagreement there was with having some first-class support for "raw" objects, which doesn't make any sense in a polyglot context.

I think having a JSON scalar type is pretty useful. Among other things I maintain a tiny library that adds exactly that for GraphQL.js: https://github.com/taion/graphql-type-json.

There was some discussion of adding this to GraphQL.js – see @leebyron's https://twitter.com/leeb/status/725723772943101953.

[bruce]

@taion Sorry if I seemed to misattribute that -- I'll read for comprehension a bit more clearly next time (and instead of editing my comment above, I'll comment here).

Since this is the analog of graphql-js (which does not have a JSON type) and a JSON type isn't in the spec -- and because there are already plenty of "JSON" scalars that people are using that JSON parse/dump instead of this (what I'll refer to as) "passthrough" variety, it won't get into Absinthe, but I'd be happy if another package extended Absinthe to provide it, similar to what you did with graphql-js.

(Personally, I still dislike the idea of scalars that serialize to what many people would think of as non-scalar values, but then again, I probably use GraphQL differently than some, and the inputs/outputs shapes from my problem space are relatively well-defined.)

[bruce]

@secretworry So, thinking about this a bit more, I think what we'll do is look at making another package that has this in it, as well as the string deserialize/deserialize version. The issue you ran into with validation is due to the fact our intermediate representation _currently_ assumes that all nodes have an associated schema node, whereas in the case of "scalars" like this, the internals need to be treated as opaque.

@benwilson512 and I went through the code today and identified a few phases that need to modified to keep this new constraint in mind. It's enough code with enough complexity that we should probably handle it directly. I'll post an update with the link to the package repo tomorrow.

[taion]

BTW, the passthroughs are convenient for working with Relay, which doesn't have support for custom scalar deserialization right now.

[gullitmiranda]

@bruce I share your thought that this really should not be part of absinthe, but you and could you instruct us which way to do this as a new package to extend absinthe?

[benwilson512]

There's a recipe for this in the Absinthe wiki: https://github.com/absinthe-graphql/absinthe/wiki/Scalar-Recipes#json-using-jason

[gullitmiranda]

@benwilson512 this is not what we need, I already use this. what we need is support to send a json instead of a json string. I want to do something similar to https://github.com/taion/graphql-type-json

as said in the comment: https://github.com/absinthe-graphql/absinthe/issues/206#issuecomment-260447164

[benwilson512]

Ah. As I recall you can do this by just having a serialize function that passes a value through:

scalar :raw_json do
  parse fn _, _ -> raise "not relevant" end
  serialize fn value -> value end
end

[gullitmiranda]

it worked for the output. but if used in the input has the problem reported in https://github.com/absinthe-graphql/absinthe/issues/206#issuecomment-260352256

[benwilson512]

It is not our plan to easily support rewriting GraphQL syntax. An input object is not json. You can add a blueprint phase or similar to hack this together yourself, but there's nothing in the spec as far as I've seen that would permit an implementation to take an input object and use it as a scalar.

[tcoopman]

Am I reading this correctly that even if you write a custom scalar that does all the parsing of the input object, this still won't work?

I've tried something and have a result in the form of {:ok, %{foo: 1}} but the input does not succeed: message: "Argument "input" has invalid value $input.... Expected type "Json!", found {foo: 1}.↵In field "foo": Unknown field."

If I use the escaped json string and use the scalar from the wiki that generates the exact same output, it works fine.

[benwilson512]

Am I reading this correctly that even if you write a custom scalar that does all the parsing of the input object, this still won't work?

Absinthe parses the input object, not your scalar. The GraphQL grammar defines what counts as an input object and what counts as a scalar. If what is provided to the Absinthe parser is an input object, Absinthe will not provide that input object data to your scalar parse function, because it isn't a scalar. Providing a string of whatever encoding you like is a valid scalar, and will be provided to your scalar parse function.

[tcoopman]

It is provided to my scalar though and my parser can parse it correctly, but the graphql endpoint returns an error. No errors are reported in the elixir console

[benwilson512]

Can you provide an example?

[tcoopman]

Given this scalar:

defmodule DashboardWeb.Schema.Types.Custom.ScalarJSON do
  use Absinthe.Schema.Notation

  scalar :json, name: "Json" do
    serialize(&encode/1)
    parse(&decode/1)
  end

  defp decode(%Absinthe.Blueprint.Input.Object{}) do
    {:ok, %{}}
  end


  defp encode(value), do: value
end

and this schema:

    field :foo, :string do
      arg(:input, non_null(:json))

      resolve(fn _,_,_ -> {:ok, "RESOLVED"} end)
    end

when I submit this mutation:

  mutation Foo($input: Json!) {
    foo(input:$input)
  }

{"input":{}}

Everything works fine, but when I provide this input: {"input":{"foo": 1}} with in the same mutation I get "Argument \"input\" has invalid value $input.\nIn field \"foo\": Unknown field."

I'm using the 1.5.0-beta.2 version

[benwilson512]

I can see how that's a little confusing, but it's actually that{"input":{}} should not work either. {"foo": 1} is not a scalar, it's the variable version of an input object.

[tcoopman]

But in this example {"foo": 1} also hits the decode function. So this is double confusing.

[benwilson512]

This

  mutation Foo($input: Json!) {
    foo(input:$input)
  }

{"input":{"foo": 1}}

should behave exactly the same way as

  mutation Foo {
    foo(input:{foo: 1})
  }

If it doesn't, that's a bug. The parse function should not be called, and the error message is correct. {foo: 1} isn't a valid value for a scalar literal, and {"foo":1} isn't a valid value for a scalar variable. If we're passing those to parse functions we shouldn't be.

[tcoopman]

They behave the same, but the parse function is called.

Changing the scalar to this shows it:

defmodule DashboardWeb.Schema.Types.Custom.ScalarJSON do
  use Absinthe.Schema.Notation

  scalar :json, name: "Json" do
    serialize(&encode/1)

    parse(fn _ ->
      IO.inspect("PARSING")
      {:ok, %{}}
    end)
  end

  defp encode(value), do: value
end

[benwilson512]

Created https://github.com/absinthe-graphql/absinthe/issues/791

[dylan-chong]

Hello, having a free-form json type would be incredible useful for the product at my company. We need to be able to store free-form json in a field called attributes in various places. It is currently very awkward/ugly to send through a graphql mutation to create a profile for example:

// variables
{
  "email": "bla",
  "full_name": "mr. tidy data",
  "greeting_name": "hello world",
  "attributes": "{\"this is\": \"awkward\", \"very\": \"awkward indeed\"}"
}

It's a little tedious to have to serialise the attributes into a json string before sending the request, especially in tests.

Please do reconsider adding a freeform json type

[benwilson512]

https://github.com/absinthe-graphql/absinthe/wiki/Scalar-Recipes#json-using-jason

[binaryseed]

That works for the stringified Scalar, but if someone wants the "escape hatch" to send in open-ended input objects, they run into some validation phases that block this.

I don't want the default Absinthe to allow this, but its something we could make easier to opt-in to...

What do you think of that @benwilson512 ?

[benwilson512]

I'm reasonably certain that a no-op scalar serialize that returns a JSON serializable value will "just work". Should test though.

[binaryseed]

Yes, it does. However, using it as an Input does not work...

[benwilson512]

Ah yeah, but that is rather different. GraphQL object syntax isn't JSON. I'm willing to reconsider my stance on whether scalars should be allowed to receive arbitrary GraphQL syntax as inputs, but there simply _is no way_ to have a raw JSON input.

[binaryseed]

Yes, I also think calling it JSON is very misleading, since JSON is just a serialization format. I would never consider literal JSON inline inside GraphQL, but more like enabling a server to opt-in to accepting open-ended input object structures (wether it's inline in GraphQL or via variables).

My server for example replaces 3 phases with modified versions to allow a small list of scalars to be open-ended, basically short circuiting some of the validation so that the raw data is allowed in...

[dylan-chong]

Oh true, about the JSON thing. I was thinking in terms of a js graphql library. But the need for an open ended input type as @binaryseed mentions still remains. It would be greatly appreciated if this could be implemented.

[dylan-chong]

Another thing to point out is that with the JSON type you submit a string as input, but the output is a json object. That's a little inconsistent.

If you would like some help on this issue I would love to help out :)