Oj: Rendering JSON with ActiveModel::Serializers returns object

[dpaluy]

I made an upgrade of _oj_ from version 2.10.2 to 2.10.4, and I got a serialized object instead of expected JSON.

{
    "object": {
        "klass": "Client",
        "table": {
            "name": "clients",
            "engine": "Client",
            "columns": null,
            "aliases": [],
            "table_alias": null,
            "primary_key": null
        },
        "values": {
            "references": [],
            "where": [
                {
                    "left": "#<struct Arel::Attributes::Attribute relation=#<Arel::Table:0x007fdeb9db0b10 @name=\"clients\", @engine=Client(id: uuid, email: string, phone: string, full_name: string, created_at: datetime, updated_at: datetime, service_provider_id: uuid, country_code: string, lang: string), @columns=nil, @aliases=[], @table_alias=nil, @primary_key=nil>, name=\"service_provider_id\">",
                    "right": "3ba32f03-0ce0-439d-a26a-8a6617d92116"
                }
            ],
            ...
}

I use _rails-api 0.3.1_ and _active_model_serializers 0.9.0_

[ohler55]

Can you provide a simple test I can run?

[dpaluy]

@ohler55 I will prepare a test by the end of this week.

[Soliah]

I can reproduce this without AMS by doing a simple select in ActiveRecord. Something like User.select("users.first_name || ' ' || users.last_name AS full_name, users.*") behaves the same.

[ohler55]

I'm about ready to make a release. There were some changes around ActiveSupport. Maybe that will help.

Anyway, if not, I don't have ActiveRecord setup with a database. A simple test would be one that uses only some number of gems and some ruby code you provide. I'm still not sure what the expected JSON is or how to reproduce what you are doing.

[Soliah]

Here's a Rails app with the behaviour: https://github.com/Soliah/oj-test. There are 2 branches, master and no-oj. The only difference betwen the 2 branches is the inclusion of oj in master. The expected output is just the properties of the User model.

Edit:

Rails 4.1.7, Ruby 2.1.4. http://localhost:3000/users is the test route.

no-oj branch:

[
   {
      id:5,
      first_name:"John",
      last_name:"Smith",
      email:"[email protected]",
      created_at:"2014-11-03T00:39:55.342Z",
      updated_at:"2014-11-03T00:39:55.342Z"
   }
]

master branch with oj:

{
   klass:"User",
   table:{
      name:"users",
      engine:"User",
      columns:null,
      aliases:[

      ],
      table_alias:null,
      primary_key:null
   },
   values:{

   },
   offsets:{

   },
   loaded:false,
   arel:{
      engine:"User",
      ctx:{
         source:{
            left:{
               name:"users",
               engine:"User",
               columns:null,
               aliases:[

               ],
               table_alias:null,
               primary_key:null
            },
            right:[

            ]
         },
         top:null,
         set_quantifier:null,
         projections:[
            "#<struct Arel::Attributes::Attribute relation=#<Arel::Table:0x007f847f84b308 @name="            users",
            @engine=User(id:integer,
            first_name:string,
            last_name:string,
            email:string,
            created_at:datetime,
            updated_at:datetime),
            @columns=nil,
            @aliases=            [

            ],
            @table_alias=nil,
            @primary_key=nil>,
            name="*">"
         ],
         wheres:[

         ],
         groups:[

         ],
         having:null,
         windows:[

         ]
      },
      bind_values:[

      ],
      ast:{
         cores:[
            {
               source:{
                  left:{
                     name:"users",
                     engine:"User",
                     columns:null,
                     aliases:[

                     ],
                     table_alias:null,
                     primary_key:null
                  },
                  right:[

                  ]
               },
               top:null,
               set_quantifier:null,
               projections:[
                  "#<struct Arel::Attributes::Attribute relation=#<Arel::Table:0x007f847f84b308 @name="                  users",
                  @engine=User(id:integer,
                  first_name:string,
                  last_name:string,
                  email:string,
                  created_at:datetime,
                  updated_at:datetime),
                  @columns=nil,
                  @aliases=                  [

                  ],
                  @table_alias=nil,
                  @primary_key=nil>,
                  name="*">"
               ],
               wheres:[

               ],
               groups:[

               ],
               having:null,
               windows:[

               ]
            }
         ],
         orders:[

         ],
         limit:null,
         lock:null,
         offset:null,
         with:null
      }
   }
}

[Soliah]

I haven't done anything special in above setup like use AMS. Just a simple render json: @users call.

[ohler55]

Since I have not used rails that much it is going to take some time for me to figure out how to get you test working and see what object class is being serialized.

[Soliah]

I'm happy to help so please let me know if you want me to try anything.

[ohler55]

Please try the latest. I just released 2.11.0.

If that does not work then if you can make a simple ruby script that creates an instance of the object you serialized that would be great. The User object from the looks of it.

[Soliah]

I've just tried this in the Rails app and the behavior is the same.

What I'm not clear on is if I'm supposed to call Oj.dump or just some_object.to_json in the context of Rails. Does the oj_mimic_json gem deal with that now that multi_json isn't in the picture?

I've tried reproducing this with ActiveRecord outside of Rails and can't get the same behaviour. This leads me to believe that it's something to do with Rails further up in the rendering section. Maybe something here: https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/metal/renderers.rb though I can't be sure.

I might just work around this and do what was done here http://brainspec.com/blog/2012/09/28/lightning-json-in-rails/

[Soliah]

For interests sake here is ActiveRecord without Rails:

source 'https://rubygems.org'


gem "activerecord"
gem "sqlite3"
gem "oj", github: "ohler55/oj"

require 'active_record'
require 'oj'

Oj.mimic_JSON()
# Requiring json here seems to stop conflicts when requiring json in other files.
begin
  require 'json'
rescue Exception
  # ignore
end

ActiveRecord::Base.logger = Logger.new(STDERR)

ActiveRecord::Base.establish_connection(
  :adapter => "sqlite3",
  :database => ":memory:"
)

ActiveRecord::Schema.define do
  create_table :users do |table|
    table.column :first_name, :string
    table.column :last_name, :string
    table.column :email, :string
  end
end

class User < ActiveRecord::Base
end

User.find_or_create_by(first_name: "John", last_name: "Smith", email: "[email protected]")

puts User.first.as_json

[ohler55]

The Oj.mimic_JSON should take care of spoofing the json gem. If it is not, let me know and I'll get it back to that state.

There are some things that will help you understand what is going on. Oj has different modes. If you are trying to serialize an object you can use the default :object mode in Oj. As you have seen that creates a JSON representation with all the active record extra stuff in it. Probably not what you want. If you can as_json() and serialize that using the lighting approach you pointed to that should work well. Notice I use "should".

Heres where it might get tricky. the as_json() method is only calling on compat mode. mimic_JSON puts Oj in compat mode though. All good so far. If you have the :use_to_json option set to true the dump should work as expected. If it is false it will most likely not. Can you verify the :use_to_json is true?

Another twist. If to_json is defined and you call that :use_to_json is automatically set to false. This is to avoid the recursive calls active makes when calling to_json. It basically calls the serializer (Oj) which is expected to call to_json. This goes on until the app dies.

The key may be to add another :use_as_json or something. Lets continue exploring.

[Soliah]

Ok it seems to me that in Rails, mode: :object is being used no matter what I do when using render json: @users. This can easily be reproduced in rails c with User.first.to_json.

Passing options to to_json doesn't change this behaviour. Calling Oj.default_options = { mode: :compat } explicilty also does nothing to change this behaviour.

However, going Oj.dump(User.first) does behave correctly, so this looks like something with Rails and how JSON is setup?

[ohler55]

That is useful information. Passing arguments to to_json() will not work. You can set the default mode like this.

  Oj.default_options = { mode: :compat }

From what you describe it sounds like the problem may be in the :use_to_json flag being set to false when to_json() is called. I may have to change that. Can you verify that on Oj.dump(User.first) the as_json() method is called?

[Soliah]

Putting a debugger here https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/metal/renderers.rb#L117

the following behaviour is observed:

json.to_json returns https://gist.github.com/Soliah/a5d21eaa77a6762235c1
Oj.dump(json) returns https://gist.github.com/Soliah/5337e30662a9e2828b1e
Oj.dump(json, mode: :object) returns https://gist.github.com/Soliah/8cf38eaec18640582af3

What interesting is json.to_json and Oj.dump(json, mode: :object) returns different values...

Edit

The json object is just:

#<ActiveRecord::Relation [#<User id: 5, first_name: "John", last_name: "Smith", email: "[email protected]", created_at: "2014-11-03 00:39:55", updated_at: "2014-11-03 00:39:55">]>

[Soliah]

From what you describe it sounds like the problem may be in the :use_to_json flag being set to false when to_json() is called. I may have to change that. Can you verify that on Oj.dump(User.first) the as_json() method is called?

Just confirmed that doing Oj.dump(User.first) does call as_json. as_json doesn't get called during a render json: @users though.

[ohler55]

okay, I guess I have to allow as_json. Let me look at some of the issue and make sure as_json does not get into an endless recursive loop with some of the other rails libraries. I the mean time I can create a branch for you with the change if that helps.

[Soliah]

One last thing, is this issue has sort of diverged potentially from the original issue with ActiveModelSerializers. I have a feeling it's somewhat related but maybe not completely. I would prefer for this to work with AMS but that might be another issue.

[ohler55]

Do you know what the AMS call? I'm sure it is related but not sure the path the AMS is taking.

[Soliah]

The current stable version 0.9.x uses as_json as far as I'm aware: https://github.com/rails-api/active_model_serializers/blob/0-9-stable/lib/active_model/default_serializer.rb#L17

master which will be 0.10 is a big refactor from what I understand, so the API will change. @kurko or @guilleiguaran might be able to offer some suggestions.

[ohler55]

Try the as_json-ok branch. I think it might be okay. I have a second level check for recursion. The problem is that active support has as_json return the object itself in some cases like for core types.

[Soliah]

Ok trying out 00b368cc33bb893bf9ae23f43f70489f7b79f791 (as_json-ok branch)

class User < ActiveRecord::Base
  def as_json(options = {})
    logger.debug("User as_json called!")
    super
  end

  def to_json(options = {})
    logger.debug("User to_json called!")
    super
  end
end

With render json: User.all, if I have the following in the User model, neither logging statements get hit.

However if I change it to render json: User.first then the logging statement occurs for to_json. Rails obviously has to do something different with User.all as that returns a collection rather than just a User object.

In both cases, as_json doesn't get called at all. Furthemore, none of these cases returns the right JSON output.

[ohler55]

Thats not good. There is a way out but it isn't very nice. A dump routine can be registered. It is not efficient though. I suspect the problem is the collection is a Hash or an Array and Active has added a to_json() method that calls the JSON.dump() which is now Oj and Oj then would call the to_json on the object if the :use_to_json was enabled. Works fine up to that point but then the use_to_json is still set to false when trying to convert the User object to json but at that point all attributes will be serialized.

Let me sleep on this and see if I can come up with a work around. I think the key will be getting the as_json methods to be called.

[guilleiguaran]

I think @chancancode can help understanding how the to_json/as_json works in the most recent versions of active support :smiley:

[ohler55]

@chancancode, can we figure out a way to get this working?

[chancancode]

I'll take a look in a few hours :+1:

[ohler55]

Thanks, I'll be offline by then but will pick it up in the morning.

[chancancode]

Hello!

The situation we got into is quite unfortunate, as we have multiple JSON libraries fighting over each other about very generic method names like to_json :disappointed:

Quick Summary

1. Rails developers should always override #as_json instead of #to_json
2. Rails expects that calling #to_json will always go through the Rails encoder (custom pure-Ruby encoder for <= 4.0, json gem for 4.1+), which considers the #as_json hooks among other things
3. To use Oj explicitly, developers should call Oj.dump(obj)
4. To use json gem explicitly, developers should call JSON.generate(obj) (this only work reliably on Rails 4.1+)
5. The #to_json definition simply call rails_json_encoder_encode(self.as_json) in all version (slightly simplified)
6. Oj can/should invoke #as_json on objects it doesn't natively know how to encode

Before we get into the details, here is some backstory and history of how we got ourselves into the current situation.

Rails JSON encoder

Ancient History (Rails 2.3 era)

Once upon a time, Ruby doesn't ship with any JSON library. As Rails needed to generate JSON objects, we wrote our own encoder, and eventualy settled on the to_json-based API as that seemed like the obvious choice at the time. You simply override to_json on your classes, construct a hash to represent your data and recursively call to_json on that Hash, and you have your desired JSON string. (Fun fact: that commit also included the initial JSON decoder that essentially worked by converting JSON into YAML using gsub so it could be parsed with Syck. Fun times!)

Rails 3 era

At some point, Rubyists started writing other JSON libraries/wrappers that are more efficient than the pure-Ruby encoder that comes with Rails, such as Oj, yajl, etc. There is a problem though – the to_json API does not offer any way for us to cooperate with these libraries. Because to_json is responsible for returning the resulting JSON string, there is no way for these alternative encoders to "hook into" this process.

The key observation here is that most application programmers don't override to_json because they want control of how Ruby objects are _encoded_ into a JSON string; they simply want to describing how they want their data _represented_ (e.g. what fields to include, whether the object should map to an Array or an Object...).

Thus, as_json was born. The idea is that developers can just override as_json to return the Ruby representation of their data (i.e. return a Hash/Array/etc) and let the encoder take care of turning that into the JSON string.

For backwards compatibility reasons, the #to_json method is kept as an "entry point" to our Ruby-based encoder (1.8.7 doesn't have an JSON encoder in the standard library). However, it should now be possible to use an alternative encoder such as Oj, e.g. by calling Oj.dump(some_rails_object_that_oj_doesnt_know_about). In this case, the oj encoder can simply invoke #as_json on the object to get a (hopefully*) simplified representation of the object that it will be able to encode.

(*the as_json hook is provided as a "hint" to the encoder – the developer is free to return any strange objects in the hook and it is up to the encoder to decide what to do with them. In practice though, the meaning of things like symbols, hashes, arrays, booleans and strings are well understood. This is not required to be recurrsive, so if the developer returned, say, an array of non-literals, the encoder should still try to call as_json on its elements. I took some detailed notes about how a reasonable encoder _should_ behave here.)

Rails 4 era

When Ruby 1.9.1 was released, the json gem officially become part of the Ruby standard library. Since 4.0 dropped support for Ruby 1.8.7, in theory everything should be nice from here on. Unfortunately, it uses the #to_json architecture, which prevented it from being very useful inside Rails because of the flaws described above. Thus things remained largely unchanged.

As the popularity of the JSON gem grew though, another annoying issue arose. As mentioned above, the json gem also defines a to_json method that is not quite the same as the one shipped with Rails. (e.g. one takes an option hash, one expects a State object; one considers as_json, the other doesn't). This created all sorts of problems and confusion.

Rails 4.1+

I picked up the task of cleaning up some of these mess. A few things changed:

• Previously, Rails and json gem would appear to play nice with each other until things suddenly blow up in some edge cases. The main issue is that the JSON gem expects and passes a hash-like-but-not-really State object as the second argument to to_json, where Rails expects a real hash. To avoid these problems, we detect and completely bypass the Rails encoder when the JSON gem is involved. So when you call ActiveSupport::JSON.encode(obj) or obj.to_json, you get the Rails stuff, whereas when you explicitly call JSON.{generate|dump}(obj) you get the "bare" result from JSON gem without any of the Rails stuff (as_json).
• Since Ruby 1.9+ ships with a json gem, there is no point maintaining the pure-Ruby encoder inside Rails anymore. I ripped out all the encoding related stuff inside Rails (i.e. code that converts Ruby object into the actual JSON strings... e.g. nil => "null"), and simply shim'ed the JSON gem. So when you call to_json, Rails would first recursively call as_json on the object and pass that to JSON.generate.
• Since the maintainers of multijson seemed uninterested in continuing to maintain that gem, it has been pulled out of Rails in favor of just using the built-in json gem. This only affects parsing! In previous versions, Rails has only used multi-json on the parsing side. The encoding side always went through Rails' own encoder (through #to_json) with or without multi-json/oj/json gem activated. Except for the spots that you explicitly used MultiJson.dump(...), you have always been using Rails' internal encoder. Installing Oj gem does not tell Rails to use the Oj encoder regardless of your MultiJson settings. Any effect on _encoding_ speed you noticed is probably psychological :)

Rails controller

When you call render json: some_object, Rails will call #to_json on the object, which should go through the Rails encoder out-of-the-box.

Active Model Serializer

v0.8 & 0.9

These two versions of AM::S has a completely different codebase, but for our purpose they are virtually the same. When you do a render json: object, it tries to wrap object in the appropriate serializer object (which implements as_json), and pass that object (still a Ruby object at this point) to the controller's json renderer, which calls to_json as described above.

If you are using Rails 4.2, you'll need 0.8.2 / 0.9.0 for things to work properly because Rails renamed some of the controller hook methods.

v0.10

This is another complete rewrite, but currently it defines to_json instead of as_json.

(@guilleiguaran we need to :scissors: these to_json in AM::S in favor of as_json as I pointed out here)

Where things went wrong with Oj + Rails

The Rails codebase relies on to_json across the board. By default, this should resolve to the Rails JSON encoder, which uses the as_json hook. It appears that somewhere in the Oj gem it overrides to_json with a definition that ignores the as_json hooks (probably here?).

Where does that leave us...

Option 1: Explicitly opt-in

1. Configure Oj to...
   
   
   
   1. Not override to_json
   
   
   2. Honor as_json
   
   
   3. (I believe the above amounts to "don't use mimic json gem mode, and enable compat mode", but I could be wrong)
   
   
2. Explicitly use render json: Oj.dump(obj) and render json: Oj.dump(WhateverSerializer.new(obj)) in your controllers

This is pretty much how things always worked with previous versions of Oj. (Again MultiJson never actually helped you for encoding.)

Presumably, you can monkey patch the json renderer to do this automatically for you (in that case the AM::S renderer additions should Just Work™ also).

Option 2: Automatically use Oj for everything...

This requires patching Oj to offer a mode that would override to_json to encode objects in a Rails-compatible way. This is super invasive and you'll probably end up dealing with a lot of the same bugs that we had dealt with before, so I don't know if I could recommend this.

Also, it's quite likely that we would deviate slightly on the encoding result, but if anything comes up I'm happy to work with you to figure out what the "correct" behavior should be (or whether that's an edge case that should be left undefined).

Option 3: Automatically use Oj for everything (Rails 4.1+ only)...

If supporting only Rails 4.1 and above is an option, there's a slightly less invasive option. (In this case it probably shouldn't try to mimic json gem.) It still has the "deviate slightly on the encoding result" problem, but it'd at least shield you from the json gem compat State object nonsense.

[chancancode]

wdyt @jeremy? :trollface:

[ohler55]

Fantastic explanation. Thanks.

I was aiming for option 1 with the exception of the to_json override. The override is there now to mitigate the rails monkey patching the core/base types. It may make sense for Oj to change some of the options and their behaviour so that it can be made to work with both Rails and the Active libraries.

I'll make a branch of Oj that follows the semi-formal description for as_json and see how it works. I believe the key difference will be in handling the primitive and base type.

[chancancode]

Let me know how it goes! Treat the gist as a "note", more than anything, we can certainly review and fix anything that doesn't make sense.

As for the immediate problem described in this ticket, I am pretty sure it's just that to_json is being overridden with an incompatible definition (as a reporter noted above, Oj.dump and to_json gives different results). So turning that off should fix the problem.

[ohler55]

I'm doing some planning and wanted to run a few things past the group. I can't remove the to_json override since many (maybe most) call JSON.dump which creates a recursive loop unless the to_json call turns off Oj calling to_json when it encountered an Object that responds to that method. The alternative would be to never call to_json from within Oj. Any opinions?

The as_json would always be called if the Object responded to that method unless it is a primitive. I will leave in the check for an object returning itself though.

[chancancode]

I'm not 100% sure if I understand your problem, can you explain...

1. What does the to_json override look like in Oj?
2. What does the problematic call sequence look like?

Here is what the to_json override looks like in recent versions of Rails (4.1+).

1. The Rails encoder do not consider to_json at all – we only use as_json for serialization, and to_json is just an entry point to the encoder.
2. Our stance is that when people do an explicit JSON.dump (instead of to_json), people probably expect to get the json gem's output (i.e. uses to_json instead of as_json), so we bypass the Rails encoder entirely.

I suspect point number 1 is why we don't have the same problem that you are dealing with.

For point number 2, we were able to do that because json gem's to_json has a unique (and incompatible) method signature – it passes a ::JSON::State object instead of a hash to the method. So when we see that the argument is a ::JSON::State object, we know that it's coming from JSON.dump or JSON.generate, in which case we just defer to the original definition of to_json without the Rails override.

[ohler55]

I think part of the problem is I'm trying got be compatible with both older and the latest rails.

Maybe the trick is the ::JSON::State object. I'm not exactly sure that works yet though.

It is clear that to_json and as_json need to be considered separately though. Let me try to put something together and see how it works. Not sure if I will get to it tonight to tomorrow.

[ohler55]

Simply not turning off as_json with to_json seems to be enough. I have to put together a test with and ActiveSupport collection though. If anyone has simple one I can use that would be great. Otherwise I'll do some web searching and write something tomorrow afternoon.

[aaronchi]

I'm wondering if yajl json's approach would work here or if it's the same issue:
https://github.com/brianmario/yajl-ruby/tree/master/lib/yajl/json_gem

[ohler55]

I'd prefer not to take the yawl/json_gem approach as it forces all object to use to_s as a json representation if ActiveSupport is not defined. Not all Oj users are using ActiveSupport.

[ohler55]

Here is a simple test that works but I'm not sure how to make sure Oj is used for the to_json call without using Oj.mimicJSON. It seems to do the right thing in the as_json-ok branch.

require 'sqlite3'
require 'active_record'
require 'oj'

#Oj.mimic_JSON()
Oj.default_options = {mode: :compat, indent: 2}

#ActiveRecord::Base.logger = Logger.new(STDERR)

ActiveRecord::Base.establish_connection(
  :adapter => "sqlite3",
  :database => ":memory:"
)

ActiveRecord::Schema.define do
  create_table :users do |table|
    table.column :first_name, :string
    table.column :last_name, :string
    table.column :email, :string
  end
end

class User < ActiveRecord::Base
end

User.find_or_create_by(first_name: "John", last_name: "Smith", email: "[email protected]")
User.find_or_create_by(first_name: "Joan", last_name: "Smith", email: "[email protected]")

puts "as_json - #{User.first.as_json}"
puts "to_json - #{User.first.to_json}"
puts "Oj.dump - #{Oj.dump(User.first)}"
puts "Oj.dump all - #{Oj.dump(User.all)}"

[ohler55]

Does the latest release fix this?

[Soliah]

I'll give it ago and report back.

[Soliah]

Is oj_mimic_json still required?

[ohler55]

the mimic_JSON call is needed if you are making JSON calls or calling to_json(). If you are using Oj directly then mimic_JSON is not needed.

[Soliah]

Working for myself with active_model_serializers using the current HEAD. Loading about 10k records in development takes about half a second off rendering:

Without oj: Completed 200 OK in 3140ms (Views: 3061.0ms | ActiveRecord: 78.4ms)
With oj: Completed 200 OK in 2483ms (Views: 2406.5ms | ActiveRecord: 76.3ms)

The objects are being serialized correctly with or without AMS.

[ohler55]

Excellent!

[Soliah]

For smaller data sets the rendering time difference isn't particularly different than the built in JSON library. Is there a way I can confirm that oj is being used?

[ohler55]

Try setting the default options indent to 2 and look at the output.

[Soliah]

Ok that confirms that oj is definitely working. :heart: :sparkles:

[ohler55]

great, are we okay to close this issue?

[Soliah]

Yup. Would be great to have a new version up on Rubygems.

[ohler55]

2.11.1 is there now. What version have you been using?

[Soliah]

Oh sorry, I was just pulling down current HEAD on master (374c1133fb412953cc52ea82884e26bbfbf7c571). The version on Rubygems doesn't give me the right JSON back.

[ohler55]

Thats strange. The code is the same except for some travis changes. And the changes to floats tonight.

[Soliah]

Sorry, looks like I was referencing the wrong version in my Gemfile.lock. Good to close!

[ohler55]

great, thanks

[stefanahman]

@chancancode @Soliah How does this fix the AMS problem?