Jsonapi-resources: Filter apply never being called

[adambedford]

I have a filter named tags, which looks like this:

filter :tags,
       verify: ->(values, _context) { values.flatten },
       apply: ->(records, values, _options) { raise("FOO BAR BAZ"); records.joins(:tags).where(tags: values)  }

Notice the raise in apply? That doesn't ever get called...

However, if I throw that, or binding.pry into a different filter, it does get called.
Is there something special with the word tags that I'm missing?

Update:

With request params of

Parameters: {"filter"=>{"tags"=>["63"], "status"=>["not_started"], "account_groups"=>["1"]}, "include"=>"account-group,tags", "page"=>{"size"=>"10"}, "sort"=>"last-name"}

and tapping into self.apply_filter, I'm seeing the tag filter coming through named tags.id, whereas other filters named as I'd expect (the param name, which correlates to the filter name)

[thomassnielsen]

Did you ever figure out what happened here?

[senid231]

@adambedford i failed to reproduce your issue
seems that all works like it does
take a look at following bug report - you can change it to repeat bug

begin
  require 'bundler/inline'
  require 'bundler'
rescue LoadError => e
  STDERR.puts 'Bundler version 1.10 or later is required. Please update your Bundler'
  raise e
end

gemfile(true, ui: ENV['SILENT'] ? Bundler::UI::Silent.new : Bundler::UI::Shell.new) do
  source 'https://rubygems.org'

  git_source(:github) do |repo_name|
    repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?('/')
    "https://github.com/#{repo_name}.git"
  end

  if ENV['RAILS_VERSION']
    gem 'rails', ENV['RAILS_VERSION'], require: false
  else
    gem 'rails', require: false
  end


  if ENV['JSONAPI_RESOURCES_PG']
    gem 'pg'
  else
    gem 'sqlite3', platform: :mri
    # gem 'activerecord-jdbcsqlite3-adapter',
    #     git: 'https://github.com/jruby/activerecord-jdbc-adapter',
    #     branch: 'rails-5',
    #     platform: :jruby
  end

  if ENV['JSONAPI_RESOURCES_PATH']
    gem 'jsonapi-resources', path: ENV['JSONAPI_RESOURCES_PATH'], require: false
  elsif ENV['JSONAPI_RESOURCES_VERSION']
    gem 'jsonapi-resources', ENV['JSONAPI_RESOURCES_VERSION'], require: false
  else
    gem 'jsonapi-resources', github: 'cerebris/jsonapi-resources', require: false
  end

end

# prepare active_record database
require 'active_record'

class NullLogger < Logger
  def initialize(*_args)
  end

  def add(*_args, &_block)
  end
end

if ENV['JSONAPI_RESOURCES_PG']
  db_name = ENV['JSONAPI_RESOURCES_PG']
  system("psql -c 'DROP DATABASE IF EXISTS #{db_name};'")
  system("psql -c 'CREATE DATABASE #{db_name};'")
  ActiveRecord::Base.establish_connection(adapter: 'postgresql', database: db_name)
else
  ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
end

ActiveRecord::Base.logger = ENV['SILENT'] ? NullLogger.new : Logger.new(STDOUT)
ActiveRecord::Migration.verbose = !ENV['SILENT']

ActiveRecord::Schema.define do
  # Add your schema here
  create_table :articles, force: true do |t|
    t.string :content
  end

  create_table :tags, force: true do |t|
    t.string :name
  end

  create_table :articles_tags, force: true do |t|
    t.integer :article_id
    t.integer :tag_id
  end
end

# create models
class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true
end
class Article < ApplicationRecord
  has_and_belongs_to_many :tags
end

class Tag < ApplicationRecord
  has_and_belongs_to_many :articles
end

# prepare rails app
require 'action_controller/railtie'
# require 'action_view/railtie'
require 'jsonapi-resources'

class ApplicationController < ActionController::Base
  include JSONAPI::ActsAsResourceController
end

# prepare jsonapi resources and controllers
class ArticlesController < ApplicationController
end

class BaseResource < JSONAPI::Resource
  abstract
end

class ArticleResource < BaseResource
  attribute :content
  has_many :tags
  filter :tags,
         verify: ->(values, _context) { values.flatten },
         apply: ->(records, values, _options) {
           raise 'qwe'
           records.joins(:tags).where(tags: values)
         }
end

class TagResource < BaseResource
  attribute :name
end

class TestApp < Rails::Application
  config.root = File.dirname(__FILE__)
  config.logger = ENV['SILENT'] ? NullLogger.new : Logger.new(STDOUT)
  Rails.logger = config.logger

  secrets.secret_token = 'secret_token'
  secrets.secret_key_base = 'secret_key_base'

  config.eager_load = false
end

# initialize app
Rails.application.initialize!

JSONAPI.configure do |config|
  config.json_key_format = :underscored_key
  config.route_format = :underscored_key
end

# draw routes
Rails.application.routes.draw do
  jsonapi_resources :articles
end

# prepare tests
require 'minitest/autorun'
require 'rack/test'

# Replace this with the code necessary to make your test fail.
class BugTest < Minitest::Test
  include Rack::Test::Methods

  def teardown
    Tag.destroy_all
    Article.destroy_all
  end

  def test_filter_tags
    another = Article.create! content: 'some'
    article = Article.create! content: 'test'
    tag1 = Tag.create! name: 'foo'
    tag2 = Tag.create! name: 'bar'
    article.tags = [tag1]
    another.tags = [tag2]
    request_params = { filter: { tags: [tag1.id] } }
    get "/articles?#{request_params.to_param}", nil, json_api_headers
    assert_equal 500, last_response.status
    # assert last_response.ok?
    # assert_equal 1, last_response_json[:data].size
    # assert_equal article.id.to_s, last_response_json[:data].first[:id]
  end

  private

  def last_response_json
    JSON.parse(last_response.body).deep_symbolize_keys
  end

  def json_api_headers
    {Accept: JSONAPI::MEDIA_TYPE, CONTENT_TYPE: JSONAPI::MEDIA_TYPE}.stringify_keys
  end

  def app
    Rails.application
  end
end

[thomassnielsen]

For me the issue turned out to be that I forgot to wrap the query param in filter[] on the client side.

[patrickemuller]

Someone knows if this is working on version 0.7.0 ??
The gem is interpreting as model's column the option filter

# records.reviewable is a scope inside the model
filter :reviewable, apply: ->(records, values, _options) {
  records.reviewable
}
...
PG::UndefinedColumn: ERROR:  column events.reviewable does not exist

I can't even binding.pry inside the filter, never stop the code there

[senid231]

@patrickemuller Can't reproduce your issue either

begin
  require 'bundler/inline'
  require 'bundler'
rescue LoadError => e
  STDERR.puts 'Bundler version 1.10 or later is required. Please update your Bundler'
  raise e
end

gemfile(true, ui: ENV['SILENT'] ? Bundler::UI::Silent.new : Bundler::UI::Shell.new) do
  source 'https://rubygems.org'

  git_source(:github) do |repo_name|
    repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?('/')
    "https://github.com/#{repo_name}.git"
  end

  if ENV['RAILS_VERSION']
    gem 'rails', ENV['RAILS_VERSION'], require: false
  else
    gem 'rails', require: false
  end


  if ENV['JSONAPI_RESOURCES_PG']
    gem 'pg'
  else
    gem 'sqlite3', platform: :mri
    # gem 'activerecord-jdbcsqlite3-adapter',
    #     git: 'https://github.com/jruby/activerecord-jdbc-adapter',
    #     branch: 'rails-5',
    #     platform: :jruby
  end

  if ENV['JSONAPI_RESOURCES_PATH']
    gem 'jsonapi-resources', path: ENV['JSONAPI_RESOURCES_PATH'], require: false
  elsif ENV['JSONAPI_RESOURCES_VERSION']
    gem 'jsonapi-resources', ENV['JSONAPI_RESOURCES_VERSION'], require: false
  else
    gem 'jsonapi-resources', github: 'cerebris/jsonapi-resources', require: false
  end

end

# prepare active_record database
require 'active_record'

class NullLogger < Logger
  def initialize(*_args)
  end

  def add(*_args, &_block)
  end
end

if ENV['JSONAPI_RESOURCES_PG']
  db_name = ENV['JSONAPI_RESOURCES_PG']
  system("psql -c 'DROP DATABASE IF EXISTS #{db_name};'")
  system("psql -c 'CREATE DATABASE #{db_name};'")
  ActiveRecord::Base.establish_connection(adapter: 'postgresql', database: db_name)
else
  ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
end

LOGGER = ENV['SILENT'] ? NullLogger.new : Logger.new(STDOUT)
ActiveRecord::Base.logger = LOGGER
ActiveRecord::Migration.verbose = !ENV['SILENT']

ActiveRecord::Schema.define do
  # Add your schema here
  create_table :articles, force: true do |t|
    t.string :content
    t.boolean :is_reviewable, null: false, default: false
  end
end

# create models
class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true
end
class Article < ApplicationRecord
  scope :reviewable, -> do
    where(is_reviewable: true)
  end
end

# prepare rails app
require 'action_controller/railtie'
# require 'action_view/railtie'
require 'jsonapi-resources'

class ApplicationController < ActionController::Base
  include JSONAPI::ActsAsResourceController
end

# prepare jsonapi resources and controllers
class ArticlesController < ApplicationController
end

class BaseResource < JSONAPI::Resource
  abstract
end

class ArticleResource < BaseResource
  attribute :content
  filter :reviewable, apply: ->(records, values, _options) {
    records.reviewable
  }
end

class TestApp < Rails::Application
  config.root = File.dirname(__FILE__)
  config.logger = LOGGER
  Rails.logger = config.logger

  secrets.secret_token = 'secret_token'
  secrets.secret_key_base = 'secret_key_base'

  config.eager_load = false
end

# initialize app
Rails.application.initialize!

JSONAPI.configure do |config|
  config.json_key_format = :underscored_key
  config.route_format = :underscored_key
end

# draw routes
Rails.application.routes.draw do
  jsonapi_resources :articles
end

# prepare tests
require 'minitest/autorun'
require 'rack/test'

# Replace this with the code necessary to make your test fail.
class BugTest < Minitest::Test
  include Rack::Test::Methods

  def teardown
    Article.delete_all
  end

  def test_filter_tags
    article = Article.create! content: 'some', is_reviewable: true
    Article.create! content: 'test', is_reviewable: false
    get_request '/articles',
                query: { filter: { reviewable: 'yeap' } },
                headers: json_api_headers

    assert last_response.ok?
    assert_equal 1, last_response_json[:data].size
    assert_equal article.id.to_s, last_response_json[:data].first[:id]
  end

  private

  def get_request(path, query: nil, body: nil, headers: nil)
    full_path = query.nil? ? path : "#{path}?#{query.to_param}"
    get(full_path, body, headers)
  end

  def last_response_json
    JSON.parse(last_response.body).deep_symbolize_keys
  end

  def json_api_headers
    {Accept: JSONAPI::MEDIA_TYPE, CONTENT_TYPE: JSONAPI::MEDIA_TYPE}.stringify_keys
  end

  def app
    Rails.application
  end
end

log

$ JSONAPI_RESOURCES_VERSION=0.7.0 RAILS_VERSION="~>4.2.0" ruby ./custom_bug_report_local_issue1147.rb
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 12.3.1
Using concurrent-ruby 1.0.5
Using minitest 5.11.3
Using thread_safe 0.3.6
Using builder 3.2.3
Using erubis 2.7.0
Using mini_portile2 2.3.0
Using crass 1.0.4
Using rack 1.6.10
Using mini_mime 1.0.0
Using arel 6.0.4
Using bundler 1.14.4
Using thor 0.20.0
Using sqlite3 1.3.13
Using i18n 0.9.5
Using tzinfo 1.2.5
Using nokogiri 1.8.3
Using rack-test 0.6.3
Using sprockets 3.7.2
Using mail 2.7.0
Using activesupport 4.2.10
Using loofah 2.2.2
Using rails-deprecated_sanitizer 1.0.3
Using globalid 0.4.1
Using activemodel 4.2.10
Using rails-html-sanitizer 1.0.4
Using rails-dom-testing 1.0.9
Using activejob 4.2.10
Using activerecord 4.2.10
Using actionview 4.2.10
Using actionpack 4.2.10
Using actionmailer 4.2.10
Using railties 4.2.10
Using sprockets-rails 3.2.1
Using rails 4.2.10
Using jsonapi-resources 0.7.0
-- create_table(:articles, {:force=>true})
D, [2018-06-23T11:23:02.320198 #12077] DEBUG -- :    (0.8ms)  CREATE TABLE "articles" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "content" varchar, "is_reviewable" boolean DEFAULT 'f' NOT NULL) 
   -> 0.0106s
Run options: --seed 26624

# Running:

D, [2018-06-23T11:23:03.029153 #12077] DEBUG -- :    (0.3ms)  begin transaction
D, [2018-06-23T11:23:03.033937 #12077] DEBUG -- :   SQL (0.2ms)  INSERT INTO "articles" ("content", "is_reviewable") VALUES (?, ?)  [["content", "some"], ["is_reviewable", "t"]]
D, [2018-06-23T11:23:03.034728 #12077] DEBUG -- :    (0.1ms)  commit transaction
D, [2018-06-23T11:23:03.035487 #12077] DEBUG -- :    (0.1ms)  begin transaction
D, [2018-06-23T11:23:03.036594 #12077] DEBUG -- :   SQL (0.2ms)  INSERT INTO "articles" ("content") VALUES (?)  [["content", "test"]]
D, [2018-06-23T11:23:03.037277 #12077] DEBUG -- :    (0.2ms)  commit transaction
D, [2018-06-23T11:23:03.057113 #12077] DEBUG -- : 
D, [2018-06-23T11:23:03.057220 #12077] DEBUG -- : 
I, [2018-06-23T11:23:03.057880 #12077]  INFO -- : Started GET "/articles?filter%5Breviewable%5D=yeap" for 127.0.0.1 at 2018-06-23 11:23:03 +0300
I, [2018-06-23T11:23:03.065714 #12077]  INFO -- : Processing by ArticlesController#index as HTML
I, [2018-06-23T11:23:03.065841 #12077]  INFO -- :   Parameters: {"filter"=>{"reviewable"=>"yeap"}}
D, [2018-06-23T11:23:03.074666 #12077] DEBUG -- :   Article Load (0.3ms)  SELECT "articles".* FROM "articles" WHERE "articles"."is_reviewable" = ?  ORDER BY "articles"."id" ASC  [["is_reviewable", "t"]]
I, [2018-06-23T11:23:03.080301 #12077]  INFO -- : Completed 200 OK in 14ms (Views: 0.8ms)
D, [2018-06-23T11:23:03.082403 #12077] DEBUG -- :   SQL (0.2ms)  DELETE FROM "articles"
.

Finished in 0.075761s, 13.1994 runs/s, 39.5981 assertions/s.

1 runs, 3 assertions, 0 failures, 0 errors, 0 skips

can you edit this bug report to reproduce the your problem?

[patrickemuller]

@senid231 I'm not filtering by some column that exists on database, I'm trying to create a "custom" filter. Is that possible?

[senid231]

@patrickemuller filter just applies a scope. You can do whatever you want there.
If your scope returns correct data when you call it in rails console then filter will do same thing

[patrickemuller]

@senid231 The problem is that in version 0.7.0 and 0.8.3 when you try to define a filter, and that filter is "custom", I always get an error saying "[filter name] is not a column on database"

[senid231]

@patrickemuller What do you mean by "customer" filter?
I understand it as a filter with a name that doesn't match any column name of a corresponding table and has :apply key with proc/lambda.
In bug report above I've created a such filter

class ArticleResource < BaseResource
  attribute :content
  filter :reviewable, apply: ->(records, values, _options) {
    records.reviewable
  }
end

There is no column named reviewable in articles table - only is_reviewable.
Can you reproduce your issue by editing bug report script that I've provided?

[patrickemuller]

@senid231 yes, that's correct, a filter that doesn't correspond to any column on model.
I'll send here the script as soon as possible 😄

[outsmartin]

I currently have the same issue on 0.9, the problem appears if I want to define a custom filter for a relationship that is defined.

has_one :branch
and then filter :branch with custom code <- this is never called

[patrickemuller]

@outsmartin I tried to create with random name like "blablabla" and it's never called too. The name of the filter is ALWAYS interpreted as some column of the model and throw an error column blablabla doesn't exist

[mizuti]

Confirmed I am able to replicate this issue when an association is the same name as the filter.

[senid231]

@mizuti could you create some bug report or snipper that reproduces the issue?

[mizuti]

@mizuti could you create some bug report or snipper that reproduces the issue?

Yes, let me test on an isolated environment and I'll fill out a bug report/snipper.

[senid231]

@mizuti thanks

[mupkoo]

@senid231 here is reproduction.

I try to overwrite the topic filter, by it never gets called.

begin
  require 'bundler/inline'
  require 'bundler'
rescue LoadError => e
  STDERR.puts 'Bundler version 1.10 or later is required. Please update your Bundler'
  raise e
end

gemfile(true, ui: ENV['SILENT'] ? Bundler::UI::Silent.new : Bundler::UI::Shell.new) do
  source 'https://rubygems.org'

  git_source(:github) do |repo_name|
    repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?('/')
    "https://github.com/#{repo_name}.git"
  end

  if ENV['RAILS_VERSION']
    gem 'rails', ENV['RAILS_VERSION'], require: false
  else
    gem 'rails', require: false
  end


  if ENV['JSONAPI_RESOURCES_PG']
    gem 'pg'
  else
    gem 'sqlite3', platform: :mri
    # gem 'activerecord-jdbcsqlite3-adapter',
    #     git: 'https://github.com/jruby/activerecord-jdbc-adapter',
    #     branch: 'rails-5',
    #     platform: :jruby
  end

  if ENV['JSONAPI_RESOURCES_PATH']
    gem 'jsonapi-resources', path: ENV['JSONAPI_RESOURCES_PATH'], require: false
  elsif ENV['JSONAPI_RESOURCES_VERSION']
    gem 'jsonapi-resources', ENV['JSONAPI_RESOURCES_VERSION'], require: false
  else
    gem 'jsonapi-resources', github: 'cerebris/jsonapi-resources', require: false
  end

end

# prepare active_record database
require 'active_record'

class NullLogger < Logger
  def initialize(*_args)
  end

  def add(*_args, &_block)
  end
end

if ENV['JSONAPI_RESOURCES_PG']
  db_name = ENV['JSONAPI_RESOURCES_PG']
  system("psql -c 'DROP DATABASE IF EXISTS #{db_name};'")
  system("psql -c 'CREATE DATABASE #{db_name};'")
  ActiveRecord::Base.establish_connection(adapter: 'postgresql', database: db_name)
else
  ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
end

LOGGER = ENV['SILENT'] ? NullLogger.new : Logger.new(STDOUT)
ActiveRecord::Base.logger = LOGGER
ActiveRecord::Migration.verbose = !ENV['SILENT']

ActiveRecord::Schema.define do
  # Add your schema here
  create_table :topics, force: true do |t|
    t.string :name
  end

  create_table :articles, force: true do |t|
    t.integer :topic_id
    t.string :content
    t.boolean :is_reviewable, null: false, default: false
  end
end

# create models
class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true
end

class Article < ApplicationRecord
  belongs_to :topic
end

class Topic < ApplicationRecord
  has_many :articles
end

# prepare rails app
require 'action_controller/railtie'
# require 'action_view/railtie'
require 'jsonapi-resources'

class ApplicationController < ActionController::Base
  include JSONAPI::ActsAsResourceController
end

# prepare jsonapi resources and controllers
class ArticlesController < ApplicationController
end

class BaseResource < JSONAPI::Resource
  abstract
end

class ArticleResource < BaseResource
  attribute :content

  relationship :topic, to: :one

  filter :topic, apply: -> (records, values, _options) {
    records.where(1)
  }
end

class TopicResource < BaseResource
  attribute :content

  relationship :articles, to: :many
end

class TestApp < Rails::Application
  config.root = File.dirname(__FILE__)
  config.logger = LOGGER
  Rails.logger = config.logger

  secrets.secret_token = 'secret_token'
  secrets.secret_key_base = 'secret_key_base'

  config.eager_load = false
end

# initialize app
Rails.application.initialize!

JSONAPI.configure do |config|
  config.json_key_format = :underscored_key
  config.route_format = :underscored_key
end

# draw routes
Rails.application.routes.draw do
  jsonapi_resources :articles
end

# prepare tests
require 'minitest/autorun'
require 'rack/test'

# Replace this with the code necessary to make your test fail.
class BugTest < Minitest::Test
  include Rack::Test::Methods

  def teardown
    Article.delete_all
  end

  def test_filter_tags
    topic = Topic.create! name: 'awesome'
    article = Article.create! content: 'some', is_reviewable: true, topic: topic
    article2 = Article.create! content: 'test', is_reviewable: false

    get_request '/articles',
                query: { filter: { topic: topic.id } },
                headers: json_api_headers

    assert last_response.ok?
    assert_equal 2, last_response_json[:data].size
    assert_equal article.id.to_s, last_response_json[:data].first[:id]
    assert_equal article2.id.to_s, last_response_json[:data].second[:id]
  end

  private

  def get_request(path, query: nil, body: nil, headers: nil)
    full_path = query.nil? ? path : "#{path}?#{query.to_param}"
    get(full_path, body, headers)
  end

  def last_response_json
    JSON.parse(last_response.body).deep_symbolize_keys
  end

  def json_api_headers
    {Accept: JSONAPI::MEDIA_TYPE, CONTENT_TYPE: JSONAPI::MEDIA_TYPE}.stringify_keys
  end

  def app
    Rails.application
  end
end

[mupkoo]

It looks like this is fixed on master, but the above test is failing on 0.9.5

JSONAPI_RESOURCES_VERSION=0.9.5 ruby filter-test.rb

[mupkoo]

So it looks like for this to work, you need to create two filters - country and countries.id. Here is an example that works

filter :country
filter :'countries.id', apply: -> (records, value, _options) {
  records.joins(:community).where(communities: { country_id: value })
}

Once I add this code to my resource, I can filter it by using filter[country]=1 and the apply block is being called

[patrickemuller]

So it looks like for this to work, you need to create two filters - country and countries.id. Here is an example that works

filter :country
filter :'countries.id', apply: -> (records, value, _options) {
  records.joins(:community).where(communities: { country_id: value })
}

Once I add this code to my resource, I can filter it by using filter[country]=1 and the apply block is being called

Odd....

[sebasjimenez10]

So it looks like for this to work, you need to create two filters - country and countries.id. Here is an example that works

filter :country
filter :'countries.id', apply: -> (records, value, _options) {
  records.joins(:community).where(communities: { country_id: value })
}

Once I add this code to my resource, I can filter it by using filter[country]=1 and the apply block is being called

Was able to reproduce on version 0.9.11

This is exactly what needs to be done in order to work around the issue. The problem is that if the filter is related to a has_many relationship, it will get added to the allowed filters just like that, without the .id part at the end. And when actually calling the apply_filter the filter key is calculated like this:

"#{_relationships[filter].table_name}.#{_relationships[filter].primary_key}"

which ends up being relationship.id. that's why the apply callable is bypassed.

I hope this helps 🙂 it _seems_ that in the latest version this issue is solved. But I have not confirmed this.