Incubator-superset: Sunburst Chart failure when using IS NOT NULL filter (regression from 0.34 to 0.35)

[elukey]

I just updated to 0.35 via pypi, and I noticed that one Sunburst Chart in a dashboard was broken, reporting this failure:

Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 418, in get_df_payload
    df = self.get_df(query_obj)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/viz.py", line 203, in get_df
    self.results = self.datasource.query(query_obj)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1377, in query
    query_str = self.get_query_str(client=client, query_obj=query_obj, phase=2)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 984, in get_query_str
    return self.run_query(client=client, phase=phase, **query_obj)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/connectors/druid/models.py", line 1355, in run_query
    client.groupby(**qry)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/pydruid/client.py", line 307, in groupby
    query = self.query_builder.groupby(kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/pydruid/query.py", line 367, in groupby
    return self.build_query(query_type, args)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/pydruid/query.py", line 295, in build_query
    query_dict[key] = Filter.build_filter(val)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/pydruid/utils/filters.py", line 119, in build_filter
    filter = filter_obj.filter['filter']
AttributeError: 'bool' object has no attribute 'filter'

Expected results

The chart correctly visualized as in 0.34.

Actual results

See stacktrace above.

How to reproduce the bug

1. Create a Sunburst char with Druid data
2. add a filter IS NOT NULL

Environment

(please complete the following information):

• superset version: 0.35
• python version: python 3.7.3
• node.js version: v10.17.0
• npm version: 6.11.3

Checklist

Make sure these boxes are checked before submitting your issue - thank you!

• [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
• [x] I have reproduced the issue with at least the latest released version of superset.
• [x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

I tried to debug it a little bit and tracked down the code responsible for this. In superset/connectors/druid/models.py, before client.groupby(**qry) I logged qry:

Nov 27 14:30:08 an-tool1005 superset[3824]: 2019-11-27 14:30:08,350:INFO:root:Query: {'datasource': 'edits_hourly', 'dimensions': ['user_is_anonymous', 'project', 'is_reverted', 'user_groups'], 'aggre
gations': OrderedDict([('SUM(edit_count)', {'fieldName': 'edit_count', 'fieldNames': ['edit_count'], 'type': 'longSum', 'name': 'SUM(edit_count)'})]), 'granularity': 'all', 'post_aggregations': Ordere
dDict(), 'intervals': '2018-07-01T00:00:00+00:00/2019-07-01T00:00:00+00:00', 'filter': True, 'limit_spec': {'type': 'default', 'limit': 50000, 'columns': [{'dimension': 'SUM(edit_count)', 'direction':
 'descending'}]}}

Focus is on 'filter': True, that later on in pydruid's build_query is not accepted. Backtracking a little bit to the following line, and then logging filter and filters:

filters = DruidDatasource.get_filters(filter, self.num_cols, columns_dict)

filters -> [{'col': 'user_is_anonymous', 'op': 'IS NOT NULL', 'val': ''}]
filter -> True

Eventually ending up in this bit of the get_filters function:

            elif op == "IS NOT NULL":
                cond = Dimension(col) is not None

            if filters:
                filters = Filter(type="and", fields=[cond, filters])
            else:
                filters = cond

It seems that cond is returned as simple bool:True in the last else. I didn't see any recent commit so I am not sure what changed, but with 0.34 I didn't see any issue.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.99. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[elukey]

As a side note I tried to downgrade pydruid to 0.34's version but the error seems still there (tried to remove a variable from the table).

From the git blame I cannot really pin-point this problem to a specific commit in the druid model.py file, but the signature of the get_filter function seems suggesting that the bool value returned is not appropriate:

def get_filters(cls, raw_filters, num_cols, columns_dict) -> "Filter"

[elukey]

@john-bodley I can confirm that the fix works on my testing environment :)