Incubator-superset: Cache warmup does not work

[EvaSDK]

The cache warmup task does not work as it tries to access the webserver without a scheme.

In commit d65b039219c9825ad50ec03ad73a1638710c73c9, use of requests was removed and replaced by urlopen. However at the same time, the get_url function was modified to not include HTTP scheme anymore. While this could make sense if the service is running behind a reverse proxy or if SSL is used, this is not currently configurable

Expected results

Cache warmup task is performed.

Actual results

Cache warmup fails with a traceback.

Screenshots

[2019-10-28 15:00:00,015: INFO/ForkPoolWorker-6] cache-warmup[4345da24-b272-4af8-a8bc-2c1ee924191c]: Loading strategy
[2019-10-28 15:00:00,015: INFO/ForkPoolWorker-6] cache-warmup[4345da24-b272-4af8-a8bc-2c1ee924191c]: Loading TopNDashboardsStrategy
[2019-10-28 15:00:00,017: INFO/ForkPoolWorker-6] cache-warmup[4345da24-b272-4af8-a8bc-2c1ee924191c]: Success!
[2019-10-28 15:00:00,031: INFO/ForkPoolWorker-6] cache-warmup[4345da24-b272-4af8-a8bc-2c1ee924191c]: Fetching 0.0.0.0:8088/superset/explore/?form_data=%7B%22slice_id%22%3A%201%7D
[2019-10-28 15:00:00,031: ERROR/ForkPoolWorker-6] cache-warmup[4345da24-b272-4af8-a8bc-2c1ee924191c]: Error warming up cache!
Traceback (most recent call last):
  File "/home/superset/superset/tasks/cache.py", line 285, in cache_warmup
    request.urlopen(url)
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.6/urllib/request.py", line 549, in _open
    'unknown_open', req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1388, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: 0.0.0.0>

How to reproduce the bug

1. Setup cache warming task in Superset configuration
2. Wait for the task to trigger
3. See error

Environment

• superset version: Installation from git commit 03b35b3
• python version: Python 3.6.9
• node.js version: v10.16.3
• npm version: 6.9.0

Checklist

• [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
• [x] I have reproduced the issue with at least the latest released version of superset.
• [x] I have checked the issue tracker for the same issue and I haven't found one similar.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.93. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[willbarrett]

@EvaSDK it looks like you've determined the exact problem. Please feel free to submit a PR with the solution!

[EvaSDK]

@willbarrett, is using requests a good option and it is better to stick to python 3 builtin modules ?

[willbarrett]

@EvaSDK I think it would be better to stick with python 3 built-in modules.

[mingqianye]

As a temporary workaround, I had to set SUPERSET_WEBSERVER_ADDRESS = 'http://0.0.0.0' in my superset_config.py.

[betodealmeida]

👀