Core: Abode not working in 0.101

[arsaboo]

Home Assistant release with the issue: 0.101.0b0

Last working Home Assistant release (if known): 0.100.3

Operating environment (Hass.io/Docker/Windows/etc.):
Ubuntu Venv

Integration:
https://rc--home-assistant-docs.netlify.com/integrations/abode/

Description of problem:
With the config flow, the abode integration does not work. If I include the following in the config:

abode:
  username: !secret abode_username
  password: !secret abode_password

I get the error:

Log Details (ERROR)
Sat Oct 26 2019 15:45:33 GMT-0400 (Eastern Daylight Time)
Error handling request
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/real_ip.py", line 40, in real_ip_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/auth.py", line 136, in auth_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/view.py", line 122, in handle
    result = await result
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/config/config_entries.py", line 154, in get
    return await super().get(request, flow_id)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/data_entry_flow.py", line 78, in get
    result = await self._flow_mgr.async_configure(flow_id)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/data_entry_flow.py", line 86, in async_configure
    if cur_step.get("data_schema") is not None and user_input is not None:
AttributeError: 'NoneType' object has no attribute 'get'

If I remove the entry from configuration.yaml and use the UI to add Abode, I get the following error:

Log Details (ERROR)
Sat Oct 26 2019 15:49:20 GMT-0400 (Eastern Daylight Time)
Error handling request
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/real_ip.py", line 40, in real_ip_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/auth.py", line 136, in auth_middleware
    return await handler(request)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/view.py", line 122, in handle
    result = await result
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/config/config_entries.py", line 163, in post
    return await super().post(request, flow_id)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/http/data_validator.py", line 48, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/data_entry_flow.py", line 90, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/data_entry_flow.py", line 89, in async_configure
    result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/data_entry_flow.py", line 132, in _async_handle_step
    result: Dict = await getattr(flow, method)(user_input)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/abode/config_flow.py", line 47, in async_step_user
    await self.hass.async_add_executor_job(Abode, username, password, True)
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 84, in __init__
    self._load_cache()
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 478, in _load_cache
    self._save_cache()
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 483, in _save_cache
    UTILS.save_cache(self._cache, self._cache_path)
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/utils.py", line 11, in save_cache
    with open(filename, 'wb') as handle:
PermissionError: [Errno 13] Permission denied: './abode.pickle'

cc: @shred86

[shred86]

I'm not having any luck duplicating this. I just did a clean install of 0.101b2 in a virtual environment in MacOS and Ubuntu. Both appear to be working fine for me setting up Abode from the UI or configuration.yaml. I noticed you are running 0.101b0, but I don't think there's been any real changes since my commits were merged except one small PR someone made.

What version of Ubuntu is this and what is it running on?

[arsaboo]

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic

on an Intel NUC. I used b2, but I guess the issue started with b0 (which is when the PR was merged). .

[arsaboo]

What version of Python are you testing this with?

[shred86]

I setup HA in a venv using Python 3.7.4, 3.7.3 and 3.6.1. I am testing on MacOS and Ubuntu 19.04. Still unable to duplicate. Will keep looking into it.

Edit: Setup Hassio in a docker in an attempt to replicate this issue but everything appears to be working fine. I'm at a loss as to what's causing those errors as everything appears to be working fine on my end on four different setups.

[MisterWil]

@arsaboo I assume you've tried deleting the abode.pickle file in the root of your config?

I did make some changes to the utils file in the last couple version releases which I think are making their way into .101 but only for handling loading issues.

[arsaboo]

@MisterWil Yes, I tried deleting the .pickle files with .101, but I still get the same errors.

There are a few things that I tried. I used the old Abode code (from 100.3) with abodepy==0.16.6 and that worked in both .100.3 and .101. With this setup, I only see abode_cache.pickle file in my config directory in .101 (not sure if I am supposed to see the abode.pickle there as well). So, the problem should be with the new integration code.

Is there any way to log the full path and permissions where the pickles files are being?

[prestoja]

I am having the exact same issue and would also like that information. I will be happy to help troubleshoot if you need additional information from a second setup.

[MisterWil]

I also didn't have this issue but I think its a simple fix. After looking through some other components I suspect it has to deal with the integration now being set up with async event calls and the defined pickle cache file path being prepended with the relative current working directory ./ - something none of the other integrations do. I've removed it in #28389 and hopefully it'll get pulled into a patch release soon.

Edit: Unfortunately it missed todays 0.101.1 release. Maybe 0.101.2? :-)

[arsaboo]

I tried the fix in my local instance and I am still getting the same error:

Log Details (ERROR)
Thu Oct 31 2019 17:32:18 GMT-0400 (Eastern Daylight Time)
Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/data_entry_flow.py", line 73, in async_init
    return await self._async_handle_step(flow, flow.init_step, data)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/data_entry_flow.py", line 132, in _async_handle_step
    result: Dict = await getattr(flow, method)(user_input)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/abode/config_flow.py", line 79, in async_step_import
    return await self.async_step_user(import_config)
  File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/abode/config_flow.py", line 47, in async_step_user
    await self.hass.async_add_executor_job(Abode, username, password, True)
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 84, in __init__
    self._load_cache()
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 478, in _load_cache
    self._save_cache()
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/__init__.py", line 483, in _save_cache
    UTILS.save_cache(self._cache, self._cache_path)
  File "/srv/homeassistant/lib/python3.6/site-packages/abodepy/utils.py", line 11, in save_cache
    with open(filename, 'wb') as handle:
PermissionError: [Errno 13] Permission denied: './abode.pickle'

Note that this is part of the abodepy.
@prestoja Can you also try the fix as a custom_component and test?

[arsaboo]

@MisterWil I do see abodepy_cache.pickle with the current DEFAULT_CACHEDB = "./abodepy_cache.pickle", so not sure if that is the problem. It sounds like the problem is with the way abodepy is saving the pickle file. Ring has a slightly different way of saving the pickle file.

[prestoja]

@prestoja Can you also try the fix as a custom_component and test?

Doesn't work for me either. When trying to add via Integrations UI I get this:

2019-10-31 14:46:25 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/opt/hass/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/opt/hass/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/opt/hass/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/http/real_ip.py", line 40, in real_ip_middleware
    return await handler(request)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/http/auth.py", line 136, in auth_middleware
    return await handler(request)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/http/view.py", line 122, in handle
    result = await result
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/config/config_entries.py", line 163, in post
    return await super().post(request, flow_id)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/http/data_validator.py", line 48, in wrapper
    result = await method(view, request, *args, **kwargs)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/helpers/data_entry_flow.py", line 90, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/data_entry_flow.py", line 89, in async_configure
    result = await self._async_handle_step(flow, cur_step["step_id"], user_input)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/data_entry_flow.py", line 132, in _async_handle_step
    result: Dict = await getattr(flow, method)(user_input)
  File "/opt/hass/lib/python3.7/site-packages/homeassistant/components/abode/config_flow.py", line 47, in async_step_user
    await self.hass.async_add_executor_job(Abode, username, password, True)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/hass/lib/python3.7/site-packages/abodepy/__init__.py", line 84, in __init__
    self._load_cache()
  File "/opt/hass/lib/python3.7/site-packages/abodepy/__init__.py", line 478, in _load_cache
    self._save_cache()
  File "/opt/hass/lib/python3.7/site-packages/abodepy/__init__.py", line 483, in _save_cache
    UTILS.save_cache(self._cache, self._cache_path)
  File "/opt/hass/lib/python3.7/site-packages/abodepy/utils.py", line 11, in save_cache
    with open(filename, 'wb') as handle:
PermissionError: [Errno 13] Permission denied: './abode.pickle'

[arsaboo]

@prestoja I am glad, I am not the only one and the bug is reproducible :)

[MisterWil]

Alrighty, worked with @arsaboo and with his help we narrowed down the issue and I believe we've got this one kicked with the PR. 💯