Core: Smart Things Integration Deletes Itself on Reboot

[aihngel]

The problem

On reboot, smart things integration re-discovers itself thereby deleting existing integration config. This forces the user to re-install smart app blah blah blah. This just started happening yesterday.

Environment

• Home Assistant Core release with the issue: Home Assistant 0.108.3
• Last working Home Assistant Core release (if known): Home Assistant 0.107.x
• Operating environment (Home Assistant/Supervised/Docker/venv): Rasbian
• Integration causing this issue: SmartThings
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/smartthings/

Problem-relevant configuration.yaml

Traceback/Error logs

Additional information

[probot-home-assistant[bot]]

Hey there @andrewsayre, mind taking a look at this issue as its been labeled with a integration (smartthings) you are listed as a codeowner for? Thanks!

[andrewsayre]

Can you post your logs when this happens? That will help narrow down what's causing the issue.

[renoye]

I have seen the same issue.
I was using 0.107.x and smart things was integrated and ran fine. I believed after I reboot yesterday. Notification showed me that HA discovered smartthings and old integration was auto-removed.
Then I tried to integrate again. Failed. I tried to upgrade to the latest 0.108.x and tried to integrate again. Still Failed.

This is the error of failed integration. I have to say it again that it was working fine before. Now I can not even integrate.

log shows some exception

Log Details (ERROR)
Logger: homeassistant.components.smartthings.config_flow
Source: components/smartthings/smartapp.py:136
Integration: smartthings (documentation, issues)
First occurred: 2:01:48 AM (1 occurrences)
Last logged: 2:01:48 AM

API error setting up the SmartApp: {'error': {'code': 'ConstraintViolationError', 'message': 'The request is malformed.', 'details': [{'code': 'TargetTimeoutError', 'target': 'https://renoye.duckdns.org:8123/api/webhook/220d020f6fd5d71445cbb88125c7bf2a09c3867422bacef51b66f8a9bd679539', 'message': 'Upstream target timed out', 'details': []}]}}
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/smartthings/config_flow.py", line 107, in async_step_user
app, client = await create_app(self.hass, self.api)
File "/usr/src/homeassistant/homeassistant/components/smartthings/smartapp.py", line 136, in create_app
app, client = await api.create_app(app)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 99, in create_app
entity = await self._service.create_app(app.to_data())
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 172, in create_app
return await self.post(API_APPS, data)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 387, in post
data=data)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 365, in request
data=data)
pysmartthings.errors.APIResponseError: Unprocessable Entity (422): {"error": {"code": "ConstraintViolationError", "message": "The request is malformed.", "details": [{"code": "TargetTimeoutError", "target": "https://renoye.duckdns.org:8123/api/webhook/220d020f6fd5d71445cbb88125c7bf2a09c3867422bacef51b66f8a9bd679539", "message": "Upstream target timed out", "details": []}]}}

Please let me know what was the issue.

Thanks

[aihngel]

I'll see if I can pull some logs. I didn't see any when this happened.

[danielgoepp]

I believe this is happening to me too, but I'm not getting an error. I wasn't running 107 before, I'm new to HA, and just getting this setup for the first time on 0.108.3. I had an issue, but it was because there was a null smartthing config. I deleted it, and was able to walk through the setup without errors. I checked the file, and the cloudhook_url was set correctly (I use nabu casa). I then restarted, and the cloudhook_url was null. Also, I see no integration with SmartThings in the UI. Happy to try anything to assist here, I have nothing to lose :) Also, I turned on debug mode logging, but am not seeing anything there either.

[danielgoepp]

This doesn't help I'm sure, but just as another data point, I've restarted again, and my configuration remained. I'm a little nervous to keep configuring though, if there is a possibility my whole setup will get wiped out.

[danielgoepp]

Well, a few restarts later, and it did happen to me... I restarted several times, so I thought I was good to go. I entered everything in, and then bam, it was all gone :( Here is a clip from the logs:

2020-04-13 09:20:54 ERROR (MainThread) [homeassistant.components.smartthings] Unable to setup configuration entry 'Home' - please reconfigure the integration
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/smartthings/__init__.py", line 104, in async_setup_entry
scenes = await async_get_entry_scenes(entry, api)
File "/usr/src/homeassistant/homeassistant/components/smartthings/__init__.py", line 183, in async_get_entry_scenes
return await api.scenes(location_id=entry.data[CONF_LOCATION_ID])
File "/usr/local/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 189, in scenes
resp = await self._service.get_scenes(params)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 314, in get_scenes
return await self.get_items(API_SCENES, params=params)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 375, in get_items
'get', self._api_base + resource, params, None)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 366, in request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url='https://api.smartthings.com/v1/scenes?locationId=b630f629-4a31-4c88-9894-024596eb6479

2020-04-13 09:20:54 ERROR (MainThread) [homeassistant.config_entries] Error calling entry remove callback Home for smartthings
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 324, in async_remove
hass, self
File "/usr/src/homeassistant/homeassistant/components/smartthings/__init__.py", line 215, in async_remove_entry
await api.delete_installed_app(installed_app_id)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 156, in delete_installed_app
result = await self._service.delete_installed_app(installed_app_id)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 255, in delete_installed_app
API_INSTALLEDAPP.format(installed_app_id=installed_app_id))
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 396, in delete
return await self.request('delete', self._api_base + resource, params)
File "/usr/local/lib/python3.7/site-packages/pysmartthings/api.py", line 366, in request
resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 946, in raise_for_status
headers=self.headers)
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url='https://api.smartthings.com/v1/installedapps/5352d6a2-7a4d-4bd0-9024-2b08033633aa

[aihngel]

I'm glad to hear it's not just me. This is the only thing I'm having issues with with 108. It just keeps blowing away my smart things integration, which thankfully is only my robot vacuum...

Please let me know if there's anything I can test to help. I don't see any relevant logs for when mine reset the config.

[andrewsayre]

It appears the ST API is periodically returning a HTTP 401 (unauthorized), which the integration assumes to mean the personal access token is no longer valid and removes the integration. This isn't an HA issue, though I can update the code not to remove the integration, but it still won't work until the a non 401 response is received. I know it's irritating, but the short-term solution is to setup the integration again when it gets removed.

[danielgoepp]

Unfortunate. I agree it's not a problem with HA specifically, and also that just gracefully handing the 401, and leaving things perhaps just disabled until a proper auth happens. Wiping out the configuration on an auth failure is a bit harsh :)

[andrewsayre]

Agreed, I'll make the change, but expect it may not come until the next release because I have another update that removes the need to use the mobile app to setup the integration that is a larger change.

[aihngel]

Both would be awesome. Thank you for jumping on this so quickly.

[renoye]

I tried to integrate today. It worked. I failed more than 10 times last night. I guess it is the smartthings api issue.. Thanks.

[toe7685]

Ive had this problem since 4/11 so just wait you think? my automatons seem to be the only thing taking a hit waiting

[danielgoepp]

Not having to use the app would be fantastic too! I run the new app, so I had to reinstall the old classic to do this, and that was a little bothersome, but whatever.

[maxi1134]

Agreed, I'll make the change, but expect it may not come until the next release

Any way to do the changes ourselves before the next release?

[VinistoisR]

same thing happening to me. it just all goes away. Same process as OP, add the new key, click through on the app, everything re-populates. Restart HA, bam, its gone, and I have a notification "discovered new devices on your network"

[olivierjag2]

same here

[olivierjag2]

do you have update when he fix problem

[andrewsayre]

Let me clarify a few things:

• If you are getting a HTTP 400 or 401 issue related to a refresh token, the only way to mitigate the issue is to completely remove the integration, clean-up your ST account using this utility, and setup the integration from scratch. Please wait 15 minutes before starting over again to give the caches time to reset.
• The refresh token gets out of sync if the HA process is not gracefully shutdown and there was a pending write. Do not kill or force terminate the HA process.

Check your logs. If you are experiencing other errors, it's a different issue.

[fryjr82]

Let me clarify a few things:

• If you are getting a HTTP 400 or 401 issue related to a refresh token, the only way to mitigate the issue is to completely remove the integration, clean-up your ST account using this utility, and setup the integration from scratch. Please wait 15 minutes before starting over again to give the caches time to reset.
• The refresh token gets out of sync if the HA process is not gracefully shutdown and there was a pending write. Do not kill or force terminate the HA process.

Check your logs. If you are experiencing other errors, it's a different issue.

Thank you very much @andrewsayre that worked great! I did have an extra HA app in my ST and could never get rid of it. Your clean up fixed that and after a reboot my ST integration is still there! Appreciate your help!

[aihngel]

I'm just super excited about not having to use the dumb ST app anymore. Can't wait for this update!

[aihngel]

Any ETA on a new release? Thanks. (No-app release). I'm testing the new version of this today.

[mwunderling]

The latest update to remove the app dependency has worked great in my experience. Any chance of getting an issue update on this ticket? Any confirmation that it is being worked on or planned would be great! For me currently, every HA reboot (beyond same day) or patch upgrade requires a complete smartthings integration reconfiguration.

[aihngel]

Mine isn't deleting itself anymore. I deleted it and readded it after the last update and now it's behaving. Has the "no app" update been pushed?

[StephaneRod]

Running Hassio, I lost the smartthings token a few days ago.
Being in a container, I can't delete Smartthings using the python code
I tried to use the command line with "ha addons remove smartthings"
but it didn't work

it seems there no way to remove and reinstall smarthings using hassio ?

[andrewsayre]

I've been investigating this issue but haven't been able to reproduce it as described. The SmartThings integration obtains a new refresh token upon startup of HA and saves that in the config entry. If the HA process is terminated or killed before the save is complete, the new token isn't persisted and the integration removes the config entry and asks you to reconfigure it (aka "deletes itself") the next time it starts up. Outside that scenario, I haven't been able to reproduce the issue through a restart or upgrade and I've done this countless times through docker as well as a venv. I'm wondering if the following are possibly the issues, which would cause the problem:
1) You aren't persisting the configuration directory in Docker. When the container is restarted, the changes to the config entries (including the new token) are lost. Solution: Ensure your configuration directory is a persisted/passed through volume.
2) You're overwriting the configuration directory on an upgrade with an older/previous version. The new token is then lost. Solution: Don't overwrite your configuration directory, or ensure it's backed up after the HA process has excited gracefully.

[mwunderling]

Yes, your first paragraph fits my issue to a T, but items 1&2 I don't consider applicable to my situation. What I've been experiencing is that restarting my docker container eventually just hangs the HA startup process (I have a separate thread on the community forum trying to understand what's going on). Basically, HA gets virtually all the way through the restarting but then doesn't output the log entry saying ...it is "Starting Home Assistant". At that point I have to shut down the container and restart it again. Sometimes this take 3-6 start/stop cycles before HA finally comes up. And yes, when it does, the smartthings integration needs to be reconfigured every time. Having to do this a few time a day when making updates/tweaks in HA then just become untenable.

But yes, this is sounding like one problem causing another problem. Fix the 1st and you shouldn't see the 2nd....

[danielgoepp]

If I follow this correctly, on start up, if previously configured, there will be an existing token, and it will be used. However, if it has expired, you'll get a 401, and need to generate a new one. This is fine and normal procedure by most APIs. If a 401 is received however, it wipes out the config. This has been my experience anyway. It has nothing to do with kill HA before it can be saved, or persisting configs, both of those work fine for me. The problem is just that after a certain period of time or reboots (with no change in config), the entire thing will just be wiped out. I thought there was an answer for this already, to just not remove the existing config on 401. Try to get a new token, and i that works, just move on, all good. Did I miss something? I honestly stopped playing with this because it was too annoying to just put all the work in to reconfigure, only to have it wiped out. That was frustrating. I would love to have this working though, so if this might get fixed, I would be happy to retest it.

[andrewsayre]

@danielgoepp A few corrections: The 401 occurs when trying to obtain the new access token / refresh token pair. When a 401 is received at that step, there is no recovery path, as the integration no longer has a valid credential to use (retrying it won't fix anything), so it initiates a new config flow for the user to setup the integration again--which is the convention for integrations in HA when the auth is no longer valid. The ST API works differently than other APIs in that access tokens have an extremely short TTL (5 min) and refresh tokens are single use only, so keys get rotated very frequently.

I personally run HA in a docker and do my development in a virtual environment and have ST setup in both. This issue has never occurred for me and I restart HA very frequently--especially my venv. There appears to be other factors/issues that cause the token to get out of sync.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.