Core: Nissan Leaf component breaks in 0.118.0b3

[Skeletorjus]

The problem

After upgrading to 0.118.0b3 the Nissan Leaf integration fails to load with the following error:

An unknown error occurred while connecting to Nissan: <class 'pycarwings2.pycarwings2.CarwingsError'>

No sensors/switches are then added to Home Assistant.

Reverting back to 0.118.0b2 fixes the problem.

Environment

Version | 0.118.0b3
-- | --
Installation Type | Home Assistant Supervised
Development | false
Supervisor | true
Docker | true
Virtual Environment | false
Python Version | 3.8.6
Operating System Family | Linux
Operating System Version | 4.19.0-9-amd64
CPU Architecture | x86_64
Timezone | Europe/Oslo

Host Operating System | Debian GNU/Linux 10 (buster)
-- | --
Update Channel | beta
Supervisor Version | 2020.11.0
Docker Version | 19.03.13
Healthy | true
Supported | true
Supervisor API | ok
Version API | ok

• Home Assistant Core release with the issue:
  0.118.0b3
• Last working Home Assistant Core release (if known):
  0.118.0b2
• Operating environment (OS/Container/Supervised/Core):
  Home Assistant Supervised
• Integration causing this issue:
  Nissan Leaf
• Link to integration documentation on our website:
  https://www.home-assistant.io/integrations/nissan_leaf/

Problem-relevant configuration.yaml

nissan_leaf:
  username: !secret leaf_user
  password: !secret leaf_pass
  region: "NE"
  update_interval:
    hours: 24
  update_interval_charging:
    hours: 24
  update_interval_climate:
    minutes: 5

Traceback/Error logs

2020-11-17 11:08:06 INFO (MainThread) [homeassistant.setup] Setting up nissan_leaf
2020-11-17 11:08:06 DEBUG (SyncWorker_7) [homeassistant.components.nissan_leaf] Logging into You+Nissan...
2020-11-17 11:08:09 ERROR (SyncWorker_7) [homeassistant.components.nissan_leaf] An unknown error occurred while connecting to Nissan: <class 'pycarwings2.pycarwings2.CarwingsError'>
2020-11-17 11:08:09 INFO (MainThread) [homeassistant.setup] Setup of domain nissan_leaf took 2.3 seconds

Additional information

[probot-home-assistant[bot]]

nissan_leaf documentation
nissan_leaf source
_{^{(message by IssueLinks)}}

[tonysprenk]

Same issue here on 0.118.0...

[BenWoodford]

Add pycarwings2 to your logger config and set it to debug

https://www.home-assistant.io/integrations/logger/

[tonysprenk]

2020-11-19 12:34:16 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/InitialApp_v2.php
2020-11-19 12:34:16 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] params: {
"RegionCode": "NE",
"custom_sessionid": "",
"initial_app_str": "9s5rfKVuMrT03RtzajWNcA",
"lg": "en-US"
}
2020-11-19 12:34:18 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 12:34:18 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":200,"message":"success","baseprm":"88dSp7wWnV3bvv9Z88zEwg"}'
2020-11-19 12:34:18 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/UserLoginRequest.php
2020-11-19 12:34:18 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] params: {
"Password": "xxxxxxxxx",
"RegionCode": "NE",
"UserId": "xxxxxx",
"custom_sessionid": "",
"initial_app_str": "9s5rfKVuMrT03RtzajWNcA"
}
2020-11-19 12:34:19 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 12:34:19 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":404,"message":"INVALID PARAMS"}'
2020-11-19 12:34:19 ERROR (SyncWorker_5) [pycarwings2.pycarwings2] carwings error INVALID PARAMS: 404
2020-11-19 12:34:19 ERROR (SyncWorker_5) [homeassistant.components.nissan_leaf] An unknown error occurred while connecting to Nissan: <class 'pycarwings2.pycarwings2.CarwingsError'>
2020-11-19 12:34:20 WARNING (MainThread) [pyatv.mrp.player_state] Received UpdateClientMessage for unknown player com.apple.TVMusic
2020-11-19 12:34:20 WARNING (MainThread) [pyatv.mrp.player_state] Received UpdateClientMessage for unknown player com.apple.TVAirPlay
2020-11-19 12:34:49 ERROR (SyncWorker_18) [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update
2020-11-19 12:34:49 ERROR (SyncWorker_33) [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update
2020-11-19 12:34:49 ERROR (SyncWorker_30) [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update

[tonysprenk]

The only thing I can see is that the password (which I've x-ed out above) does not match the actual password, not sure if that's expected behaviour.

[Skeletorjus]

Add pycarwings2 to your logger config and set it to debug

https://www.home-assistant.io/integrations/logger/

Here you go:

2020-11-19 13:42:47 INFO (MainThread) [homeassistant.setup] Setting up nissan_leaf
2020-11-19 13:42:47 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/InitialApp_v2.php
2020-11-19 13:42:47 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] params: {
   "RegionCode": "NE",
   "custom_sessionid": "",
   "initial_app_str": "9s5rfKVuMrT03RtzajWNcA",
   "lg": "en-US"
}
2020-11-19 13:42:48 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 13:42:48 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":200,"message":"success","baseprm":"88dSp7wWnV3bvv9Z88zEwg"}'
2020-11-19 13:42:48 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/UserLoginRequest.php
2020-11-19 13:42:48 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] params: {
   "Password": "MyPassword",
   "RegionCode": "NE",
   "UserId": "MyUser",
   "custom_sessionid": "",
   "initial_app_str": "9s5rfKVuMrT03RtzajWNcA"
}
2020-11-19 13:42:50 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 13:42:50 DEBUG (SyncWorker_5) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":404,"message":"INVALID PARAMS"}'
2020-11-19 13:42:50 ERROR (SyncWorker_5) [pycarwings2.pycarwings2] carwings error INVALID PARAMS: 404
2020-11-19 13:42:50 ERROR (SyncWorker_5) [homeassistant.components.nissan_leaf] An unknown error occurred while connecting to Nissan: <class 'pycarwings2.pycarwings2.CarwingsError'>
2020-11-19 13:42:50 INFO (MainThread) [homeassistant.setup] Setup of domain nissan_leaf took 2.3 seconds

This is on 0.118.0

For reference, here is the log from 0.118.0b2 (working):

2020-11-19 13:55:00 INFO (MainThread) [homeassistant.setup] Setting up nissan_leaf
2020-11-19 13:55:00 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/InitialApp_v2.php
2020-11-19 13:55:00 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] params: {
   "RegionCode": "NE",
   "custom_sessionid": "",
   "initial_app_str": "9s5rfKVuMrT03RtzajWNcA",
   "lg": "en-US"
}
2020-11-19 13:55:01 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 13:55:01 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":200,"message":"success","baseprm":"88dSp7wWnV3bvv9Z88zEwg"}'
2020-11-19 13:55:01 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] invoking carwings API: https://gdcportalgw.its-mo.com/api_v190426_NE/gdc/UserLoginRequest.php
2020-11-19 13:55:01 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] params: {
   "Password": "MyPassword",
   "RegionCode": "NE",
   "UserId": "MyUser",
   "custom_sessionid": "",
   "initial_app_str": "9s5rfKVuMrT03RtzajWNcA"
}
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] Response HTTP Status Code: 200
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] Response HTTP Response Body: b'{"status":200,"vehicleInfo":[{"vin":"SJNXXXXXXXXXXXXXX","nickname":
"CarName","charger20066":"true","telematicsEnabled":"true","custom_sessionid":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=="}],"vehicle":{"profile":{"vin":"SJNXXXXXXXXXXXXXX","gdcUserId":"","gdcPassword":"","encAuthToken":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","dcmId":"XXXXXXXXXXXX","status":"true","statusDate":"2019\\/05\\/16 22:00","nickname":"CarName"}},"EncAuthToken":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","CustomerInfo":{"UserId":"MyUser","Language":"no_NO","Timezone":"Europe\\/Paris","RegionCode":"NE","OwnerId":"XXXXXXXXXX","EMailAddress":"[email protected]","Nickname":"CarName","Country":"NO","VehicleImage":"\\/content\\/language\\/default\\/images\\/img\\/ph_car.jpg","UserVehicleBoundDurationSec":"946771200","VehicleInfo":{"VIN":"SJNXXXXXXXXXXXXXX","DCMID":"XXXXXXXXXXXX","SIMID":"XXXXXXXXXXXXXXXXXXXX","NAVIID":"XXXXXXXXXXXX","EncryptedNAVIID":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","MSN":"XXXXXXXXXXXXXXX","LastVehicleLoginTime":"","UserVehicleBoundTime":"2020-10-18T10:30:46Z","LastDCMUseTime":"2020\\/11\\/19 12:39","NonaviFlg":"false","CarName":"LEAF","CarImage":"carimg6.png"}},"UserInfoRevisionNo":"1","ngTapUpdatebtn":"300000","timeoutUpdateAnime":"300000","G1Lw":"5","G1Li":"2","G1Lt":"20","G1Uw":"15","G1Ui":"2","G1Ut":"20","G2Lw":"15","G2Li":"2","G2Lt":"20","G2Uw":"15","G2Ui":"2","G2Ut":"20","resultKey":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}'
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] gdc_user_id: 
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] dcm_id: XXXXXXXXXXXX
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] tz: Europe/Paris
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] language: no_NO
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] vin: SJNXXXXXXXXXXXXXX
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] nickname: CarName
2020-11-19 13:55:06 DEBUG (SyncWorker_12) [pycarwings2.pycarwings2] created leaf SJNXXXXXXXXXXXXXX/CarName
2020-11-19 13:55:06 WARNING (SyncWorker_12) [homeassistant.components.nissan_leaf] WARNING: This may poll your Leaf too often, and drain the 12V battery.  If you drain your cars 12V battery it WILL NOT START as the drive train battery won't connect. Don't set the intervals too low

Sorry for sanitizing all the info, but I don't know what's sensitive or not.

@tonysprenk The password is hidden behind some kind of obfuscation, so I think it's expected that the password displayed in your logs is not equal to the one in your settings.

I noticed that pycarwings2 depends on the requests-package which was bumped in 0.118.0b3, but this could of course be completely unrelated to this issue.

[andbad]

Same issue posted here: https://github.com/home-assistant/core/issues/43194

[BenWoodford]

This could be some weird change in the requests library then, however I wouldn’t rule out the Nissan servers being flakey. If you can sometimes get it to login without changing then it’s probably then, as their servers are downright terrible

[Skeletorjus]

This could be some weird change in the requests library then, however I wouldn’t rule out the Nissan servers being flakey. If you can sometimes get it to login without changing then it’s probably then, as their servers are downright terrible

I'm sure it's not Nissan's servers as this can be consistently reproduced by switching between 0.118.0b2 and newer versions. It's also possible to check Nissan's servers simply by testing with the official app. If it's not working there it's won't work in HA and vice versa.

[andbad]

Questo potrebbe essere uno strano cambiamento nella libreria delle richieste, tuttavia non escluderei che i server Nissan siano instabili. Se a volte riesci a farlo accedere senza cambiare, probabilmente è allora, poiché i loro server sono decisamente terribili

I just made a lot of test, is not a Nissan server problem (see also the other issue and the thread: https://community.home-assistant.io/t/nissan-leaf-component-s-platform/38663/407).

By(t)e

[BenWoodford]

Just thought I’d check, I’ve had it not connect one restart and then be fine the next

[tonysprenk]

Still not working for me. Maybe it’s a regional thing? I’m in Europe. The app is working fine for me btw.

[marinopalsson]

+1 Broken for me after upgrade to 0.118 while Nissan service is working

[Torstein80]

+1 Broken here to. Rolling back to 0.117xx versions and all is fine. My log appears to be similar to the above issue

2020-11-21 15:30:13 ERROR (SyncWorker_1) [pycarwings2.pycarwings2] carwings error INVALID PARAMS: 404
2020-11-21 15:30:13 ERROR (SyncWorker_1) [homeassistant.components.nissan_leaf] An unknown error occurred while connecting to Nissan:

System Health

Home Assistant Core Integration

version: 0.118.2
installation_type: Home Assistant OS
dev: false
hassio: true
docker: true
virtualenv: false
python_version: 3.8.6
os_name: Linux
os_version: 5.4.77
arch: x86_64
timezone: Europe/Oslo

Home Assistant Cloud

logged_in: false
can_reach_cert_server: ok
can_reach_cloud_auth: ok
can_reach_cloud: ok

Hass.io

host_os: HassOS 4.17
update_channel: stable
supervisor_version: 2020.11.0
docker_version: 19.03.12
disk_total: 48.5 GB
disk_used: 9.4 GB
healthy: true
supported: true
board: ova
supervisor_api: ok
version_api: ok
installed_addons: Samba share (9.2.0), Visual Studio Code (2.8.2), InfluxDB (3.7.9), Grafana (5.3.5), Mosquitto broker (5.1), Node-RED (7.2.10), Terminal & SSH (8.9.1)

Lovelace

dashboards: 2
mode: auto-gen
resources: 0

[TomTorger]

Broken here as well: 0.118.2

[andbad]

Did anyone try this method?

[Torstein80]

Hi, and thank you @andbad for the suggestion.

1. Downgrade to previous version
2. Upgrade to newest
3. Restart HA
4. test integration

I followed the procedure above, but unfortunately no success. Here is my experience:
I used "ha core update --version=x.xxx.x" to change between versions
In my setup, it appears to have broken between version 0.118.0b2 and 0.118.0b3. Perhaps someone with more programming skills than me can find a change between those versions that may cause this?

0.117.6 ok
0.118.0b0 ok
0.118.0b1 ok
0.118.0b2 ok
0.118.0b3 Broken
0.118.0 Broken
0.118.1 Broken
0.118.2 Broken

[remyderuysscher]

I concur, same issue with 0.118.0 and up.

[oskretc]

Same issue on my setup, 0.118.x is broken

[andbad]

See that: https://github.com/home-assistant/core/issues/43194#issuecomment-732280642

[Skeletorjus]

I tried digging some more into this issue.
A quick test of pycarwings2 with the get-leaf-info.py (found under examples in the repo) is working fine.
There has been no version changes of the integration between 0.118.0b2 and 0.118.0b3.
I also tried debug logging of urllib3 to see if there were some differences, but there's not.
The second request to the Nissan servers simply returns {"status":404,"message":"INVALID PARAMS"} in the newer versions, like what you get if you just try to open the URL with no, or invalid, POST data.

As previously stated, https://github.com/home-assistant/core/pull/43279 is the only PR I can see that has any relation to this integration, as the requests package is used by pycarwings2, but I have no idea how, and _if_, it could be related.

[Kalkran]

By simply changing the URL to a netcat listener, it seems the newer version has a User-Agent header while the old one does not send one.

New:

POST /InitialApp_v2.php HTTP/1.1
Host: 172.18.0.1:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded
User-Agent: python-urllib3/1.26.2

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=^C

Old:

POST /InitialApp_v2.php HTTP/1.1
Host: 10.0.0.50:8888
Accept-Encoding: identity
Content-Length: 79
Content-Type: application/x-www-form-urlencoded

RegionCode=NE&lg=en-US&initial_app_str=9s5rfKVuMrT03RtzajWNcA&custom_sessionid=

It seems that setting the User-Agent to None in older versions of the requests library would get rid of the header altogether, but in newer versions it defaults to using the python-urllib3/(version) header.

I might be way off track and will have to test some more to figure out if this even is the issue.

[prilly-dev]

It would probably be the case here what you have discoverd. It might be better to set the user agent to the same as the official app if it has one.

[Kalkran]

Yeah, monkeypatching urllib3 to not add the default User-Agent lets me use the API pycarwings2 library properly. I'm not sure about what the original User-Agent would be from the app, quickly grepping through the Android app decompiled source-code does not show any references to User.Agent (. as wildcard) so it may not send one.

[prilly-dev]

Ok it seems urllib3 1.26.x has some interesting changes:

https://pypi.org/project/urllib3/

Default user-agent is now added

Added default User-Agent header to every request (Pull #1750)
Added urllib3.util.SKIP_HEADER for skipping User-Agent, Accept-Encoding, and Host headers from being automatically emitted with requests (Pull #2018)

[Kalkran]

I already sent out a pull request to the pycarwings2 as there didn't seem to be a way to suppress the header (setting it to None, which worked before still added the header), however setting it to an empty string "" will still suppress the header. I've closed my pull request, but I'll set up another one. Let's hope it gets accepted soon.

[prilly-dev]

Maybe take this upstream? Maybe get pyrequest to adopt the skip_header from urllib3? Currently there is no easy way to disable header in pyrequest, you need to build the hole header structure and leave out user-agent to disable it. Its seems easy to change user-agent with header=.

The best is to probably invesrigate what user-agents is accepted and set up a list and rotate it.

Anyway good work, thank you!

[Kalkran]

As mentioned in my previous note, it seems that simply setting the User-Agent in the headers= parameter to an empty string ("") instead of None will prevent urllib from adding its own and does not send it out along with the request.

I run Home Assistant in a docker container and by editing the /usr/local/lib/python3.8/site-packages/pycarwings2/pycarwings2.py file and adding the headers={"User-Agent": ""} parameter on line 121, the integration starts working again. This change is also what I (tried to) put into a pull request to filcole/pycarwings2 which seems to be the used version.

[prilly-dev]

I agree with you, this is a good fix at the moment.

[andbad]

I change the line 121 with:
req = Request('POST', url=BASE_URL + endpoint, headers={"User-Agent": ""}, data=params).prepare()

and seems to work.

By(t)e