Core: Mikrotik integration not tracking devices

[tjorim]

The problem

After HA 0.105.0 the Mikrotik device_tracker integration is not tracking my network devices anymore. However, there are no errors either! I enabled debugging for both the component and library used by it: no errors, librouteros even logs the found devices. Reverting back to HA 0.104.0 fixes the issue.

Environment

• Home Assistant release with the issue: 0.105.x
• Last working Home Assistant release (if known): 0.104.x
• Operating environment (Hass.io/Docker/Windows/etc.): Home Assistant on generic Linux (I mean the old Hass.io, not HA Core)
• Integration causing this issue: mikrotik
• Link to integration documentation on our website: https://rc.home-assistant.io/integrations/mikrotik/

Problem-relevant configuration.yaml

mikrotik:
  - host: !secret mikrotik_host
    username: !secret routeros_username
    password: !secret routeros_password
    login_method: plain

Traceback/Error logs

None

Additional information

Related PR's https://github.com/home-assistant/home-assistant/pull/30800 https://github.com/home-assistant/home-assistant/pull/30810
Related issue: #30745

[probot-home-assistant[bot]]

Hey there @engrbm87, mind taking a look at this issue as its been labeled with a integration (mikrotik) you are listed as a codeowner for? Thanks!

[balloob]

@springstan you think it could be because of your PR?

[springstan]

Yeah, probably. Unfortunately I cannot debug it myself. @engrbm87 could you please take a look at this? Thank you very much 😊

[onkelbeh]

Same problem here.
Worked without problems up to 0.104.3

First I already ran into a problem with Home Assistant 0.105.0b0, because I still was using the old token based authentication:

mikrotik:
  - host: 172.16.3.134
    username: homeassistant
    password: !secret mikrotik_pw
    ssl: false
    login_method: token

login_method seems to be ignored since 0.105.0b0:

2020-01-30 17:55:32 ERROR (SyncWorker_14) [homeassistant.components.mikrotik] Mikrotik 172.16.3.134 error not logged in
2020-01-30 17:55:32 INFO (MainThread) [homeassistant.setup] Setup of domain mikrotik took 0.1 seconds.
2020-01-30 17:55:32 ERROR (MainThread) [homeassistant.setup] Setup failed for mikrotik: Integration failed to initialize.

Anyway, I upgraded my Caps AC (RouterBOARD cAP Gi-5acD2nD) to RouterOS 6.45.8 and removed login_method: token

Now, the integraton loads again, still with Home Assistant 0.105.0b4

2020-02-01 10:18:24 INFO (SyncWorker_8) [homeassistant.loader] Loaded mikrotik from homeassistant.components.mikrotik
2020-02-01 10:18:25 INFO (MainThread) [homeassistant.setup] Setting up mikrotik
2020-02-01 10:18:25 INFO (SyncWorker_5) [homeassistant.components.mikrotik] Mikrotik Connected to MikroTik (172.16.3.134)
2020-02-01 10:18:25 INFO (MainThread) [homeassistant.setup] Setup of domain mikrotik took 0.4 seconds.

But no tracked devices from mikrotik show up, I have >30 items in Wireless -> Registration:

root@g18-hass:/var/log/homeassistant # hass-cli entity list tracker
ENTITY_ID                                                            NAME    DEVICE_ID                         PLATFORM    CONFIG_ENTRY_ID                   DISABLED_BY
device_tracker.android6                                              None    d1afca50cb5f49e9bd6affb59bc602d9  mobile_app  e4683b186b99476ebc4323c67ea70a70  None
device_tracker.iphone_von_michaela                                   None    0ed06111aff5445d9feb0cdfbf4cce98  mobile_app  ed87a4f14fe64835b3679a26046108a0  None
device_tracker.behsiphone_cc73dfff_6816_4bdb_8d15_48568ef1c0b9       None    7843f679200a4343aeb0c31db6fb59c6  owntracks   ebbf5e006cf74a01b926b772883d4d74  None
device_tracker.michaelasiphone_f612220b_8cf5_47bd_8ce5_e60bc1bf38f2  None    974376ea7ae04818b4f6cbc648bb945e  owntracks   ebbf5e006cf74a01b926b772883d4d74  None
root@g18-hass:/var/log/homeassistant #

Next, i added

logger:
  default: info
  logs:
    homeassistant.components.mikrotik: debug

but no additional lines are logged.

[tjorim]

Since it's not working currently and is already a breaking change (because of the changed login-method), can't we cherry-pick this PR? https://github.com/home-assistant/home-assistant/pull/27484
I know it was merged after the beta-cutoff (by a few hours) but to me it looks cumbersome if some users have 2 breaking changes for the same platform just 2 release after each other.

[onkelbeh]

Can't you just roll back to the old code? I would expect that only tested (and working) code should be merged, but if broken code has been already merged, just go back.
Once it has been fixed, it can be merged again in 0.106

[springstan]

I actually just did that with #31385 which will be in 0.105. 😅

[balloob]

@springstan what was the fix for this?

[tjorim]

I tested #27484 and works perfectly, config flow is smooth and devices are getting tracked again.
Imports my existing platform from configuration.yaml and got one error for login_method,
but that's already a breaking change with Home Assistant 0.105.0 beta anyway...

[springstan]

Sorry my bad, the PR I mentioned was completely unrelated 😅

[onkelbeh]

I just took the 'new' component from https://github.com/home-assistant/home-assistant/pull/27484, and can confirm it's working.
I'm on 0.105.0b4.
Seems configuration changed a bit, is there an updated documentation?

Had:

mikrotik:
  - host: 172.16.3.134
    username: homeassistant
    password: !secret mikrotik_pw
    ssl: false
    method: wireless
    # login_method: token

got:

[ssl] is an invalid option for [mikrotik]. Check: mikrotik->mikrotik->0->ssl.
[method] is an invalid option for [mikrotik].

Comment both out, now it got the devices back again. Thanks.

[tjorim]

@onkelbeh Not sure about the options you mentioned, but this is the accompanied documentation change: https://github.com/home-assistant/home-assistant.io/pull/11927

[balloob]

Alright there seems to be confusion about mikrotik and there is a breaking change scheduled for 106. To make it smoother I'll pull in the breaking change into 105. Please test beta when it's pushed.

[balloob]

Please try out 0.105.0b6 which has #27484 included.

[onkelbeh]

Good morning,

made a quick shot (not more time) and can confirm the code coming with 0.105.0b6 is adding tracked devices to my entities. These look a bit different compared to those with 0.104.3. But I am running out of time now. This ticket already was closed anyway, so I'll take a deeper look in some hours.

Thanks a lot, guys.

[fahmietalife]

The integration doesn't work for new release. No error in log. Reverting back to 0.104.3

[veno77]

The new integration constantly changes the state of my devices from away to home and back.
My configuration:
mikrotik:

• name: Mikrotik Router
  host: 192.168.1.1
  username: homeassistant
  password: password
  #verify_ssl: true
  #arp_ping: true
  force_dhcp: true
  #detection_time: 30

I tried changing the detection_time and enable arp_ping with no success. The lease time for these devices is set to 10 minutes in my mikrotik router. This was working flawlessly before 0.105.1
Also ssl is not working, followed the exact steps to setup a certificate on the mikrotik router, but continue to get in the logs of Hassio - connection refused for api-ssl.

[engrbm87]

Hello @veno77 , I see you are using force_dhcp. Can you enable arp_ping and then change the logging level for mikrotik component to debug so that you can see if the devices that are changing to away are timing out or not.

[engrbm87]

I tested api-ssl and it is working. I have firmware 6.46 installed.

[veno77]

Hi @engrbm87, I enabled arp_ping and debug on the mikrotik but could not see anything related to api in the log. I suppose you are forcing the mikrotik to do arp_ping via the api, right?

[veno77]

I am running latest stable, 6.46.3 on the mikrotik router.

[fahmietalife]

Below is my error on log

Fri Feb 07 2020 16:30:46 GMT+0800 (Singapore Standard Time)
Error setting up entry Mikrotik for mikrotik
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 215, in async_setup
hass, self
File "/usr/src/homeassistant/homeassistant/components/mikrotik/__init__.py", line 68, in async_setup_entry
if not await hub.async_setup():
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 375, in async_setup
await self.hass.async_add_executor_job(self._mk_data.update)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 272, in update
self.update_devices()
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 168, in update_devices
wireless_devices = self.get_list_from_interface(interface)
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 151, in get_list_from_interface
result = list(self.command(MIKROTIK_SERVICES[interface]))
File "/usr/local/lib/python3.7/site-packages/librouteros/api.py", line 28, in __call__
yield from self.readResponse()
File "/usr/local/lib/python3.7/site-packages/librouteros/api.py", line 67, in readResponse
raise MultiTrapError(*traps)
librouteros.exceptions.MultiTrapError: no such command or directory (caps-man), no such command prefix

Below is my configuration yaml

mikrotik:

• host: 192.168.0.1
  username:
  password:
  arp_ping: true
  force_dhcp: true

device_tracker:

[engrbm87]

The above error says setup error for mikrotik. But you mentioned previously that it was setup and showing away then home. can you please clarify what is the current status exactly?

[veno77]

Hi @engrbm87 , you are answering to somebody else who posted after me.
Here is what I found regarding api_ssl. Current my configuration:
mikrotik:

• name: Mikrotik Router
  host: 192.168.1.1
  username: homeassistant
  password: pass
  verify_ssl: true
  arp_ping: true
  force_dhcp: true
  detection_time: 300

But I found after restart of hassio. It tries connecting via api. So it looks like it is working...
But when I disable api on the mikrotik and reboot again it says it could not connect.
verify_ssl looks like not working...

Also I suspect for some reason hassio is not reading the exact configuration from configuration.yaml
Because when I go:
Configuration->Integrations->Mikrotik and select settings
"Consider Home Interval" was set to 30 , though I set it to 300 from the configuration.yaml
I changed it to 300 via the interface and now the things look more stable.... and the devices not change from away to home and back every 30 seconds....

[engrbm87]

Hello @veno77 , sorry for the mix up. If the integration is already imported you can't update it by editing configuration.yaml, it will import only the first time. You can change the options later using the UI.
changing to ssl means you need to remove the integration and create it again using the UI, or remove it using the UI and restart HA to import again from configuration.yaml

[fahmietalife]

Below is my error on log

Fri Feb 07 2020 16:30:46 GMT+0800 (Singapore Standard Time)
Error setting up entry Mikrotik for mikrotik
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 215, in async_setup
hass, self
File "/usr/src/homeassistant/homeassistant/components/mikrotik/init.py", line 68, in async_setup_entry
if not await hub.async_setup():
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 375, in async_setup
await self.hass.async_add_executor_job(self._mk_data.update)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 272, in update
self.update_devices()
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 168, in update_devices
wireless_devices = self.get_list_from_interface(interface)
File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 151, in get_list_from_interface
result = list(self.command(MIKROTIK_SERVICES[interface]))
File "/usr/local/lib/python3.7/site-packages/librouteros/api.py", line 28, in *call
yield from self.readResponse()
File "/usr/local/lib/python3.7/site-packages/librouteros/api.py", line 67, in readResponse
raise MultiTrapError(*traps)
librouteros.exceptions.MultiTrapError: no such command or directory (caps-man), no such command prefix

Below is my configuration yaml

mikrotik:

• host: 192.168.0.1
  username:
  password:
  arp_ping: true
  force_dhcp: true

device_tracker:

so can anyone help me with this ? i already tried to reconfigure both using yaml and UI still having same error log as above. not sure what's happening here.

[engrbm87]

Hi @fahmietalife , I was debugging and found the issue. I will submit a PR to fix. This is happening if the router doesn't support wireless.

[fahmietalife]

Hi @fahmietalife , I was debugging and found the issue. I will submit a PR to fix. This is happening if the router doesn't support wireless.

thank you so much, yeah my Mikrotik 750G r3 doesnt support Wifi. Thanks i'll wait for the fix.

[veno77]

@engrbm87 Probably this should be explained in the Release Notes...
I deleted and reloaded from configuration.yaml .
I confirm ssl is working. My config is below:
mikrotik:

• name: Mikrotik Router
  host: 192.168.1.1
  username: homeassistant
  password: password
  verify_ssl: true
  port: 8729
  arp_ping: true
  force_dhcp: true
  Now the question is how can I add pictures to the devices, like before with known_devices.yaml ?

[fahmietalife]

@engrbm87 Probably this should be explained in the Release Notes...
I deleted and reloaded from configuration.yaml .
I confirm ssl is working. My config is below:
mikrotik:

• name: Mikrotik Router
  host: 192.168.1.1
  username: homeassistant
  password: password
  verify_ssl: true
  port: 8729
  arp_ping: true
  force_dhcp: true
  Now the question is how can I add pictures to the devices, like before with known_devices.yaml ?

Should be under entities customization UI.

[kernehed]

I got the same problem, I think @engrbm87 PR will fix it.

[veno77]

@engrbm87 Probably this should be explained in the Release Notes...
I deleted and reloaded from configuration.yaml .
I confirm ssl is working. My config is below:
mikrotik:

• name: Mikrotik Router
  host: 192.168.1.1
  username: homeassistant
  password: password
  verify_ssl: true
  port: 8729
  arp_ping: true
  force_dhcp: true
  Now the question is how can I add pictures to the devices, like before with known_devices.yaml ?

Should be under entities customization UI.

I could not find such option to add picture...

[onkelbeh]

Had some problems during the 0.105.0 beta phase here, too. But, since 0.105.0b6 the new components works for me. Have 2 CAP AC (RouterBOARD cAP Gi-5acD2nD), no capsman.
Found out that the old token based auth is not working anymore, so I had to upgrade the AP's to v6.45.8, then I removed the configuration from my Yaml and used the config flow in integrations. I have a user group with read, web and api allowed (as described in the old manual).
I am tracking two mobile phones (which re-appear with a trailing _2).
Everything is now working like it should.

[veno77]

anyone have an idea if possible to add picture to entity ?

[kernehed]

anyone have an idea if possible to add picture to entity ?

I do it in customize.yaml

device_tracker.my_iphone:
  friendly_name: MyName
  entity_picture: /local/MyName.jpg

[balloob]

No discussion on closed PRs please. Use forums or chat for support questions.