Pyatv: Not able to connect to Apple TV in different subnet

[ITTV-tools]

Describe the bug
I can not add ATV to my Home Assistant.
My network is separated in multiple VLANs. The ATV is in VLAN 1 and Home Assistant in VLAN 2.
Both can talk to each other.
I use a MDNS repeater to use for example Homekit in other VLANs.

In older versions of the integration, HA could see the Apple TV and I got the text field to enter the PIN code.
But the PIN was always wrong.

To Reproduce
Place ATV in different subnet. (I guess)
Expected behavior

System Setup (please complete the following information):

• OS: Debian
• Python: 3.7.3
• pyatv: 0.7.0
  
  
  
  • Apple TV: Apple TV 4K tvOS 13.4.6
  
  

Additional context
Traceback (most recent call last):
File "/home/homeassistant/.homeassistant/custom_components/apple_tv/config_flow.py", line 144, in async_step_user
await self.async_find_device()
File "/home/homeassistant/.homeassistant/custom_components/apple_tv/config_flow.py", line 218, in async_find_device
self.unique_id, self.hass.loop, cache=self.scan_result
File "/home/homeassistant/.homeassistant/custom_components/apple_tv/config_flow.py", line 76, in device_scan
scan_result = await scan(loop, hosts=hosts, timeout=timeout)
File "/srv/homeassistant/lib/python3.7/site-packages/pyatv/__init__.py", line 49, in scan
devices = (await scanner.discover(timeout)).values()
File "/srv/homeassistant/lib/python3.7/site-packages/pyatv/support/scan.py", line 154, in discover
*[self._get_services(host, timeout) for host in self.hosts]
File "/srv/homeassistant/lib/python3.7/site-packages/pyatv/support/scan.py", line 167, in _get_services
self.loop, str(host), ALL_SERVICES, port=port, timeout=timeout
File "/srv/homeassistant/lib/python3.7/site-packages/pyatv/support/mdns.py", line 448, in unicast
f"address {address} is not in any local subnet"
pyatv.exceptions.NonLocalSubnetError: address 192.168.178.49 is not in any local subnet

[postlund]

This might be a bug with the new scanning implementation, I have seen some issues myself that I will look in to. Adding via unicast (specifying IP address) will not work between VLANs. That's just how zeroconf is designed.

If you could not add because of invalid PIN, that's a separate issue we can deal with once scanning works. Can you provide some output from atvremote --debug scan?

[ITTV-tools]

2020-07-15 16:06:58 DEBUG: Sending multicast DNS request to 224.0.0.251:5353 (Data=35ff012000040000000000000b5f6170706c6574762d7632045f746370056c6f63616c0000ff80010b5f746f7563682d61626c65045f746370056c6f63616c0000ff80010e5f6d6564696172656d6f74657476045f746370056c6f63616c0000ff8001085f616972706c6179045f746370056c6f63616c0000ff8001)
2020-07-15 16:06:59 DEBUG: Sending multicast DNS request to 224.0.0.251:5353 (Data=35ff012000040000000000000b5f6170706c6574762d7632045f746370056c6f63616c0000ff80010b5f746f7563682d61626c65045f746370056c6f63616c0000ff80010e5f6d6564696172656d6f74657476045f746370056c6f63616c0000ff8001085f616972706c6179045f746370056c6f63616c0000ff8001)
2020-07-15 16:07:00 DEBUG: Sending multicast DNS request to 224.0.0.251:5353 (Data=35ff012000040000000000000b5f6170706c6574762d7632045f746370056c6f63616c0000ff80010b5f746f7563682d61626c65045f746370056c6f63616c0000ff80010e5f6d6564696172656d6f74657476045f746370056c6f63616c0000ff8001085f616972706c6179045f746370056c6f63616c0000ff8001)
Scan Results

[postlund]

Ok, so no response at all. I guess other services are discovered in Home Assistant? Are you running in docker?

[ITTV-tools]

Yes other services work. I use python virtual environment

[postlund]

Ok, so, I have a theory. Been looking at the code for python-zeroconf to try to understand what is happening. The problem I have is that proxies set up by atvproxy are not picked up by a scan. I use python-zeroconf to publish services for these proxies. What I can see is that requests are received just fine, but no responses are sent. The reason for this is that the source port is not one of 53 (DNS) or 5353 (MDNS), which is a pretty strict constraint (but probably according to standard). Since I just send a unicast request to a multicast address, a random source port are used so this is reasonable. When implementing a full zeroconf client, it would also listen on port 5353 in which case it would make sense to use 5353 as source port. The Apple TV's does not seem to care about this and treat my requests as unicast requests and answer. But as I said, python-zeroconf does not and your MDNS responder probably does not either. This is likely the reason why it doesn't work. So, I guess I will have to investigate how much work is needed to fix this.

Sorry about the long post. Here, have a 🌮

[postlund]

Did some testing last night and I think my theory might be right. It seemed to work when binding to port 5353, but there will be problems if another zeroconf client is doing the same. It's no problem binding to the same address and port, assuming the other client did the same, but received messages will be load balanced between the processes. So pyatv will miss some responses and receive other responses. It will only work properly if one zeroconf client is used at the same time. Will have to think about this a bit...

[loopway]

I've got the same problem, although running in a docker container (NOT host network mode).

pyatv.exceptions.NonLocalSubnetError: address X.X.X.X is not in any local subnet

Used to work with older versions though.

[postlund]

What are you entering when adding the device? Not sure exactly how I ended up doing it, but I think you need to specify either device name or identifier when using a repeat. IP address will not work. Since current scanning is somewhat broken, it might not work still. Looking into that.

[loopway]

I'm pretty sure I used to add it with entering the IP address. Now it doesn't work anymore, neither does it with the device name. (probably as expected, since I don't use a repeater)

[giricgoyal]

So i am also having similar issue except that my Apple TV (4K) and home assistant server are on the same subnet. I tried adding the Apple TV via name (for which it said "Device could not be found on network, please try again."). But when I tried adding through the IP address it gave me the following error:

Unexpected exception
Traceback (most recent call last):
  File "/config/custom_components/apple_tv/config_flow.py", line 144, in async_step_user
    await self.async_find_device()
  File "/config/custom_components/apple_tv/config_flow.py", line 218, in async_find_device
    self.unique_id, self.hass.loop, cache=self.scan_result
  File "/config/custom_components/apple_tv/config_flow.py", line 76, in device_scan
    scan_result = await scan(loop, hosts=hosts, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/pyatv/__init__.py", line 49, in scan
    devices = (await scanner.discover(timeout)).values()
  File "/usr/local/lib/python3.7/site-packages/pyatv/support/scan.py", line 154, in discover
    *[self._get_services(host, timeout) for host in self.hosts]
  File "/usr/local/lib/python3.7/site-packages/pyatv/support/scan.py", line 167, in _get_services
    self.loop, str(host), ALL_SERVICES, port=port, timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/pyatv/support/mdns.py", line 448, in unicast
    f"address {address} is not in any local subnet"
pyatv.exceptions.NonLocalSubnetError: address 192.168.1.247 is not in any local subnet
Home Assistant has started!

[postlund]

How are you running your Home Assistant instances? The edge case I can think of where it would potentially work is if the network is in NAT mode. When the Apple TV receives the request it must come from an IP address on the same network or it will be ignored.

[giricgoyal]

I am running HA in docker on windows and used --net=host while bringing up the container.

[giricgoyal]

Just checked the docker and it was in "bridge" mode. So I recreated the docker container using host net config, but now I now I cant seem to access HA itself. And since host network is not supported on windows I would need to go back to bridge mode.
So is there a way to integrate apple tv with HA in bridged mode?

[postlund]

Right, so host mode should of course work as long as the host is on the same network as the Apple TV. But it definitely uses NAT as means for network access (it has another IP address inside the container which is translated to the host address outside of the container). This is a problematic edge case where my check will fail. I might have to remove the check to remedy this.

[postlund]

Removing local subnet check is probably the safest bet until I get scanning to work as expected. I can try to make that happen and make a release later tonight.

[giricgoyal]

That would be great @postlund . I can try again once you release the update. Thanks

[postlund]

@giricgoyal I'll let you know when it's done. Created #775 for the change.

[postlund]

@giricgoyal I have not forgotten about you, just wanted to spend some additional time fiddling with the mdns implementation. I think I have hacky prototype that solves the base problem. Will need some time to make it presentable, but hopefully within the next couple of days. Time to get some sleep.

[radhus]

@postlund is it possible for you to create a release with #775 in the mean time, and include it in a version of the hass component? I have applied that change locally and it works for me, but it would be nice to have it included automatically so I don't have to do it again 😄

[postlund]

@radhus There will be a release very soon 😊 Just had a short break ("vacation") from pyatv, will take up development again albeit with a lower pace. I'm just finishing up #777 (will merge that tonight) and after that I will try to sort out #780. That is what I'm planning for next release, should be able to finish that today or tomorrow. After that I will continue updating the integration and also re-submit PR so I can finally have it updated in Home Assistant. Been putting that of intentionally due to bugs.

[radhus]

@postlund great! No stress :) Let me know if there's anything special to test and I'll help out :)

[postlund]

@radhus Just pushed an update to the component, give it a go and let me know if it works 😊

[ITTV-tools]

With the newest version I get no connection at all

[postlund]

@ITTV-tools Interesting that you get that. Please enable logging and provide the pyatv log, perhaps I can implement that.

[geekofweek]

I see the same behavior

[radhus]

@postlund upgrade worked, then removed and added the apple-tv again (with IP address) and it worked fine. 👍

[postlund]

@geekofweek Do you have the possibility to reproduce this with PYATV_BINARY_MAX_LINE=10000 atvremote --debug -s <ip> playing? I started to create a fix in #802, but I'm not sure that fixes the problem. I'm afraid that it removes the exception but no device would still be found. So complete logs are needed for debugging.

[geekofweek]

Home Assistant sits on 192.168.101.0/24, it has all ports opened to the AppleTVs through the firewall so nothing should be blocking it.

# PYATV_BINARY_MAX_LINE=10000 atvremote --debug -s 192.168.100.11 playing
2020-08-17 15:39:58 DEBUG: Knocking at port 3689 on 192.168.100.11
2020-08-17 15:39:58 DEBUG: Knocking at port 7000 on 192.168.100.11
2020-08-17 15:39:58 DEBUG: Knocking at port 49152 on 192.168.100.11
2020-08-17 15:39:58 DEBUG: Knocking at port 32498 on 192.168.100.11
2020-08-17 15:40:00 DEBUG: Knocking at port 3689 on 192.168.100.11
2020-08-17 15:40:00 DEBUG: Knocking at port 7000 on 192.168.100.11
2020-08-17 15:40:00 DEBUG: Knocking at port 49152 on 192.168.100.11
2020-08-17 15:40:00 DEBUG: Knocking at port 32498 on 192.168.100.11
2020-08-17 15:40:01 DEBUG: Cleaning up after UDNS request
2020-08-17 15:40:01 DEBUG: Cleaning up after UDNS request
2020-08-17 15:40:01 ERROR: Could not find any Apple TV on current network

[postlund]

Yeah, that won't work. The Apple TV will drop packets from other subnets. That's part of the specification (although it's possible to be lax about it, that's how Apple Implemented it). Only way for this to work is to use an mdns relayer and rely on regular scanning (not unicast like was used here).

[geekofweek]

I have mdns forwarding configured, which I assume is how it works on the previous versions.

[postlund]

Yeah, most likely. In that case you can run the same command but switch the -s part to --id <id> and use an identifier from atvremote scan. That should hopefully replicate the issue.

[geekofweek]

On version 0.7.1:

atvremote scan provides no results

scanning the host directly provides no results:

atvremote --scan-hosts 192.168.100.11 scan

When attempting to connect with the ID, it reports back other devices, mostly Sonos and other AirPlay stuff:

# PYATV_BINARY_MAX_LINE=10000 atvremote --debug --id <ID> playing
2020-08-17 16:19:50 DEBUG: Binding on *:5353
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('0.0.0.0', 5353)>
2020-08-17 16:19:50 DEBUG: Binding on 127.0.0.1:0
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('127.0.0.1', 46576)>
2020-08-17 16:19:50 DEBUG: Binding on 192.168.101.240:0
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('192.168.101.240', 41518)>
2020-08-17 16:19:53 DEBUG: Auto-discovered Surround Sound at 192.168.101.201:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Basement Move at 192.168.101.213:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Surround Sound Receiver  at 192.168.101.10:5000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Office Move at 192.168.101.212:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 ERROR: Could not find any Apple TV on current network

On version 0.6.1 (Last working version with VLANS):

*atvremote scan finds all devices

can connect with both IP and ID

[jjlawren]

Willl an mDNS repeater/reflector be necessary to discover across subnets? Can a subset of the ports listed above be used for scanning/discovery?

[postlund]

On version 0.7.1:

atvremote scan provides no results

scanning the host directly provides no results:

atvremote --scan-hosts 192.168.100.11 scan

When attempting to connect with the ID, it reports back other devices, mostly Sonos and other AirPlay stuff:

# PYATV_BINARY_MAX_LINE=10000 atvremote --debug --id <ID> playing
2020-08-17 16:19:50 DEBUG: Binding on *:5353
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('0.0.0.0', 5353)>
2020-08-17 16:19:50 DEBUG: Binding on 127.0.0.1:0
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('127.0.0.1', 46576)>
2020-08-17 16:19:50 DEBUG: Binding on 192.168.101.240:0
2020-08-17 16:19:50 DEBUG: Adding socket <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('192.168.101.240', 41518)>
2020-08-17 16:19:53 DEBUG: Auto-discovered Surround Sound at 192.168.101.201:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Basement Move at 192.168.101.213:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Surround Sound Receiver  at 192.168.101.10:5000 (Protocol.AirPlay)
2020-08-17 16:19:53 DEBUG: Auto-discovered Office Move at 192.168.101.212:7000 (Protocol.AirPlay)
2020-08-17 16:19:53 ERROR: Could not find any Apple TV on current network

On version 0.6.1 (Last working version with VLANS):

*_atvremote scan_ finds all devices

can connect with both IP and ID

Huh, that's strange. How is the mdns repeater set up? Where is it running? Looks like only devices on the same subnet as you run atvremote responds. Which repeater are you using btw? Maybe I'm doing something that isn't compliant.

[postlund]

Willl an mDNS repeater/reflector be necessary to discover across subnets? Can a subset of the ports listed above be used for scanning/discovery?

It is necessary simply because the Apple TV will drop the packets if they arrive from a different subnet. So it's not possible to work around with port forwarding or things alike.

[jjlawren]

Sorry, my question was if discovery could work with only repeating/reflecting some of the ports above instead of all.

[postlund]

Sorry, my question was if discovery could work with only repeating/reflecting some of the ports above instead of all.

MDNS uses port 5353 UDP via multicast. You could try that.

[postlund]

But I feel that I don't really know what you mean by port forwarding in this context. From and to where are you forwarding?

[geekofweek]

@postlund I'm using Protocol-Independent Multicast (PIM) that is implemented with the Debian Wheezy package running on my router (Unifi USG).

It works for other use cases: device discovery across VLANs, AirPlay across VLANs, etc. It worked fine on the previous pyatv versions with discovery when doing a scan finding all of the devices. It was the best implementation of MDNS forwarding I've found thus far that seemed to work for all use cases I had.

[postlund]

Hmm, interesting. That should work of course. Can you re-run the command but also add --mdns-debug? It will produce a lot of output probably. Shouldn't be much sensitive information in there, but I would still suggest that you email the log to me rather than upload it here.

[geekofweek]

@postlund sent

[postlund]

@geekofweek I can't see any answers from devices in the other VLAN, which I kinda guessed. Would it be too much of a hassle for you to verify if the MDNS requests actually reach the other VLAN? Maybe run wireshark, filter on port 5353 UDP (or "mdns" probably works too). Not sure, but maybe it's possible to look at some logs in PIM? I'm quite interested in where the traffic stops.

[geekofweek]

@postlund I just tested with another device on the VLAN and it can discover the mdns devices from all the other networks, so I know that it forwards the traffic. Might be difficult to wireshark it from within the Home Assistant container. It also would discover them with mdns on all the previous versions being on separate VLANs so it's curious as to what it's looking for that would terminate or not find it. I have the firewall between HA and the Apple TVs wide open so I don't think it's port related.

[postlund]

@geekofweek I re-wrote the multicast scanning code from scratch, replacing the zeroconf library I used before. So there are likely a million reasons to why it wouldn't work 😉 I have tried to replicate much from zeroconf, but it's likely not 100%. This is mainly the reason why I would like to see the traffic, to hopefully see where I'm failing.

The easiest way to capture traffic inside the Home Assiatant docker container is probably to add tcpdump (apk add tcpdump) and dump with tcpdump -w output.pcap mdns and then open the pcap file in wireshark.

I'm trying to think of all possible ways to debug this and I'm ears to other ideas. Not really sure what is going on, but I certainly would like to know.

[geekofweek]

@postlund took me a few to realize after forgetting the container is based on alpine how to use apk to install the tcpdump package, so that's on me for using apt-get so often.

Sent a few captures under a few scenarios. The only thing that really stuck out is that it doesn't seem to traverse past it's own subnet / VLAN when doing a scan or connecting by ID. It seemed to stay fairly isolated. I'll look them over some more and see if anything pops out.

[jjlawren]

I'm still unable to add my Apple TV on another subnet with the latest release with mDNS traversing subnets correctly. Is more information needed to understand what's happening?

[janegilring]

I’m running Home Assistant in a Docker container which is behind a NAT-network (172.x.x.x while the real network is on 192.x.x.x). Since the Docker container is running on Windows, host networking is not supported. This haven`t been a huge blocker, as most integrations supports manually adding devices to the configuration.

I am able to scan my Apple TV from within the container (172.x.x.x network):

atvremote --scan-hosts 192.168.1.231 scan
Scan Results
========================================
       Name: Hjemmekino
   Model/SW: 4K tvOS 14.2 build 18K57
    Address: 192.168.1.231
        MAC: 90:DD:5D:CA:0F:9B
 Deep Sleep: False
Identifiers:
 - 12BDC2D1-0E81-4444-A448-DAC2B49E9DC5
 - 90:DD:5D:CA:0F:9B
Services:
 - Protocol: MRP, Port: 49152, Credentials: None
 - Protocol: AirPlay, Port: 7000, Credentials: None

In order to be able to perform a pair, I also had to use --scan-hosts:

atvremote --scan-hosts 192.168.1.231 --protocol mrp --id 90:DD:5D:CA:0F:9B pair
Enter PIN on screen: 3065
Pairing seems to have succeeded, yey!
You may now use these credentials: 3826efa2d85f43dbf8fc4416489aad396f...

I think everything seems to be working. Added the device to HA using @postlund `s excellent custom integration.
Next step is to add a nice looking card for remote controls. Any guidance on that topic?

Update: Found this - everything seems to be working =)

[joshtbernstein]

Looks like I'm having this issue as well. MDNS is working with other devices, but to be sure I've triple checked my firewall rules and I still can't get this to work. @postlund Can I send you any logs?

[geekofweek]

Looks like this issue carried over to the official integration with 2020.12.0

[postlund]

@joshtbernstein Yes, please do. I will take a look when I have time.

@geekofweek Yeah, I haven't figured out what is wrong. This is top of my bad conscious list...

[joshtbernstein]

@postlund Is there anything specific you need/want?

[postlund]

@joshtbernstein I know that the problem is at least two-fold. One part is this issue, where responses to the requests doesn't seem to come at all. It used to work when I depended on aiozeroconf for the scanning implementation, so comparing what aiozeronf (or python-zeroconf) is doing network-wise is one thing. Maybe scan with latest pyatv and last release in the 0.6.x series and compare traffic. Might also be something that is different when it comes to what addresses I bind to locally and such, so that might also be a good idea to verify.

The second part is reported in #878. Here the MDNS responder responds with records from multiple devices (I expect each response to originate from one device), so the service parsing fails and potentially also miss devices. In the response in the issue, one of the responses has all the required records to parse the service except for the SRV record. So there's no IP address or port. That record however comes in a response a bit later, so if all records are collected over time and then parsed, it would work.

[postlund]

@joshtbernstein @geekofweek Would you mind testing if #899 improves the situation?

[joshtbernstein]

@postlund Hopefully I did this right, but I cloned down https://github.com/paxswill/pyatv/tree/mdns and ran an atvremote scan. Same results.

[geekofweek]

I did a test installing the code via pip3:

pip3 install https://github.com/paxswill/pyatv/archive/mdns.zip

Same results, no devices found via atvremote scan or trying to add the Apple TV by IP address

[paxswill]

@geekofweek @joshtbernstein Sorry to keep bothering you with this, but could you try with a new branch I just pushed up (multicast-join)?

The mdns branch worked for me on a Linux laptop, but I found a FreeBSD host was still unable to scan properly across subnets. It appears that since I had avahi installed on the laptop, it was already part of the multicast group. It also looks like Linux doesn't require each socket to be joined individually to a group before it'll be given packets. With the changes in that branch it works on my FreeBSD host, so hopefully that's the only thing keeping scanning from working.

If that doesn't work, could you check (something like tcpdump -n host 224.0.0.251 or ip proto 2 will show mDNS and IGMP packets) if mDNS responses are even getting to the host?

[joshtbernstein]

@paxswill Not a bother! I won't be able to test this for a a few weeks unfortunate. I'll report back when I do.

[geekofweek]

No luck so far, the only traffic I see in the tcpdump from the same LAN on the Home Assistant container is my phone. Lots of noise though as I have a few mDNS devices on that network.

[geekofweek]

I might just move my Apple TVs to the same VLAN, which at this point has 90+ % of my devices since this is an ever present issue with IoT things. Firewall rules might be easier to figure out.

[geekofweek]

I ended up just moving my Apple TVs to the same VLAN for now, I added 2 more so I have a total of 6 at this point. Was able to sort out the firewall rules I needed pretty easily. Sometimes I question my sanity doing multiple VLANs, I design that stuff for a living I don't want to have to support it either.

[joshtbernstein]

@paxswill Sorry for the delay. I tired the multicast-join branch and I'm having the same results. Running tcpdump -n host 224.0.0.251 or ip proto 2 on my macMini (different VLAN from my AppleTV) shows some traffic when I try to initiate a connection to output the computer's audio to the AppleTV.

[joshtbernstein]

@paxswill or @postlund Anything else I can try to troubleshoot? Also is there a way to pick which interface is binded when scanning?

[postlund]

I'm pretty much out of ideas. If you enable debug logs (--debug) you will see which IP addresses that it binds too.