Universalmediaserver: VLC on iOS doesn't always browse first directory

[SubJunk]

It shows up as recognized and connected, then when I click the UMS icon in the VLC app, it displays an empty directory. Other times it works like normal.

[Sami32]

http://www.universalmediaserver.com/forum/viewtopic.php?f=9&t=4060 ;-)

[SubJunk]

It seems like this has been fixed in #975 though I'm not sure I know which part of that fixed it. I think it was something from the initial commit 6b38ab1

[SubJunk]

@UniversalMediaServer/developers that was a false alarm, it is still happening. It's just inconsistent. It may just be that it expects us to respond within a certain time and we aren't.

I would appreciate some help on this because I've tried using Wireshark to spot differences between us and Serviio (which works 100% of the time) and I can't see what the difference is.

Ideas? Comments? @atamariya ?

By the way I have traced this bug all the way back to PMS so it has always existed in UMS, and I think always in PMS too though although I haven't tried the first release of that.

[Sami32]

@SubJunk Some ideas, but before i would like ask you if you tried UMS with only one shared folder containing only 1 or 2 files ? did you get this issue as well ?

EDIT: better try with little sized files.
I suspect that it's a VLC CDS linked issue, that we could implemented or tricked in UMS.

[Nadahar]

As per DLNA spec you have quite a while to answer a browsing request (30 seconds or something like that from what I remember), so I doubt that's the problem. I'd guess that there's some small different in the response when it works and doesn't, I'd try to pinpoint that. With trace logging the whole response is dumped right? Copy a working and a not working response into two separate text files and compare them in a diff tool. If they are identical, compare debug timestamps timings (the time diff from receiving the request until the response is sent).

[SubJunk]

@Sami32 tried it with just one folder with one small file in it and it still happened

@Nadahar I've been trying to do that for weeks and maybe I'm just not good at it.
I think it always happens when VLC is being run a second+ time since UMS started (so UMS already remembers connecting to VLC), I guess that is a good clue too.

[Nadahar]

@SubJunk I understand that you've compared them, I just thought that using a diff tool could be helpful for picking up tiny details that could be enough for VLC to dismiss it. I don't remember the details, but I looked at some "fishy" calculations in the response message a long time ago - some calculation of remaining items, bytes or something of that kind that looked flawed to me - but I lacked the overview to do something about it. My guess is that while many renderers accepts such small errors and just "makes the best out of it", others might not be so forgiving.

[Sami32]

@SubJunk IMO, your 2 Wireshark comparison session captures could be usefull for the ones that don't have iOS and VLC, and more efficient than a guessing game...

P.S. Did you tried with the VLC settings and changing "Network Caching Level" to "Lowest Latency" ?
Eventually also make "Text Encoding for FTP connections" set to "Lowest Latency".
Some users reported that it solved their empty folder issue.

EDIT: Did you got some errors in your UMS log when you was not able to browse folder with VLC on iOS ?
Or in your Wireshark capture ?

[SubJunk]

I have 2 logs from Wireshark and UMS. One where the browse failed and one where it succeeded:

vlc.zip

I use DiffMerge for looking at differences in text files and I haven't seen anything significant so far, maybe one of you will see a relevant difference.

In those UMS logs there are some random numbers which you can ignore, they were just custom logs to try to trace the path the code was taking.

[atamariya]

@SubJunk VLC 2.2.2 doesn't work on my win7 box. VLC recognises it but doesn't send CDS browse requests.

[Nadahar]

VLC for Windows has never worked with UMS for me. If finds the server, but no content is ever found. I thought that was because VLC tried to fetch all media first and then sort/organize and display them. UMS with it's virtual folders doesn't really work well in such a scenario, the whole logic is based on "one folder at a time browsing". My guess has been that the request has timed out before UMS has finished basicly parsing the whole media tree, but if you say that the CDS browse request is never sent at all, something else is going on..

[SubJunk]

The VLC computer program and the phone app are quite different - this is specifically for the phone app, which does usually work for me (and for other users according to the forum thread that @Sami32 posted)

[atamariya]

@SubJunk and @Nadahar You might want to check the requested count. For windows, that's the issue.
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <s:Body><u:Browse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1"> <ObjectID>0</ObjectID> <BrowseFlag>BrowseDirectChildren</BrowseFlag> <Filter>id,dc:title,res,sec:CaptionInfo,sec:CaptionInfoEx,pv:subtitlefile</Filter> <StartingIndex>0</StartingIndex> <RequestedCount>0</RequestedCount> <SortCriteria></SortCriteria> </u:Browse> </s:Body> </s:Envelope>
My first level tests didn't show the issue as VLC caches the discovery results. :(

[Sami32]

@SubJunk Thanks for all the log, without that i could be wondering much more ;-)
Sorry to be late but i'm running after time.

It's definitively a DLNA issue: SUBSCRIBE is not sended correcltly.
It sended after the GET request so just forget about CDS as it cannot be without a correct subscription.

7.3.2.18.1
[GUIDELINE] The SUBSCRIBE response shall include the Content-Length: 0 HTTP
header/value pair, if the response is not encoded with Chunked Transfer Coding.
The only exception to this rule is if the device can guarantee that a TCP FIN packet is sent before
the initial event message is sent to the subscribing control point.
[COMMENT] In order for a control point to receive the initial event from a UPnP device, a control
point needs to know the Subscription ID (SID) value.
The SID is obtained in the response to a SUBSCRIBE request.
Therefore, a control point will receive the entire SUBSCRIBE response before it receives the
first event.
The HTTP clients of control points only have two ways to know when the SUBSCRIBE response
has finished. The first is to complete the transaction when the Content-Length:0 values are
specified. The second is to receive the TCP FIN flag in the TCP stream.

In the purpose to narrow the possible root of this problem:

• Did you get the same issue when UMS is runnning before VLC and after VLC ?
• Did you get the same issue with request handler V1 ?

[Nadahar]

@SubJunk I agree that the response sent by UMS is identical, but it is also sent in both instances which means there must have been a query.

@Sami32 You might have a point, but it's very difficult (for me at least) to understand what you think is the problem. Do you mean that SUBSCRIBE is sent too late by UMS?

[SubJunk]

@Sami32 it usually happens when UMS is running before VLC. If I open VLC and then UMS, it is more reliable. Yes it seems to be the same with v1 and v2 unfortunately.

[SubJunk]

I've just been playing around with the latest master and it has been working tonight if I wait for about 30 seconds after a failed browse attempt. I still can't see anything very useful in Wireshark

[Sami32]

@SubJunk Sorry, i loose my Internet access yesterday...

I was waiting something like that.
You give one part of the answer, in your Wireshark session capture, we don't see any GET requests.
Looking at your log, we can see that SUBSCRIBE is not correctly done

Request: HTTP/1.1 : SUBSCRIBE : upnp/event/content_directory.
After the BYEBYE sending, you don't get the SUBSCRIBE request as it should but directly the GET request, so no events, no CDS are possible. When we compare your log with your Wireshark capture we see a monologue.

Your waiting of 30s is conform to the DLNA standard:

Requirement [7.4.20.8]: An HTTP Client Endpoint must wait at least 30 seconds before
closing the TCP connection if it has not received any response from the HTTP Server
Endpoint to an HTTP GET request for content.

Requirement [7.4.45.1]: HTTP Client Endpoints should close persistent (HTTP/1.1)
connections after completing all outstanding HTTP transactions and within 30
seconds of inactivity has passed.

I hope that you are understanding what i'm trying to explain ?

If you could make an other Wireshark capture when it's not working, but this time with Serviio totally shutdown as it interferred the last time.

[SubJunk]

Okay here's a debug log and wireshark capture when I did the following, after shutting down Serviio:

1) Started UMS
2) Started Wireshark capturing
3) Started the VLC app on iOS
4) Went to the Local Network section in VLC
5) Waited for the UMS icon to be displayed in VLC
6) Tapped the icon/server name in VLC, nothing happens (when it is working, this displays the root folders)
7) Stopped capturing after a few seconds

ums-vlc4ios-failed.zip

[SubJunk]

I would think it was a bug with VLC but it works with Serviio all the time so it must be something we are doing differently

[atamariya]

@Sami32 UMS doesn't have GENA implemented. The subscription request is effectively ignored.

[Nadahar]

I despise aPple so I don't have any iOS devices, but I just tested the Andoid app quickly and it seemed to work as good as one can expect. It was deteced as Chromecast (as all Android apps do because of the user-agent), so supported formats is obviously wrong, but I found no problems while browsing.

[SubJunk]

@Nadahar can't say I'm a huge fan of corporations whether they're Apple or Google.
Did you try a few things like closing the app completely then re-opening and re-connecting? The bug only appears infrequently so I will have nights where it seems bug-free.

[Nadahar]

@SubJunk I agree, but I have a special hatered for Apple because of the way they do things. I simply think they are much worse at swindeling people than most others (EA might be at their level though). They have a below average product, the manage to create a huuge hype by capitalizing on the psycological weaknesses of the less intelligent part of the population and then they sell that at very high prices. I see it as close to a scam what they do.

I didn't do any thorough testing in any way, I did open it a coulple of times but it could have been a coincidence. That said, iOS applications are written i C while Android application are written in Java so it can't be the same code - and the exact behaviour is propably not the same either.

[Sami32]

@SubJunk Yes, incorrect subscription is definitively the main issue there.
So, i see 2 things that could explain it:

• one, as advisely pointed out by @atamariya GENA is not correctly implemented in UMS as it should be since Cling offer this possibility: http://4thline.org/projects/cling/core/apidocs/org/fourthline/cling/model/gena/GENASubscription.html.
• secondly, it could be that GET /images/ request interfer the subscription ?
  You could verify and confirm that point in checking all the "failed" Wireshark session captures.

[SubJunk]

Oh cool. It would be good to get GENA implemented

[SubJunk]

I just added logging locally to the eventReceived function in SubscriptionCallback (in UPNPControl.java:611) and it doesn't appear to be called when I browse via VLC. Maybe VLC doesn't do subscriptions.
@atamariya anything to add?

[atamariya]

@SubJunk UMS subscribes to RENDERER events (that you are tracing) but DOESN'T generate CDS events. A compliant CDS client is supposed to cache results till it receives a notification from server that server content has changed (systemUpdateID).

[Sami32]

@SubJunk As you know i'm not skilled for coding it, but your implementation test look strange to me.

I leave here some links, just in case it could be helpful:
http://4thline.org/projects/cling/core/manual/cling-core-manual.html#section.BindingDevice
http://4thline.org/projects/cling/core/manual/cling-core-manual.html#javadoc.example.controlpoint.EventSubscriptionTest
http://www.programcreek.com/java-api-examples/index.php?api=org.fourthline.cling.model.gena.RemoteGENASubscription

EDIT: Btw, M-SEARCH is incorrectly done: ST field should be ssdp:all and not ...:device:MediaServer:1
https://github.com/UniversalMediaServer/UniversalMediaServer/blob/cce913b70fd8fcefa16eda090f6113656250202d/src/main/java/net/pms/network/UPNPHelper.java#L486-L491
_Cling_ could do it as well:
http://4thline.org/projects/cling/core/apidocs/org/fourthline/cling/protocol/async/SendingSearch.html

[Sami32]

@SubJunk What is this crappy thing ?
https://github.com/UniversalMediaServer/UniversalMediaServer/blob/cce913b70fd8fcefa16eda090f6113656250202d/src/main/java/net/pms/network/UPNPHelper.java#L291
The old specification asked for a TTL of 4, and the new one for a value of 2.

To limit network congestion, the time-to-live (TTL) of each IP packet for each multicast
message should default to 2 and should be configurable.

Why Cling classes was not used ? It's was many years ago though...before this Github repository.
http://4thline.org/projects/cling/core/apidocs/index.html?org/fourthline/cling/model/DiscoveryOptions.html
https://github.com/4thline/cling/blob/d9c3b7d188b744e7ae894fddb0d7251ca1bdd961/core/src/main/java/org/fourthline/cling/UpnpServiceConfiguration.java

A rarely used setting of DiscoveryOptions is byeByeBeforeFirstAlive: If enabled, Cling will send a byebye NOTIFY message before sending the first alive NOTIFY message. This happens only once, when a LocalDevice is added to the Registry, and it wasn't registered before.

http://4thline.org/projects/cling/core/manual/cling-core-manual.html

https://github.com/4thline/cling/blob/master/core/src/manual/basicapi/upnpservice.xhtml
https://github.com/4thline/cling/blob/master/core/src/main/java/org/fourthline/cling/protocol/async/SendingSearch.java
https://github.com/4thline/cling/blob/master/core/src/main/java/org/fourthline/cling/UpnpServiceConfiguration.java

I will check deeper the subscription step, but as SSDP M-SEARCH was the anterior step...

[Nadahar]

@Sami32 To give you a simple answer, UMS didn't use to have Cling but implemented its own UPnP handling. Cling was added to provide DLNA remote control capability but the main functionality has never been moved to Cling. Thus the current state. In my perfectionistic view it's a mess, but I guess it was a lot easier than to integrate Cling properly.

If you look in the log you will see that UMS constantly "finds itself", that is because there's actually two UPnP endpoints running in parallel each doing their part.

[Sami32]

@Nadahar I share your opinion on that point, sad that it was never fully used, it could avoid many UPnP/DLNA issues imho...I checked the Cling code and it's much standard than UMS one.

Ooooh, that was the meaning ! Yes, i saw that double entries in the log and wondered why and for what purpose it was. Thanks for the explanation :-)

[SubJunk]

Considering that we had the issue recently where we had to roll back Cling code because it used too much CPU (that was during an ALIVE message every 30 seconds, which is a valid use of Cling AFAICT) I'd say it's a good thing we don't use it for everything

[Sami32]

@SubJunk Or i become Alzeimer or my answer was deleted 8-/ Alzeimer symptome i guess :-)
Thanks for the explanation, i didn't know about that CPU issue.
I find that strange as many other media servers use Cling as well, but anyway the WebOS TV, SONOS and IPTV have a proprietary ServiceType that are not yet implemented in Cling, so yes it cannot be used for everything, but it's could help UPnP/DLNA improvements imho ;-)

[Nadahar]

@SubJunk I don't know enough of the details to assert anything about this, but my guess is that using Cling in a "half" way without studying how to use it properly can lead to issues that's not an issue if used as intended. I'd also guess that we could get "feedback" issues between Cling and UMS' own implementation that could trigger things like the CPU issue.

That said, any external lib that is used pose a threat when it comes to regressions or design changes that impact us. That's not unique for Cling, and that's one of the reasons I'm not a fan of just throwing in a lot of external libs. Ideally one should do proper testing every time an external library version is updated, but that would become very time consuming. Failing to do so means that we're basically accepting the risk of issues introduced as a consequence of this.

@Sami32 Do you have any documentation/links to this proprietary ServiceType and is that considered to be DLNA complient (that is, within the DLNA "rules")? How does UMS handle those devices today?

In any case, my guess is that a lot of custom things can be implemented by subclassing Cling classes - if the code is well designed that is the great advantage of OOP languages like Java.

[Sami32]

@Nadahar Yes, i have some documentation but i don't want post it now, as i'm afraid that someone would try to add them before having fixed the UPnP/DLNA issues already there ;-)

EDIT: Yes, there are autorized by the standard, as vendor specific.
I don't have these devices personnaly to test, but the UMS users on the forum speak by themself: not handled or errors...JRiver is good in that kind of implementation, Serviio did implemented SONOS serviceType.

I didn't saw request length detection or erroneous ssdp handling, that need to be ignored as per standard, and smartly fixed by Cling as some devices do it incorrectly...so i come to think that it will much better to use Cling, where skilled peoples already work on that since many years, and improve or add what need to be. Faster and better imho.
Like we say: "No need to reinvent the wheel" ;-)

@atamariya I will appreciate your comment on socket timeout:
https://github.com/UniversalMediaServer/UniversalMediaServer/blob/master/src/main/java/net/pms/network/UPNPHelper.java#L296
In UMS a value of 0 is used, meaning keeping indefinitively open AFAIU, but if i understood correctly the UPnP and DLNA standard, 30 seconds is the recommended value (and 60 s the default value).
Did i understood correctly ?

[SubJunk]

This seems to have been fixed in VLC itself, I can't reproduce it anymore. The app still sometimes crashes when you go into it but when it doesn't crash it works, so that's a different bug to this one.

[SubJunk]

This actually still happens

[SubJunk]

There is a thread on the VLC forums about it too which I have posted in https://forum.videolan.org/viewtopic.php?f=36&t=134806

[shaljam]

My VLC for iOS lists folders only once after I restart UMS.

[SubJunk]

I did some testing of this and reproduced the problem with Plex too. Just noting it for diagnostic purposes since I had thought it wasn't a problem when using them.