Streamlink: Please fix the ustream plugin

[TheSneakySniper]

The plugin seems to not be able to detect desktop streams... (This was ported directly from livestreamer, where it was never fixed. using librmtp never worked to download desktop streams, i am linking the post)
https://github.com/chrippa/livestreamer/issues/1122

Some streams that are in 720p don't show up, due to it only detecting mobile versions of the streams.
See here:

[cli][info] Found matching plugin ustreamtv for URL http://www.ustream.tv/nasahdtv Available streams: mobile_240p (worst), mobile_360p, mobile_480p, mobile_720p (best)
While this specific stream has 720p (not sure why), other streams do not.

This is a problem with the plugin only detecting mobile versions of streams.

Ustream Livestreamer plugin history:
https://github.com/chrippa/livestreamer/commits/develop/src/livestreamer/plugins/ustreamtv.py

[TheSneakySniper]

From the streamlink ustream plugin:
HLS_PLAYLIST_URL = ( "http://iphone-streaming.ustream.tv" "/uhls/{0}/streams/live/iphone/playlist.m3u8" )
https://raw.githubusercontent.com/streamlink/streamlink/master/src/streamlink/plugins/ustreamtv.py
This is my proof for saying that it only supports mobile streams.

Here is a hopefully helpful link(s):
http://forum.videohelp.com/threads/371350-Getting-rtmpdump-URL-for-USTREAM-feed

http://stream-recorder.com/forum/help-rip-rtmp-ustream-t16162.html?s=04c8d48d96e8cacec74abb7a6b5705aa&

[tp0]

Have the non-mobile streams ever worked in past <2 years? I remember before the plugin became completely broken, the streams opened and played, but there was constant buffering (segments) = unwatchable.

[TheSneakySniper]

No, they have not. From what I have read in the code (I may be wrong), all it does is get the ustream channel ID and then plug that into a pre constructed text specifically for mobile; and then downloads it using some method that I am not educated of.
http://pastebin.com/raw/ULNWCqX9

Ustream (and i think twitch VOD's too) changes the direct stream URL every second, (every chunk it changes) and it is probably parsed into a video using ffmpeg or some other method (i don't know how it downloads it).

e.g
http://http-cachelevel3.ustream.tv/sjc-uhs50/sjc-uhs01/streams/httpflv/ustreamVideo/8707800/streams/live_1_1365950097_1876974286.flv?start=195509

changes to:
http://http-cachelevel3.ustream.tv/sjc-uhs50/sjc-uhs01/streams/httpflv/ustreamVideo/8707800/streams/live_1_1365950098_1876974286.flv
the digits at the end change

Source:
http://stream-recorder.com/forum/help-rip-rtmp-ustream-t16162.html

[xlal0523]

Desktop ustream uses a combination of websockets and segmented flv now.

See websocket dump below:

{"args":[{"stream":{"hashes":[{"1475476210":"1124380824","1475476200":"1324067429"}],"streamType":"flv/segmented","providers":[{"name":"uhs_tcdn","url":"http://uhs-live-default.ustream.tv/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/"},{"name":"uhs_akamai","varnishUrl":"http://uhs-akamai.ustream.tv/sjc/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/","url":"http://uhs-akamai.ustream.tv/sjc/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/"},{"name":"uhs_llnw","varnishUrl":"http://uhs-llnw.ustream.tv/a/sjc/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/","url":"http://uhs-llnw.ustream.tv/a/sjc/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/"}],"keyframe":[{"offsetInMs":4991,"offset":32867,"chunkId":1475476207}],"streams":[{"chunkTime":5000,"isTranscoded":false,"width":384,"bitrate":348,"streamName":["streams/live_1_%_%.flv"],"height":216,"videoCodec":"H264"}]}}],"cmd":"moduleInfo"}

Using ["streams/live_1_%_%.flv"], the FLV segments are created using the hashes:

• Segments 1475476200 to 1475476209 will be created with hash 1324067429 i.e. streams/live_1_1475476204_1324067429.flv
• Segments after 1475476210 will be created with hash 1124380824 i.e. streams/live_1_1475476212_1124380824.flv

The full link is a combination of 'providers + segment'

• http://uhs-live-default.ustream.tv/sjc-uhs-omega14/streams/httpflv/ustreamVideo/19600300/streams/live_1_1475476204_1324067429.flv

[FredThompsonII]

Desktop streams worked until about a month ago using this ustreamtv.py file in livestreamer. Now all I see are the mobile streams.

The + is the mobile streams generally come across without the end each program segment being truncated. The - is the bitrate is lower than the desktop streams.
ustreamtv.zip

Goal, I think, is no truncation and desktop streams, broken into the individual program segments.

The youtube-dl guys state CreativeLive.com ustream feeds are live. I assume by that they mean the stream is only accessible as a stream, not as a file to be retrieved such as a YouTube or Vimeo static file. The streams from CreativeLive appear to be requanted as the K-frames are always in exactly the same locations.

[TheSneakySniper]

Are desktop streams working for you with the ustream file you posted? Do you have librmtp installed? Please give us more information.
https://github.com/chrippa/livestreamer/commits/develop/src/livestreamer/plugins/ustreamtv.py
Here is the history of the livestreamer ustream plugin, might be useful.

Interesting comment added in this commit... (in the code)
https://github.com/chrippa/livestreamer/commit/958a969d67e2797b15dfc74433b2bf8c598c1a2c

[FredThompsonII]

As I stated, I'm only seeing mobile streams now. Never used librmtp. You can see my notes in the issues area of livestreamer. Time cutting them is minimal compared to annoyance of lost portions. Current plugin drops every so often. This happens to all streams at the identical time. Might be a system issue here. Haven't looked into it. I just keep recording then cut and append clips.

[beardypig]

For desktop streams to work you need python-librtmp (another @chrippa project) installed, this might be tricky if you are using python 3.5+ on Windows because some C extensions for librtmp needs to be compiled. As @xlal0523 alluded to, desktop streams on ustream.com use their custom streaming protocol. Previously there were RTMP streams available but it looks like these have disappeared. It is possible the desktop stream could have worked if you were using a version of livestreamer from 2013, which would have used the old ustream API.

Even if you have python-librtmp installed the Desktop streams will not work due to a bug in the latest version of the plugin. In commit 958a969d67e2797b15dfc74433b2bf8c598c1a2c @paulguy made a change that dropped "uhs_akamai" streams which probably worked at the time, but something has since changed and for some streams (nasahdtv at least) there are no other valid streams.

With this hack-patch, I was able to get a "720p+" stream for http://www.ustream.tv/nasahdtv working - there is still some pausing and dropping.

diff --git a/src/streamlink/plugins/ustreamtv.py b/src/streamlink/plugins/ustreamtv.py
index 5f15ffa..6610ba6 100644
--- a/src/streamlink/plugins/ustreamtv.py
+++ b/src/streamlink/plugins/ustreamtv.py
@@ -508,7 +508,7 @@ class UStreamTV(Plugin):

         streams = {}
         for provider in channel["stream"]:
-            if provider["name"] == u"uhs_akamai":  # not heavily tested, but got a stream working
+            if "url" not in provider:
                 continue
             provider_url = provider["url"]
             provider_name = provider["name"]

[beardypig]

Desktop streams a hybrid RTMP/HTTP streaming protocol is used. The RTMP server is used to give out metadata about the channel, including a list of streams and a base url for all of the streams. Each of the streams has metadata, including bit rate, resolution.

The streams are divided in to chunks (or segments), each chunk is found under the base URL for the channel. The exact path for each chunk is given as a combination of a file pattern, chunkId, and token.

For example, these are some fields from a stream entry.

{
    'isTranscoded': False, 
    'streamName': u'streams/live_1_%_%.flv', 
    'chunkTime': 5000.0, 
    'height': 720.0, 
    'chunkRange': {u'1475815790': u'797895358', u'1475815780': u'640805711'}, 
    'offset': 681.0, 
    'chunkId': 1475815783.0, 
    'bitrate': 1513.0, 
    'offsetInMs': 4985.0
}

The important fields are chunkRange, streamName, and chunkId. chunkRange tells the client the "tokens" that will be used for a range of chunkIds, and chunkId tells the client the current chunk that is available.

In the example stream we have chunk tokens for 1475815780 and 1475815790, and the current chunkId is 1475815783
Using the streamName the client can determine the urls for the chunks 1475815780-1475815790, but that only up to chunk 1475815783 exists

• streams/live_1_1475815780_640805711.flv
• streams/live_1_1475815781_640805711.flv
• streams/live_1_1475815782_640805711.flv
• streams/live_1_1475815783_640805711.flv

Subsequent requests to the moduleInfo endpoint may return a more recent chunkId, but with the same chunkRange.

{
    ...
    'chunkId': 1475815787.0, 
    ...
}

This indicates to the client that chunks up to 1475815787 are avilable (streams/live_1_1475815784_640805711.flv ... streams/live_1_1475815787_640805711.flv).
Further requests will update the current chunkId, when the current chunkId crosses the boundary in a chunk range the token will change.

eg. streams/live_1_1475815787_640805711.flv ... streams/live_1_1475815793_797895358.flv

There is a bug in the current implementation where some segments can be missed, which can lead to chopping/skipping playback.

[stevek123]

The HTML5 only player uses a different format.

[beardypig]

@stevek123 do you have to enable some option to get the HTML5 player or do specific channels use the HTML5 player?

[stevek123]

For example, this one: http://www.ustream.tv/leolaporte is always HTML5 for me, even if flash is installed (Win10)

[beardypig]

Perfect, cheers!

[TheSneakySniper]

Does librmtp need to be installed for this to work? (I think it does, but I just wanted to verify)

[beardypig]

@TheSneakySniper currently yes, however, with the information from @stevek123 it should be easy to create a version of the plugin that does not use python-librtmp - in fact, I'm working on it right now...

[TheSneakySniper]

@beardypig Any progress?

[beardypig]

Yes, progress has been made, you can follow it on my fork: https://github.com/beardypig/streamlink/commits/ustream-websocket

This is almost a total rewrite of the plugin. I have a working websockets client that is getting the fragment information, the appear to be two different types of of stream; fragmented flv (as before) and fragmented mp4 (very similar to MPEG DASH). Most of the old code for concatenating the flv stream still worked, so it was very straightforward to get the fragmented flv stream working.

The fragmented mp4 stream is a bit different, the video and audio come in different streams (again, like some DASH streams). I am currently working on a pure Python implementation of a muxer for the video and audio streams. After the muxer is complete it should be very little work to finish the plugin.

Currently there is no support for the mobile streams, however it is relatively easy to add them.

Additionally, this work will be useful for supporting MPEG DASH and could be useful in supporting SmoothStreaming. Which are two adaptive streaming protocols that I am interested in adding support for.

[beardypig]

I have pushed some commits to that branch (https://github.com/beardypig/streamlink/commits/ustream-websocket) that include a basic MP4 muxer. Some help testing mp4 streams would be welcome :)

[gravyboat]

@beardypig do you just want to open the PR for that so we can get it merged in? Also if you haven't already and want a free shirt you should sign up for https://hacktoberfest.digitalocean.com/, you've got way more than enough PRs to get it.

[beardypig]

Thanks for the tip about the t-shirt :-)

I will open a PR, but I want to tidy some things up and move the MP4 parser in to a separate library. I will do that over the next couple of days :-)

[RuKeBo]

Not sure if this is the right place for this but it says you need python-librtmp to access desktop streams. I tried installing this on Windows 10 using "pip install python-lbrtmp" per instructions but it gives me an error every time about not finding "/librtmp/rtmp.h". Any ideas?

[beardypig]

@rukebo; I'm going to assume you're using Python 3.5...

tldr; use an earlier version of python, eg. 2.7 or 3.4

Unfortunately it is not as simple as just installing python-librtmp with pip after 3.4. python-librtmp is a Wrapper around the library librtmp and includes some C extentions that need to be compiled at install time. There is no wheel (pre-compiled) package, so you need to compile it yourself - pip is trying to do that for you, but it cannot the header files for librtmp. You either need to install the header files and the correct version of MSVC (the same one that python is compiled with), or switch to an earlier version of python (2.7 - 3.4). I'm not much of a Windows user, but if you choose to try and compile it yourself (via pip) you will need to find a general guide for compiling python libraries with C extentions in Windows, and where you need to store any extra headers (librtmp files in this case).

[RuKeBo]

That is correct, version 3.5. I had 3.3 but had problems with that and upgraded. I will try a lower version.

Thanks.

[RuKeBo]

I went back to 2.7 since it seemed the easiest way to get it to work. When I install python-librtmp it seems to install without error but when I run streamlink, it says it is not installed. I have a librtmp folder in Python27\Lib\site-packages, not sure what this means. There is also a streamlink folder there. I think I must have tried to compile Streamlink before, I removed it.

[sbstp]

Is there an issue with using the mobile stream? I'm using it right now and it's pretty decent. It would also be easier to maintain because the HLS code is already implemented.

[RuKeBo]

I have no issue using the mobile stream. I just wanted to get the other working since it should and used to at one time. I am not advocating changing anything to get it working though.

[sbstp]

I think I found an easy way of grabbing a m3u8 stream out of ustream. Every channel page has a <meta name="ustream:channel_id" content="..."> tag, and you can just put this id into the mobile url, http://iphone-streaming.ustream.tv/uhls/$ID_HERE/streams/live/iphone/playlist.m3u8

EDIT: The logic for this seems to already be implemented, but it's not using it...

[RuKeBo]

How does it know that is a ustream channel?

[RuKeBo]

Can someone please tell me what the "._" is/does in the following from ._librtmp import ffi, lib as librtmp. I got it to at least see where librtmp is by putting the "librtmp" folder in the C;\Program Files\Streamlink\pkgsstreamlink folder, but it seems to get caught on the line above in __init__.py file.

[beardypig]

That is a relative import of a python package called _librtmp, which is a C extension library that needs to be compiled. In Linux/Mac it will be called _librtmp.so and in Windows I think (not a Windows user) it's two files, _librtmp.dll and _librtmp.lib.

[beardypig]

@sbstp The plugin does test for mobile streams, but the mobile streams are created on-demand and are transcoded to mpeg-ts when they are requested. There is a retry loop when looking for mobile streams; it tries 10 times unless some desktop streams have been found, then it only checks once and might fail. Any failures finding mobile streams are hidden if there are desktop streams.

Edit: It does use that URL you referenced, but it finds the channel_id using a different method :)

[RuKeBo]

So it is not a relative path to the librtmp folder? I have a librtmp.dll file there that I copied from a rtmpdump-2.3-windows download, there is no librtmp.lib file. I guess it is doing something. If I remove the .dll file then it just fails, if I leave it in, I am getting a Windows C++ Runtime Library error and also fails, but at least I know where the current problem is.

[beardypig]

@RuKeBo I'm not very familiar with streamlink under Windows, I'm not sure how you installed it. If you have Python2.7 installed and with python-librtmp installed for that Python2.7 install, you can try and install streamlink using pip and then run it like that? I don't actually know how you run streamlink under Windows if you install it with pip :)
The main problem is python-librtmp doesn't come precompiled for versions of Python greater than 3.4... I am working on a new version of the plugin that doesn't use python-librtmp, so there shouldn't be any more of those problems once I'm done... Unfortunately I didn't get time to finish it yet, but maybe tomorrow I will finally create that PR I promised... :)

[RuKeBo]

Ok. This is sort of just something I was trying to get to work, not a need. I used the Windows installer for version 0.0.2. I uninstalled all the Python versions I had and then installed 2.7. I used pip to install python-librtmp but that did nothing as far as streamlink was concerned. I found out that streamlink was using the Python version, 3.5 I think, that came with the install. I moved all the files over to a location by trial-and-error and that is where I am now. So maybe I am mixing things. If librtmp was compiled using 2.7 maybe it won't work with the 3.5 streamlink uses. I might try using pip to install streamlink just to see what it does. Thanks!

[beardypig]

Fair enough :-) It's need to be compiled not only with the same Python version but the same version of MSVC :-) If pip (pip3) came with streamlink maybe you can use that version to install python-librtmp.

[RuKeBo]

This is probably too far beyond me. I didn't realize it would all be in Python. With the installer, it gives me an .exe file, I don't know Python enough to get that to work. I will probably just wait until there is a new version, it would take me longer than that to figure it out.

The streamlink I installed using pip does not appear to have Python with it. The Windows version does. But that is 3.5 and I think you said before python-librtmp could not be installed above 3.4.

[sbstp]

RTMP is kind of a mess, I tried installing it on Linux and it's pretty annoying, can't use setup.py only, you need to install librtmp and libffi. I tried installing python-librtmp on Windows and it didn't work either, but I haven't tried debugging it much.

I see 2 possible solutions to this problem:
1) We implement the RTMP protocol in Python, no native extensions. It's probably a big task, so I don't know about doing this.
2) We avoid RTMP everytime it's possible, like in the ustream case. We might have cases where only RTMP is available, so we should study how much RTMP is used in the plugins. Also @beardypig I'm not sure what HLS being "on demand" implies. Can you clarify that?

[beardypig]

As far as I can tell python-librtmp is only used in the ustream plugin. It is used to talk to rtmp servers, doing rpc, etc. which is generally not required rtmpdump is usually sufficient. I don't see any point trying to support it at the moment.

Ustream desktop streams are either fragmented flv or fragmented mp4, which are very similar, both are h264 and aac I think. It seems like their source of truth for each stream is the mp4/flv, and if you want to watch it on mobile (iOS) then you need an HLS stream which needs to be mpeg ts (iOS 10, supports mp4 in HLS). They don't appear to generate the HLS streams, m3u8 and the mepg ts fragments, unless one is requested and it can take a while for the HLS stream to appear. I'd imagine they lag behind the desktop streams as well, but I haven't tested that to confirm. The video content is likely exactly exactly the same, just in a different container format. Although I think the desktop streams have 720p in higher bit rate.
It's true that the HLS stream handling code is already implemented, which is good and we can continue to expose the mobile streams for ustream.

The fragmented mp4 streams have separate video and audio streams and need to be muxed. They are created to the MPEG DASH spec, adding support for the fragmented mp4 streams (the flv streams are already supported) will be good for the project as a whole. It paves the way for MPEG DASH support, the next thing on my list...

[sbstp]

The HLS mobile stream I watched on ustream last night was ahead of the desktop stream played on the website. Anyway it doesn't really matter, when librtmp is not available, it just should use HLS stream, or at least ask for it. We could print a warning message saying that the mobile streams can take time to start and to try again in a few seconds/minutes.

[beardypig]

Fair enough :-) It doesnt mean we shouldn't support the desktop streams though. The good this is librtmp is actually not required any more, they have replaced their rtmp severs with a websockets equivalent. We can still print a warning about the mobile streams.

[RuKeBo]

Sorry, can I have clarification on my question above concerning from ._librtmp import ffi, lib as librtmp. I just want to understand what this is doing. I understand that in linux that two dots ".." will move up one directory and one dot "." basically means the directory where you are. But what is the dot and underline doing together? I ask because if I remove the lib a lbrtmp from the line, it still fails, no error but it exits __init__.py in the librtmp directory and continues on in the ustreamtv.py plugin. I am just trying to track down what in this statement is causing the issue. I was assuming this meant that it was looking in the librtmp directory and importing ffi, but I put a print statement in ffi.py and it never gets there, which may be the issue.

[beardypig]

.. and . are the same as in linux (mostly). The _ is actually part of the package name. ffi is part of the python library cffi which is used to interface with external libraries.
You can see in the python-librtmp source code where it sets up the librtmp._librtmp "package" https://github.com/chrippa/python-librtmp/blob/master/librtmp/ffi.py.

[RuKeBo]

Ok, so would this (_librtmp) be an internal thing, I mean it is not a physical file or directory? If so, then I would assume it is not compiled correctly. I don't know how any of that works, how it (Python) knows where librtmp is when it is looking for it. I was hoping it was just looking for the files somewhere (in the librtmp directory) and I could move them around until it worked.

There is an ffi.py in the librtmp directory and then I see that it references a cffi directory in from cffi import FFI, I didn't have the cffi directory so that will not work as I understand it. There is one in the Python site-packages folder I copied over. All the problems with this seem to be that streamlink wants everything to be relative to the streamlink directory and Python (when I installed things), put them all in the Python27\Lib\site-packages directory. I am not looking for any resolution on this, just mentioning it if it means something to you. I am interested in the _librtmp item above.

[TheSneakySniper]

@beardypig We don't know what ustreams plans are, and if they intend to drop the mobile 720p streams support in the future. It is best if we continue development, if we don't then they could drop it, and we would have to start another issue.

[beardypig]

@TheSneakySniper who said anything about stopping? :)

[TheSneakySniper]

@beardypig Sorry, I thought that because the mobile streams got 720p support you would drop the project. I guess not.

[TheSneakySniper]

@beardypig Any updates?

[beardypig]

When the ffmpeg muxer stuff is merged I will update the ustream branch, hopefully this week :-)

[RuKeBo]

Not trying to be the squeaky wheel but this was slated to be worked on long before Christmas.

[Tristanx]

I have put together a temporary fix of the current ustream plugin's desktop streams for my own purposes. I know beardypig is working on a new version but if there is demand for it in the mean time, I can send a pull request. see my fork: https://github.com/Tristanx/streamlink (note: only tested on linux)

[TheSneakySniper]

@beardypig Any updates?

[tp0]

@Tristanx Broken or am I just using the wrong url? I tried and http://www.ustream.tv/embed/nnnnnnnn and http://www.ustream.tv/channel/nnnnnn-nnnnnn-nnnnnnn/.

streamlink "URL" best
[cli][info] Found matching plugin ustreamtv for URL http://www.ustream.tv/channel/nnnnnn-nnnnnn-nnnnnn/
Traceback (most recent call last):
  File "/usr/local/bin/streamlink", line 9, in <module>
    load_entry_point('streamlink==0.4.0', 'console_scripts', 'streamlink')()
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink_cli/main.py", line 1007, in main
    handle_url()
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink_cli/main.py", line 479, in handle_url
    streams = fetch_streams(plugin)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink_cli/main.py", line 391, in fetch_streams
    sorting_excludes=args.stream_sorting_excludes)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink/plugin/plugin.py", line 325, in get_streams
    return self.streams(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink/plugin/plugin.py", line 240, in streams
    ostreams = list(ostreams)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink/plugins/ustreamtv.py", line 550, in _get_live_streams
    streams = self._get_desktop_streams(channel_id)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-0.4.0-py2.7.egg/streamlink/plugins/ustreamtv.py", line 513, in _get_desktop_streams
    provider_url = provider["url"]
KeyError: 'url'

[Tristanx]

@tp0 hmm, maybe I should have also added I've only tested it with python 3.
"http://www.ustream.tv/channel/..." should be the right url. It got to the ustream plugin so I don't think there are any problems there.

Have you tried different streams? some 24hr streams you could try are "http://www.ustream.tv/channel/iss-hdev-payload" and "http://www.ustream.tv/channel/apl-puppies"

also, are you only getting this problem with my fork? if so & you are still having trouble make a new issue here.

[tp0]

My mistake, as I first installed it with python2.7. Now works.

[gravyboat]

Closing this since #754 was merged.