Newpipe: refactor download manager

[theScrabi]

• [5] I carefully read the contribution guidelines and agree to them.
• [no] I checked if the issue/feature exists in the latest version.

Gigaget wase once a downloader distributed on fdroid. However it is discontinued, tho we copypasted their code into NewPipe, and made gigaget our downloader.
Now we have to get rid of it, since its causing a lot of trouble.
Related issues:
https://github.com/TeamNewPipe/NewPipe/issues/961
https://github.com/TeamNewPipe/NewPipe/issues/779
https://github.com/TeamNewPipe/NewPipe/issues/355
https://github.com/TeamNewPipe/NewPipe/issues/345
https://github.com/TeamNewPipe/NewPipe/issues/315

Maybe we could replace it with a Download Manager available on Android Arsenal. Feel free to browse a little and find a downloader you think fits to NewPipe.

A new download manager must have these abilities:

• using SAF (for downloading to sdcard)
• Be singe threaded (one thread per download)

Nice to have:

• Prevent/restrict download on mobile connection
• download files with external download manager
• easy accessing videos form download manager
• make it a lib and put it back on android arsenal
• show download status in titlebar/notification

[ghost]

@theScrabi From your link, Fetch looks like the most active and up to date project.

[theScrabi]

Uuuh fetch also has a wifi only switch :)

Does it also support notification?

[ghost]

@theScrabi No idea. There's not a lot of open source download managers for Android out there. Gigaget and uGet are pretty much the most well known ones. The ones you listed are more like side projects that aren't ready to be posted on the Play Store or even F-Droid.

You may have to stick with Gigaget for a while longer or even find somebody to fork Gigaget and fix the issues you mentioned. Or perhaps uGet can be ported to Newpipe? Not sure how easy that would be compared to porting Gigaget.

[LosEagle]

uGet is just frontend for CLI downloader aria2. Perhaps aria2 can be used in newpipe?

[ghost]

@LosEagle Actually, its a frontend for curl and aria2, and only curl is supported by default. aria2 support is included in uGet but you have to enable it in the settings before you are able to use it.

However, aria2 on it's own might be a good option but I'm not sure how easy it will be to add to Newpipe.

[ghost]

Well, using a native library would add another level of hell to NewPipe.
For example Android's PackageManager is totally not reliable with
installing native libraries, which can be avoided with ReLinker, and some
other stuff. I'd just write something using OkHttp.

[theScrabi]

Well, using a native library would add another level of hell to NewPipe.

Oh yes. Well it was nice if there is already some external downloader available.
@wb9688 could you please write the downloader as a module for for gradle, like the NewPipeExtractor.

[ghost]

could you please write the downloader as a module for for gradle, like the NewPipeExtractor.

@theScrabi: I don't think we should do that, since we'll need to have things specific for NewPipe (for example getting a new stream URL from NewPipeExtractor when the URL it's using to download expires). I'll write a new downloader for NewPipe though.

[karyogamy]

Has anyone looked at this? It looks quite mature and modular.

[ghost]

@wb9688 Thanks for your efforts and be sure to take your time on this downloader project. Please make sure the downloader makes multiple simultaneous connections while downloading like Gigaget does, though, or I predict many people will complain.

[theScrabi]

So I worked myself into this a bit further, and what i found out is, that accessing files via Unix file system (File class) is not possible any more. Instead you would want to use the Android Storrage Access Framework, which comes with its on classes and methods to handle files.

Uri instead of file names

So instead of filenames SAF uses Uri, to access files. What we would want to do in order to access SD card files is getting the Uri for the root directory, and cooncan the path to our file onto the end of it.

InputStream/OutputStream

Files can not be handled with File class anymore, however from android we can request InputStream/OutputStreams and create/delete fiels.

gigaget

Our current download manager is bad rubbish, it's old code someone wrote to test his andorid skills, and therefore its not quite reliable and up to date. Also it is based ontop of the idea that you can download one file with parallel threads.

What I've seen so far is that there is no real replacement for a download manager that uses SAF on android arsenal (notify me if you find something different).

So we have two options here:

1. Refactor gigaget, so it does not use threads, and supports SAF
2. Write a downloadmanager from scratch

File picker

Our filepicker is cool, but it does also not use SAF, and on top of that its sometimes a big buggy, here it would be cool to use the android SAF file picker, but that is not relay usable until android 6.0 ... so meh, we need to stick to it, and try to translate from filepath to uri.

[ghost]

Didn't @wb9688 already say he was writing a new downloader? I don't think Gigaget is salvagable. Fixing it will probably be more work than wring a barebones downloader from scratch, and all of the mature alternatives have already been mentioned in this issue.

[ghost]

Yeah, I'll write a new download manager. I'll throw away that GigaGet
garbage and then write something new that uses OkHttp and SAF. I'll also
make a better UI and integrate jCodec.

[placidcasual98]

So what would the time scale will I be before we can download to SD cards?

[theScrabi]

This is hard to tell. It's done when its done, however it got requested so often that I think things will get rolling soon :D

@wb9688 I don't want to sound unpolide, but you also said you where going to implement support for AndroidTV, and PipeCast. So far you started those projects than abandon them.
I think it's ok if you work on things, but notify us about them once you have a decent amount of code, because otherwise people might not consider giving it a shot as well, and when you abandon them again the project is basically blocked.

[ghost]

@theScrabi: I'm sorry, I just don't have as much time anymore. Others are
free to do it too, since I don't have much implemented of the new download
manager yet.

[mauriciocolli]

I'm experimenting with this library: okdownload, which is a rewrite of the popular one that @karyogamy posted previously (with improvements in documentation, testing and other aspects).

I may have a working proof-of-concept in the next weekend.

[karyogamy]

@mauriciocolli Sorry I didn't notice this before, just found out the next major release for ExoPlayer will include a downloader service with support for adaptive streams such as HLS. Here's the release notes and diff. Hopefully this can be useful for the new downloader implementation.

[placidcasual98]

Well this all sounds promising. That will the ETA be for it's deployment?

[placidcasual98]

Take it this is never getting fixed

[ghost]

@placidcasual98 Give it time. Lots of issues have taken way more time to fix.

[TobiGr]

@placidcasual98 We are working on this project in our free time. This means, NewPipe's developers are students or/and have a daily job, family, friends and other hobbies. Mauricio said, he started to implement a solution. Therefore a fix will be provided in the future.
The download mechanism is complex and thus there is a lot code to review and rewrite. There is no reason to do this as fast as possible, because working in great haste always introduces bugs.
Please keep this in mind.
If you think, this lasts too long, dig into the code and help us with implementing it.

[theScrabi]

@placidcasual98 if you want this to go on faster you can support us. Either through money via bountysource or librapay, or much better help with the actual code :)

[TobiGr]

We got a really detailed comment about the download handling on the blog.

Just to sum up the most important parts:

• Add confirmation dialog before deleting a downloaded video

• Allow rows instead of tiles in the downloads view. This would be good for video with long names (e.g. series which even have the same beginning)

• Add download queue to limit the number of parallel downloads

Maybe we can address some of these suggestions since they'd improve the UI and UX significantly.

[dreamsyntax]

On it.

(09/08/2018) I've been busy with other projects, however will push before the end of the month

[voidc]

Would it be an option to use the android system download manager? [1] It would not require an external dependency and provides a couple useful features out-of-the-box.

[lemnos]

Given that youtube throttles connections having the ability to use multiple threads with the current downloader allows you to max out your connection, it is a feature not a bug. I'm not sure why you want to confine downloads to a single thread. As has been mentioned if all you want is a single threaded downloader that integrates nicely with storage why not just use the system download manager?

[theScrabi]

The throtteling is a good point. Maybe we should leave a multi thread download manager then, but jus implement it better than gigaget.

[voidc]

I'm working on this now, too. I tried to replace GigaGet with the system download manager. You can have a look at my attempt here. So far, it looks promising, however I only started this weekend...

[theScrabi]

@voidc i think after what @lemnos said with the throtteling we should still take a look at a multi threaded manager. The internal one can not do that. @mauriciocolli @TobiGr @karyogamy what do you think?

[voidc]

We could benchmark the old (muti-threaded) download manager vs the system download manager in order to determine the effective loss of download speed. If the difference turns out to be not significant I would suggest still considering the system downloader. Every dependency on an external manager comes with the risk to end in a situation similar to GigaGet.

[placidcasual98]

Yeah but wee need to be able to download to SD cards!.

On Sun, 9 Sep 2018, 19:47 Dominik Stolz, [e-mail] wrote:

We could benchmark the old (muti-threaded) download manager vs the system
download manager in order to determine the effective loss of download
speed. If the difference turns out to be not significant I would suggest
still considering the system downloader. Every dependency on an external
manager comes with the risk to end in a situation similar to GigaGet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-419736292,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuPdF904amC8Pdnn-WpTWjCaSTrBtks5uZWJegaJpZM4RUWCP
.

[Blauelf]

What android versions have this problem with the current GigaGet implementation?
I am able to download to SD card on Android 5.0

Doesn't work for me after an update to Android 6 Marshmallow. Usually now applications request access by asking me to select a directory, and only after I complete the dialogue they write. NewPipe does not request such access, but fails.

[placidcasual98]

You not able to dl to SD!

On Tue, 11 Sep 2018, 21:37 dreamsyntax, [e-mail] wrote:

Yeah but wee need to be able to download to SD cards!.
… <#m_-3098015029238380638_>
On Sun, 9 Sep 2018, 19:47 Dominik Stolz, @.*> wrote: We could
benchmark the old (muti-threaded) download manager vs the system download
manager in order to determine the effective loss of download speed. If the
difference turns out to be not significant I would suggest still
considering the system downloader. Every dependency on an external manager
comes with the risk to end in a situation similar to GigaGet. — You are
receiving this because you were mentioned. Reply to this email directly,
view it on GitHub <#962 (comment)
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-419736292>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuPdF904amC8Pdnn-WpTWjCaSTrBtks5uZWJegaJpZM4RUWCP
.

What android versions have this problem with the current GigaGet
implementation?
I am able to download to SD card on Android 5.0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-420416911,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuMOIBvOg4qXs___u1ghk1gKiRPcJks5uaB8MgaJpZM4RUWCP
.

[lemnos]

Youtube definitely throttles (although it may not do this for every video/format). Looking at the youtube_dl source it looks like it only does this for range requests > 10M (see https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/youtube.py#L1866). I use multiple connections to get around this in my own code (not java/android related) and gigaget automatically does this for newpipe, but the approach taken by youtube_dl is probably better (i.e creating a sequence of requests < 10M). I don't know how much flexibility download manager offers you, but I doubt it will allow you to chunk the requests in this way, in which case a custom download manager is needed anyway. If you like I can take a crack at it, but it appears there is already a lot of interest in the implementation.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thabubble]

Would downloading to internal storage with gigaget and then moving it to the external storage work as a way to enable download to the SD card for the time being?

[theScrabi]

Yes it would. But wasnt this kind of hacky?

[thabubble]

It is hacky. But this is a feature a lot of people want. So while the downloader gets sorted out it will at least work for those with limited internal storage. Currently this is what I do with the file manager manually since I don't have root access and fbind/magic bind.

It would be greatly appreciated :)

[theScrabi]

But im more of a fan of propper solutuons even if it takes longer.

[placidcasual98]

Even if it takes a bit longer god damnit it's been like 3 years .

On Sun, 3 Feb 2019, 22:43 Christian Schabesberger <[email protected]
wrote:

But im more of a fan of propper solutuons even if it takes longer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-460096013,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuAFc9AmrL-jDTREPWlmsfsEk5Vc1ks5vJ2XzgaJpZM4RUWCP
.

[theScrabi]

I've tried to replace java file foo with android storrage management, it is possible but i didnt have time to get through.

If we can get help with this it might work even the better way.

[liberforce]

As a workaround of not being able to directly download on the SDCard, you may download them on the internal memory with NewPipe, then use an Android file manager to move the files on the SDCard, and read them with a media player.

[placidcasual98]

Workarounds are good enough it just needs to do what it's supposed to do
should really get that sorted.

On Thu, 7 Feb 2019, 16:16 liberforce <[email protected] wrote:

As a workaround of not being able to directly download on the SDCard, you
may download them on the internal memory with NewPipe, then use an Android
file manager to move the files on the SDCard, and read them with a media
player.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-461492823,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuPtwNMfpIkSVnkoLrOD4BuHqnRPXks5vLFFfgaJpZM4RUWCP
.

[gkeegan]

I agree with @liberforce. The temporary solution of moving the files manually is good enough while a proper solution like @theScrabi said is being worked on. I could try coding something like was mentioned by downloading directly to the SD card, but I am not the best, so no promises can be made.

[placidcasual98]

It not really good enough, I'll pass, incomplete software. Not been sorted
for 2 years. Call your self half arsed coders.....

On Thu, 7 Feb 2019, 19:21 Keegan Given <[email protected] wrote:

I agree with @liberforce https://github.com/liberforce. The temporary
solution of moving the files manually is good enough while a proper
solution like @theScrabi https://github.com/theScrabi said is being
worked on. I could try coding something like was mentioned by downloading
directly to the SD card, but I am not the best, so no promises can be made.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-461561130,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuMLz9F0hYJmCEoCKUKnPv7MyVEfFks5vLHzEgaJpZM4RUWCP
.

[gkeegan]

If you wanted it that bad: a. Why didn't you make an issue before now? b. You could always pay for YouTube Premium, while this app is free. :)

[yilmzfurkan]

@placidcasual98 instead of whining about these why don't you contribute to these issues? 2 years is pretty good time to learn Java.

[placidcasual98]

I did report this years and I just got a response like this.

On Fri, 8 Feb 2019, 03:29 Keegan Given <[email protected] wrote:

If you wanted it that bad: a. Why didn't you make an issue before now? b.
You could always pay for YouTube Premium, while this app is free. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-461678950,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuMIvXCUu26gBXdMuZhmGEXIGM_B5ks5vLO8EgaJpZM4RUWCP
.

[liberforce]

@placidcasual98 : This is Free/Libre software. Developers do their best, on their free time. If you don't like it, then don't use it, or face the fact that they owe you nothing you have paid for. Also, this is a bug report, not a forum, so please refrain from non-technical content. And BTW, insulting people you're asking to work for you for free had never motivated anyone to get any work done.

[placidcasual98]

Come on chop chop people get this sorted. You're only letting yourselfs
down, with your inability to solve this. Chop chop

On Sun, 10 Feb 2019, 18:20 liberforce <[email protected] wrote:

@placidcasual98 https://github.com/placidcasual98 : This is Free/Libre
software. Developers do their best, on their free time. If you don't like
it, then don't use it, or face the fact that they owe you nothing you have
paid for. Also, this is a bug report, not a forum, so please refrain from
non-technical content. And BTW, insulting people you're asking to work for
you for free had never motivated anyone to get any work done.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-462157818,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuOv_97HHQ2aNYxX7mUU7Z0TH3HnPks5vMGLegaJpZM4RUWCP
.

[theScrabi]

@placidcasual98 pay me 50 grand in Avancen than we can talk.

[placidcasual98]

Yeah man in the post.....

On Wed, 13 Feb 2019, 10:13 Christian Schabesberger <[email protected]
wrote:

@placidcasual98 https://github.com/placidcasual98 pay me 50 grand in
Avancen than we can talk.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/TeamNewPipe/NewPipe/issues/962#issuecomment-463140363,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiiVuPoLaOzgZnaQ7exYR-5psvZTbhnjks5vM-VIgaJpZM4RUWCP
.

[opusforlife2]

@theScrabi There are now 2 proper download managers available on F-Droid, Aria2App and Download Navi.

https://github.com/devgianlu/Aria2App
https://github.com/TachibanaGeneralLaboratories/download-navi

Maybe they might be of use?

[snappyapple632]

So with this issue seemingly fixed in 0.17, can this issue be finally closed, that is, when @kapodamy claims his bounty money?

[PeterHindes]

would youtube-dl be a good candidate? It's a separate open source program primarily used from the command line. It could likely be embedded in NewPipe.

[Stypox]

We know about youtube-dl (we even take inspiration from there for the extractor). But youtube-dl uses Python, which can't be run on Android unless with a heavy and slow emulator

[wb9688]

@PeterHindes and @Stypox: Plus youtube-dl isn't even a download manager and won't solve any of the issues mentioned here.

[PeterHindes]

Fair enough. I don't know much about youtube-dl, but I believe it has a binary form, IDK if it still packages python there though.

[wb9688]

@PeterHindes: The issue here is not retrieving the URL of the file, which NewPipeExtractor does wunderfully. The problem is actually downloading the file from the URL that NewPipeExtractor provides. youtube-dl would be an alternative for NewPipeExtractor, which we don't need, besides that youtube-dl lacks a lot of features.

[PeterHindes]

Maybe I'm not understanding. Does NewPipe Extractor provide a URL that you could for example download in a browser? I'm not familiar with the details of the process of downloading a YouTube video.

[opusforlife2]

Yep. It's Newpipe code that downloads the video. Look at the source file names.