Libtorrent: Not working without a sparse mode

[master255]

I checked. The library shows that there are peers, but it doesn't download anything.
Most devices in the world do not have sparse mode. It's an obvious library bug.

[arvidn]

what is?

[master255]

@arvidn Scenario:
I download a 10GB file to an SD card in Android. Write speed to SD card 10 Megabyte\sec.
10Gb/10Mb = 1000 sec. = 17 min. I have to wait for the library to create the whole file and start downloading, although it can download data as soon as I start downloading. But it doesn't do that. It's a bug.
Of course, it still takes 17 minutes to download the file. Ideally. So I spend 17+17 = 34 min to download 10Gb of data. Instead of 17 min.

[FranciscoPombal]

@master255
libtorrent does have a sparse mode, which is the default:
https://libtorrent.org/manual-ref.html#storage-allocation
https://libtorrent.org/reference-Storage.html#storage_mode_t
Ctrl+F storage_mode in this page for more examples: https://libtorrent.org/single-page-ref.html

[master255]

@FranciscoPombal In the most popular operating system in the world Android - no sparse mode for SD cards and external drives (USB flash drives and disks).

[FranciscoPombal]

@master255 well, that's an Android limitation. What do you propose libtorrent do about that?

[master255]

@FranciscoPombal I already wrote about it.
This can be fixed by allowing developers to download and save files from a certain byte.
It is necessary to add the event of writing data to disk from the cache. To know exactly which data is written and which data should be downloaded.
This will allow to download the files in parts and then combine them into one file.

Maybe you have some other suggestion. I'm willing to discuss it.

[master255]

@FranciscoPombal That's why I invented file storage technology.
Okay. If the library uses it, it will work much faster.
It's the technology:
We save a file named name_file.avi. Size 17000000.
When saving, the first file is called "name_file.avi-bt0_17000000".
Second, third, fourth...: "name_file.avi-bt100000."

In the first file, "0" means that it is the first. Then the number 17000000 is the size of the whole file.
In the second file, the number at the end is "100000". - means the byte from which this piece begins. (100000 + file size) = means the final byte of this piece.
Knowing the full size of a file and having all necessary parts, you can build an entire file. Or you may not splice it into one file, but use it addressing to pieces.
This is a way to store files on media that do not support sparse mode.
And what's more, you can resume downloading files.

[master255]

My media player already supports this technology and can play from multiple files, as from one.

[FranciscoPombal]

@master255

Ok, so you split the file in smaller chunks - but doesn't each chunk have to be pre-alliocated as well? If that is that is the case, then you will spend at least as much time preallocating space for the whole file, as if you had done it all in the beginning. The preallocation will be interspersed with the download, instead of occurring all at the start at once. Please correct me if I'm wrong.

However, I can still see an advantage in doing it that way: the case where "streaming" is a priority (popcorn time style) - in this case you'd want the download to start as soon as possible. Is that what you are aiming for?

[master255]

@FranciscoPombal No. The pieces are downloaded from a certain byte and are not pre-allocated. The size of chunks is equal to the size of incoming traffic. There's no pre-allocation process at all.

in this case you'd want the download to start as soon as possible. Is that what you are aiming for?

I need to download the file in multiple threads. If I download it sequentially, I will not be able to use rewind. However, sparse mode is not available for saving data.

[FranciscoPombal]

@master255
Ok, in that case, no doubt libtorrent should provide better ergonomics for saving to these kinds of filesystems. But personally, I dislike the approach of creating multiple small files to achieve that. It's just a pain to deal with, once you take storage moves, unclean terminations in the middle of the download, select/deselect a file for download, etc into account. Just speaking from experience about qBittorrent and the whole mess surrounding .part files and .unwanted folders.

Hopefully the same effect as what you describe can be achieved, but without creating such files. @arvidn thoughts? Or am I looking at this totally the wrong way?

[master255]

@FranciscoPombal

But personally, I dislike the approach of creating multiple small files to achieve that.

In practice, they're not small files. I'm afraid it's not that simple.
The main thing is to do it in libtorrent:

1. Add the ability to download files sequentially and from a certain byte.
2. Add event about writing data from cache to disk.
3. Make it possible to download one file multithreaded from different peers (different part of one file).

My technology can use libtorrent, as a minimum. But it's much better if it's used at a higher level by developers who are using the library.

[FranciscoPombal]

@master255

1. Add the ability to download files sequentially and from a certain byte.

2. Add event about writing data from cache to disk.

3. Make it possible to download one file multithreaded from different peers (different part of one file).

I'm not sure I understand what you mean. The way you are describing it, sounds like libtorrent can already do all of this:

1. sequential_download
2. flush_cache(), cache_flushed_alert
3. libtorrent already downloads different pieces from different peers, regardless of whether or not those pieces belong to the same file.

If you want to increase the number of disk I/O threads, that would be aio_threads. By default it is 4, but it works in a somewhat unexpected way: for every 4 aio_threads there is 1 hashing thread. So if you have a CPU with 4 logical cores, you probably want to set aio_threads to 16, because that way libtorrent will spawn 16/4 = 4 hashing threads. With fast storage and network speed, a single hash thread can become the bottleneck if the CPU can't SHA-1 hash very fast. There has been talk of decoupling the configuration of the number of hashing threads from the number of I/O threads, but for now it's how it works.

[master255]

@FranciscoPombal No. No. It's not that simple.

1. Sequential download from a specific byte.
2. Need information about how much data has been written and into which file.
3. I want to manually create pieces to load from a certain byte and define my own file name for each piece.

[arvidn]

I believe this is supported by the custom storage interface https://libtorrent.org/reference-Custom_Storage.html#overview

[arvidn]

as far as I can tell, @master255 is using the phrase "not working" to mean "has poor performance".

Btw, HFS+, which used to be the main filesystem on MacOS (and probably is very popular) also doesn't support sparse files, but still you don't have these kinds of problems.. hm..

[master255]

@arvidn

I believe this is supported by the custom storage interface

It's easier for me to write your entire library in Java (which I almost did) than to make small changes to your library.

You can call it anything you like, but it's an obvious bug. Most data storages in the world don't support sparse mode. That means your library has a bug, not a problem.

"not working" to mean "has poor performance"

I talked to the LibreTorrent developer and tested the application. The application does not tell the user in any way that the file is being created. Any user will decide that the torrent is not being downloaded. The developers do not understand that, at the minimum, they should tell the user what is going on.
Waiting time for the download to start increases dramatically due to file size and luck. It can be as long as hours. If your library may hang on for an hour at some point, it means it is not working :-)

And i'm use Win10 LTSC Compact and recommend it.

I've already done it at my application level:
https://github.com/arvidn/libtorrent/issues/4616#issuecomment-625876774
I need to do it at the level of your library:
https://github.com/arvidn/libtorrent/issues/4616#issuecomment-625942212

[arvidn]

It's easier for me to write your entire library in Java (which I almost did) than to make small changes to your library.

I don't see what the point of this ticket is then. It sounds like it should be a PR. Or are you asking for guidance?

[master255]

@arvidn I want you to fix this bug. Any way you want.

[master255]

@arvidn Okay. I see you changed the api. But I don't know how to use it.
I found new commands: map_file(), file_slice, map_block(), read_piece_alert and others...
Maybe you can tell me how best to implement my technology?
What is the best way to download\save to disk a file from a certain byte?

[arvidn]

Maybe you can tell me how best to implement my technology?
What is the best way to download\save to disk a file from a certain byte?

I think it's misguided. I think you are optimizing something that is not very relevant towards your end goal. Also, if you find it prohibitively difficult to implement your ideas, you may want to question whether they're worth the effort.

[master255]

@arvidn Ok. Maybe you can figure out how this works. I can't get the piece data. I have a pointer from alert ((ReadPieceAlert)Alerts.cast(a)).bufferPtr()), but what command can I use to get the data?
From what object? The names aren't obvious. I've already spent a day trying to figure it out.

[arvidn]

I'm surprisingly ignorant of the java bindings, if that's what you're asking about. the read_piece_alert has a boost::shared_array pointer member (called buffer) which points to the piece data (unless there's an error, in which case error is set to indicate what went wrong). documented here.

[master255]

@arvidn Okay, this message can't be used.
What about the other commands?
map_file(), file_slice, map_block()
What exactly are they doing? Can I specify a byte before downloading torrent?
I don't know what all this means. It's very unclear what it says:

"This function will map a range in a specific file into a range in the torrent. The file_offset parameter is the offset in the file, given in bytes, where 0 is the start of the file. See peer_request.
The input range is assumed to be valid within the torrent. file_offset + size is not allowed to be greater than the file size. file_index must refer to a valid file, i.e. it cannot be >= num_files().".

You can explain in more detail what "This function will map a range in a specific file into a range in the torrent" means.
map a range - what is this process? At what time is it possible to do it? Before downloading the file or after?
Is it .torrent file or a file that is downloaded?

[arvidn]

"map" as in a function taking a value in domain A returning its counterpart in domain B. that kind of mapping. It will convert a file+offset into a torrent_file_offset

[master255]

@arvidn What's this for? Can I set an offset before I start downloading? How?
The data before the offset will not be loaded?

I think you're speaking too technically. You need to explain in a simpler language what that means.

[arvidn]

What's this for?

To convert a file index and offset into that file, to the offset in the torrent "space". Determining which piece a specific byte is in for example.

Can I set an offset before I start downloading? How?

No, it doesn't "set" anything. it converts numbers.

[master255]

@arvidn Ok. I looked at the updated manual. Now I know for sure that map_file unfortunately doesn't do what I need but returns peer_request.
Thank you for the clarification and correction.
The description of map_block() and file_slice is more or less clear to me.
Now I do not understand how it works.
For example, I perform map_file() and get peer_request.
Can I somehow use peer_request to get data from peers?
If yes, how?

[master255]

@arvidn I only have one way to do what I need to do? Am I right in understanding that? Use a read_piece_alert that's not working right now?

[arvidn]

Can I somehow use peer_request to get data from peers?

No

I only have one way to do what I need to do? Am I right in understanding that?

I don't know what you need to do.

Use a read_piece_alert that's not working right now?

What's wrong with it? (or was that a question whether it works or not? if so, as far as I know, read_piece() works)

[master255]

@arvidn I've tried all the relevant classes in this library.
I tried to convert the read_piece_alert answer to byte_vector and entry. But it's not working. A developer hasn't answered for three days now. It looks like it doesn't work in Java now.
https://github.com/frostwire/frostwire-jlibtorrent/issues/232
https://github.com/frostwire/frostwire-jlibtorrent/issues/231

[elgatito]

@master255 Most of your questions come from misunderstanding the nature of torrent file. You see it like transferring files between peers.
But in real it is a transfer of bytes, length X, where you have an information that bytes belong to files, where file A has the range of bytes between {0, M}, files B has the range {M+1, N} and so on.
All the torrent is splitted into pieces and peers send/receive those pieces of bytes, not files.

When you speak about streaming and having parts of file, you should not operate files at all, you operate bytes, if you have a file ranged {A .. B} and you need only from 10th Mb, then you calculate which pieces belong to range {A+10 * 1024 * 1024 .. B}, and set piece_priority to > 0, then libtorrent will download this piece.

With storage_interface Arvid means overriding default storage interface, that stored file, described in the torrent. You can write your read/write functions that will do what you want. Write to custom files, write to "parts" files, write to memory, write to mmap file, whatever you need.

[master255]

@elgatito Thank you for trying to answer that. But of course, the problem is much more complicated than you think.
I have already written my own implementation of the UTP protocol and a Java-based DHT module. My program has the most complicated algorithms to convert TCP and UDP traffic to pure data. Yes, not UTP, but UDP, without converting to UTP and message torrent.
I invented the technology of saving files to media without sparse mode.
Arvidn added a new Piece_read_alert, which could help but doesn't help because it doesn't work in Java.
And the developers of the library's Java implementations are silent for the third day.
And so we sit here, coryphaeus of torrents... and you write us about pieces and messages. Thank you, but you might have the wrong topic.

[arvidn]

I tried to convert the read_piece_alert answer to byte_vector and entry. But it's not working. A developer hasn't answered for three days now. It looks like it doesn't work in Java now.

OK. Good thing it's open source then

[master255]

@arvidn I have tested all the possibilities of getting data from read_piece_alert in FrostByte library. I tend to think they haven't added this feature yet. Studying how to add and compile it is a very big job. It's easier for me to write it in pure Java.

It's open code. Yes. But it's in C++, which needs to be implemented in Java. To do that, I need to study two projects in C++ and Java. Then study how to compile them. And all this without normal instructions.

[arvidn]

It's easier for me to write it in pure Java.

Yet, you keep coming back here :)

But it's in C++, which needs to be implemented in Java. To do that, I need to study two projects in C++ and Java. Then study how to compile them. And all this without normal instructions.

Yes, it's a lot of work, and not enough engineers in the world.

[master255]

@arvidn I might be able to help, but your library is already written and working. All I have to do is win a disk write from a certain byte. Otherwise, the library is perfect. Everything's done. My involvement doesn't make sense.
Studying two projects due to one small improvement is too much.
I can help with the algorithms. With complex algorithms, of course.

[arvidn]

I'm pretty confident read_piece() and read_piece_alert works as expected in libtorrent. I use it regularly in a program at least. So, if jlibtorrent doesn't expose it to java, that's really the only place it needs to be fixed (so, not two projects, just one I think)

[master255]

@arvidn Look at my message, maybe you know how to help?
https://github.com/frostwire/frostwire-jlibtorrent/issues/232#issuecomment-631447827.
If not, don't reply. We're waiting for Gubatron.

[master255]

@arvidn Am I doing everything right?
I start downloading with these parameters:
addTorrentParams.swig().set_disabled_storage();
addTorrentParams.flags(TorrentFlags.SEQUENTIAL_DOWNLOAD);

after : sessionHandle.addTorrent(addTorrentParams, errorCode).
I'm waiting for the TORRENT_CHECKED_ALERT, and in it I do:
session.get_torrents().get(0).pause();
session.get_torrents().get(0).read_piece(0);

then the event of successful reading of the first piece READ_PIECE_ALERT is triggered.
and I'm trying to read buffer.
Is that right?

[arvidn]

Are you sure you want to disable storage?

[arvidn]

http://libtorrent.org/reference-Storage.html#disabled_storage_constructor()

You probably need to disable hash checking as well if you disable storage

[master255]

@arvidn

Are you sure you want to disable storage?

Not sure.
I have a problem with the sparse mode. You know that very well.
My goal is to download the end of 100G file to disk without sparse mode.
I want to request the last piece and save it from a certain byte to disk.
What's the best way for me to do that?

You probably need to disable hash checking as well if you disable storage

Why? I'm downloading the data. The data has to be checked in RAM. I need this.

[master255]

@arvidn I just tested it. Disabling the hash check (disable_hash_checks = true) does not change anything. I still have an old error.

2020-05-20 20:51:15.918 1471-2440 E/AndroidRuntime: FATAL EXCEPTION: SessionManager-alertsLoop
Process: com.media.library, PID: 1471
java.lang.IndexOutOfBoundsException: vector index out of range
at com.frostwire.jlibtorrent.swig.libtorrent_jni.byte_vector_get(Native Method)

[arvidn]

my point about the hash checking is that if you throw away all the data you download, and read back garbage, all hash checks that needs to read back from the disk will fail anyway.

I wouldn't expect throwing away the data to be an acceptable solution to your sparse file issue. The way you solve it is by implementing a custom storage. I'm pretty sure I've suggested this before, so it's possible we're heading back into a circle (because I can't remember where we went from there).

[arvidn]

it sounds like you rely on the data being stored in the disk cache before being passed on to the storage (and thrown away). The disk cache is just a cache though, and you're not guaranteed that you will be able to read the piece back before any of it is flushed to disk.

Instead, you can implement a custom storage that stored everything in ram or does whatever you want, with the data.

[master255]

@arvidn Okay. I can try to help with the library if you think I can do it.
I've thought long and hard about what I need to do exactly.
My goal is more than I described in the message above.
Details:
It is necessary to add STORAGE_MODE_NO_SPARSE mode for disk storage (to AddTorrentParams). In this mode for torrent_handle we can set Map (table). When adding a new line to this table, the library must start loading torrent data from the specified byte into the specified file.
If the first piece should be saved from the middle, then load the whole piece into RAM, check the hash and save it from a certain byte specified in the table to disk (the remaining data will have to be wiped and downloaded again by different range). After the actual writing to disk, an event should be generated about saving data to disk with the current position and file name from the table.
When downloading data, it is necessary to check for collisions with other parts of the file and the end of the torrent. If a collision occurs, it is necessary to stop the downloading this part and generate an alert.
The data should be downloaded in strict (byte to byte) sequential mode. (Is it possible? Is that how it works now?). All files must be downloaded in parallel.
After the external command torrentHandle.checkShareTorrent(), the library should check the completeness of the file (ranges of bytes). If it's fully downloaded, then (and generate alert about result):

1. Mark all the pieces as verified hash.
2. Sharing all downloaded chunks with other peers on the Internet as a single file.

After that, I want it to be updated when the library is updated.
It's, as I understand it, not a little work. And what you're talking about is just a small part of what needs to be done.
Do you seriously think that a person who doesn't know much about C++ and doesn't know your project can do it? If yes, I can give it a try.

What do you think about this algorithm?

_This algorithm is ready in my player and even more. But I need a good DHT (bad available), channel aggregation and hash checking now. I'm thinking of getting out of this by downloading without channel aggregation. It will significantly reduce the need for hash checking.
I don't have the encryption yet, Pex is not ready, Holepunch is not.
If you help me, we can make a Java torrent library together. Maybe drop this C++? Let's go to Java ? :-D_

[master255]

@arvidn ?

[arvidn]

It is necessary to add STORAGE_MODE_NO_SPARSE mode for disk storage (to AddTorrentParams).

Can you explain why a custom storage is insufficient?

The data should be downloaded in strict (byte to byte) sequential mode. (Is it possible? Is that how it works now?). All files must be downloaded in parallel.

With the strictest definition of "strict"; no, that is not theoretically possible.

[arvidn]

I think it would be reasonable to have a piece picker mode where individual files are picked in order. Right now there's a simpler "sequential-download" mode, where the whole torrent is requested in order.

I think it could probably be done from the client-side too, with piece priorities. If there are some significant improvements that can be made by adding a feature to the piece picker, that might make sense.

I think the main challenge are pieces that span the start of one file and the end of another. You want to download it early bu you don't want to write the end of the previous file to disk right away.

[master255]

@arvidn

With the strictest definition of "strict"; no, that is not theoretically possible.

Ok! It means that it is necessary to reduce the file size to the last byte that has been downloaded sequentially. Is it possible?

I think the main challenge are pieces that span the start of one file and the end of another. You want to download it early bu you don't want to write the end of the previous file to disk right away.

Why do you have to load them onto a disk? We can download the last piece to RAM and check the hash there. The number of ranges will be very small (1-5 in one time). It won't take much RAM.

Can you explain why a custom storage is insufficient?

The first piece of each range I need to download to disk is not complete, but the previous piece must be downloaded to the byte I specify and collide. It turns out that the last piece will not be fully downloaded, but it must be marked as a successfully tested hash. I don't know if it's possible to do it in custom storage?
I also want to receive library updates after this modification. Is it possible?
If you do this (in main branch), all users can use it, not just me.

[arvidn]

Ok! It means that it is necessary to reduce the file size to the last byte that has been downloaded sequentially. Is it possible?

I don't know what you mean by "reduce the file size of the last byte", nor how that follows from what I said.

Why do you have to load them onto a disk?

I don't think I said you have to. I said it was a challenge.

The first piece of each range I need to download to disk is not complete, but the previous piece must be downloaded to the byte I specify and collide. It turns out that the last piece will not be fully downloaded, but it must be marked as a successfully tested hash

I don't think I follow that. It sounds like you want to partially download a piece and then assume that it passed the hash check. That doesn't seem like a terribly useful feature for most people. I think you would need a very convincing argument to land that in libtorrent.

[master255]

@arvidn

I don't know what you mean by "reduce the file size of the last byte",

Okay. I got it. It's against the hash check. But we have to do it so we don't lose the data.
For example, if the file download is broken in the middle of a piece, we need to save the part that was downloaded to disk.
If the download is restored, then the last piece must be checked for hash. But if during one download we first download the end of the piece and check its hash, and then download the beginning of the piece, then we need to check the hash of that piece a second time. (There's an error in my algorithm, yes.)

That doesn't seem like a terribly useful feature for most people.

Now for the truth. I don't think the hash check will be used to stream the video. If any data goes missing, the player will continue playing. But turning on the hashing algorithm would be even better. Your library cannot be used for serious streaming right now. I think even the developers of frostByte don't know how to modify it for that.
Only I can tell you how. And you, as a library developer, should do it. This will not only fix this bug, it will also increase the popularity of the library.

I answered a long time because I found another Libtorrent written in go and Snail lib. Now I'm writing my DHT module. (with blackjack and...)

[arvidn]

This will not only fix this bug

I don't know what "this bug" refers to. but if you're off to something else, that's fine.

[master255]

@arvidn The bug is described in this scenario: https://github.com/arvidn/libtorrent/issues/4616#issuecomment-625835184.
I haven't seen a way to fix it yet.
Bug can only be fixed with my algorithm: https://github.com/arvidn/libtorrent/issues/4616#issuecomment-632403972.
Bug exists and the developers who use this library don't know about it. For example libretorrent app.

[arvidn]

ah, I see. that's not a bug. That's by design.

[master255]

@arvidn If most users in the world do not have sparse mode, this is a bug, not a design.
If you call each bug a design, you can not fix anything.

[arvidn]

I think you are conflating libtorrent's users with your users.

[master255]

@arvidn My audience is all Android devices in the world. Android is the most popular operating system in the world.
Who is your audience?

[elgatito]

@master255 Sparse mode is not controlled by software. Software tells OS to open file with N size.
If filesystem/OS can operate sparse files - it will instantly open that file and reserve storage space.
If not - it will physically write file with zeroed bytes.

As it was said few times before, if you want to overcome that - you need your custom storage implementation, which can:

• Create part files, instead of full files. It will allow you to quickly allocate storage space when it is needed.
• Can remove non-needed part files if you need to clean up space.
• Can act like a file cache, to open file, that can fit N*{piece_size} space, and can save needed pieces to that file, overwrite non-needed pieces with something like LRU strategy.

Again, it is not a task for libtorrent library.

This issue is already 21 days old.
You could easily make own storage implementation, debug and have something that others don't have.

[master255]

@elgatito Okay, using custom storage I can do:
1) Specify for the library, which pieces have already been tested for hash?
2) Load the pieces into RAM and check the hash without writing them to disk?
3) Start loading from a specific byte if half of the required piece is already loaded?
4) Sharing the files loaded in parts (different files) as one torrent?
5) Load different parts of files at the same time, but in sequential mode?

[elgatito]

@master255

Specify for the library, which pieces have already been tested for hash?

You have a read/write methods in your storage implementation, so you decide where and how you store and how to read. Library needs byte array to calculate it's hash and verify it is fine.
Each piece has a hash and if it does not match, you cannot verify you have correct bytes for specific piece.

Load the pieces into RAM and check the hash without writing them to disk?

Up to you. Create a byte slice of a piece length, and you can save piece content to the memory.

Start loading from a specific byte if half of the required piece is already loaded?

When you add torrent, set all file priorities to 0, to avoid downloading them automatically.
Then call prioritize_piece(piece_index, priority) for each needed index, or create an array of priorities, to set pieces that you need, and unset all other pieces.

If you create 10 byte arrays to store pieces in the memory, and set 10 pieces priority > 0, then 10 pieces will be downloaded, you can store them in the memory. Once piece is read by someone (like media viewer software), you can request next piece and write in on top of non-needed piece.
That is a default strategy of streaming from the memory without any usage of disk.

Sharing the files loaded in parts (different files) as one torrent?
Load different parts of files at the same time, but in sequential mode?

Torrent libraries operate torrent as a byte array, where byte regions can be split by N bytes (piece length) to M pieces. Library needs piece, you know which byte region is that, then it is up to you, get it from memory, get from some file, or any other method.
If you have torrent length 10gb.
If you split part files to 500Mb, then if you download all files from the torrent, you will have 20 part files.
If you need only one file with length 0.7Gb, which has byte range [4_500_000_000 .. 5_200_000_000] (approximally). Then you know it is saved in files [10 .. 11], so when libtorrent asks your storage to get some piece bytes, you read from appropriate part file.

Load different parts of files at the same time, but in sequential mode?

If you enable sequential mode for a torrent and ask libtorrent to download 10 pieces with the same priority, then libtorrent will request piece from peers from lower to higher piece. This does not guarantee that they will come in that order, because peers can send slower/faster. You only know that request for piece will come sequentially.

[master255]

@elgatito https://github.com/arvidn/libtorrent/issues/4616#issuecomment-630879352
I understand your level of knowledge. I'm not interested in your answers. Thank you, but I need real answers.

[master255]

@arvidn Can you please answer these questions? If you can, shortly: yes/no and a few words why.

[arvidn]

@master255 you really should demonstrate that you've at least made the effort of reading the documentation.

[master255]

@arvidn Ok. I did the best I could. Bye.

[master255]

@arvidn Okay. I figured out how to compile frostByte libtorrent. I can compile c++ to java Android. Now I'm studying the c++ project.
I've got a new idea how to change the library.
I need to dynamically (during the download) specify a map of files from the torrent file.
For example: I only have infohash. The library finds the torrent file in dht and downloads it.
But the beginning and end of files to save to disk does not take from the torrent file, but from the table.
And I need to set the table dynamically during the download.
And is it possible to do that several files are loaded sequentially and at the same time?

[arvidn]

And is it possible to do that several files are loaded sequentially and at the same time?

Probably. I don't know what you mean by "sequentially and at the same time". That sounds like a contradiction.

[master255]

@arvidn What's the contradiction? Is it impossible to load sequentially from different places in the file?
If the speed of peers is enough?
For streaming, it is necessary to limit the maximum speed. For example, if the speed of the main thread is more than necessary for playback by 2 times, it is possible to start downloading other parts of the file.

It's complicated logic. I doubt we can do it in C++.

[arvidn]

What's the contradiction?

"sequentially" means one part before the other, in some order they appear.
"at the same time" means in parallel, which is different from one part before the other. When doing things in parallel there is no ordering.

[master255]

@arvidn There may be several files in the torrent.
Why can't I download several files at once? What's the problem?
Let's be honest, sequential downloading is only necessary for streaming, and for streaming it is necessary to download one file as several files.

[arvidn]

Why can't I download several files at once?

What happens when you try?

I'm sure you can figure out how to do what you want from the documentation. Please let me know if you think the description of any specific piece of functionality is imprecise and we can try to fix that.

[master255]

@arvidn If you don't want to help, then write it. Don't make up ridiculous questions.

[arvidn]

If you don't want to help, then write it

I'm not interested in writing your program. I'm happy to clarify details about libtorrent if the documentation is unclear.

[master255]

@arvidn This is the choice:

1. Either you help me finish your library and I use it (and everyone uses it).
2. Either you don't help me and I'll finish your library myself. And I'm the only one who uses it.
3. Either I finish my library and I use it. I'm the only one who uses it.
4. Either you help me finish my Java library and I will publish it. (and everyone is using it)

Now I already have UTP, DHT on one port. Aggregation of peers in development.

[arvidn]

patches are welcome

[master255]

@arvidn Ok. Now I understand why so few programmers help with the project.

[master255]

@arvidn bug still exists

[arvidn]

Assuming the alleged bug you're referring to is the title of this ticket ("not working without sparse mode") I have not seen any compelling evidence that anything breaks, or isn't working. As recent as a few days ago I verified with client_test that I can download to a drive formatted with FAT. It completed without errors.

You have spent a lot of time not showing any interest in providing any such evidence, in fact, you have not shown any good faith attempt at helping to fix anything. At this point I seriously doubt you will. You mostly seem interested in trying to talk me into helping you write your program, and it's not going very well.

[master255]

@arvidn I may have misunderstood. I have a lot of proof!
Download this 4k.zip file to an external usb flash drive or sd card via libretorrent (which runs on your libtorrent library). No sequential mode. And you will see that the download will hang, even if there are peers.
For a very long time!

[arvidn]

I prefer to use my own client, to know what's going on.
Is there any important difference between that torrent and an ubuntu image? They are both single-file torrenst.
Are there any specific settings that triggers this problem?

If the issue you are referring to is that if you close the file before completing it, one disk I/O thread will be blocked while waiting for the kernel to fill in the zeroes, I don't think it's accurate to characterize it as "Not working". If there's some other problem, could you be a bit more precise in your description of the symptom?
You're a programmer, so maybe you could even dig up a stack trace or a log.

[master255]

@arvidn The problem is that in the first minutes of the file download one of the last blocks of the file comes and the application tries to create the whole file, which takes several hours. The application and data download hangs for several hours.

Is there any important difference between that torrent and an ubuntu image?

A very big difference is the size.

Are there any specific settings that triggers this problem?

Just don't use sequential mode.

At this point I seriously doubt you will.

I'm not stopping until I solve this problem. This problem is much more serious than it looks. I need a result - that's what matters.

[arvidn]

application tries to create the whole file, which takes several hours. The application and data download hangs for several hours.

More precision please. It sounds like there's a call to write() which hangs while the kernel is filling the file with zeroes. Do all other threads in the process halt too? If so, where and why?

If this is in fact the behavior, it would seem somewhat unique to Android. Mac OS and Linux does not block in write() calls.

[master255]

@arvidn

Do all other threads in the process halt too? If so, where and why?

I don't have c++ debugging in Android. That's not possible. But obviously all the threads are hanging. The download is not going on at all. I've been waiting over 10 minutes. At the same time, on the internal memory, the file starts downloading immediately.

What if peer one? And he only has the end of the file? Does the download start in 2 hours? When will the file be created? It's an obvious bug to me.

[arvidn]

I don't have c++ debugging in Android. That's not possible.

I'm positive gdb exists for android. You may have to run gdbserver with remote debugging, but it's definitely possible.

What if peer one? And he only has the end of the file? Does the download start in 2 hours? When will the file be created?

You're the one experiencing this, you're the one who can answer those questions, not me. How can you know what the solution is if you don't have a solid understanding of the problem?

[master255]

@arvidn

You're the one experiencing this, you're the one who can answer those questions, not me.

I see a 100% bug. The library is trying to create a 50GB file on a drive that does not support sparse mode to write 10MB of data to the end of the file.
Not only does it take a very long time, it also creates a giant file that the user doesn't need.
In fact the traffic is 200Mb, but it creates a file of 50Gb, do you think it's normal??
I don't think it's normal. What happens if a user runs 100 such files? 100*50Gb? On a phone with 10 GB free? (That's sarcasm)
The size of the downloaded files should be equal to the size of the traffic.

What if peer one? And he only has the end of the file? Does the download start in 2 hours? When will the file be created?

It's not questions, it's statements. Like I said, it's a very serious problem. Your library will ignore peers under certain conditions. It caused me to stop loading completely with 20 peers.

How can you know what the solution is if you don't have a solid understanding of the problem?

I've tested the problem many times. It's not loading. And in my player, that' s not working out like in your library. My player saves files on a disk that are equivalent in size to the incoming traffic. It is reasonable and correct, and the main thing is that you don't create 50GB files.

I'm positive gdb exists for android.

What do I need this for now? Do you expect this problem to be solved by multithreading?)).
It's funny... but no. While the file is filled with zeros, other threads cannot write data into it.

[elgatito]

@arvidn you are too much polite and patient for this case.

[arvidn]

The size of the downloaded files should be equal to the size of the traffic.

Tell the people developing Android.

What do I need this for now? Do you expect this problem to be solved by multithreading?)).

Maybe multi-threading will solve it, maybe not. It seems we don't know for sure right now. If you feel confident that it won't you don't have to try it (although, it would just be a matter of changing the settings_pack::aio_threads setting).

If it is absolutely unacceptable to write to the end of sparse files for you, there are lots of options for you. All supported by libtorrent. It's not clear to me why you seem to ignore them. In libtorrent you can control which pieces are requested. You can control exactly how those pieces are written to disk. In fact, just write every single piece to a separate file; problem solved.

It's hard to believe you're actually an engineer. it seems you spend more time complaining in github, trying to make other people develop your program, than actually experiment, investigate and come up with solutions to your challenges.

It's funny... but no. While the file is filled with zeros, other threads cannot write data into it.

I'm pretty sure that's not the case on Linux, in my experience. I know because I use debuggers.

[arvidn]

@master255 how about you don't respond to this thread until you've actually gone and tested some possible solution and have some new discoveries or understanding to report?

[master255]

@arvidn The problem is your misunderstanding. I write a big program that has a lot of libraries like yours (8-10). If I write these libraries myself, I'll never write a big program.
Because you didn't want to help me adapt your library, I started writing my library in pure Java. I spent a lot of time on it. At the same time, I continue to encounter your unfriendliness.
I haven't had a chance to study the library because your instructions are not complete and clear.
I need more than just to save each piece to a separate file.
I need to start downloading the torrent from a certain byte.
If saving each piece to a separate file will help you, then add this mode and then there is no bug!
But this is a very primitive solution to the problem.
If you want me to try and add a new mode without sparse file support, I need a chat and I have a lot of questions.

[arvidn]

Because you didn't want to help me adapt your library,

as I said, patches are welcome

I haven't had a chance to study the library because your instructions are not complete and clear.

I'm happy to improve any documentation if you can point out what is unclear or under specified.

If saving each piece to a separate file will help you, then add this mode and then there is no bug!

This is already supported by libtorrent, via a custom storage.

If you want me to try and add a new mode without sparse file support, I need a chat and I have a lot of questions.

I don't have a particularly strong opinion whether you do this or not. You're more than welcome to, but if you were serious about it, I would have expected you to have started a month ago.

[master255]

@arvidn And "no sparse mode" is not a type of storage. It is a type of file. There are small and large files in the torrent. This mode is not needed for small files, but for large files.

but if you were serious about it, I would have expected you to have started a month ago.

It is impossible to learn torrents and c++ language - in one month.

This is already supported by libtorrent, via a custom storage.

Storage model is really not badly written. But without my logic, it's useless.

[master255]

@arvidn New information about sparse mode in Android TV.
If to use usb external drive as an external drive, besides absence of sparse mode there is no possibility to create directories. Directories cannot be created on NTFS and ExtFat file systems, but on Fat32 it is possible. But on Fat32 - there is a limit of maximum file size 4gb. All these file systems are not suitable.
I beat the lack of sparse mode, but how to beat the lack of directories I do not know. This is not possible.
But there is only one way to get around this limitation.
Move all applications to an external drive. In this mode, the file system is formatted in Ext4, which supports files of any size and what is more epic sparse mode.
Flash drive for this conversion is not suitable - it will work very slowly, because it does not support multi-threading.
So the ideal external drive for this is an external ssd drive. The maximum capacity that supports Android TV 256 GB., which is more than enough for torrents and other applications.
Support for sparse mode is more, I think, not relevant.

[master255]

@arvidn upd:
For Android TV is not relevant, but relevant for Android smartphones with external usb otg flash drives.
Maybe for some Android TV consoles, which do not allow you to transfer applications to an external drive.