Srt: using stransmit for file transfer

[nitrat7]

hello,
I've used a while ago sample-apps from UDT for file transfer and I was quite astonished about performance in comparison to SCP/FTP on high-latency networks.

So I thought that I might could use stransmit-process for file delivery too, but connection drops.

Sender
./stransmit file:///Users/test/source.file srt://40.74.114.99:4000?mode=caller

Receiver
./stransmit srt://:4000/?mode=listener file:///home/test/destination.file

Issue/question
I see that connection breaks without any reason, should given setup work for file-transmission?

[ReubenM]

SRT is not a generic transport. It is specific to MPEG2-TS. If you want to transfer files use UDT.

[nitrat7]

@ReubenM ok thanks, just thought, it could be a nice "side"-feature (as you refer to FILE-input/output in documentation).
As far as I know SRT is agnostic from either using MPEG2-TS or any other ...

[ghost]

SRT is completely content agnostic and not at all limited to MPEG-TS. I will check whether the demo app is somehow limited. Regarding UDP MultiCast, i could imagine the app didn't implement it. I will also check that. Again , SRT has nothing to do with the UDP stream coming in. If there is a limitation, it's the sample app implementation.

[ghost]

@nitrat7 We are actively working on file transfer support in order to use SRT as TCP replacement. That functionality should be ready soon.

[ReubenM]

For some reason I was under the impression that it was video specific similar to RTP. I guess I've never seen it used outside of a video transport context. I have used stransmit with file IO, but it was always a .ts file or a stdio pipe.

[ethouris]

I think there are currently two problems with the file transfer:

• As this is predicted to be used for live streaming only, there's no EOF situation predicted and it's not handled properly by stransmit, which will result in dropping the last part of the file. AFAIR there's also a small problem with close() function, which doesn't do kinda "sync on termination complete" (on live streaming transmission it doesn't matter, but it makes the application break before the sender worker thread finishes flushing out the output buffer)
• When you use encryption, the KMX must be done first, and it's currently triggered with the first portion of data sent. The receiving party can't decrypt withoug KM, so some of the initial part will be dropped or turned into a garbage.

I'm working on that currently, but it needs to almost completely reinvent the handshake procedure, so it will take a bit more time.

[ReubenM]

There is a UDT flush implementation that might be useful:

Implemented Flush

[ethouris]

Thanks, but I think we cannot exactly take advantage of it. As I understand correctly, this code exposes the part of close() that does the flushing without closing the socket. In our case the problem is different: even the close() function doesn't exactly do everything that is necessary for complete flushing the data.

What I most suspect is - and this is then the problem that also exists in UDT - that some final job must be done by the queue worker thread, so close() should simply make sure to join this thread. This isn't done, even in this fixed version, this guy probably thought that "sleeping for 1 second" should suffice. Such a workaround can be done also in stransmit by just waiting 5 seconds after breaking the connection, but it's not a solution.

In the current situation the application exits before the thread finishes the job, so the thread gets silently abnormally terminated. This is so in POSIX threads, not sure whether this is intended and formally defined, but so it is. Were these C++11 threads, the attempt to abnormally terminate a running thread would result in std::terminate() and the software wouldn't even make it to the surface without having that fixed.

[nitrat7]

Hello,
I have checked out current DEV-branch and tested file-transfer.
https://bitbucket.org/martingrohme/stransmit-ubuntu
https://bitbucket.org/martingrohme/stransmit-ubuntu/downloads/stransmit-ubuntu-filesupport.zip

Setup running sender AWS instance [email protected] receiver AWS instance [email protected]

Sender:
__
root@ip-172-31-45-13:/usr/local/bin# dd if=/dev/zero of=/usr/local/bin/output.dat bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.648694 s, 162 MB/s
root@ip-172-31-45-13:/usr/local/bin# ./stransmit file:///usr/local/bin/output.dat srt://54.157.243.34:5002
(DEBUG) EOF when reading file. Looping until the sending bufer depletes.
(DEBUG)... still 4409952 bytes
(DEBUG)... still 4409952 bytes
(DEBUG)... still 4409952 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
(DEBUG)... still 4053316 bytes
terminate called after throwing an instance of 'std::runtime_error'
what(): Watchdog bites hangup
Aborted (core dumped)
__

Receiver:
__
root@ip-172-31-25-67:/usr/local/bin# ./stransmit srt://:5002 file:///usr/local/bin/output.dat

ERROR #-1000: Unknown error.
STD EXCEPTION: error in recvmsg: Unknown error.
Waiting 5s for possible cleanup...
root@ip-172-31-25-67:/usr/local/bin# ls -al output.dat
-rw-r--r-- 1 root root 18154220 Sep 19 23:15 output.dat
__

=> There's missing huge amount of data on receiver, if you compare file on sender.
This loss seems to grow as bigger file.

=> is my test-setup valid?
=> do you need more info to reproduce issue?

[ethouris]

The stransmit application isn't predicted for file transfer at all (although probably it may work this way at some point). There's another application for this purpose: sf. It utilizes also the "stream ID" facility, through which the filename is passed - if listener is receiver, then it will use the filename passed by caller as a target filename.

There are two reasons as to why this needs another application:

• Live and file (VOD) transmission modes require two different sets of parameters. For file transfer there are several options that must be set together. The default values are suitable for live transmission, the sf application sets them correctly for file transmission.
• There's still the strict API alignment for particular transmission mode, as it was in UDT: srt_send/srt_recv functions are for file transfer, whereas srt_sendmsg/srt_recvmsg for live transfer. That's an unfortunate legacy of UDT which is going to be changed soon, but for the time being it's still in force. The sf application uses the first set.

The most important difference between Live and File transmission is that in Live transmission the library relies on the input data sending speed, whereas in File transmission the input speed is considered potentially infinite, so the SRT library must split the data into single UDP-sendable units and define the sending speed by itself. Passing the data from the "infinite speed" source to a medium prepared for live transmission may result in undefined behavior.

[nitrat7]

Hi @ethouris
I did not see any reference to sf in CMakeLists.txt, but for siplex in dev-branch.
Here I see multiplexing-option, you have referred to.

Unfortunately still I see issue on file transfer.

Sender:

./siplex srt://54.242.161.87:5002 -i output.dat?id=1

Receiver (54.242.161.87):

./siplex srt://:5002 -o output.dat?id=1
__
ERROR #2001: Connection was broken.
__

Source file on Sender has 100MBytes, on Receiver-Side there's only 22 or 18 or... MBytes

Do I refer to wrong branch, where's app sf?

Thanks Martin :-)

[ethouris]

Ah, I'm not on exactly right track, doing too many things simultaneously :)

The branch that contains the file transfer support - and the sf application - is currently in "pending" state, it's in the pull requests for the time being. Should be merged to dev branch soon.

[nitrat7]

Hello,

I have now tested dev-branch Version 1.3 and app "srt-file-transmit".
Apparently now file-transfer is working 👍

e.g. Sender
./srt-file-transmit file:///home/martin/srt/output10m.dat srt://34.226.205.121:5002
e.g Receiver
./srt-file-transmit srt://:5002 file:///usr/local/bin/output.dat

=> File-Transfer is very fast between e.g two AWS-Regions - 100MBytes in <7 seconds vs. SCP about 25 seconds.

BUT

If I choose e.g. DSL-Connection (at home) to AWS-Region far away, performance is not good.
e.g.
SCP about 3 seconds for 1.5MBytes and > 55 seconds for SRT.
=>
Why is performance so bad in this usecase?

If I compare verbose output on very slow transmission with fast transmission I see a lot of entries "Sending buffer still: ..." before end, this takes a lot of time.

---

Martins-MacBook-Pro-FAST:srt martingrohme$ ./srt-file-transmit file:///Users/martingrohme/aha22.ts srt://34.226.205.121:5002 -r:1000 -v
SOURCE type=file, TARGET type=srt
Extract path '/Users/martingrohme/aha22.ts': directory=/Users/martingrohme filename=aha22.ts
Parameters:
transtype = 'file'
Passing 'aha22.ts' as stream ID

Connect with requesting stream [aha22.ts]
Connecting to 34.226.205.121:5002 ... connected.
Extracted outgoing port: 59200
USING ID: aha22.ts
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
....
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1456 --> 1456
Upload: 1368 --> 1368
Sending buffer still: bytes=1582584 blocks=1087
Sending buffer still: bytes=1559288 blocks=1071
Sending buffer still: bytes=1557832 blocks=1070
Sending buffer still: bytes=1554920 blocks=1068
Sending buffer still: bytes=1553464 blocks=1067
Sending buffer still: bytes=1549096 blocks=1064
Sending buffer still: bytes=1541816 blocks=1059
Sending buffer still: bytes=1540360 blocks=1058
Sending buffer still: bytes=1537448 blocks=1056
Sending buffer still: bytes=1533080 blocks=1053
Sending buffer still: bytes=1519976 blocks=1044
Sending buffer still: bytes=1492312 blocks=1025
Sending buffer still: bytes=1482120 blocks=1018
Sending buffer still: bytes=1476296 blocks=1014
Sending buffer still: bytes=1447176 blocks=994
Sending buffer still: bytes=1429704 blocks=982
Sending buffer still: bytes=1420968 blocks=976
Sending buffer still: bytes=1418056 blocks=974
Sending buffer still: bytes=1388936 blocks=954
Sending buffer still: bytes=1370008 blocks=941
Sending buffer still: bytes=1355448 blocks=931
Sending buffer still: bytes=1335064 blocks=917
Sending buffer still: bytes=1298664 blocks=892
Sending buffer still: bytes=1278280 blocks=878
Sending buffer still: bytes=1271000 blocks=873
Sending buffer still: bytes=1257896 blocks=864
Sending buffer still: bytes=1222952 blocks=840
Sending buffer still: bytes=1205480 blocks=828
Sending buffer still: bytes=1198200 blocks=823
Sending buffer still: bytes=1188008 blocks=816
Sending buffer still: bytes=1183640 blocks=813
Sending buffer still: bytes=1148696 blocks=789
Sending buffer still: bytes=1137048 blocks=781
Sending buffer still: bytes=1129768 blocks=776
Sending buffer still: bytes=1125400 blocks=773
Sending buffer still: bytes=1096280 blocks=753
Sending buffer still: bytes=1084632 blocks=745
Sending buffer still: bytes=1070072 blocks=735
Sending buffer still: bytes=1059880 blocks=728
Sending buffer still: bytes=1046776 blocks=719
Sending buffer still: bytes=1036584 blocks=712
Sending buffer still: bytes=1026392 blocks=705
Sending buffer still: bytes=1022024 blocks=702
Sending buffer still: bytes=997272 blocks=685
Sending buffer still: bytes=985624 blocks=677
Sending buffer still: bytes=978344 blocks=672
Sending buffer still: bytes=976888 blocks=671
Sending buffer still: bytes=955048 blocks=656
Sending buffer still: bytes=939032 blocks=645
Sending buffer still: bytes=933208 blocks=641
Sending buffer still: bytes=924472 blocks=635
Sending buffer still: bytes=899720 blocks=618
Sending buffer still: bytes=882248 blocks=606
Sending buffer still: bytes=876424 blocks=602
Sending buffer still: bytes=869144 blocks=597
Sending buffer still: bytes=840024 blocks=577
Sending buffer still: bytes=829832 blocks=570
Sending buffer still: bytes=822552 blocks=565
Sending buffer still: bytes=815272 blocks=560
Sending buffer still: bytes=813816 blocks=559
Sending buffer still: bytes=777416 blocks=534
Sending buffer still: bytes=767224 blocks=527
Sending buffer still: bytes=758488 blocks=521
Sending buffer still: bytes=757032 blocks=520
Sending buffer still: bytes=727912 blocks=500
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=706072 blocks=485
Sending buffer still: bytes=674040 blocks=463
Sending buffer still: bytes=669672 blocks=460
Sending buffer still: bytes=647832 blocks=445
Sending buffer still: bytes=644920 blocks=443
Sending buffer still: bytes=637640 blocks=438
Sending buffer still: bytes=614344 blocks=422
Sending buffer still: bytes=598328 blocks=411
Sending buffer still: bytes=596872 blocks=410
Sending buffer still: bytes=589592 blocks=405
Sending buffer still: bytes=567752 blocks=390
Sending buffer still: bytes=553192 blocks=380
Sending buffer still: bytes=548824 blocks=377
Sending buffer still: bytes=543000 blocks=373
Sending buffer still: bytes=522616 blocks=359
Sending buffer still: bytes=512424 blocks=352
Sending buffer still: bytes=509512 blocks=350
Sending buffer still: bytes=502232 blocks=345
Sending buffer still: bytes=487672 blocks=335
Sending buffer still: bytes=477480 blocks=328
Sending buffer still: bytes=471656 blocks=324
Sending buffer still: bytes=465832 blocks=320
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=454184 blocks=312
Sending buffer still: bytes=433800 blocks=298
Sending buffer still: bytes=429432 blocks=295
Sending buffer still: bytes=426520 blocks=293
Sending buffer still: bytes=419240 blocks=288
Sending buffer still: bytes=410504 blocks=282
Sending buffer still: bytes=410504 blocks=282
Sending buffer still: bytes=395944 blocks=272
Sending buffer still: bytes=393032 blocks=270
Sending buffer still: bytes=385752 blocks=265
Sending buffer still: bytes=384296 blocks=264
Sending buffer still: bytes=374104 blocks=257
Sending buffer still: bytes=368280 blocks=253
Sending buffer still: bytes=362456 blocks=249
Sending buffer still: bytes=361000 blocks=248
Sending buffer still: bytes=355176 blocks=244
Sending buffer still: bytes=343528 blocks=236
Sending buffer still: bytes=337704 blocks=232
Sending buffer still: bytes=334792 blocks=230
Sending buffer still: bytes=331880 blocks=228
Sending buffer still: bytes=321688 blocks=221
Sending buffer still: bytes=315864 blocks=217
Sending buffer still: bytes=314408 blocks=216
Sending buffer still: bytes=310040 blocks=213
Sending buffer still: bytes=304216 blocks=209
Sending buffer still: bytes=298392 blocks=205
Sending buffer still: bytes=292568 blocks=201
Sending buffer still: bytes=291112 blocks=200
Sending buffer still: bytes=280920 blocks=193
Sending buffer still: bytes=275096 blocks=189
Sending buffer still: bytes=270728 blocks=186
Sending buffer still: bytes=267816 blocks=184
Sending buffer still: bytes=259080 blocks=178
Sending buffer still: bytes=254712 blocks=175
Sending buffer still: bytes=248888 blocks=171
Sending buffer still: bytes=247432 blocks=170
Sending buffer still: bytes=234328 blocks=161
Sending buffer still: bytes=231416 blocks=159
Sending buffer still: bytes=222680 blocks=153
Sending buffer still: bytes=221224 blocks=152
Sending buffer still: bytes=209576 blocks=144
Sending buffer still: bytes=205208 blocks=141
Sending buffer still: bytes=200840 blocks=138
Sending buffer still: bytes=196472 blocks=135
Sending buffer still: bytes=192104 blocks=132
Sending buffer still: bytes=183368 blocks=126
Sending buffer still: bytes=177544 blocks=122
Sending buffer still: bytes=173176 blocks=119
Sending buffer still: bytes=170264 blocks=117
Sending buffer still: bytes=160072 blocks=110
Sending buffer still: bytes=154248 blocks=106
Sending buffer still: bytes=148424 blocks=102
Sending buffer still: bytes=145512 blocks=100
Sending buffer still: bytes=135320 blocks=93
Sending buffer still: bytes=129496 blocks=89
Sending buffer still: bytes=123672 blocks=85
Sending buffer still: bytes=119304 blocks=82
Sending buffer still: bytes=110568 blocks=76
Sending buffer still: bytes=106200 blocks=73
Sending buffer still: bytes=98920 blocks=68
Sending buffer still: bytes=97464 blocks=67
Sending buffer still: bytes=85816 blocks=59
Sending buffer still: bytes=82904 blocks=57
Sending buffer still: bytes=77080 blocks=53
Sending buffer still: bytes=75624 blocks=52
Sending buffer still: bytes=65432 blocks=45
Sending buffer still: bytes=61064 blocks=42
Sending buffer still: bytes=56696 blocks=39
Sending buffer still: bytes=53784 blocks=37
Sending buffer still: bytes=45048 blocks=31
Sending buffer still: bytes=40680 blocks=28
Sending buffer still: bytes=36312 blocks=25
Sending buffer still: bytes=33400 blocks=23
Sending buffer still: bytes=29032 blocks=20
Sending buffer still: bytes=21752 blocks=15
Sending buffer still: bytes=14472 blocks=10
Sending buffer still: bytes=10104 blocks=7
Sending buffer still: bytes=7192 blocks=5
Sending buffer DEPLETED - ok.
SrtCommon: DESTROYING CONNECTION, closing sockets (rt%421174334 ls%-1)...
SrtCommon: ... done.

---

[metairie]

I re-test in the same network (Azure).
My first tests failed between a Linode server and Azure server

[metairie]

with srt-file-transmit
everything is fine !

[rmsoto]

I can say it works fine with AWS and local machines. Great job!

[ghost]

Nice!

[ethouris]

Just to mention - all these messages about sending and depleting the sender buffer appear due to -v option. Transferring a file (unlike live) uses internal speed control and tries to push the speed to the limit, so printing anything on the screen while transmitting might really matter.