Cockpit: File Upload feature to cockpit

[ghost]

Page:

Is there any way to upload files to cockpit server from remote system browser? if yes please let me know. Thank you so much,

[marusak]

No, unfortunately there is no such feature implemented. This could be a application that would made it possible to manipulate with files (see https://github.com/cockpit-project/starter-kit)

[mckenziec]

I second this. I'd like have a custom plugin be able to import files for a docker container volume mount.

I was planning to see if I could use browser JS and cockpit.js to write a file as part of an upload style test.

[martinpitt]

Sorry, this isn't a good goal for the cockpit project itself. As pointed out above, this would be an excellent target for a standalone project based on starter-kit. We are happy to help interested people with the bootstrapping, CI, etc., but we won't create that ourselves.

[stefwalter]

The general facility to upload a file through the various javascript APIs and/or cockpit channels does not exist. @martinpitt This is something that people ask for every year or so. There's another post on the mailing list about this.

[martinpitt]

Oops, I mis-read that as "file management", as in "nautilus in a browser" (there was an issue for that as well). Sorry!

[gsanchietti]

We managed to upload a file using cockpit.spawn: the JavaScript code sends the file in base64 format into the stdin of the command.
The JS part execute something like:

var args = ["/usr/libexec/nethserver/api/system-backup"];
var process = cockpit.spawn(args);
process.input(JSON.stringify({ data: our_base64_encoded_data })) // our_base64_encoded_data 
process.done(function (successResp) {}).fail(function (errorResp, errorData) {})

As for now, we can upload only files smaller than 128KB which is the web socket payload size limit: https://github.com/cockpit-project/cockpit/blob/dee6324d037f3b8961d1b38960b4226c7e473abf/src/websocket/websocketconnection.c#L154

Is there any chance to have such limit exposed inside cockpit.conf?

[mckenziec]

I was going to get around to working out a large file upload technique using the JS FileReader object: https://www.javascripture.com/FileReader

There's a readAsArrayBuffer() method that should get you any file in a byte array. You can then attempt to encode or write it to the server using cockpit.spawn and piping (input) the data to it, or using the cockpit.file() method.

Small files aren't a big problem, its larger files and increasing the MAX_PAYLOAD size is probably not a great idea. However making it configurable if you're willing to change it is totally reasonable.

What I was hoping to work out was a way of reading a local file in a browser and chunking it up to be streamed to the server. If FileReader actually returned a iterator of buffered data as the file is read until the end, then you could make repeated calls to append the data (via cockpit.spawn) or cockpit.file() write it.

For that I was going to look at the File.slice() method: https://riptutorial.com/javascript/example/16626/slice-a-file

"This is useful especially in cases where you need to process files that are too large to read in memory all in once. We can then read the chunks one by one using FileReader."

So all the pieces are there (theoretically) to read in chunks/slices of a local file passed into FileReader. Here's a jsfiddle example that reads a file in 1KB chunks. It should be enough to clue you in on how it could be done. http://jsfiddle.net/mw99v8d4/

I do not know if this approach will reduce the amount of memory the client browser will have to allocate for the file read. Technically is overrides the FileReader().onload function to parse the result. That doesn't mean that the file isn't 100% read into the result before onload is called. If reading the whole file can't be avoided (due to available browser technology), then this is the best you can hope for (a method for chunking the read file ASAP, so you can write it else where in pieces).

The only hang up is how to write it to the server in a compatible way. The async promise callback approach to everything means a long series of promises chained together to get 10 chunks appended into a file using spawn.

_IF_ you went on the assumption that you would be reading/parsing the file in order, then promise writes would be performed in order, and the chances of an out of order write were slim (but not impossible because of the async nature off making a series of non-blocking spawn calls to "echo $STUFF >> $FILE") you could just try firing off a bunch of spawns, and if the write fails, junk the file altogether.

What cockpit.js could do is give us a clear mechanism for appending to an existing file, or the best would be a file write method via a callback, where you can construct it (point to a desired file target), then call a .write() method when you need to, then a .close() method to finish the write. Then you could construct it, do whatever you need to to generate/obtain the data, and write a series of chunks, finally closing it. Under the hood, cockpit.js could track the writes with an incrementing ID, so cockpit/bridge could ensure that the data was all written correctly/inorder.

Good luck and I'll post something if I work it out. Ideally I'd like to get to a point where I can upload a docker image tar file.

[mckenziec]

Oh, I didn't know about the "await" operator! https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await

So you could use FileReader with an overridden .onload method to slice chunks, then call spawn or cockfile.file to write, but block/await for the write to complete before reading the next chunk. I don't know how smooth/elegant that would be, but you could read chunks, write to the server in order.

[stefwalter]

We managed to upload a file using cockpit.spawn: the JavaScript code sends the file in base64 format into the stdin of the command.
The JS part execute something like:

var args = ["/usr/libexec/nethserver/api/system-backup"];
var process = cockpit.spawn(args);
process.input(JSON.stringify({ data: our_base64_encoded_data })) // our_base64_encoded_data 
process.done(function (successResp) {}).fail(function (errorResp, errorData) {})

As for now, we can upload only files smaller than 128KB which is the web socket payload size limit:

https://github.com/cockpit-project/cockpit/blob/dee6324d037f3b8961d1b38960b4226c7e473abf/src/websocket/websocketconnection.c#L154

Is there any chance to have such limit exposed inside cockpit.conf?

There's really no need to reduce this limit. I suggest changing the code as follows:

var args = ["/usr/libexec/nethserver/api/system-backup"];
var process = cockpit.spawn(args);
var data = JSON.stringify({ data: our_base64_encoded_data });
for (var i = 0; i < data.length; i += 65535)
    process.input(data.substr(i, 65535), true);
process.input()
process.done(function (successResp) {}).fail(function (errorResp, errorData) {})

I know we need to make file uploads less awkward ... but until then the above should scale beyond 128K. Hope that helps.

[gsanchietti]

Thank you @stefwalter this is an excellent solution for us!
We just added a final call to process.input() otherwise no EOF was sent.
For any one interested this is our working code:

var process = cockpit.spawn(args);
var data = JSON.stringify(input)
for (var i = 0; i < data.length; i += 65536) {
    process.input(data.substr(i, 65536), true);
}
process.input() // send EOF
process.done(function (successResp) {}).fail(function (errorResp, errorData) {})

[stefwalter]

Aha, good catch. I've updated the snippet above in case anyone else uses it and runs into same thing.

A next step for me would be to follow up and have process.input() break up long content without forcing the caller to have to do it.

[martinpitt]

Thanks @gsanchietti and @stefwalter! So we could use this in conjunction with calling a command like dd of=path to get a generic file upload way, and at least document this now, so that it's easier to find?

[stefwalter]

@martinpitt That should already work with cockpit.file() and file.replace(). But yes the FileReader part on the client side needs documentation.

[martinpitt]

It of course would be nice if the cockpit.file() API would support sending the contents in multiple chunks, but cockpit.js doesn't currently do that.

However, the protocol does allow it. I. e. you can open a raw channel with the payload="fsreplace1" type, and feed the data using channel.send() in multiple blocks. I'm not aware of an existing example there, but that's IMHO
the most elegant and efficient way.

[stefwalter]

I hope to do some work to factor in the flow control into the javascript to send multiple blocks ... so that the transfer is memory efficient. But in principle, I agree with @martinpitt

[aravindj]

Hi,
I am fairly new to cockpit.
I am trying to achieve the file upload functionality similar to the way discussed above.
I have spawned a 'dd' process and trying to stream chunks of file data read via FileReader.

The code for the same is in
https://gist.github.com/aravindj/4c2842659ff437ec13fba4fd043ef413

However, I am getting a lots of "cockpit.js:647 sending message on closed channel" warnings in my console logs and not more than 1kb of data is being stored in the target file. (The behaviour is inconsistent. Sometimes I could see around 8Kb of data being stored in the target file).

Any help would be appreciated.

P.S: Not sure if it is fine posting here. But since the discussion is still on, thought of posting the query here.

[massey101]

I haven't tried the above JSReader option yet, has anyone given it a go for large files? >4GB?

I have done a workaround by making a small Flask application that's installed alongside the cockpit plugin. Then one of the components uses cockpit.spawn() to start it when required. Really we just need an endpoint that accepts POST requests and saves the attached file somewhere. Is this something that could be added to cockpit-bridge? Something like:

const fileServer = cockpit.fileUploadServer('/url/route/', '/tmp');
fileServer.stop();