Panel: Duplicate Files on SFTP

[LeCanadienOP]

Hello,

• Daemon Version: Lasted (Installed Today)

• Panel Version: 0.7.6

• Server Version: Ubuntu Server 16.04 "Xenial Xerus" LTS (64bits)

When users try to connect in SFTP connection, they have duplicate files of all based files. Like 4 x garrysmod, 4 x bin, 4 x addons... I send 2 screens for more examples.
.

I try to launch a diagnostics, Results:

at process._tickDomainCallback (internal/process/next_tick.js:218:9)
16:51:12.850Z ERROR wings: An exception was encountered while handling the SFTP subsystem. (server=2f6772de-8fac-4dca-bf86-4224fc70dabb)
exception: {
"code": "ECONNRESET",
"errno": "ECONNRESET",
"syscall": "read"
}

Informations about Daemon

--| Docker Information

Containers: 8
Running: 2
Paused: 0
Stopped: 6
Images: 4
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-72-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 128.63GiB
Name: rowlt-dedicated-1
ID: CLMR:LSXL:GTX2:DAZJ:YLOC:5C3T:5DPQ:EQIT:GAFE:XBMN:ZYPK:DVUR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8

Logs of FileZilla Nothing annormal.

Status: Connected to panel.-----.com
Status: Retrieving the contents of the folder ...
Status: Ad Directory /
Status: Content of the folder "/" displayed successfully

[LeCanadienOP]

I found 2 errors:

Error: Can not understand the SFTP response packet from the server: request ID mismatch
Error: Unable to recover the contents of the folder

TypeError: Cannot read property 'server' of undefined
    at Client.client.on.on.on.err (/home/daemon/src/http/sftp.js:496:31)
    at emitOne (events.js:116:13)
    at Client.emit (events.js:211:7)
    at SSH2Stream.<anonymous> (/home/daemon/node_modules/ssh2/lib/server.js:311:10)
    at emitOne (events.js:121:20)
    at SSH2Stream.emit (events.js:211:7)
    at onDISCONNECT (/home/daemon/node_modules/ssh2-streams/lib/ssh.js:2204:10)
    at SSH2Stream.<anonymous> (/home/daemon/node_modules/ssh2-streams/lib/ssh.js:203:5)
    at emitMany (events.js:147:13)
    at SSH2Stream.emit (events.js:224:7)

[vipesz]

Contribution:

I've had this issue in my server deployment over at my hosting company. I've been attempting to find a fix, and the only thing that currently works is to restart the sshd service - then run

docker stop $(docker ps -aq) && docker rm $(docker ps -aq)

The files then seem to go back to normal. My servers run on CentOS 7.4

If there's a way Dane could clarify how the new sftp system works in regards to connections, I may be able to find a patch.

[DaneEveritt]

The current system doesn't touch the system's sshd service, so I really don't see how that has anything to do with it. I highly doubt its a docker issue either, as the files live outside, and the current system doesn't do anything with docker.

Finally, as far as I have seen this only affects Filezilla, which has always done things in the most roundabout way possible. Its really no surprise this is broken on it, and switching to a different client resolves this problem for others with no additional changes necessary.

[codesquid]

FileZilla developer here. I had a look at your SFTP implementation, the bug in your daemon is quite obvious.

Please pay attention to section 6.1 of https://tools.ietf.org/html/draft-ietf-secsh-filexfer-02:

The protocol and implementations MUST process requests relating to
the same file in the order in which they are received. In other
words, if an application submits multiple requests to the server, the
results in the responses will be the same as if it had sent the
requests one at a time and waited for the response in each case.

Your implementation does not ensure this guarantee when processing SSH_FXP_READDIR packets.

[DaneEveritt]

@codesquid Thanks for the tip, should be an easy fix once I have my environment setup, so hopefully in the coming weeks...

[codesquid]

Move line 183 in sftp.js up 16 lines to work around the issue.

For a proper fix, keep state of the listing operation, synchronize the requests currently handled asynchronously and while you're at it, implement paging to keep memory consumption down server-side should a directory contains a few million files [*].

[*] Can easily happen with programs stuck in a crash, restart loop, creating a new log file each time.

[lancepioch]

Thanks @codesquid for the hotfix.

@DaneEveritt https://github.com/pterodactyl/daemon/pull/63

[DaneEveritt]

@codesquid So trying to fix whatever is going on with Filezilla ends up breaking a bunch of other SFTP clients in the process. Granted, I'm probably approaching this wrong, but is there any reason why FileZilla would be sending 4 simultaneous calls for a READDIR on the same directory?

[codesquid]

So trying to fix whatever is going on with Filezilla ends up breaking a bunch of other SFTP clients in the process

How so?

Is there any reason why FileZilla would be sending 4 simultaneous calls for a READDIR on the same directory?

Yes, pipe-lining. I took a very conservative approach when deciding on the 4. In the future I might make this dynamic, increasing it exponentially on large listings.

[DaneEveritt]

I am not fixing this at this time. FileZilla is the only SFTP client with this issue, and fixing it would require overhauling a significant amount of code from what I can see. Theres no reason it should send 4 simultaneous, identical requests and expect to get something other than the same response back with each.

No other clients out there perform this way, and they all work fine.

If someone wants to make a PR that addresses this inconsistency and maintains support across the other clients, you're welcome to. But as the sole developer I cannot devote that kind of time to supporting one specific piece of software that chooses to do things in a non-conforming manner.

[lancepioch]

It seems like the most conservative solution will require our daemon to hold a command queue for each request that it receives and then act on each request in order. This will almost certainly fix any duplicate file issues with Filezilla. However that will take a decent amount of time to simply fix only a single SFTP client.

However that being said, the fact that Filezilla sends more requests than necessary as a way to speed up large directory reads is just not a good solution to that particular issue. That issue being that a client should try to reduce the large number of files in a single directory.

[codesquid]

Theres no reason it should send 4 simultaneous, identical requests and expect to get something other than the same response back with each.

There is one very good reason to expect to get something _different_ with each response: The SFTP specifications. They are very explicit on this one.

But as the sole developer I cannot devote that kind of time to supporting one specific piece of software that chooses to do things in a non-conforming manner.

It's your server that does things in a non-confirming manner. Read the SFTP specifications.

That issue being that a client should try to reduce the large number of files in a single directory.

I don't believe jokes belong in this discussion.

[BrainStone]

While I don’t have any concrete numbers I’d also like to point out that FileZilla can essentially be considered the default option.

So while things like missing compression are not a big deal, an implementation bug that creates severe usability issues with one of the most popular SFTP clients is definitely an issue that should be taken care of.

While I certainly understand that you do not have a lot of time, just closing this issue and pretending it doesn’t exist is a questionable choice at best.
Secondly I believe bug fixes should always take priority over developing new features.

[lancepioch]

It is unreasonable to rewrite a large portion of the nodejs daemon just to support a single SFTP client that skirts around the edge of the specifications.

The fact is that there are no compliant SFTP nodejs libraries out there. We are not going to write a fully compliant one ourselves. It's more unreasonable to accomplish that than to arbitrarily send out 4 equivalent requests that no other client does.

As far as I'm aware, it's impossible to have multiple files with the same name in a single directory on both DOS and UNIX. With this in mind, there's also no reason for an SFTP client to show a list of these impossible files as well.

[codesquid]

It's more unreasonable to accomplish that than to arbitrarily send out 4 equivalent requests that no other client does.

The 4 requests aren't equivalent. The first request asks the server for the first, if any, set of files. The second request asks for the second, if any, set of files. The 3rd request asks for the third, if any, set of files, and so on: The n-th request asks for the n-th, if any, set of files.

It is not okay for the server to reply with the first set of files on the n-th request for n > 1.

[lancepioch]

@codesquid As far as I was aware, the requests were identical. What part of the request is indicating the different sets of files?

[codesquid]

It's the order of reception.

[lancepioch]

So what you're saying is that the requests are identical. Their order being different doesn't make the requests unique.

After reading over the specs that you linked to, here's my notes:

The protocol and implementations MUST process requests relating to the same file in the order in which they are received.

A directory is not the same thing as a file.

SSH_FX_EOF indicates end-of-file condition; for SSH_FX_READDIR it indicates that no more files are contained in the directory.

Our application returns that correctly. Because of that, it looks like Filezilla should be waiting for the end of each request before it makes another regarding directory scanning.

The client should call SSH_FXP_READDIR repeatedly until it has found the file it is looking for

Because I believe you will bring this up, repeatedly does not mean that you should make multiple requests at the same time. You should wait until each request is done in order. There is nothing that burdens this on the server implementation. I think this is where your misunderstanding lies.

[BrainStone]

Well in the UNIX universe directories are considered files. Special files, but files

Regardless, I think it would be best to reopen this issue, as it is an issue nevertheless. It could be marked as needing help/ a PR, to make it clear to others.

[DaneEveritt]

@codesquid so does Filezilla send an arbitrary number of requests, or is it always 4?

[lancepioch]

@BrainStone While a directory object is technically a file in nix, the specifications clearly separate out what to do for files and directories in multiple different places meaning that in the specification's context, they are treated differently.

[codesquid]

So what you're saying is that the requests are identical. Their order being different doesn't make the requests unique.

No, not at all. The are not identical: There can only be one first request, there can only be one second request and so on. By virtue of being ordered they are unique.

Oh, and they all have a request id, at any given point in time the set of ids of all in-flight requests is unique.

it looks like Filezilla should be waiting for the end of each request before it makes another regarding directory scanning.

No, the protocol allows to there being any number of outstanding requests. The specifications mandate that the server must process multiple requests from the client such that "the results in the responses will be the same as if it had sent the requests one at a time and waited for the response in each case."

Filezilla send an arbitrary number of requests, or is it always 4?

Doesn't matter.

[WolfSkin0]

Just wanted to mention this since my earlier comment was deleted, this issue isn't isolated to filezilla but also appears when I use bitvise as a sftp client

[DaneEveritt]

@codesquid, I think I am just misunderstanding what you're saying here. Things seem to be getting a little heated, which is not my intention and could just be because text can be a difficult way of communicating issues sometimes.

No, the protocol allows to there being any number of outstanding requests. The specifications mandate that the server must process multiple requests from the client such that "the results in the responses will be the same as if it had sent the requests one at a time and waited for the response in each case."

By this logic what is being done is not necessarily incorrect. The results in the responses _are_ the same as if they were sent one by one. It just happens that they all have the same response? The only way I was able to address this from the standpoint of multiple requests that are sent is to respond only once to the request and then send empty result sets back on subsequent requests. If thats the approach that needs to happen then I can continue fiddling with the logic surrounding it, but clients like Cyberduck do not operate this way. Cyberduck sends a single request, for example with an ID of 100. It gets the files, and then request 101 is sent back _after_ that request is returned, and expects an EOF.

From what I remember during my testing with Filezilla, it does not operate that way, and instead sends multiple requests, regardless of the EOF sent with the first. Thus, the only working approach for that was to keep responding to those repeated requests, first sending nothing back to any of them, and then only sending data to the final request (on loop 2), and then EOF on loop 3 (for all of them). I believe this is the wrong approach however?

[codesquid]

The results in the responses are the same as if they were sent one by one. It just happens that they all have the same response?

No, there is only one case where all responses can be the same and that is after EOF, in which case the responses have to be EOF.

The meaning of any SSH_FXP_READDIR request is "get me the next set of files on that handle, or reply with EOF if there are no more files". From the specs:

   Each SSH_FXP_READDIR request returns one
   or more file names with full file attributes for each file.  The
   client should call SSH_FXP_READDIR repeatedly until it has found the
   file it is looking for or until the server responds with a
   SSH_FXP_STATUS message indicating an error (normally SSH_FX_EOF if
   there are no more files in the directory).

Basically this allows to send the listing in small chunks, which is quite important if you consider the age of the protocol; it dates back to 2001, machines with only a handful of megabytes of memory were still wildly used, predating even Windows XP.

Imagine you're in early 2001, on a computer still using Windows ME and only 32 MB of RAM. Processing a large directory listing with tens of thousands of files without chunking would completely overwhelm such a small machine.

This gets further illustrated in the following excerpt from the SFTP specs:

   The maximum size of a packet is in practice determined by the client
   (the maximum size of read or write requests that it sends, plus a few
   bytes of packet overhead).  All servers SHOULD support packets of at
   least 34000 bytes (where the packet size refers to the full length,
   including the header above).  This should allow for reads and writes
   of at most 32768 bytes.

In other words, unless a client has in some way indicated support for larger packets, a server should only send directory listings in 32KB chunks.

Your SFTP server implementation is quite unusual in that it sends the entire listing, regardless of size, in a single chunk many Gigabyte in size given an sufficiently large directory.

So the semantics of any SSH_FXP_READDIR request are "Give me the next small chunk of files from this listing you have not sent before, unless you have no more data to send".

Cyberduck sends a single request, for example with an ID of 100. It gets the files, and then request 101 is sent back after that request is returned, and expects an EOF.

No, as explained above it does not expect EOF. It expects either the next chunk of files or EOF.

Were you to reply to the second request with another set of files, Cyberduck would send a third request, until you eventually send EOF.

text can be a difficult way of communicating issues sometimes.

Text is very precise, every word counts. I prefer it for this very reason.

[DaneEveritt]

Alright, I have a better idea for approaching this problem now. Thanks @codesquid 👍

[lancepioch]

Each SSH_FXP_READDIR request returns one or more file names with full file attributes for each file. The client should call SSH_FXP_READDIR repeatedly until it has found the file it is looking for or until the server responds with a SSH_FXP_STATUS message indicating an error.

@codesquid I'm referencing the bolded part. Why is Filezilla sending all of the requests at the same time instead of repeatedly (one after another)? Filezilla isn't waiting for a response before sending out multiple concurrent open requests. Multiple other projects such as WinSCP all wait for a response until "repeatedly sending out more calls". Are they not following the specifications correctly?

[DaneEveritt]

closed in https://github.com/pterodactyl/daemon/commit/98d4892a295c5b40ec24c9706179c366350b3f7d and https://github.com/pterodactyl/daemon/commit/b753e462b560c168d58f6f3a4605aff8de972a9d

[schrej]

@lancepioch
Because the spec allows it.

The protocol and implementations MUST process requests relating to
the same file in the order in which they are received. In other
words, if an application submits multiple requests to the server, the
results in the responses will be the same as if it had sent the
requests one at a time and waited for the response in each case.

[codesquid]

Why is Filezilla sending all of the requests at the same time instead of repeatedly (one after another)?

It's a lot faster. Try listing a directory with a million files on a machine across the Atlantic that sends listings in small chucks (as recommended by the specs). Also it's allowed by the protocol.

until the server responds

Since data transfer isn't instantaneous, this applies to the point of reception, a lot of time can pass between sending out the reply and it being received. Even after the client has received the reply, the server can still receive requests sent by the client from before the server itself has sent the reply to the previous command.

Multiple other projects such as WinSCP all wait for a response until "repeatedly sending out more calls". Are they not following the specifications correctly?

They are following the specifications, just as FileZilla is. The protocol allows doing things in different ways and as such any compliant implementation must be able to handle all these different ways.