Rclone: [Feature Request] rclone mount for windows

[ajkis]

Hopefully its doable..

[ncw]

This relies on there being a stable FUSE for windows. There have been movements in that direction but I don't think we are there yet. It also would require working from go which makes things a little trickier.

[ajkis]

This project is active and should be good for it https://github.com/dokan-dev/dokany.

p.s. I see rclone is mentioned quite a lot here: https://github.com/dokan-dev/dokany/issues/406

[qnorsten]

Well they actualy fixed a bug in dokany today that made it incompatible with file handeling in go. So it might be worth looking into if something can be done with dokany, as it is the most stable fuse I found for windows.

[naeloob]

It seems that the bug was using the Fuse wrapper, i'm not sure there was any bug using dokan interface.
Well, now we have two options. I supose it must easy to use Fuse wrapper.

Excited to be able to mount on windows too.

Edit : Here the info to use the Dokan Fuse Wrapper -> https://github.com/dokan-dev/dokany/wiki/FUSE
"Just" linking with the dynamic library dokanfuse.dll. I would love it could be so easy.

[naeloob]

I will love to be wrong but AFAIK, you can´t.

[ajkis]

I dont think so eg rclone mount will work once fuse is working on windows.

[nicko88]

There is a new Windows FUSE driver recently released as a supposedly more stable and better library than Dokany.

From the developer's blog:

_"My research led to me to Dokany and after some work I ported the core file system to Windows and the FUSE layer to the Dokany API. Unfortunately I soon discovered that Dokany did not implement correct file system semantics and was very unstable and slow. I briefly tried to correct some of these problems, but soon decided that I needed to start from scratch with a project that did FUSE on Windows "right"."_

Info:
http://www.secfs.net/winfsp/blog/files/winfsp-2017.html

Repo:
https://github.com/billziss-gh/winfsp

Example projects using the library:
https://github.com/billziss-gh/sshfs-win
https://github.com/billziss-gh/nfs-win
https://github.com/billziss-gh/redditfs

Other language bindings for reference:
https://github.com/billziss-gh/fusepy
https://github.com/DuroSoft/fuse-bindings

[AggelosM]

This is very good news!

[billziss-gh]

Hello, I am the author of WinFsp. I would be happy to help.

[justusiv]

I just played around with WinFsp and am amazed. While i dont understand what would be involved in implementing WinFsp into rclone, i am intently watching this thread and commits for any info on it. Its awesome that bill said he would help. Thank you bill, and great job on the tool. I really hope you and Nick can work together.

[billziss-gh]

@justusiv thanks for the kind words, re: WinFsp.

A few caveats:

• I do not know Go (this has not stopped me before).

• Rclone appears to use bazil/fuse. Unfortunately porting this project to WinFsp is not possible. From the bazil/fuse README:

  It is a from-scratch implementation of the kernel-userspace communication protocol, and does not use the C library from the project called FUSE.

  WinFsp provides a FUSE compatibility layer, but it does not emulate the FUSE "kernel-userspace communication protocol". Doing so would make little sense on Windows.

• Rclone appears to use "low-level" FUSE. The primary difference is that file lookups are done by (parent_ino, name) tuples rather than path. WinFsp only implements the high-level FUSE API (i.e. lookups by path). The low-level FUSE API is not very well suited to Windows (but of course can be implemented/emulated).

With this in mind I see two approaches:

1. Rclone could be changed to support alternative FUSE for GO implementations. If there is a FUSE for GO implementation that supports the FUSE high-level API, it could be ported to WinFsp-FUSE.

2. Rclone could support the native WinFsp API.

[ncw]

@billziss-gh - WinFsp is a very impressive project. I read about your difficulties with undocumented APIs and debugging - quite a challenge!

Unfortunately I'm not much of a Windows programmer - I did a fair amount of Win32 programming mostly to port cross platform programs to windows, but I've never done windows first development. I've tried to keep the non cross platform code in rclone to the absolute minimum.

Actually rclone uses the higher level path interface that bazil/fuse provides not the lower level inode based interface. rclone is very much path based internally.

I think the path of least resistance for rclone would be to use the C based FUSE compatible api and use cgo to interface with it.

That would probably mean making an alternative mount command which used cgo to interface with libfuse, and re-use the internals of the current mount command.

[vampywiz17]

That would probably mean making an alternative mount command which used cgo to interface with libfuse, and re-use the internals of the current mount command.

It means, that similar with mount encrypted filesystem now?

[billziss-gh]

@ncw wrote:

@billziss-gh - WinFsp is a very impressive project. I read about your difficulties with undocumented APIs and debugging - quite a challenge!

Thank you.

Unfortunately I'm not much of a Windows programmer - I did a fair amount of Win32 programming mostly to port cross platform programs to windows, but I've never done windows first development. I've tried to keep the non cross platform code in rclone to the absolute minimum.

I sympathize. Obviously I understand Windows internals quite well, but it is not my primary platform. I am writing this on a Mac :)

I expect that for a port like this, one does not have to know a lot about Windows native development. You might end up learning more than you like about its file system semantics though :)

Actually rclone uses the higher level path interface that bazil/fuse provides not the lower level inode based interface. rclone is very much path based internally.

Fantastic. This will make things much easier.

I think the path of least resistance for rclone would be to use the C based FUSE compatible api and use cgo to interface with it.

That would probably mean making an alternative mount command which used cgo to interface with libfuse, and re-use the internals of the current mount command.

I spent a couple of hours this morning doing the Go tour and playing with examples/exercises. It looks solid and interesting.

I also looked at cgo. Looks like the right approach although I note that according to this it requires gcc (mingw) on Windows.

I have also found this: https://github.com/golang/go/wiki/WindowsDLLs. Looks like a rather crude mechanism for doing DLL calls. However in the case of FUSE we also need to implement fuse_operations, so it probably is not a good solution.

---

I like the cgo approach and wondering whether it would be possible to create a generic cross-platform GO/FUSE package that uses the C based API regardless of platform.

[billziss-gh]

Hi, @ncw:

Actually rclone uses the higher level path interface that bazil/fuse provides not the lower level inode based interface. rclone is very much path based internally.

I am trying to reconcile this statement with what I am seeing in cmd/mount. For example, Lookup appears to implement the bazil/fuse interface NodeStringLookuper which is a variant of FUSE lowlevel/VFS lookup.

Please let me know if I misunderstand.

[billziss-gh]

In the last few days I have been working on a FUSE layer using the cgo approach. The result of this work is cgofuse.

Cgofuse currently runs on OSX. There is a sample file system named passthrough that currently passes most of fstest. I plan to also port this to Linux and then to WinFsp on Windows.

The API is a simplified version of FUSE and fuse_operations.

User mode file systems are expected to implement fuse.FileSystemInterface. To make implementation simpler a file system can embed ("inherit") a fuse.FileSystemBase which provides default implementations for all operations. To mount a file system one must use fuse.FileSystemHost and fuse.NewFileSystemHost.

Here is an example:

    host := fuse.NewFileSystemHost(&ptfs)
    host.Mount(args)

This API is not compatible with bazil/fuse. I could not find a path based API for bazil/fuse.

[ncw]

@billziss-gh wrote:

I could not find a path based API for bazil/fuse.
I expect you are right and I'm confused about what exactly is meant by the path based interface.

cgofuse looks great! I had a look through the code - very impressive for a first go program :-)

I had a go with it on linux and a brief smoke test with it worked (I didn't try fstest or anything like that, just a bit of ls).

I'll send you a PR with some fixes I had to make to get it to compile. I suspect you have symlinks in your GOPATH or something like that.

As for rclone, I guess what is needed is a new mount command to use cgofuse - the internals of mount need factoring out into a package, then rclone could use cgofuse directly, and hopefully WinFsp in due course.

I already have an alternate mount system (not released yet) using the other go fuse implementation hanwen/go-fuse so that internal refactoring is overdue.

Looks very promising :-)

[billziss-gh]

@ncw wrote:

cgofuse looks great! I had a look through the code - very impressive for a first go program :-)

Thanks. Feel free to make any suggestions for improvement. In particular I am wondering if the way I am mapping C pointers to Go interfaces and back (in handle.go) is the best way of doing things.

I'll send you a PR with some fixes I had to make to get it to compile. I suspect you have symlinks in your GOPATH or something like that.

This is just me resisting the way Go package paths work. I basically do git clone repo in the go/src directory rather than go get repo.

My main problem with go get repo is that I do not fully undestand the src/github.com/user/repo scheme. What if, for example, I move the cgofuse repo to bitbucket.org? Are users of cgofuse expected to change their github.com imports to bitbucket.org imports?

Still if this is the way it has to be done in order to play in the Go ecosystem that's fine.

As for rclone, I guess what is needed is a new mount command to use cgofuse - the internals of mount need factoring out into a package, then rclone could use cgofuse directly, and hopefully WinFsp in due course.

I've been away for the weekend so progress has been slow in the last couple of days. However I did the initial port to WinFsp over the weekend and intend to test the port over the next couple of days and make any necessary changes.

Looks very promising :-)

Glad to be of help :)

[billziss-gh]

Cgofuse is now ported to OSX, Linux and Windows (WinFsp). The OSX and Linux ports are being tested using fstest. I have not completed fully testing the Windows port yet.

Once Windows testing is complete, I will set up CI through Travis for Linux and AppVeyor for Windows. I note that although Travis supports OSX, it does not do so for Go. Any other CI suggestions for Go on OSX?

Finally cgofuse contains two example file systems:

• Passthrough which runs on OSX and Linux. Unfortunately I used POSIX calls from the syscall package to implement this file system and I am feeling rather lazy to port it to Windows.
• Memfs which runs on OSX, Linux, and Windows.

One final note: Cgofuse is currently licensed under the GPLv3. I will probably change it to a more liberal license to ease adoption, likely the BSD.

[qnorsten]

I just tried some other projects listed above that uses WinFsp and it seem to be an improvement over dokany. Keep up the good work billziss-gh:)

Hope CgoFuse will be stable enough to be used in rclone in the near future. Will be nice to bypass the need for a linux VM just to mount the cloud shares on Windows:)

Only thing needed after that is some sort of unionfs-fuse or similar for Windows. If that can be accomplished with WinFsp? But I guess that is for an other project and not strictly rclone or even WinFsp related:)

[billziss-gh]

I just tried some other projects listed above that uses WinFsp and it seem to be an improvement over dokany. Keep up the good work billziss-gh:)

My goals with WinFsp were stability, correctness and performance (in that order). Other goals were secondary; for example, I could have simplified the native WinFsp API but at the cost of not supporting some Windows file system features.

Hope CgoFuse will be stable enough to be used in rclone in the near future. Will be nice to bypass the need for a linux VM just to mount the cloud shares on Windows:)

Cgofuse is a simple layer over the C FUSE layer. It does not have any complicated code paths. The only potential problems are in how marshaling is done between the C and Go worlds or if some Go object that gets passed to C gets garbage collected. If there are any such remaining problems, we should be able to catch them with some testing.

Only thing needed after that is some sort of unionfs-fuse or similar for Windows. If that can be accomplished with WinFsp? But I guess that is for an other project and not strictly rclone or even WinFsp related:)

I do not use unionfs, but I understand the general concept. I believe it should be simple to create on Windows.

[ncw]

@billziss-gh wrote

Feel free to make any suggestions for improvement. In particular I am wondering if the way I am mapping C pointers to Go interfaces and back (in handle.go) is the best way of doing things.

That code will certainly work - it is very conservative.

I'd have probably just used an index into a slice rather than the result of C.Malloc(1) - it is an opaque pointer after all. You'd need to change the unsafe.Pointer to a uintptr though as unsafe.Pointers are supposed to point to valid memory.

All those type assertions .(FileSystemInterface) are ugly though, so I'd probably make two versions one specialized for FileSystemInterface and the other forFileSystemHost.

This is just me resisting the way Go package paths work. I basically do git clone repo in the go/src directory rather than go get repo.

My main problem with go get repo is that I do not fully undestand the src/github.com/user/repo scheme. What if, for example, I move the cgofuse repo to bitbucket.org? Are users of cgofuse expected to change their github.com imports to bitbucket.org imports?

Still if this is the way it has to be done in order to play in the Go ecosystem that's fine.

I felt the same way when starting go. Go is very opinionated about some things and this is one of them, so it is best to do it the Go way otherwise you'll be fighting a losing battle! Yes if you move your source code repo you need to change the code :-( There are ways of making a vanity url you can use, but it turns out not to be much of a problem in practice.

Cgofuse is now ported to OSX, Linux and Windows (WinFsp). The OSX and Linux ports are being tested using fstest. I have not completed fully testing the Windows port yet.

Great - well done :-)

Once Windows testing is complete, I will set up CI through Travis for Linux and AppVeyor for Windows. I note that although Travis supports OSX, it does not do so for Go. Any other CI suggestions for Go on OSX?

I test rclone on OSX via Travis. Check out rclone's .travis.yml - I test all go version on linux, but only the latest on OS X.

One final note: Cgofuse is currently licensed under the GPLv3. I will probably change it to a more liberal license to ease adoption, likely the BSD.

:-)

Cgofuse is a simple layer over the C FUSE layer. It does not have any complicated code paths. The only potential problems are in how marshaling is done between the C and Go worlds or if some Go object that gets passed to C gets garbage collected. If there are any such remaining problems, we should be able to catch them with some testing.

I expect you've seen the docs on pointer passing which has a section on runtime flags you can set to do more checking.

Also note that go vet does some checking of unsafe.Pointer use.

...

Next step for me to experiment with cgofuse and rclone :-)

[billziss-gh]

All cgofuse builds are now GREEN! :tada:

• OSX/Linux: Travis CI with fstest and fsx for testing.

• Windows: AppVeyor with winfsp-tests for testing.

Now that the basic framework is complete we can concentrate on improvements.

@ncw wrote:

Feel free to make any suggestions for improvement. In particular I am wondering if the way I am mapping C pointers to Go interfaces and back (in handle.go) is the best way of doing things.

That code will certainly work - it is very conservative.

I'd have probably just used an index into a slice rather than the result of C.Malloc(1) - it is an opaque pointer after all. You'd need to change the unsafe.Pointer to a uintptr though as unsafe.Pointers are supposed to point to valid memory.

Interesting idea. Perhaps this could be used to eliminate the handle table mutex (and the table itself of course).

All those type assertions .(FileSystemInterface) are ugly though, so I'd probably make two versions one specialized for FileSystemInterface and the other for FileSystemHost.

I probably made this too generic because I was not sure how I would end up using it. But I am wondering: is there a performance penalty to using type assertions? Or is that suggestion more of a stylistic issue?

I test rclone on OSX via Travis. Check out rclone's .travis.yml - I test all go version on linux, but only the latest on OS X.

Yes, I certainly looked at rclone to get ideas/inspiration :) [E.g. godoc reference badge]

I expect you've seen the docs on pointer passing which has a section on runtime flags you can set to do more checking.

I had seen those docs, but had not groked their significance. I will look at them more closely.

Next step for me to experiment with cgofuse and rclone :-)

Fantastic. Note that I made a couple of changes to the public API:

• Create now takes an extra flags argument (contains the open flags O_CREAT, etc.)
• Error is a new type that encapsulates a FUSE error. This Error type should be rarely used as returning error codes from the file operations should be sufficient. In some rare circumstances however it is convenient to throw an Error that contains a boxed error code; prior to returning to the OS the cgofuse layer will recover and unwrap the Error to get the error code to return. For example, during late testing of memfs I discovered that I needed to return ENAMETOOLONG in some circumstances. Rather than retrofit the whole file system to handle the new error path, I simply use: panic(fuse.Error(-fuse.ENAMETOOLONG)).

[ncw]

I probably made this too generic because I was not sure how I would end up using it. But I am wondering: is there a performance penalty to using type assertions? Or is that suggestion more of a stylistic issue?

Using an interface is effectively a pointer lookup, so same penalty as a virtual method in C++. Doing a type assertion is more expensive as it involves scanning a table for type information. However it is pretty quick... This SO question indicates it is 5x slower than the direct interface call - I think it depends on how many possible types there could be though.

Fantastic. Note that I made a couple of changes to the public API:

Noted!

I've been factoring the current mount code into a library so I can re-use it with cgofuse. I've made cgofuse work as far as directory listings which means I'll probably get the rest working over the weekend :-)

I haven't tried building on Windows yet, but I'll have a go with your instructions in a bit.

[billziss-gh]

Doing a type assertion is more expensive as it involves scanning a table for type information. However it is pretty quick... This SO question indicates it is 5x slower than the direct interface call - I think it depends on how many possible types there could be though.

Interesting. The comments in that SO question suggest that in later versions of Go type assertions have been optimized. I will try the included benchmark.

I haven't tried building on Windows yet, but I'll have a go with your instructions in a bit.

I have found that Mingw-builds works well.

Let me know if there is something else I can help with.

[ncw]

I have put the first version (not quite finished) of cgofuse for rclone in the cmount branch. It has a cmount command which works just like the mount command except it uses libfuse under linux/osx and WinFsp under Windows. I haven't tried building it for Windows yet though so that might not actually work yet!

Got to go to bed now - will try building and testing it with Windows tomorrow!

PS This project https://github.com/karalabe/xgo looks like an excellent way of doing a cross compile build for Windows...

[billziss-gh]

@ncw, fantastic. I will try to test it later tonight (I am in the PDT time zone).

As for Windows builds I have thought about providing binary releases for cgofuse. It is not clear to me that library binary releases are well supported with Go however.

[billziss-gh]

WORKS! :tada: :tada: :tada:

It also worked on my macbook. Great job!

I had to hackpatch it to get it to compile and run on Windows:

diff --git a/cmd/cmount/fs.go b/cmd/cmount/fs.go
index c9bd8ab..fb05f45 100644
--- a/cmd/cmount/fs.go
+++ b/cmd/cmount/fs.go
@@ -1,5 +1,5 @@
 // +build cgo
-// +build linux darwin freebsd
+// +build linux darwin freebsd windows

 package cmount

diff --git a/cmd/cmount/mount.go b/cmd/cmount/mount.go
index 52469fb..ebee92d 100644
--- a/cmd/cmount/mount.go
+++ b/cmd/cmount/mount.go
@@ -3,7 +3,7 @@
 // This uses the cgo based cgofuse library

 // +build cgo
-// +build linux darwin freebsd
+// +build linux darwin freebsd windows

 package cmount

@@ -21,7 +21,7 @@ import (
    "github.com/ncw/rclone/fs"
    "github.com/pkg/errors"
    "github.com/spf13/cobra"
-   "golang.org/x/sys/unix"
+   //"golang.org/x/sys/unix"
 )

 // Globals
@@ -39,8 +39,8 @@ var (
    writebackCache                   = false
    maxReadAhead       fs.SizeSuffix = 128 * 1024
    umask                            = 0
-   uid                              = uint32(unix.Geteuid())
-   gid                              = uint32(unix.Getegid())
+   uid                              = ^uint32(0)
+   gid                              = ^uint32(0)
    // foreground                 = false
    // default permissions for directories - modified by umask in Mount
    dirPerms  = os.FileMode(0777)
@@ -48,8 +48,8 @@ var (
 )

 func init() {
-   umask = unix.Umask(0) // read the umask
-   unix.Umask(umask)     // set it back to what it was
+   umask = 000//unix.Umask(0) // read the umask
+   //unix.Umask(umask)     // set it back to what it was
    cmd.Root.AddCommand(commandDefintion)
    commandDefintion.Flags().BoolVarP(&noModTime, "no-modtime", "", noModTime, "Don't read/write the modification time (can speed things up).")
    commandDefintion.Flags().BoolVarP(&debugFUSE, "debug-fuse", "", debugFUSE, "Debug the FUSE internals - needs -v.")
@@ -195,12 +195,14 @@ func mount(f fs.Fs, mountpoint string) (<-chan error, func() error, error) {
    fs.Debugf(f, "Mounting on %q", mountpoint)

    // Check the mountpoint
-   fi, err := os.Stat(mountpoint)
-   if err != nil {
-       return nil, nil, errors.Wrap(err, "mountpoint")
-   }
-   if !fi.IsDir() {
-       return nil, nil, errors.New("mountpoint is not a directory")
+   if runtime.GOOS != "windows" {
+       fi, err := os.Stat(mountpoint)
+       if err != nil {
+           return nil, nil, errors.Wrap(err, "mountpoint")
+       }
+       if !fi.IsDir() {
+           return nil, nil, errors.New("mountpoint is not a directory")
+       }
    }

    // Create underlying FS
@@ -260,7 +262,7 @@ func Mount(f fs.Fs, mountpoint string) error {
    }

    // This isn't needed under Windows as it gets unmounted by the cgofuse
-   if runtime.GOOS != "windows" {
+   if true || runtime.GOOS != "windows" {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

[qnorsten]

Is there any plan to release a beta version that includes the cmount command? Would love to try it out:) But have not had the energy to set up a go environment and build it myself. Even though it might be a good exercise.

[vampywiz17]

I think if it will reach the right development level, they will implement it.

[qnorsten]

Just managed to compile and run it by using the hackpatch changes that bill posted above.
Looking good I must say, nice to be able to mount encrypted drives on Windows.
Even though I guess it is not finished and stable yet.

Some things that do not seem to work (but you proably know that all ready).

• Read only flag not working as file can still be renamed or deleted
• Trying to move or copy a file in mount causes a panic and crashes (even though I know the recommended way is to use rclone move or rclone copy commands instead of performing operations against the mount)

[vampywiz17]

Hi guys,

I try to compile the "cmount" branch(on Manjaro):

git clone https://github.com/ncw/rclone.git
git checkout cmount
GOOS=windows GOARCH=amd64 go build

it do it successful, but after if i try mount it on windows, i get this:

E:\>rclone cmount myshare: M:
Error: unknown command "cmount" for "rclone"
Run rclone --help for usage.
2017/05/07 21:46:45 Fatal error: unknown command "cmount" for "rclone"

What do you think, what is the problem?

[ncw]

@billziss-gh thanks for your hackpatch!

I've added that in in a cross platform way in 3fe64e44e41e7ec02027c1c124c34c9938e3e33b - thank you!

I didn't add this chunk as I didn't seem to need it, unmount happened when I CTRL-C the rclone process. Am I missing something?

@@ -260,7 +262,7 @@ func Mount(f fs.Fs, mountpoint string) error {
    }

    // This isn't needed under Windows as it gets unmounted by the cgofuse
-   if runtime.GOOS != "windows" {
+   if true || runtime.GOOS != "windows" {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

[ncw]

For anyone who would like to play with mount on Windows here is a binary. You'll need WinFSP installed. Run it with rclone cmount remote:path X: where X: is an unused drive letter.

It likely has a lot of bugs still so don't use it for anything you care about!

rclone-v1.36-48-g3fe64e4-cmount-windows.zip

I built this against WinFSP v1.0 - I probably should have used 1.1Beta...

[ncw]

@vampywiz17 you'll need to compile it on Windows as it needs cgo. I installed the mingw compiler, the WinFsp dev and copied the include files into the mingw include directory. There is a binary above if you just want to have a go with it.

[vampywiz17]

@ncw Thanks the binary and the instructions!

[billziss-gh]

@ncw wrote:

I didn't add this chunk as I didn't seem to need it, unmount happened when I CTRL-C the rclone process. Am I missing something?

Nothing really. Recall that when I did this quick patch cgofuse was providing less guarantees regarding Mount / Unmount than it does now. But this code (signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM), etc.) is no longer needed on any platform.

I've added that in in a cross platform way in 3fe64e4 - thank you!

I recommend also setting uid = ^uint32(0) and gid = ^uint32(0) on Windows. This is a special instruction to WinFsp-FUSE to do the equivalent of uid = uint32(unix.Geteuid()) and gid = uint32(unix.Getegid()) on Windows.

I built this against WinFSP v1.0 - I probably should have used 1.1Beta...

My recommendation is to stick with v1.0 for now. The v1.1 adds a .NET layer (which is not yet fully stable). It does not have any additional fixes on the WinFsp core.

@vampywiz17 you'll need to compile it on Windows as it needs cgo. I installed the mingw compiler, the WinFsp dev and copied the include files into the mingw include directory. There is a binary above if you just want to have a go with it.

@ncw, if you have any ideas on how to streamline the Windows build for cgofuse (and consequently rclone) I will be happy to modify cgofuse. At some point I added a Makefile, but removed it when I found out that Mingw-builds does not come with make! I also thought about adding a winbuild.bat batch file to avoid having to set Mingw PATH and CPATH, but somehow it felt "wrong" to do something like this on Go.

[sbr481]

Hi, i'd like to tell you, that i've been testing rclone with new cmount feature in Windows, and i've been able to copying from my computer to a remote (Google Drive Account) using Cygwin.
With proper Microsoft commands, it fails copying to remote (it says access denied), but it can download from remote to my computer.
At this moment, windows only can see mounted unit in cmd, but at least, with Cygwin it works. When copying with windows commands, file is created in the remote, but transfer fails (file stores as 0 bytes).
Another thing, i've noticed that making directories work, but it notices a fail (but directory is created).
I've attached a screenshot.
I've mount the remote with flags --allow-root and --allow-other.
Thanks for your great effort, and i'm sure you'll get everything working. I'll continue testing, and i'll tell you everything i think is useful.
EDIT: Tested a crypted remote and working too.
Congrats, and best regards.

[billziss-gh]

@sbr481 The current cmount code does not appear to pass the proper uid, gid to WinFsp-FUSE yet (it just uses zero (0)). This is likely why you are getting "access denied" errors.

Cygwin likes to enable the backup and restore privileges (when it can) so things often work there when cmd.exe does not.

[ncw]

@billziss-gh have fixed up uid.gid in dc0f9087463e42e2dfcd6d42f92a869498f53f48

if you have any ideas on how to streamline the Windows build for cgofuse (and consequently rclone) I will be happy to modify cgofuse. At some point I added a Makefile, but removed it when I found out that Mingw-builds does not come with make! I also thought about adding a winbuild.bat batch file to avoid having to set Mingw PATH and CPATH, but somehow it felt "wrong" to do something like this on Go.

I'm not sure of the best way... Normally go builds take no configuration at all which is kind of relaxing once you've got used to it! go-sqlite3 might be interesting to study.

I'm using this little .bat file to compile rclone on Windows (thanks for the hint about CPATH).

@echo off
echo Setting environment variables for mingw+WinFsp compile
set GOPATH=X:\go
set PATH=C:\Program Files\mingw-w64\i686-7.1.0-win32-dwarf-rt_v5-rev0\mingw32\bin;%PATH%
set CPATH=C:\Program Files\WinFsp\inc\fuse

TBH the whole mingw install seems a bit of a mess, and given that Windows doesn't seem to have standard places for compilers or include files, trying to do anything other than show a bat file like the above as an example is probably too hard.

As for making a release build for rclone I'll try to cross compile it if possible so it can fit in with the rest of the builds, failng that I'll build it on AppVeyor.

I had a look at the rclone.exe binary and it doesn't seem to depend on a WinFsp DLL. Squinting at the code confirms that it loads the WinFsp DLL dynamically. So this means I can make the Windows release include the cmount code but not require WinFsp to be installed unless mount is needed. Is that right?

@sbr481 here is a new version with the fix for the uid, gid as suggested by @billziss-gh - it works for write in my very brief testing.

rclone-v1.36-50-gdc0f908-cmount.zip

[vampywiz17]

Now, i see the files size in CMD. the list and the basic features working well. the folder create is good.

If i try to check a file descpription in GUI , the network traffic going to hight, but not open the desc. windows. (need to force close the mount)

[sbr481]

Hi ncw, billziss,
I've been testing the new rclone release (with Google Drive account, and crypted remote):

· At least in my tests, it continues saying 'access denied' when i try to use 'copy' command.

· When i try to create a folder, it is created but always notifies an error (both: mkdir or md). With crypted remote, i got another error related to I/O operation (only happened once).



· Deleting is OK, files or directories ('rm' and 'del' commands).
· Listing works perfectly ('dir' and 'ls').
· Mounted units only are visible by CMD.

I'm using W10 x64 bits, with CMD in Administrator mode. Sometimes, when an error happens, rclone gives an output of the error, giving info (a lot), an output like that:
output_error.txt
I don't remember what exactly threw that exception/error, i'll try to replicate the situation. Which verbosity level is useful for you?
Hope all above be useful. Regards.

EDIT: I'm noticed i wasn't using beta release of Winfsp (using 1.0.17072), so, i'll retry everything...
Now tested with updated winfsp (1.1.17107), but still same problems mentioned above.

[vampywiz17]

I get exactly same errors. i try beta and stable Winfsp.

[ncw]

Try this version - note that you need to use rclone mount now (not rclone cmount)

rclone-v1.36-58-g824c66e-cmount.zip

If you are still getting the same error (I didn't when I tried it) then run the mount with -vv and post the relevant logs please!

Note that if you are asking for properties on a directory then it will need to scan the whole directory which may take some time and network bandwidth.

[nicko88]

Does this new mount command in these builds here support parameters like --max-read-ahead ?

[billziss-gh]

@ncw wrote:

I had a look at the rclone.exe binary and it doesn't seem to depend on a WinFsp DLL. Squinting at the code confirms that it loads the WinFsp DLL dynamically. So this means I can make the Windows release include the cmount code but not require WinFsp to be installed unless mount is needed. Is that right?

Yes, that is correct. The intent was to give maximum flexibility and not make WinFsp a hard requirement for rclone on Windows.

[sbr481]

Hi again,
I've been doing some testing with last rclone (in the same scenario and similar commands). Mounted now with mount instead of cmount, and flags --allow-other and --allow-root. Still no luck, same issues:

· 'mkdir' still throws as output 'incorrect function' after creating directory (this 'really' is a problem only if launching a recursive copy, because makes to stop copying when needs to create a directory in the recursion).
mkdir_debug.txt
· 'copy' command still has the same behaviour:


copy_debug.txt
· As a curiosity, i've noticed this little difference when viewing information of script.sh (the file i've been using for tests); relative with uid, gid and permissions when launch 'ls -li' from local, and launching inside the mounted remote:

Thanks again for your effort and support.

[ncw]

@nicko88 wrote

Does this new mount command in these builds here support parameters like --max-read-ahead ?

It should support all the same parameters

[ncw]

@billziss-gh wrote:

Yes, that is correct. The intent was to give maximum flexibility and not make WinFsp a hard requirement for rclone on Windows.

Perfect :-)

[ncw]

@sbr481 wrote:

'mkdir' still throws as output 'incorrect function' after creating directory (this 'really' is a problem only if launching a recursive copy, because makes to stop copying when needs to create a directory in the recursion).

I haven't managed to replicate this - can you give me a screenshot of the command you used and the result.

Also the exact command line you are using for rclone and the type of remote.

I have managed to replicate using copy to overwrite a file giving access denied messages.

[billziss-gh]

@sbr481 wrote:

'mkdir' still throws as output 'incorrect function' after creating directory

"Incorrect function" is likely the error message for ERROR_INVALID_FUNCTION. This can be produced for a variety of reasons on Windows, but in our case the primary culprit is the Windows kernel status code STATUS_INVALID_DEVICE_REQUEST. This can also be produced if someone returns -ENOSYS from the FUSE layer. @ncw do you return -ENOSYS anywhere in the latest version?

For what it's worth mkdir foo works fine for me over an Amazon S3 bucket using @ncw's latest build.

(this 'really' is a problem only if launching a recursive copy, because makes to stop copying when needs to create a directory in the recursion).

I cannot quite parse the meaning of this. What do you mean by recursive copy?

As a curiosity, i've noticed this little difference when viewing information of script.sh (the file i've been using for tests); relative with uid an gid and permissions when launch 'ls -li' from local, and launching inside the mounted remote:

I notice in your screenshot that Cygwin reports uid and gid as "no+body". I think this may be the crux of the problem, because the uid and gid should be that of your local account. I am unsure why this is so, because rclone appears to fix uid and gid ^uint32(0).

---

@ncw: does rclone pass -o uid=-1,gid=-1 to WinFsp-FUSE or some other value? For example, I am not sure the value -o uid=4294967295,gid=4294967295 will work.

[sbr481]

Here are the screenshot of the command and the output of rclone mount:

mount_debug.txt

It only contains:
1 - mount of remote (Google Drive account crypted) on u:
2 - copy of the file script.sh from local unit to u:

If i use 'cp' command from Cygwin, no problems. But if i use 'cp -r' to copy recursively, it fails/abort copying, when try to make the first subdirectory, related to problem of making directories, mentioned before.

@billziss-gh

I cannot quite parse the meaning of this. What do you mean by recursive copy?

A recursive copy, is when i want to copy a tree of directories and files, keeping the original structure. So, that replicates the original structure in the destination path/folder, with same tree structure (files and directories). Hope this explanation will be more understandable by imagining how works 'cp -r' command.

EDIT: tested with a normal remote (Google Drive without crypt):
· Same problems using 'mkdir' or 'md' (output: incorrect function). But directories are made.
· Same problems using 'copy' (output: access denied), and file is created only (without content), with 0 bytes.
It seems is nothing related to crypted or not crypted remote. Same behaviour in both remote types.

@ncw

I have managed to replicate using copy to overwrite a file giving access denied messages.

My tests are without overwrite, and with overwrite, both cases give the same error. But what i've been talking about is without overwriting. I suppose first is needed solve problem without overwrite, although maybe it's related to overwrite... I don't know...

[vampywiz17]

I try the latest release, but it same. If i try to right click on file it freeze.

Here my log (at the end, i try to copy a file)
log.txt

Now i try copy a file with Far Manager in Windows. (CLI file manager, similar with MC) and it first "thinking" about 1-2 minute, but after the file copy is really fast, similar with mount in my Ubuntu server.

Edit:

Now is really fast copy files from Amazon to local machine. (with Far Manager) Unfortunately with GUI, it slow...

other thing. If i mount it with --buffer size flag, the copy job freeze if it to reach end of pre-set buffer. (now 1 GB)

2017/05/08 23:20:24 DEBUG : Filmek/Hegylakó 1080p/Hegylakó Dircut BDRemux 2xHUN - Janus.mkv: ReadFileHandle.Read OK
2017/05/08 23:20:24 DEBUG : Filmek/Hegylakó 1080p/Hegylakó Dircut BDRemux 2xHUN - Janus.mkv: ReadFileHandle.Read size 1048576 offset 963641344
runtime: out of memory: cannot allocate 1048576-byte block (1073741824 in use)
fatal error: out of memory
2017/05/08 23:20:24 DEBUG : Filmek/Hegylakó 1080p/Hegylakó Dircut BDRemux 2xHUN - Janus.mkv: ReadFileHandle.Read OK

[sbr481]

@vampywiz17 For that purpose, i'd prefer to use directly rclone-browser. I'd like to use rclone as a mount option (in Windows) instead other third party software (like netdrive, expandrive...).
Anyways is good to know there is some CLI software able to work with rclone mount without problems.

[vampywiz17]

Of course, i think the same. But now it is a very unstable release and i think any aspect and log is useful to devs to solve problems :)

[ncw]

@billziss-gh wrote

"Incorrect function" is likely the error message for ERROR_INVALID_FUNCTION. This can be produced for a variety of reasons on Windows, but in our case the primary culprit is the Windows kernel status code STATUS_INVALID_DEVICE_REQUEST. This can also be produced if someone returns -ENOSYS from the FUSE layer. @ncw do you return -ENOSYS anywhere in the latest version?

Ah,.. Yes... I haven't implemented every FUSE callback, so if the fuse.FileSystemBase returns -ENOSYS for some things then rclone will. I've probably forgotten something important!

@sbr481 Passing the --debug-fuse flag to rclone will set -o debug which should show which

EDIT: tested with a normal remote (Google Drive without crypt):
· Same problems using 'mkdir' or 'md' (output: incorrect function). But directories are made.
· Same problems using 'copy' (output: access denied), and file is created only (without content), with 0 bytes.
It seems is nothing related to crypted or not crypted remote. Same behaviour in both remote types.

I seem to be able to make directories just fine using google drive (no crypt). I wonder if this is because I'm using Windows 7 for my tests?

@ncw: does rclone pass -o uid=-1,gid=-1 to WinFsp-FUSE or some other value? For example, I am not sure the value -o uid=4294967295,gid=4294967295 will work.

rclone doesn't pass uid or gid at all, it sets them in the Attrs returned.

[sbr481]

@ncw

I seem to be able to make directories just fine using google drive (no crypt). I wonder if this is because I'm using Windows 7 for my tests?

You're right, in Windows 7 it works but not well (at least in my case)... 'copy' command says a file is copied correctly, but rclone mount, throws an error: 'Can't truncate files'. I'm going to try to debug it, and give you the output.
Also, I'm going to test crypted remote.
EDIT: I tried to replicate the above error, but now it seems it works well... Maybe was a spontaneous error...
So, it seems a problem related to W10. In W7:
· No problems using copy command.
· No problem using mkdir command to create directories.
I'm going to test with crypted remote, but i think it works correctly.
EDIT2: As expected, it works with crypted remote too:


So i'll try to debug fuse in W10 and see if there is something relevant.

[billziss-gh]

@ncw: does rclone pass -o uid=-1,gid=-1 to WinFsp-FUSE or some other value? For example, I am not sure the value -o uid=4294967295,gid=4294967295 will work.

rclone doesn't pass uid or gid at all, it sets them in the Attrs returned.

Aha! I think we have found our problem. WinFsp-FUSE treats uid==-1 and gid==-1 specially only on the command line interface (i.e. if passed as -o uid=-1,gid=-1. It does not treat uid, gid returned from getattr specially (i.e. uid==-1 and gid==-1 do not have special meaning there).

I see a couple of different solutions:

• Pass -o uid=-1,gid=-1 as options to WinFsp-FUSE. I recommend this approach.

• Implement Geteuid and Getegid on Windows. Unfortunately this is not straightforward because Windows uses SID's (not UID/GID's). We would have to determine the SID of the account that launches the file system and then translate that to the UID/GID. [Here is how this is done in WinFsp-FUSE.] The mapping between SID's and UID/GID is well-defined, but is not straightforward (it is the same as the one used by Cygwin).

• Hardcode some uid and gid that work on Windows. I have found the values uid==11 and gid==65792 to work well on Windows. Keep in mind that uid==11 makes the file appeared to be owned by "Authenticated Users" and gid==65792 makes the file appear to belong to the group "Everyone".

[sbr481]

Incredible... We are focusing in some things, when problem was other... Administrator CMD is the problem...
I've tested with normal CMD, and now everything is working... OMG!! xD
Basically, i've thought about that before, but i didn't thought that could be the reason... And now, with W7 working, i've thought before anything more, test with normal CMD... That's the answer...
Let's go to another thing... xD

[billziss-gh]

@sbr481

I cannot quite parse the meaning of this. What do you mean by recursive copy?

A recursive copy, is when i want to copy a tree of directories and files, keeping the original structure.

I understand now. Sorry for being thick for a moment there.

EDIT: tested with a normal remote (Google Drive without crypt):
· Same problems using 'mkdir' or 'md' (output: incorrect function). But directories are made.
· Same problems using 'copy' (output: access denied), and file is created only (without content), with 0 bytes.

I strongly suspect that all this is a permissions related problem.

@sbr481 can you run rclone mount with the --debug-fuse option and then try your mkdir scenario? This will show us the Windows level file operations and we should be able to see there which operation fails with STATUS_INVALID_DEVICE_REQUEST or similar.

[sbr481]

@ncw @billziss-gh read above... xD

[billziss-gh]

Incredible... We are focusing in some things, when problem was other... Administrator CMD is the problem...

@sbr481 I am glad that you have found that this works for you when not an Administrator, but I also think you have helped us identify a real problem with the uid, gid used on Windows. It would also be helpful if you still run rclone mount --debug-fuse and posted the log as described above to make sure we catch some unimplemented operation that should be implemented.

[sbr481]

No problem, i'll test what you say. Give me some minutes.

[billziss-gh]

Thanks @sbr481. In the meantime I tested myself as an admin now. I can confirm your findings:

Y:\>mkdir bar
Incorrect function.

The log says:

rclone[TID=19d4]: 85A32BA0: >>Create [UT---C] "\bar", FILE_CREATE, CreateOptions=200021, FileAttributes=10, Security=NULL, AllocationSize=0:0, AccessToken=00000518, DesiredAccess=100001, GrantedAccess=0, ShareAccess=3
rclone[TID=19d4]: 85A32BA0: <<Create IoStatus=c0000010[0]

Status IoStatus=c0000010 is STATUS_INVALID_DEVICE_REQUEST. I am not sure why this error code is returned. As explained earlier, this gets translated by Windows to ERROR_INVALID_FUNCTION which is the "Incorrect function" message that we have been seeing.

[sbr481]

First test as administrator mode (needed to get the errors), mkdir command, normal remote (no crypt):

Now that type of error is new. After that, i tried to run a dir command in root of remote, and gets hunged, i need to force close rclone mount command. Here is the output until done mkdir command. I mean, the 'dir' command i've running after that is not included:
debug_fuse_mkdir_administrator.txt
After closing CMD, i've remounted the remote, and directory was created although it threw the error, as usual behaviour known. I'll continue testing with 'copy' command, and more test with crypted remote.
UPDATING:
Here is the 'copy' command, and the expected error:

And here is the output relative to the command:
debug_fuse_copy_administrator.txt
And the expected result, a file of 0 bytes:

If you need something special, tell me. I'll continue with crypted remote.
UPDATING 2:
Admin CMD, mkdir command, same error as normal remote:

After that, if i try to run a 'dir' command, it hungs up:

Here is the output (this time i recopiled both outputs, with mkdir command, and dir command hunged, and separated to see easily):
debug_fuse_mkdir_crypted_remote.txt
I'm going to force close, and remount, to make copy scenario.
UPDATING 3:
Admin CMD, copy command, same error as normal remote:

The output related:
debug_fuse_copy_crypted_remote.txt
Maybe the errors are caused by a layer of privileges related to Administrator mode. Hope this help guys.
Regards.

[billziss-gh]

@sbr481 thank you very much. I can confirm that I have also seen the alternative mkdir problem. To summarize here are the problems that I am aware of and have confirmed so far:

• Mkdir sometimes results in "the request could not be performed because of an I/O device error". This is the Windows user mode error ERROR_IO_DEVICE, which is a translation of the Windows kernel mode error STATUS_IO_DEVICE_ERROR, which comes from the FUSE layer returning -EIO. This is likely because an exception is thrown within the Go code during Mkdir.

• Mkdir sometimes results in "incorrect function". This is the Windows user mode error ERROR_INVALID_FUNCTION, which is a translation of the Windows kernel mode error STATUS_INVALID_DEVICE_REQUEST. I have debugged this and have found that it is due to WinFsp-FUSE calling Chown under some circumstances. Rclone mount does not currently implement Chown and hence FUSE returns -ENOSYS.

• The uid, gid values are not properly set as discussed in an earlier post.

[sbr481]

Great work guys, at least you already know some things to correct, and do a better and stable tool. Here we are to help you as we can. Time to sleep xD
Best regards.

[vampywiz17]

What do you think guys, what he prob. the GUI operations? (copy, move is extreamly slow) on CMD, it is relally fast, but the GUI command buggy now. you need some log?

[ncw]

Here is the latest version

• Statfs: reduce max size of volume for Windows
  
  
  
  • this should make the volume size be 15.9 TB not 0!
  
  
• allow extra options to pass to fuse with -o
  
  
  
  • for testing
  
  
• Add -o uid=-1 -o gid=-1 for Windows/WinFsp
  
  
  
  • this should fix the permissions issues - thanks @billziss-gh
  
  
• implement no-ops for Fsync, Chmod, Chown, Access, Fsyncdir and stop using fuse.FileSystemBase
  
  
  
  • this should fix one of the Mkdir issues - the "incorrect function one" thanks again @billziss-gh
  
  
• Add function tracing
  
  
  
  • this should provide enough logging to track down the other Mkdir issue
  
  
  • I'm not quite sure what is causing it so logs needed please!
  
  

rclone-v1.36-66-g2cb5a60-cmount.zip

Testing in CMD and the cygwin shell (it should work in both) appreciated!

@vampywiz17 wrote

What do you think guys, what he prob. the GUI operations? (copy, move is extremely slow) on CMD, it is really fast, but the GUI command buggy now. you need some log?

The GUI seems to do this approximately 35 times after copying a file which won't be helping the performance. I have no idea why!

2017/05/09 18:22:18 DEBUG : /zz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/09 18:22:18 DEBUG : /zz: >Getattr: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/09 18:22:18 DEBUG : /zz: >Getattr: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Open: flags=0x0
2017/05/09 18:22:18 DEBUG : /zz: >Open: errc=0, fh=0x2803
2017/05/09 18:22:18 DEBUG : /zz: Flush: fh=0x2703
2017/05/09 18:22:18 DEBUG : /zz: >Flush: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Release: fh=0x2703
2017/05/09 18:22:18 DEBUG : zz: ReadFileHandle.Release closing
2017/05/09 18:22:18 DEBUG : /zz: >Release: errc=0

[vampywiz17]

thanks @ncw

I tested it, the volume size is good now!

Folder creaton and delete now is really good on GUI. no speed and permission problem. Good!

i try again the --buffer size flag, but it FC, if it will arrive the pre-set buffer size. I attach the log:

log.txt

The GUI problem: i hope you will found some solution this. 👍 But in this release the copy is really good on GUI (much better, with previous release)

Edit: And the --read only flag still not working.

Now I try to open (stream) a video, it opened about 1-1.5 min, but after it play it relatively flawlessly. (of course the buffer size is missing)

One idea of GUI problem:

Now, Rclone Mount it a "Local Drive" not a "Network Drive". Not possible, that the problem is that the Win handle different way to the local and the network storage? (Indexing, Availability, Latency, etc..) and it is the reason that the GUI is much slower?

[sbr481]

Hi, i've been doing some tests of new version. It works fine, no errors copying and no errors making directories. Overwrite works fine too.
Now i've been testing about opening some videos, maybe next step is work in the performance of transferring.
As vampywiz said, --read-only does not work.
I'm doing some tests of performance (rate transfer), but now i'm uploading to the same remote (on the same computer), and my tests are not 'reliable' at this moment.
Mounted 2 remotes (one crypted, one not crypted), working fine:

As soon as possible i'll give you feedback of how is transfers and their relatives logs.

Great job, thanks for your support.

[billziss-gh]

@ncw wrote:

• allow extra options to pass to fuse with -o
  
  
  
  • for testing
  
  

Thanks. This will be very useful.

• implement no-ops for Fsync, Chmod, Chown, Access, Fsyncdir and stop using fuse.FileSystemBase
  
  
  
  • this should fix one of the Mkdir issues - the "incorrect function one" thanks again @billziss-gh
  
  

Thanks for doing this, although I have come to consider this a WinFsp-FUSE problem (i.e. a problem with the FUSE layer of WinFsp).

The problem is that LIBFUSE allows sensible defaults when a file system does not implement a particular operation. WinFsp-FUSE does so as well. But they somewhat disagree on what an unimplemented operation is.

LIBFUSE considers an operation unimplemented when:

• Either the operation is not specified in fuse_operations.
• Or the operation is specified but it returns -ENOSYS.

WinFsp-FUSE considers an operation unimplemented only when:

• The operation is not specified in fuse_operations.

If the operation returns -ENOSYS it will consider it implemented and will translate this to the Windows equivalent ("Incorrect function"). Unlike a general FUSE file system written in C, cgofuse "implements" all FUSE operations in fuse.FileSystemBase. Hence we are getting some unexpected problems.

So I need to revisit WinFsp-FUSE regarding its handling of -ENOSYS. I must also reconsider whether Chown is mandatory for this case or not.

• Add function tracing
  
  
  
  • this should provide enough logging to track down the other Mkdir issue
  
  
  • I'm not quite sure what is causing it so logs needed please!
  
  

Thanks. I will test myself later today and see if we can catch that exception.

@vampywiz17 wrote:

Now, Rclone Mount it a "Local Drive" not a "Network Drive". Not possible, that the problem is that the Win handle different way to the local and the network storage? (Indexing, Availability, Latency, etc..) and it is the reason that the GUI is much slower?

How were you mounting the file system as a network drive before? Was it being exported as a samba drive from another OS (e.g. Linux)?

All other things being equal exposing a file system as a "Local Drive" file system will always be theoretically faster than exposing it as a "Network Drive". The reason is simple, file operations must go through an extra driver in the "Network Drive" case (called the MUP). In practice though I have never seen any real difference in performance, other than the file system taking longer to mount when mounted as a "Network Drive".

It is actually possible to mount a file system as a network drive by using the special WinFsp-FUSE option:

--VolumePrefix=\server\share
               ^
               |
               + [Notice the single backslash.] This will mount on \\server\share

Unfortunately this option is not currently supported by rclone mount.

[sbr481]

I think it will be useful to implement an option to mount as a network drive. As my experience, network drives work better than a local storage in other softs (per example, netdrive). I know the wrapper is different in network drives, but maybe is more efficcient and realistic with latencies and other stuff.
Local drive wrapper is designed to work as a local unit, and the specs of the storage of cloud services are far away from local units (in general terms).
It would be interesting to see how is performance with both modes.
I mentioned Netdrive, because it offers 2 ways of mount a storage cloud service, as a local drive, and a network drive. And network mode works better than local mode.
Said only as a suggestion.

[sbr481]

Hi, i have some info, cause i've been testing how it's functionality on windows.
I've recently had a problem, trying do download a file from remote to local , and it seems that 'rclone mount', had a FC.
Scenario: Google Drive Account remote (crypted), withou --debug-fuse (i forgot launchig command with the flag).
error copy.txt
This log starts before copyig the file to my local storage. The file is Nerve.mkv (i was simply testing performance in downloading). But maybe is interesting to see what's happening before start the copying. Also, i think it's important to say, that the copy process start, and in 2 seconds aproximately, process was interrupted.
Uploading is very unstable, it has a lot of peaks.

As a curiosity, i've noticed, windows need to recover attribs from files in the directory, and maybe, is one of possible causes that affects performance, related to this mentioned before:
@ncw

The GUI seems to do this approximately 35 times after copying a file which won't be helping the performance. I have no idea why!

2017/05/09 18:22:18 DEBUG : /zz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/09 18:22:18 DEBUG : /zz: >Getattr: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/09 18:22:18 DEBUG : /zz: >Getattr: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Open: flags=0x0
2017/05/09 18:22:18 DEBUG : /zz: >Open: errc=0, fh=0x2803
2017/05/09 18:22:18 DEBUG : /zz: Flush: fh=0x2703
2017/05/09 18:22:18 DEBUG : /zz: >Flush: errc=0
2017/05/09 18:22:18 DEBUG : /zz: Release: fh=0x2703
2017/05/09 18:22:18 DEBUG : zz: ReadFileHandle.Release closing
2017/05/09 18:22:18 DEBUG : /zz: >Release: errc=0

And maybe with a network drive mode, this behaviour is different.

I have another remote, also mounted, and it continues mounted (i mean a crash doesn't affect the other), that's good, as independent processes.
I know some problems are happening too, so i'll try to recover info, and test with --debug-fuse, and replicate the problems.

[billziss-gh]

Now that rclone mount appears to be more stable I attempted to run the WinFsp test suite on it. Unfortunately I hit a snag almost immediately captured by this rclone mount log:

2017/05/10 13:47:04 DEBUG : /file0: Open: flags=0x2                      
2017/05/10 13:47:04 ERROR : /file0: Can't open for Read and Write        
2017/05/10 13:47:04 DEBUG : /file0: >Open: errc=-1, fh=0xFFFFFFFFFFFFFFFF

Checking the mount code I find:

    case rdwrMode == fuse.O_RDWR:
        fs.Errorf(path, "Can't open for Read and Write")
        return -fuse.EPERM, fhUnset
    }

So this is of course done on purpose. I am assuming that it is an artifact of rclone having to work with cloud storage systems that provide object-level rather than file-level interfaces (e.g. no ranged PUT as we have discussed on a different thread with @ncw).

I wanted to mention this because it took me by surprise at first. I think it might be worthwhile to consider implementing caching functionality for rclone akin to the Andrew File System (again as discussed elsewhere). This would solve many of the problems and it can perhaps be done in a way that solves (or minimizes) the problems with eventual consistency in cloud storage systems.

[ncw]

Here is the latest code

rclone-v1.36-74-g0a020f88-cmount.zip

Changes are

• on read only open of file, make open pending until first read
  
  
  
  • This fixes a problem with Windows which seems fond of opening files just to read their attributes and closing them again.
  
  
• Wait for mountpoint to appear on Windows before declaring mounted
• pass --FileSystemName under windows
• implement --fuse-flag to pass commands to fuse library directly
  
  
  
  • Useful for --fuse-flag -h to see exactly which options the library supports.
  
  
• fix openFile leak

@vampywiz17 wrote

i try again the --buffer size flag, but it FC, if it will arrive the pre-set buffer size. I attach the log:

That is out of memory (as is your other log).

Try again with the latest beta - I fixed a couple of things which could have caused that.

If not please send a log!

@sbr481 wrote

As vampywiz said, --read-only does not work.

I haven't fixed that yet, but I've received a steer from @billziss-gh as to the best way

@billziss-gh wrote

It is actually possible to mount a file system as a network drive by using the special WinFsp-FUSE option: --VolumePrefix=\server\share

You should now be able to pass that option to rclone like this

--fuse-flag --VolumePrefix=\server\share

You can alsu use --fuse-flag -h which shows you exactly what options you can pass.

@billziss-gh wrote about rclone not supporting read/write files

I wanted to mention this because it took me by surprise at first. I think it might be worthwhile to consider implementing caching functionality for rclone akin to the Andrew File System (again as discussed elsewhere). This would solve many of the problems and it can perhaps be done in a way that solves (or minimizes) the problems with eventual consistency in cloud storage systems.

Yes! I plan to spool uploads to disk quite soon. The main reason is to make uploads more reliable. Cloud storage systems being connected over the Internet are quite unreliable (in filesystem terms). It is easy to retry read operations, but write operations are impossible to retry as none of the cloud storage systems have a ranged PUT (some of them do do resume, but rclone doesn't support that yet). Getting the file onto local storage would then mean uploads can be retried at leisure.

A full solution like AFS would involve downloading files so they could be r/w modified - I'd like to get this far eventually :-)

[vampywiz17]

Hello @ncw

• Now, the GUI is much-much faster!!! Very nice work!!!!
  I dont know, that the reason your changes, or the network option (need to test it).

• buffer-size bug: It still happen :( i will post log soon...

• Network share: Working well!

Edit: now i change the folder view in windows to "detailed list" and i get this:

log1.txt

The mount is crashed.

Other thing. I i go to three level folder depth, the file list will be more slow (does not matter, that it contain only one file)

And i not understand this:

It say that the copy is really slow, (348KB/sec) but the network traffic go to maximum (21 MB/Sec)

(it FC about 20%...)

[sbr481]

Hi, i want to share a new problem i've been detected.
The problem is related to a runtime error and the cause is by out of memory. Here is the full log, since the remote is mounted, until FC happened.
log mount.txt
The log is large, ---debug-fuse flag is also being used. But i left the entire log, because maybe it's useful to see the behaviour of Windows and rclone mount function. Here is what i've done:
·Mount remote in CMD.
·Open file explorer
·Navigate to root of remote(t:) > Media folder > Peliculas folder
·At the moment i'm inside of the last folder, i've seen the crash of rclone.

Also, i've been testing new function --fuse-flag to test the remote as network drive. It seems to work well, but also crash by the same error mentioned in this post, runs out of memory. I'm going to make more tests with this flag, but i have the feeling that it crashes earlier than mounting as a local drive.
This remote is not crypted, and is a Google Drive Account.

[billziss-gh]

@ncw wrote:

• on read only open of file, make open pending until first read
  
  
  
  • This fixes a problem with Windows which seems fond of opening files just to read their attributes and closing them again.
  
  

Yes, Windows opens file for everything. Even a simple GetFileAttributesW will result in CREATE, QUERY_INFORMATION, CLEANUP and CLOSE IRP's to be sent to the file system driver (FSD). The FSD will only forward the CREATE (Open) and CLOSE to user mode and will satisfy the QUERY_INFORMATION call from its metadata cache, but it is still a lot of traffic.

[By default WinFsp-FUSE (and LIBFUSE) use a metadata timeout of 1 second. This can be controlled by the -o attr_timeout option.]

The FUSE layer will also add overhead of its own. For example, it is fond of doing Getattr as you probably already know :)

@sbr481 wrote:

The problem is related to a runtime error and the cause is by out of memory. Here is the full log, since the remote is mounted, until FC happened.

I looked at this too and the cause is not obvious (to me).

Golang is a garbage collected language so I would expect that collection would happen before running out of memory. However it is possible to "leak" memory in garbage collected languages, by simply stashing somewhere object references that the rest of the program "forgets" about.

OTOH we also have a number of native components in the mix now (WinFsp, cgofuse). This log is only 39 seconds long so it is rather unlikely that they are the culprit but still.

[sbr481]

Maybe i didn't explain well this:

The problem is related to a runtime error and the cause is by out of memory. Here is the full log, since the remote is mounted, until FC happened.

I mean the output says runtime error by out of memory (and i was talking about the reason of the runtime error), but what is the reason that throws this error, is a good question (or maybe the question). I don't know what's the reason, but i've noticed a lot of scans while i am navigating through folders (in that log i navigate in 2 subfolders).
Maybe the number of files, and queries to those files have some type of relationship with the error and gets out of memory. Intereting to note that this error was happening at least to me, with the 2 last versions of rclone-DEV (i mean it's not related to new and last compiled binary).
In fact, in my yesterday's last post, i got the same type of error, while i started to copying a file from the remote to local. I don't think the error was related to the copy process.
Probably the reason is simply Windows behaviour, but we know this is a very simple answer to a question that we know it has a more concrete answer.
These are only assumptions, i know it's difficult to identify and debug this errors, so we'll be here to help.
Regards.

[vampywiz17]

so we'll be here to help.

Yes :)

[billziss-gh]

so we'll be here to help.

Yes :)

@sbr481 and @vampywiz17 from my perspective the help is very much appreciated and recognized. Thank you for helping to improve rclone and (indirectly) WinFsp.

[vampywiz17]

log.txt

Here my new log. (what I promised earlier @ncw )

It is a copyjob with GUI. I mount it like a network drive.

Two serious bug.

• it FC if it will raise the buffer-sizesize.

• Real network speed and copyjob speed. It happen with and without enable buffer-size. The real bandwith usage it much higher that the copyjob say to me. Seems to be the data come to my PC, but the file system and/or rclone cant recognise it correctly.

Edit: A other copyjob. buffer-size enabled, After i create the screenshot it FC. but i show you that the difference the copy speed and the real network speed:

[sbr481]

It seems like it crashes because it starts to use a lot of RAM, at least in my tests; always crash when arrives near 850 MB. I've observated how it starts to grow it's consumption, and always crash when arrives near to that amount of RAM.
Here is the first screenshot, mounted as a network-drive:


Here is other screenshot when crashed as local drive:


Here is what rclone eats when navigating to a folder (using Total Commander as a simple test):

And always when reaches about 850 MB it crashes. If you see, all the crashes happen with the same amount of memory (see previous logs too). Maybe here is the main problem, with the stability. If not, at least it's curious.

[billziss-gh]

@sbr481 this definitely points to a memory leak IMO. Out of curiosity do you have any equivalent tests on Linux?

[ncw]

I found a major memory leak in the pending open code! This should also fix really slow transfers. Here is a fixed version

rclone-v1.36-89-g6352557e-cmount-rebase.zip

[ncw]

@sbr481 wrote

It seems like it crashes because it starts to use a lot of RAM, at least in my tests; always crash when arrives near 850 MB. I've observated how it starts to grow it's consumption, and always crash when arrives near to that amount of RAM.

Ah, I see I've been compiling rclone as a 32 bit binary which explains the 850 MB limit.

Here is a 64 bit binary for you to try

rclone-v1.36-90-ga8f76175-cmount-rebase-amd64.zip

[vampywiz17]

Nice work!!

With buffer-size 1G

Edit: something wrong with 1.36-90:

The 1.36-89 working well.

Edit2:

I found a another bug.

If i create a new .txt file, open it, write something and i try to save it, it say that no write rights. But i able to delete/rewrite the file. I attach the log.

log.txt

[ncw]

@vampywiz17 excellent!

I found a another bug.

If i create a new .txt file, open it, write something and i try to save it, it say that no write rights. But i able to delete/rewrite the file. I attach the log.

The error is this from the log

2017/05/12 11:33:56 ERROR : /Sorozatok/Új szöveges dokumentum.txt: Can't open for Read and

Write

This is a known limitation of the linux/OSX version too. It will get fixed by caching files locally #711

[vampywiz17]

This is a known limitation of the linux/OSX version too.

I see. Thanks the info!

Do you know about the 1.36-90 mount problem?

[ncw]

Do you know about the 1.36-90 mount problem?

That is the 64bit version. I haven't tried it at all (don't have a 64bit VM) so I wouldn't be suprised if it doesn't work. What did it do?

[vampywiz17]

What did it do?

https://cloud.githubusercontent.com/assets/14281430/25991887/2e6421e0-3705-11e7-8bab-49c90c8bcff3.png

[ncw]

@vampywiz17 thanks!

I forgot to enable CGO when cross compiling....

Try this version (again untested by me!) built with

X:\go\src\github.com\ncw\rclone>go env
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=386
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=X:\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_386
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Dev
\AppData\Local\Temp\go-build370856747=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

rclone-v1.36-91-g91ac31c4-cmount-rebase.zip

[vampywiz17]

Try this version (again untested by me!)

same error :(

[sbr481]

Hi,
@billziss-gh i didn't test with linux, but it would be interesting make some tests and see if it is affected. I think is different, and in linux won't be a problem, but it's a thought only... xD

I've been testing last rebase in x64 bits, and i have the same problems as @vampywiz17, mount option is not recognized. Something strange, but sure it's a little problem easy to correct.
So I've done some tests with 1.36-89 version. And it works great, i've been copy from remote to local at maximus speed of my bandwidth. Upload as normal with an individual file with rclone.
And it's very stable with RAM now, it gets at maximum of 210MB:

Working as a network drive, RAM is about 100 MB only... A great change!!
That's a great job guys. I've been testing to open a MKV file, and it takes some seconds to start, but seeking is very stable. And as i said, now RAM it's very stable at 210MB. I think main problem now is solved :)
Today it will be very difficult for me to test, but as soon as possible, i'll try everything xD
Once more, congrats guys!!

[vampywiz17]

Slowly Rclone do the same with Netdrive/Expandrive but it totally free!

[billziss-gh]

@ncw wrote:

I forgot to enable CGO when cross compiling....

I can create a build for the thread if needed as I have 64-bit Windows. However I do not think all commits are in the cmount branch yet?

Going forward we should definitely figure out a good way for doing builds. Xgo looks interesting, but it seems to be Linux only? [I am on OSX.] I would recommend AppVeyor, which is definitely capable of doing it.

We should also consider that this may be a cgofuse problem. It introduces a difficult problem to solve for projects that want to use it by requiring a C compiler. I would be happy to arrange it so that cgofuse has binary distributions. I am unclear on how to achieve this in the Go environment though.

[billziss-gh]

Seems the binary only packages proposal is already part of Go 1.7. I could arrange that cgofuse is also delivered as a binary-only package. This would eliminate the need for cross-compilation of rclone and would shift the burden to cgofuse (where it should be).

@ncw thoughts?

[billziss-gh]

After a bit of experimentation I got AppVeyor to produce Windows binaries for both x86 and x64 architectures: https://ci.appveyor.com/project/billziss-gh/cgofuse/build/135/artifacts

[NOTE: this is completely untested.]

Thinking more about this it might be worthwhile trying to integrate xgo into a Travis CI build for cgofuse, so that we get a single zip with all architectures in it.

[ncw]

A bit more reading convinced me that what I was trying to do by cross compiling 64 bits on 32 will never work :-( You can't do cgo while cross compiling without a lot more work.

I managed to build rclone using xgo -targets windows/* . I had to copy the WinFsp header files into cgofuse/fuse to get it to compile which isn't a good long term solution as it breaks all the other builds!

This has a 32 bit and 64 bit binary which should both have a mount subcommand.

rclone-v1.36-93-gfafd0512-cmount-rebase-xgo.zip

@billziss-gh interesting idea on binary only packages. It would solve part of the problem - rclone would still have to be compiled with cgo though which would mean a cross compile environment was needed. I'm not sure it is worth doing just for not having to install the WinFsp headers.

If we were to squeeze the WinFsp development parts into the form as described in the xgo docs that would fix the cross compile issues. It would presumably be quite easy to make a configure which did nothing and a make install which copied the header files into the right place.

I've been pushing commits to the cmount-rebase branch (which I shall force push so beware!) in preparation for merging it back to master. I want to get mountlib back into master as people keep sending me patches I have to forward port!

[ncw]

@billziss-gh wrote

After a bit of experimentation I got AppVeyor to produce Windows binaries for both x86 and x64 architectures: https://ci.appveyor.com/project/billziss-gh/cgofuse/build/135/artifacts

Nice!

You need to add package fuse to the two .go files BTW.

Relaxing the cgo build constraint I get this cross compiling on linux

cmd/cmount/fs.go:12: import /home/ncw/go/pkg/windows_386/github.com/billziss-gh/cgofuse/fuse.a: object is [windows 386 go1.8 X:framepointer] expected [windows 386 go1.8.1 X:framepointer]
g

So it is nearly working, and it is going to need a build for each go compiler version which is annoying.

(short pause to build go1.8)

and I get this

/opt/go/go1.8.0/pkg/tool/linux_amd64/link: cannot open file /opt/go/go1.8.0/pkg/windows_386/runtime/cgo.a: open /opt/go/go1.8.0/pkg/windows_386/runtime/cgo.a: no such file or directory

Which is kind of what I was expecting. It is probably not impossible to build that cgo.a.

However that is the problem xgo solves. I think it is probably less effort if we think of a way of getting the WinFsp headers into xgo. I can do this by copying them in as part of a build step and deleting them afterwards which isn't too bad, or we can make an xgo "cross platform" package.

Thinking more about this it might be worthwhile trying to integrate xgo into a Travis CI build for cgofuse, so that we get a single zip with all architectures in it.

Yes that would work.

Hmm, I wonder if you can run docker on Travis... Yes. Have to use the non-container build but that would be OK if a bit slower. The xgo container is massive though!

$ docker images
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
karalabe/xgo-latest           latest              0532a33f402b        4 weeks ago         4.322 GB

I think the optimal solution might be to build the windows builds on Appveyor (provided I can figure out how to build the 32 bit version and the 64 bit version) and push them up separately. Each release is a separate zip for each os/arch so that would work ok without too much work.

Anyway in summary I think there are 2 possibilities for automated builds

• Use xgo on Travis
• Build the windows builds on Appveyor, and the rest on Travis (like they are at the moment)

Using xgo to build locally is convenient for me running on linux.

[billziss-gh]

@billziss-gh interesting idea on binary only packages. It would solve part of the problem - rclone would still have to be compiled with cgo though which would mean a cross compile environment was needed. I'm not sure it is worth doing just for not having to install the WinFsp headers.

Interesting that cgo is still needed. Is it because a program that contains Cgo libraries requires to be linked with the native linker?

You need to add package fuse to the two .go files BTW.

Done. [I think -- I am only modifying travis/appveyor scripts and have not tested the produced builds at all.]

However that is the problem xgo solves. I think it is probably less effort if we think of a way of getting the WinFsp headers into xgo. I can do this by copying them in as part of a build step and deleting them afterwards which isn't too bad, or we can make an xgo "cross platform" package.

I probably need to bite the bullet and really look into xgo. I think somehow making the WinFsp headers into an xgo package makes sense.

I would also like to see this problem solved in a general fashion so that other projects that might want to use cgofuse will not hit the same walls.

---

FWIW, I can now produce cgofuse builds on OSX and Windows. Linux still evades me because it does not look like you can build a 32-bit FUSE program on 64-bit Linux without major work.

Branch where I am currently experimenting: link
Possibly related stackoverflow article: link

[vampywiz17]

Now ,the 64 bit binary is working!

But i found a other interesting thing.

I only just browse in folders, check the right click menu speed, etc.. and once the download speed come to the maximum and start to fill up my full memory...

I tried to close the explorer and all app but it was continued

here my log:
log.txt

[sbr481]

Hi, i'm testing new rclone 64 bits, and i can't mount a remote as a local drive, i get this error:

C:\Temp>rclone-windows-4.0-amd64.exe -vv mount --allow-other --allow-root --debug-fuse UDrive:/ t:
2017/05/13 14:02:34 DEBUG : rclone: Version "v1.36-DEV" starting with parameters ["rclone-windows-4.0-amd64.exe" "-vv" "mount" "--allow-other" "--allow-root" "--debug-fuse" "UDrive:/" "t:"]
2017/05/13 14:02:34 INFO : Google drive root '': Modify window is 1ms
2017/05/13 14:02:34 DEBUG : Google drive root '': Mounting on "t:"
2017/05/13 14:02:34 DEBUG : Google drive root '': Mounting with options: ["-o" "fsname=UDrive:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "debug" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "allow_other" "-o" "allow_root"]
2017/05/13 14:02:34 DEBUG : Google drive root '': Init:
2017/05/13 14:02:34 DEBUG : Google drive root '': >Init:
2017/05/13 14:02:34 DEBUG : /: Statfs:
2017/05/13 14:02:34 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/13 14:02:34 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 14:02:34 DEBUG : /: >Getattr: errc=0
The service rclone-windows-4.0- has been started.
2017/05/13 14:02:45 ERROR : mountpoint "t:" didn't became available after 10s - continuing anyway

I forgot to say that it doesn't crash, the drive appears in file explorer, but it can't be accessed.

By other way, mounting as a network drive, works perfectly, at least for me. I've tried to calculate a hash of a file of 10GB (on the fly), and observating RAM is not over 60MB (using 64bits version).

The problem i've noticed it's when navigating to some folders, it takes time to show content, because it analyzes the files, instead list the content first, and get attribs later. Per example, listing a folder with 15 Files (with a total of 200GB) takes some time to see the content in file explorer of Windows... xD And RAM is over 66 MB (nothing bad, is great).

I'll test i386 version to see it's behaviour.
UPDTATE:
Testing i386 binary:
· Same problem if trying to mount as a local drive.
· I've noticed it's greedy with RAM (mounted as a network drive):

That was only RAM consumption while navigating. When tried to hash the same file as my test in 64bit version:

Crash like previous version at the same amount of RAM (850MB).

A friend is using 64bit binary, while hashing a lot of files (10TB), and for now, it's OK, working very well (hashing 20 files at the same time, with about 670MB of RAM consumption only).

UPDATE2:
I've continuing testing and now, i was trying to upload directly, and there is a problem. Doesn't matter the size of the file (tried with 10GB, 4GB, 1GB, 400MB), this is what happens:
File start to upload as normal. Some seconds later it seems like counter of upload reset (i mean in Windows copy resume window), and then, it crash saying the file is too large to upload to the destination file system.
Here is an example:

And here is the log:
example_uploading_error.txt
Scenario: x64 binary, GDrive account (crypted), as network drive.
This time i've tested with a crypted remote, but i'm going to confirm that problem in a not crypted remote.
UPDATE3:
Confirmed, also happens in a not crypted remote.

[ncw]

I've been working on the build system...

Can someone with a win64 system have a go with this? I managed to produce it direct from Appveyor. Still haven't figured out how to build a 32 bit version on a 64 bit platform :-( Probably installing a 32 bit go and using a 32 bit C compiler will do it. I need to test on my 64 bit VM, but I had to restore it from a backup of 3 years ago and it has a LOT of updates to install!

https://beta.rclone.org/test/v1.36-98-g78b7fff4/rclone-v1.36-98-g78b7fff4-windows-amd64.zip

@billziss-gh wrote

Interesting that cgo is still needed. Is it because a program that contains Cgo libraries requires to be linked with the native linker?

I'm not sure. It is certainly missing the cgo.a library which probably could be sourced.

@vampywiz17 I can't really see what is going on in that log. Is it possible you had lots of files open - from the file handle numbers it looks like you did. If you are using --buffer-size 1G then each open file can potentially use 1G of memory (though only if the file is that big). Can you try agian with much smaller --buffer-size and see if that makes a difference?

[sbr481]

Same uploading problem with this version:

https://beta.rclone.org/test/v1.36-98-g78b7fff4/rclone-v1.36-98-g78b7fff4-windows-amd64.zip

@ncw i'm going to try what you say, compile a 32 bit binary, in a 64 bit system
EDIT:
It seems it works, in a W7 64 bit system, installed go lang in 32 bits, downloaded branch, and compiled:
rclone-cmount-rebase.zip
Only as a test (mount or cmount option doesn't work, i don't know why, it says unknown command with mount or cmount).

[vampywiz17]

Now i try to set to buffer size 200M and after try to open a video file. about 3 min later, the video is freeze. It possible that happen on 1 G buffer size, just until this moment i never wait it in the past.

It seems that only play the buffered size and not continue the download.

Edit: now, that i write this post, it resume play! (it stop about 3 min) :)

I attach the log:

log.txt

It is the "break" part of the log:

2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=336686224, fh=0x603
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=1538748
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=338224972, fh=0x603
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=1513652
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=339738624, fh=0x603
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=25096
2017/05/13 21:13:08 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=339763720, fh=0x603
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Opendir: 
2017/05/13 21:13:53 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 21:13:53 DEBUG : /: Statfs: 
2017/05/13 21:13:53 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/13 21:13:53 DEBUG : /: Releasedir: fh=0x1
2017/05/13 21:13:53 DEBUG : /: >Releasedir: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Opendir: 
2017/05/13 21:13:53 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 21:13:53 DEBUG : /: Releasedir: fh=0x1
2017/05/13 21:13:53 DEBUG : /: >Releasedir: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Opendir: 
2017/05/13 21:13:53 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 21:13:53 DEBUG : /: Releasedir: fh=0x1
2017/05/13 21:13:53 DEBUG : /: >Releasedir: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:53 DEBUG : /: >Getattr: errc=0
2017/05/13 21:13:53 DEBUG : /: Opendir: 
2017/05/13 21:13:53 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 21:13:53 DEBUG : /: Releasedir: fh=0x1
2017/05/13 21:13:53 DEBUG : /: >Releasedir: errc=0
2017/05/13 21:13:57 DEBUG : /desktop.ini: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:13:57 DEBUG : : Re-reading directory (4m56.9413468s old)
2017/05/13 21:13:57 DEBUG : amazon drive root 'data': Reading ""
2017/05/13 21:13:58 DEBUG : amazon drive root 'data': Finished reading ""
2017/05/13 21:13:58 DEBUG : /desktop.ini: >Getattr: errc=-2
2017/05/13 21:15:40 ERROR : Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: ReadFileHandle.Read error: low level retry 1/10: read tcp 192.168.31.102:57698->52.216.80.48:443: wsarecv: A létező kapcsolatot a távoli állomás kényszerítetten bezárta.
2017/05/13 21:15:40 DEBUG : Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: ReadFileHandle.seek from 339763720 to 339763720
2017/05/13 21:15:40 DEBUG : 0dac77pcik3i8pnual6j1v3lmg/ho0qlhelc32s5n02il4jcsrgn1r6b77bjqcgrjgjlh5irocovfd549tvud4mkggvpfjeeonjtjahe/0f03e790i0qkop21c7lo8gb50kt9brji7ovupl6otchvv141700g: Downloading large object via tempLink
2017/05/13 21:15:41 DEBUG : 0dac77pcik3i8pnual6j1v3lmg/ho0qlhelc32s5n02il4jcsrgn1r6b77bjqcgrjgjlh5irocovfd549tvud4mkggvpfjeeonjtjahe/0f03e790i0qkop21c7lo8gb50kt9brji7ovupl6otchvv141700g: Downloading large object via tempLink
2017/05/13 21:15:43 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=1513652
2017/05/13 21:15:43 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=341277372, fh=0x603
2017/05/13 21:15:43 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=1538748
2017/05/13 21:15:43 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=342816120, fh=0x603
2017/05/13 21:15:44 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=1116808
2017/05/13 21:15:44 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=343932928, fh=0x603
2017/05/13 21:15:44 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: >Read: n=421940
2017/05/13 21:15:44 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-HyperX/arrival-1080p-hyperx.mkv: Read: ofst=344354868, fh=0x603
2017/05/13 21:15:44 DEBUG : /Filmek/Arrival.2016.1080p.BluRay.DD5.1.x264.HuN-H

@sbr481 need more part of to compile that rclone with mount feature (gcc and etc... @ncw write it in the past the necessary modules)

[sbr481]

@vampywiz17 Make a try without --buffer-size flag.

[vampywiz17]

I use latest version (1.36-98)

Now i try to upload a about 1 gb size iso. (i test with @sbr481 problem) but i finished successfully. the speed is a little hectic, but no problem at all. Here the log:

log.txt

Edit: well i'm wrong... now i check the uploaded file and it say that the size is 0b...

the log say:

2017/05/13 21:37:33 DEBUG : manjaro-deepin-17.0.1-stable-x86_64.iso: Error detected after finished upload - waiting to see if object was uploaded correctly: HTTP code 409: "409 Conflict": response body: "{\"logref\":\"9b643863-3813-11e7-9902-a74d0fa82336\",\"message\":\"Node with the name g8tn0e7e0ta53b4dcf0ep905qh4dldbh1cp1th7r0kt20c2g6jni5k1i534iqle6mvhsth5emunam already exists under parentId DPDWHldaQWWME9CGIfvaDg conflicting NodeId: 5pAr5IPPTHOSQcSVg_H2kQ\",\"code\":\"NAME_ALREADY_EXISTS\",\"info\":{\"nodeId\":\"5pAr5IPPTHOSQcSVg_H2kQ\"}}" ("409 Conflict")
2017/05/13 21:37:34 DEBUG : manjaro-deepin-17.0.1-stable-x86_64.iso: Object found with correct size 32 after waiting (1/1) - 0s - returning with no error
2017/05/13 21:37:34 DEBUG : /manjaro-deepin-17.0.1-stable-x86_64.iso: >Flush: errc=0
2017/05/13 21:37:34 DEBUG : /manjaro-deepin-17.0.1-stable-x86_64.iso: Release: fh=0x2
2017/05/13 21:37:34 DEBUG : manjaro-deepin-17.0.1-stable-x86_64.iso: WriteFileHandle.Release nothing to do
2017/05/13 21:37:34 DEBUG : /manjaro-deepin-17.0.1-stable-x86_64.iso: >Release: errc=0
2017/05/13 21:38:28 DEBUG : /desktop.ini: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 21:38:28 DEBUG : : Re-reading directory (1m15.7178708s old)
2017/05/13 21:38:28 DEBUG : amazon drive root 'data': Reading ""
2017/05/13 21:38:29 DEBUG : amazon drive root 'data': Finished reading ""
2017/05/13 21:38:29 DEBUG : /desktop.ini: >Getattr: errc=-2

Now i try a little file (about 200 M) without problem.

I upload it, after download it i do a hash check, but no problem.

Here the log:

log2.txt

[nicko88]

Strange, I am trying all these builds but I am not seeing a drive get mounted.

2017/05/13 15:06:44 DEBUG : rclone: Version "v1.36-DEV" starting with parameters ["rclone" "mount" "gdrivecrypt:" "Z:" "--log-file=c:/log.txt" "-vv"]
2017/05/13 15:06:45 INFO  : Encrypted drive 'gdrivecrypt:': Modify window is 1ms
2017/05/13 15:06:45 DEBUG : Encrypted drive 'gdrivecrypt:': Mounting on "Z:"
2017/05/13 15:06:45 DEBUG : Encrypted drive 'gdrivecrypt:': Mounting with options: ["-o" "fsname=gdrivecrypt:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone"]
2017/05/13 15:06:45 DEBUG : Encrypted drive 'gdrivecrypt:': Init: 
2017/05/13 15:06:45 DEBUG : Encrypted drive 'gdrivecrypt:': >Init: 
2017/05/13 15:06:45 DEBUG : /: Statfs: 
2017/05/13 15:06:45 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
The service rclone has been started.
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Opendir: 
2017/05/13 15:06:45 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 15:06:45 DEBUG : /: Statfs: 
2017/05/13 15:06:45 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Opendir: 
2017/05/13 15:06:45 DEBUG : /: >Opendir: errc=0, fh=0x101
2017/05/13 15:06:45 DEBUG : /: Releasedir: fh=0x1
2017/05/13 15:06:45 DEBUG : /: >Releasedir: errc=0
2017/05/13 15:06:45 DEBUG : /: Releasedir: fh=0x101
2017/05/13 15:06:45 DEBUG : /: >Releasedir: errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/13 15:06:45 DEBUG : /: >Getattr: errc=0
2017/05/13 15:06:45 DEBUG : /: Opendir: 
2017/05/13 15:06:45 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/13 15:06:45 DEBUG : /: Releasedir: fh=0x1
2017/05/13 15:06:45 DEBUG : /: >Releasedir: errc=0

Must be something simple I am doing wrong.

[sbr481]

Here is a video explaining what i was saying, at least for me this happens at 99%:
https://www.anonfiles.cc/file/7e1b9ee351b16fa8fd16be2c0097d124
I can't upload the file here, so i left it in anonfiles, it's a wmv zipped (i don't know why i can't attach that zip here...)
@vampywiz17 I know maybe mount function needs more stuff and dependencies to solve, but i wanted to test only compiling a 32 bit binary in a win64 system. And there are different ways to compile, if i have time, i'll try to see that stuff. xD

I was only able to complete a transfer, 2 times of all my tests (i made a lot of tests, encrypted and unencrypted). I don't know if it's related to Google Drive. Download it's not a problem, it uses all my bandwidth.

[sbr481]

@nicko88 Don't use Administrator CMD.

[nicko88]

@sbr481 Oh thanks, yeah it works.

[billziss-gh]

@ncw wrote:

Can someone with a win64 system have a go with this? I managed to produce it direct from Appveyor.

I tried it but got the usual rclone: command not found when I tried rclone mount.

Still haven't figured out how to build a 32 bit version on a 64 bit platform :-( Probably installing a 32 bit go and using a 32 bit C compiler will do it.

I believe that is the ticket. At least that is how I was able to build a 32-bit version of cgofuse under AppVeyor. Here is the relevant part in the cgofuse AppVeyor script:

build_script:
- set CGO_ENABLED=1
- set PATH=C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin;%ORIGPATH%
- set GOARCH=386
- go install -v ./...
- set PATH=C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin;%ORIGPATH%
- set GOARCH=amd64
- go install -v ./...

So at first this uses the 32-bit gcc from Mingw-w64 to compile for GOARCH=386. It then uses the 64-bit gcc from Mingw-w64 to compile for GOARCH=amd64.

---

Unfortunately I have not had the time to look more into these build issues and probably will not have a lot of time in the coming days (as something non-opensource related has come up).

[billziss-gh]

Quick note: I managed to install xgo on my mac and am able to use it using Docker for Mac. Things seem to work a bit, but of course:

$ ~/Projects/go/bin/xgo --targets=darwin/*,linux/*,windows/* ./fuse
Checking for required docker image karalabe/xgo-latest... found.
Cross compiling github.com/billziss-gh/cgofuse/fuse...
Building locally github.com/billziss-gh/cgofuse/fuse...
Compiling for darwin-10.6/amd64...
# github.com/billziss-gh/cgofuse/fuse
osxcross: warning: possibly dangerous include path specified: '-I /usr/local/include/osxfuse/fuse'
osxcross: info: you can silence this warning via 'OSXCROSS_NO_INCLUDE_PATH_WARNINGS=1' (env)
./host.go:35:10: fatal error: 'fuse.h' file not found
#include <fuse.h>
         ^
1 error generated.
2017/05/13 16:18:30 Failed to cross compile package: exit status 2.

So this means (I think) that we have to somehow create a cross-platform "FUSE" that can be used with xgo's --deps. Looking at the xgo source such a package has to support configure and make install. [[link](https://github.com/karalabe/xgo/blob/master/docker/base/build_deps.sh)]

But this is non-trivial IMO. Even if we tried to run individual xgo command lines with the different versions of FUSE, this clearly would require work.

$ ~/Projects/go/bin/xgo --targets=darwin/* --deps=https://github.com/osxfuse/fuse/archive/master.zip ./fuse
Checking for required docker image karalabe/xgo-latest... found.
Downloading new dependency: https://github.com/osxfuse/fuse/archive/master.zip...
New dependency cached: /Users/billziss/.xgo-cache/master.zip.
Cross compiling github.com/billziss-gh/cgofuse/fuse...
Building locally github.com/billziss-gh/cgofuse/fuse...
Compiling for darwin-10.6/amd64...
# github.com/billziss-gh/cgofuse/fuse
osxcross: warning: possibly dangerous include path specified: '-I /usr/local/include/osxfuse/fuse'
osxcross: info: you can silence this warning via 'OSXCROSS_NO_INCLUDE_PATH_WARNINGS=1' (env)
./host.go:35:10: fatal error: 'fuse.h' file not found
#include <fuse.h>
         ^
1 error generated.
2017/05/13 16:23:12 Failed to cross compile package: exit status 2.

Unfortunately I do not think I will have the necessary time to meaningfully contribute towards this goal in the coming days. OTOH It might just be a question of a few hours fiddling with configure scripts and Makefiles to make this happen.

[billziss-gh]

Unfortunately I do not think I will have the necessary time to meaningfully contribute towards this goal in the coming days.

I actually ended up spending time this Sunday on this and I believe I have it (almost) working. I can now produce cross platform builds for cgofuse using xgo on CircleCI. Here is an example artifact produced from CircleCI: link. This artifact contains the linux and windows, 32-bit and 64-bit builds.

The OSXFUSE build currently does not work. I am getting some error messages documented in karalabe/xgo#80.

To make this happen I used this Dockerfile, which is based on the xgo docker image. Then a relatively simple CircleCI configuration file does the rest.

Please note: I have not tested any of the resulting cgofuse binaries. I have no idea if they work or not.

[I chose CircleCI, because I found that it has good support for docker and because it supports uploading artifacts. Unfortunately the artifacts page is private to the project maintainer, even for open source projects. So the artifact URL is not easily discoverable.]

[billziss-gh]

I have resolved the OSXFUSE issue and now have prebuilt binaries of cgofuse in this location:

https://github.com/billziss-gh/cgofuse/releases

Please note that I still have not had the chance to verify that these binaries actually work as advertised (i.e. that the cross compilation actually works and does not produce nonsense). I am hoping to do so later tonight or tomorrow.

[vampywiz17]

@vampywiz17 I can't really see what is going on in that log. Is it possible you had lots of files open - from the file handle numbers it looks like you did. If you are using --buffer-size 1G then each open file can potentially use 1G of memory (though only if the file is that big). Can you try agian with much smaller --buffer-size and see if that makes a difference?

@ncw

I tested the stream with disabled (default) buffer size and no problem at all. But, if set a highter buffer size, it freeze.

I check a logs and seems to if i set a hight buffer, after the data pre-download, the connection is disconnected and after i reach the end of buffer size (and need to continue the dowload/buffering) it need some time to reconnect it.

On Linux (with FUSE) Rclone first fill up the buffer, after it contstantly clean the unnecessary data and download the new part of the file. (ergo not need to disconnect the connection)

I think this is missing or sometimes work wrong on WinFsp.

[ncw]

@billziss-gh thanks for your help - your appveyor examples have been really useful. I have finally managed to make travis and appveyor build windows, osx and linux versions with a great deal of messing around. The os x / linux builds I am building without cmount and the windows build with.

I'd like to make the cgofuse fit into the xgo framework in due course, but having travis and appveyor doing the builds is good enough for the moment.

I've now merged the cmount branch to master so beta builds will appear. Here is the latest.

https://beta.rclone.org/v1.36-98-g7ee3cfd7/

I'm going to go through this thread and pick up bugs to fix, after that I'm going to close the issue as it has got rather long!

[vampywiz17]

Yes, the main goal is finished now, the possible bugs need to create separate threads. (This one a little dense now.)

[billziss-gh]

@ncw I agree. It may even be safer to do the builds on travis/appveyor rather than xgo, because they are done natively and may avoid cross-compiler bugs.

As for isolating the cgo dependency in cgofuse and consequently making rclone builds easy, this does not seem easily possible. The go tooling is just not mature enough (and somewhat amateurish in some respects IMO).

Here is what I get trying to build rclone for Windows on my OSX after installing the prebuilt cgofuse and making any necessary changes in rclone (disabling cgo build constraints, deleting the cgofuse vendor directory, etc):

$ GOOS=windows GOARCH=amd64 go build rclone.go
# github.com/ncw/rclone/cmd/cmount
cmd/cmount/fs.go:92: undefined: fuse.EBADF
cmd/cmount/fs.go:97: undefined: fuse.EBADF
cmd/cmount/fs.go:103: undefined: fuse.EBADF
cmd/cmount/fs.go:141: undefined: fuse.ENOTDIR
cmd/cmount/fs.go:161: undefined: fuse.EISDIR
cmd/cmount/fs.go:203: undefined: fuse.Stat_t
cmd/cmount/fs.go:253: undefined: fuse.Stat_t
cmd/cmount/fs.go:276: undefined: fuse.Stat_t
cmd/cmount/fs.go:327: undefined: fuse.Statfs_t
cmd/cmount/fs.go:543: undefined: fuse.Timespec
cmd/cmount/fs.go:161: too many errors

[Why are symbols missing? I thought I provided a prebuilt-binary? And if some symbols are not in the binary (nm -a confirms that), do we now also need a .go file that defines some symbols and acts somewhat like a C header file?]

After closing the thread I may not always receive notifications about a discussion of relevance. Please feel free to ping me any time you think I might be able to help.

Thank you for using WinFsp in rclone and for teaching me about Go.

[sbr481]

Hi, i'll only add a comment to this version i've been testing:
https://beta.rclone.org/v1.36-98-g7ee3cfd7/

Mounting remotes as a local drive works as well as downloading as uploading.(GREAT!!)
Mounting remotes as a network-drive works well downloading, but the same problem uploading that i commented 4 days ago.

[qnorsten]

Glad to see that this has made so much progress:) Will play around and test the beta now.
Just a question is it possible to mount this to a folder on an other drive (similar to how it works on linux) or to a network drive, without it needing to take up one drive letter (e.g X:) as I have several cloud drives that I would like to mount and do not want to clog up my computer with everyone as drive letter.
I read that --fuse-flag --VolumePrefix=\server\share could be use to mount it as a network drive, but still can not figure out how to do it without assigning a drive letter to it.

[billziss-gh]

@qnorsten it is actually possible to create a WinFsp file system without mounting it. However for compatibility reasons that functionality is not available to FUSE.

You can however mount the file system on a directory. For example, the following command line should do it:

rclone mount remote:path localpath

After this gets executed a new localpath directory will be created and you will be able to access your files in there. Contrary to UNIX the localpath directory must not exist prior to mounting.

There is one caveat. Windows does not really like file systems that are case-sensitive when mounted on directories. Sometimes it tends to get the case of file names wrong, which makes most things work, but some things may not. This is a Windows limitation that I do not believe can be addressed by WinFsp.

If the user mode file system is case-insensitive then everything just works. I am not sure whether rclone mount is case-sensitive or case-insensitive, I suspect that it depends on the particular backend.

[qnorsten]

Thanks for that info @billziss-gh that is interesting news. Even though that do not seem to work that good with rclone right now.
The mount folder is created and I can see the root folder, but can not access any other subfolder without an "Location is not available" error and if I try to create a new file I get an "Could not find item" error message but the folder is created anyway. (similar to as reported earlier in this issue)

This is the command I run. rclone mount educrypt: G:\test -vv --log-file=g:\rclonelog
Where educrypt is a rclone crypt on a gdrive.

Here is a log. Perhaps ncw can fix that sometime in the future, but not a big deal as mounting on a drive letter seem to work good.

rclonelog.txt

[billziss-gh]

A quick look at the log reveals the problem I was referring to earlier: that Windows treats a file system as case-insensitive when mounted on a NTFS directory regardless of what the file system reports to the operating system.

Notice for example, that the directory Tecknat is accessed under the names /Tecknat and /TECKNAT.

Returns ENOENT (file not found):

2017/05/17 22:59:57 DEBUG : /TECKNAT: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/17 22:59:57 DEBUG : /TECKNAT: >Getattr: errc=-2

Succeeds (errc=0):

2017/05/17 22:59:57 DEBUG : /Tecknat: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/17 22:59:57 DEBUG : /Tecknat: >Getattr: errc=0

This is a well-known Windows problem. The fix would be to make rclone case-insensitive, which is probably not feasible across all backends.

[qnorsten]

Thanks for the information. Guess I will have to live without the mount in folder functionality. It is too bad that windows have all these special quirks.

[ettorep]

When ever I try to open a file in windows, by double clicking on it in explorer it fails to open and the file size becomes 0.

My mount command is
mount -o uid=-1 -o gid=-1 --fuse-flag --VolumePrefix=\server\share gc: g:

I see the following error on the command line:
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 1/10: couldn't reopen file with offset: bad response: 416: 416 Requested range not satisfiable
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 2/10: EOF
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 3/10: EOF
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 4/10: EOF
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 5/10: EOF
...
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: low level retry 10/10: EOF
ERROR : music/#084 - book1.jpg: ReadFileHandle.Read error: EOF
ERROR : IO error: EOF

[fabioneves]

How can we mount to a "local drive" that doesn't exist?
I tried: rclone mount gdrive: x: but nothing happens, it just says rclone service started but I don't see any "X" drive on my PC.

[ghost]

@fabioneves
do you have winfsp installed?

[fabioneves]

Yes, I have @azureblaze. Here's the log when I try to mount to h:, nothing happens basically.. I don't see any new drive, but it doesn't appear to throw any error either.
I have WinFsp (1.0.17072) installed.

c:\Tools>rclone mount boda: h: -vv
2017/05/18 16:42:57 DEBUG : rclone: Version "v1.36-98-g7ee3cfd7" starting with parameters ["rclone" "mount" "boda:" "h:" "-vv"]
2017/05/18 16:42:58 INFO : Google drive root '': Modify window is 1ms
2017/05/18 16:42:58 DEBUG : Google drive root '': Mounting on "h:"
2017/05/18 16:42:58 DEBUG : Google drive root '': Mounting with options: ["-o" "fsname=boda:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone"]
2017/05/18 16:42:58 DEBUG : Google drive root '': Init:
2017/05/18 16:42:58 DEBUG : Google drive root '': >Init:
2017/05/18 16:42:58 DEBUG : /: Statfs:
2017/05/18 16:42:58 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
The service rclone has been started.
2017/05/18 16:42:58 DEBUG : /: Statfs:
2017/05/18 16:42:58 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:8796093022207 Bfree:8796093022207 Bavail:8796093022207 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Opendir:
2017/05/18 16:42:58 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/18 16:42:58 DEBUG : /: Releasedir: fh=0x1
2017/05/18 16:42:58 DEBUG : /: >Releasedir: errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Opendir:
2017/05/18 16:42:58 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/18 16:42:58 DEBUG : /: Releasedir: fh=0x1
2017/05/18 16:42:58 DEBUG : /: >Releasedir: errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2017/05/18 16:42:58 DEBUG : /: >Getattr: errc=0
2017/05/18 16:42:58 DEBUG : /: Opendir:
2017/05/18 16:42:58 DEBUG : /: >Opendir: errc=0, fh=0x1
2017/05/18 16:42:58 DEBUG : /: Releasedir: fh=0x1
2017/05/18 16:42:58 DEBUG : /: >Releasedir: errc=0

[fabioneves]

Finally managed, mounting with admin cmd prompt doesn't work!

[ghost]

Interesting, I tried mounting with an elevated prompt just to test and it won't show up in powershell/cmd when I'm running as normal user, but it works when elevated, not sure if that's something to be fixed or not.

[billziss-gh]

Interesting, I tried mounting with an elevated prompt just to test and it won't show up in powershell/cmd when I'm running as normal user, but it works when elevated, not sure if that's something to be fixed or not.

This is by (unfortunate) design on Windows. The drive namespace that a user sees when elevated and when not is different. See DefineDosDeviceW and this link.

The way to add a global drive (visible to all users) is to start the file system under the SYSTEM account.

I should add an FAQ entry on this to WinFsp.

[32kHz]

Please excuse my poor English.

Files and directories with names longer than 128 characters are not displayed.
As a result of trial, it seems that only the number of characters less than half the value of "MaxComponentLength" seem to be displayed.
(I tried it with "Google Drive", "encrypted Google Drive" "encrypted local directory".)

Is this a problem only for me?

OS: Windows 7 64bit
rclone: v1.36-98-g7ee3cfd7
WinFsp: 1.0.17072 and 1.1.17131

[ghost]

I can confirm the same issue as 32kHz, as a result folders with filenames that are for example 119 characters when decrypted but in excess of 255 when encrypted are inaccessible both through windows explorer and powershell/cmd.exe

[billziss-gh]

Interesting. @ncw is there any reason that rclone would restrict the number of characters in a file name?

There are a couple of additional possibilities although from the wording of the reports above sound less likely:

• Most Windows apps are coded with a hard limit of 260 paths for the overall path (i.e. C:\path\to\my\files must be less than 260 chars). So you may be actually hitting that limit rather than MaxComponentLength limit. [The 260 limits is not a hard limit for Windows, but its a historical limit for Win32 apps deriving from the MAX_PATH constant.]

• This may also be a bug with WinFsp-FUSE. I know there are tests that specifically test the maximum file name length for the native WinFsp API. However I don't think I have such tests currently for the FUSE layer.

[32kHz]

Thanks for replying.

I tried the MAX_PATH restriction under the following conditions.

Directory structure: Z:\aaa...\bbb...\testfile ("aaa..." is 127 characters, "bbb..." is 127 characters, totaling 267 characters)
File Manager: 7-zip File Manager

Google Drive (non crypt) - OK
Google Drive (crypt) - OK

If the file manager supports long path names, it seems to be accessible without problems.
(However, as mentioned above, it becomes impossible to access if the name of a file or directory exceeds 127 characters.)

By the way, about the 127 character limitation, it seems that restrictions become more severe when using non-ASCII characters.
As far as I confirmed, it was not displayed unless it had to be 85 characters or less.
(This was the same whether crypt or non crypt.)

[billziss-gh]

I have done some tests and I do not believe this to be a problem with WinFsp-FUSE.

For my tests I used a simple "passthrough" file system built on top of FUSE. This file system is actually based on POSIX and it runs on Cygwin, but also on native Windows using the simple Windows/POSIX layer in winposix.c. I used the Cygwin version of it, because winposix.c uses CreateFileA (and other "ANSI" API's) which have internal limits of MAX_PATH. Cygwin uses NTOS or Win32 Unicode API's which have no MAX_PATH limits.

Here is a session that proves that passthrough-cygfuse (which is a WinFsp-FUSE file system) supports filenames up to (and exactly) 255 characters.

billziss@windows:~/Projects/winfsp/tst/passthrough-fuse [release/1.1]$ make cygfuse
gcc passthrough-fuse.c -o passthrough-cygfuse -g -Wall `pkg-config fuse --cflags --libs`
billziss@windows:~/Projects/winfsp/tst/passthrough-fuse [release/1.1]$ ./passthrough-cygfuse.exe . y: billziss@windows:~/Projects/winfsp/tst/passthrough-fuse [release/1.1]$ cmd
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\billziss\Projects\winfsp\tst\passthrough-fuse>y:
y:

Y:\>echo hello>123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345
echo hello>123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345

Y:\>echo hello>123456789012345678901234567890123456789012345678901234567890123456789012345678901234568901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
echo hello>1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
The filename, directory name, or volume label syntax is incorrect.

Y:\>dir
dir
 Volume in drive Y has no label.
 Volume Serial Number is 659E-20B9

 Directory of Y:\

05/17/2017  08:12 PM                60 .gitignore
05/22/2017  12:02 PM                 7 123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345
05/17/2017  08:12 PM               620 Makefile
05/17/2017  08:12 PM               341 README.md
05/22/2017  12:01 PM            83,836 passthrough-cygfuse.exe
05/17/2017  08:12 PM             8,562 passthrough-fuse.c
05/17/2017  08:12 PM             1,313 passthrough-fuse.sln
05/17/2017  08:12 PM            10,298 passthrough-fuse.vcxproj
05/17/2017  08:12 PM               708 passthrough-fuse.vcxproj.filters
05/17/2017  08:12 PM            14,440 winposix.c
05/17/2017  08:12 PM             2,126 winposix.h
              11 File(s)        122,311 bytes
               0 Dir(s)  18,505,936,896 bytes free

Y:\>exit
exit
billziss@windows:~/Projects/winfsp/tst/passthrough-fuse [release/1.1]$ pkill passthrough

The first file name written is exactly 255 characters. This succeeds. The second file name is 256 characters. This fails with The filename, directory name, or volume label syntax is incorrect. I believe this to be correct.

[32kHz]

Thank you for the detailed report.
I think that it is not a problem of WinFsp.

Well, there is one correction.
I told you in the previous post that "Google Drive (not crypt)" has 127 character limit, but this was a mistake.
Up to 255 letters were OK, so-called ASCII letters only.
Other cases are described above.
I'll apologize and correct that.

The following is a summary.

1. Google Drive (not crypt)
   ASCII characters only - Up to 255 characters
   Non-ASCII characters only - Up to 85 characters
2. Google Drive (crypt)
   ASCII characters only - Up to 127 characters
   Non-ASCII characters only - Up to 85 characters

• Non-ASCII characters used "∀".

[billziss-gh]

@32kHz I think this makes more sense now.

The character ∀ is Unicode character 0x2200, which translates to the UTF-8 byte sequence e2 88 80. Thus 1 character takes 2 bytes to encode in UTF-16 (Windows "Unicode") and 3 bytes to encode in UTF-8 (which FUSE and POSIX uses).

You can see now how 85 of those characters will eat up the limit pretty fast: 85*3=255

[32kHz]

Thanks for your reply.
In other words, when using rclone mount, does the file (and directory) name have a limit of 255 bytes instead of 255 characters?
(I'm poor in English and programming knowledge, so I may be misunderstanding it.)

[powareverb]

I believe here that Windows internals themselves have this limit, along
with one around the total path length. I believe that there are a few ways
around it but not great in general. Note this is not an issue with
filesystems themselves e.g. NTFS supports 4000+ file lengths but the win
API is just limited, and of course backwards compatible. There are some
github issues clearly identifying this but would have to find them for you.

Gav

On Wed, 24 May 2017, 14:38 32kHz, [e-mail] wrote:

Thanks for your reply.
In other words, when using rclone mount, does the file (and directory)
name have a limit of 255 bytes instead of 255 characters?
(I'm poor in English and programming knowledge, so I may be
misunderstanding it.)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/1018#issuecomment-303594171, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAsS67dn2Snws6OFbDDTn8Gu3l3KvwTtks5r84rogaJpZM4LgTTa
.

[powareverb]

Here, this may be enlightening:
https://stackoverflow.com/questions/530109/how-to-avoid-system-io-pathtoolongexception

I'm not 100% sure it's what you're currently talking about, but pretty sure it applies here.
Although the post is .NET specific, the issue is with the Win32 internals and pretty sure is the same as you're discussing.

[billziss-gh]

In other words, when using rclone mount, does the file (and directory) name have a limit of 255 bytes instead of 255 characters?

You are spot on.

[32kHz]

@ Poolareverb
Thank you for the advice.
I will read the introduced article from now.

@ Billziss-gh
Thank you for your reply.
I understood that it was not a misunderstanding, I was relieved.

By the way, once I tried it, more than 128 characters (ASCII) of directories and files are no longer displayed in "Google Drive (not crypt)".
As a result of various trials, I understood the cause.
By doing this, I have to correct the summary.

If the following conditions are satisfied, the directory or file will be displayed with a name of 128 characters (ASCII) or more.
(Tested with "Google Drive (not crypt)" "Google Drive (crypt)")

1. There are multiple directories or files in the same directory.
2. The directory or file is not the last in the name order. (This is insufficient verification)

Therefore, if I summarize what I have learned so far, it will be as follows.

1. Google Drive (not crypt)
   ASCII characters only - Up to 127 characters (but up to 255 characters if the above conditions are met)
   Including non-ASCII characters - up to 255 bytes
2. Google Drive (crypt)
   ASCII characters only - Up to 127 characters (but up to 255 characters if the above conditions are met)
   Including non-ASCII characters - up to 255 bytes

ASCII characters are 1 byte in UTF-8, aren't they?
Why is it sometimes that only 127 characters can be displayed?

I'm sorry to be so inquisitive.

[cjnama]

I use EncFSMP mount drive it can play 4k video (TS) very good
but use rclone-v1.36-98-g7ee3cfd7-windows-amd64 (mount drive)
even SD .mp4 it cannot open it
open pdf doc files is ok
so rclone cannot support EncFS ?
BTW
if someday Google Drive API stop support Rclone like ACD
The all encrypts files must download to PC and how to see it ?

[cjnama]

sorry EncFSMP + ExpanDrive on Google Drive

[ghost]

@cjnama download the files using some other tool, then create a new crypt remote pointing to the local path, using the exact same passwwords (better yet just change the path in the rclone.conf) and do something like rclone move local_crypt: /path/to/decrypted/files

[billziss-gh]

@32kHz

I'm sorry to be so inquisitive.

An inquisitive mind is a great thing :) Keep it up!

ASCII characters are 1 byte in UTF-8, aren't they?

Yes.

Why is it sometimes that only 127 characters can be displayed?

Unfortunately I cannot answer this directly and without digging into the rclone source and I do not have time for that currently.

[32kHz]

Thanks for replying me.
I think I will investigate it a little more on my own.
I thank everyone who gave me a reply.

[TechTrooper]

You've done amazing work here. Mounting actually works and I'm able to stream very high bitrate video.

Issues so far

Plain & Crypt GDrive mounts
copy compressed files from local disk to mount via file explorer:
2017/06/07 11:14:16 ERROR : /User/Downloads/madVR.zip: Can't truncate files

Explorer window randomly hangs when right click on a file.
Explorer window randomly hangs when accessing mount
System freezes when remote is mounted (Freezes are 1s or less in duration, massive DPC latency spikes)

Remotes mounted with standard command: rclone mount remote:path/to/files /path/to/local/mount &
OS: Windows 10 Pro 1703 64-bit
Rclone: v1.36-159-g3fe94482-windows-amd64
WinFSP: winfsp-1.0.17072

Should I be using pre-release versions instead? At the moment WinFsp 2017.1 B3 seems to be available.

[billziss-gh]

Plain & Crypt GDrive mounts
copy compressed files from local disk to mount via file explorer:
2017/06/07 11:14:16 ERROR : /User/Downloads/madVR.zip: Can't truncate files

My understanding is that rclone mount does not support seeking in a file, so such problems should be expected.

Explorer window randomly hangs when right click on a file.
Explorer window randomly hangs when accessing mount
System freezes when remote is mounted (Freezes are 1s or less in duration, massive DPC latency spikes)

These are most likely unrelated to WinFsp, and are perhaps related to your network or system setup. WinFsp does not really use DPCs as it is a file system driver and does not service interrupts. It does use some internal timers, but I would be surprised if this issue was related to them.

Do you have any additional evidence regarding those DPC spikes? How do you measure/collect them?

Should I be using pre-release versions instead? At the moment WinFsp 2017.1 B3 seems to be available.

There is no need to use the prerelease versions. The prerelease versions contain a .NET layer, but no additional changes for the core WinFsp.

[TechTrooper]

This happens very often when I access a folder:

It seems to be caused by thumbnail caching. Can it be disabled? Uses a lot of bandwidth.
I guess for now it's best to disable thumbnails in folder settings.

[Jdogzz]

@billziss-gh Regarding your earlier problem with Windows and the file name case issue when mounting to a path, I found an interesting loophole when using my Android phone:

1. Mount drive on Windows to a path.
2. Connect to computer on phone through shared Windows folders which contain the path in Step 1 as a subfolder (the app I tried is ES File Explorer, for posterity).
3. In the app, navigate to folder inside mounted drive at the path location specified.

All my files and folders (including subfolders) (so far) are accessible. Just thought it would be an interesting tidbit of information.

[billziss-gh]

@Jdogzz thanks for the data point.

If I understand your scenario correctly, in this case accesses to the underlying user mode file system are done via the SRV2 driver (the Windows networking aka SMB driver) and not directly from the NT kernel. It is possible that SRV2 is more careful at constructing I/O requests to the backing FSD (file system driver) than the Windows kernel is.

Definitely good to know.

[darkneo29]

Quick question everyone. Alot of different version being used. Since this isn't in release version what version of rclone and winfsp should I be using for Windows 10 mounting?

[ghost]

@darkneo29 Ideally the latest beta I would think so that you don't have issues that may already have been fixed.

[billziss-gh]

@darkneo29 for WinFsp just use the latest release available.

[darkneo29]

Got it working but command prompt doesn't go away. If I close it mount closes. I am using & at end. How would this work in background?

[billziss-gh]

The & ampersand does not work on Windows as it does on Unix. The closest equivalent is the start command.

It is possible to run a WinFsp file system as a service (i.e. in the background), but I do not believe that rclone mount supports this at the moment.

[ncw]

This has been working well for some time now, so I'm going to close this issue.