Minikube: Should minikube mount be deprecated?

[idcmp]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Bug Report.

If you go through the auto-closed issues, you will find a lot of people who are unable to use minikube mount to bring a host path into a pod running in minikube. It looks like there's a maximum number of files per directory before some of the moving parts fall over (even with --msize set really high). In some cases they get a specific error message, in others there are simply files missing, in others there's simply a generic Invalid argument returned.

Since it's reasonable to expect people to be using minikube for development, and a very common reason to want to mount a host directory into a pod is to speed up the development cycle when there are lot of files, it stands to reason that the current implementation of minikube mount isn't well-suited for its most common use case.

Please provide the following details:

Environment:

Minikube version : v0.28.2 (tried on 0.28.1 too)

• OS (e.g. from /etc/os-release): macOS 16.7.0 (but others have been on other systems)
• VM Driver I've tried with both virtualbox, hyperkit
• ISO version 0.28.1
• Install tools: brew cask,

What happened:

ls: reading directory '.': Invalid argument

What you expected to happen:

A list of the 1181 sub directories.

How to reproduce it (as minimally and precisely as possible):

cd some_project_that_has_a_huge_node_modules
minikube mount --msize $[250*1024*1024] **$(pwd):/mount-workspace
minikube ssh
cd /mount-workspace/node_modules
ls

Output of minikube logs (if applicable):

No logs are generated at the default log level when I run the above commands. Others have posted huge log files (+20Mb).

Anything else do we need to know:

I realize there's a lot of moving parts here, but I'm stuck (just as others have been) because this doesn't work the way one would expect.

[Jokero]

I have the same problem but even with small number of directories and files. It happens at random moment without any visible reason.
I'm also getting Invalid argument message when trying to do ls. virtualbox is used as VM driver.
This behaviour is very annoying and does not allow to use minikube for development on a daily basis. That's why I have decided to use docker-compose for dev purposes

[idcmp]

I've switched do using the Kubernetes built in to Docker and have found it doesn't have this problem and basically don't use Minikube anymore. This lets me continue to use k8s primitives which is handy.

[Jokero]

Cool. But looks like Docker still supports Kubernetes only on Mac and Windows

[fabiand]

Couldn't this be switched to an NFS over IP based solution?
With ganesha used by minikube (host sided)?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[idcmp]

Closing. This problem is still present.

[balopat]

I'll reopen this. I agree that this is a very important piece of local developer experience. I would love to see this prioritized for the next quarter. @tstromberg what do you think?

[fabiand]

A userspace NFS server like ganesha might also be helpful here - at least
for some platforms.

[tstromberg]

My gut instinct says "no" to deprecating mount, though replacing the underlying technologies with ones which are much more reliable is definitely something I would like to see. Help wanted!

[Jokero]

I've switched from ubuntu to mac and it's awful. If before I was able to develop with minikube somehow (it worked from time to time on ubuntu), now I can't do this due to problems with mounting.

When I execute ls inside minikube machine (minikube ssh), it results in different outputs, although I did it in a row:

$ ls -al /app
total 14
-rw-r--r-- 1 501 20 8196 Feb 27 12:57 .DS_Store
drwxr-xr-x 1 501 20  576 Feb 27 12:48 app-api
drwxr-xr-x 1 501 20  544 Feb 27 12:48 app-service
drwxr-xr-x 1 501 20  896 Feb 27 12:57 app-web-client
drwxr-xr-x 1 501 20  320 Jan 27 21:38 app-words
$
$ ls -al /app
ls: cannot access '/app/.DS_Store': Invalid argument
ls: cannot access '/app/app-words': Invalid argument
total 5
-????????? ? ?   ?    ?            ? .DS_Store
drwxr-xr-x 1 501 20 576 Feb 27 12:48 app-api
drwxr-xr-x 1 501 20 544 Feb 27 12:48 app-service
drwxr-xr-x 1 501 20 896 Feb 27 12:57 app-web-client
d????????? ? ?   ?    ?            ? app-words
$ 
$ ls -al /app
ls: cannot open directory '/app': Invalid argument

I'm using minikube start --mount --mount-string to run cluster.

If the problem caused by tons of mounted files, it will be nice to support multiple --mount-string arguments so there will be a way to specify all needed directories to be mounted and avoid other non-necessary directories (for example, heavy node_modules)

[afbjorklund]

Depending on what you are using it for, you can get a better experience by leaving all the files on the VM and then using sshfs to access only the ones that you need to edit. Unfortunately it's a bit of trouble to set it up with minikube (unlike docker-machine mount), but if you run ssh in verbose mode it shows the flags...

[idcmp]

@tstromberg Considering the initial report and the one from @Jokero, what additional evidence are you seeking?

[tstromberg]

minikube mount should not be deprecated: it should be improved. There are still use cases where it is quite handy.

mount has recently been refactored to support other filesystems. Instead of deprecating mount, we should look into adding support for faster filesystems, such as nfs and virtio-fs. We should default to using the best filesystem for the given environment we are operating in.

[MarkBennett]

minikube mount should not be deprecated: it should be improved. There are still use cases where it is quite handy.

Support this 100% @tstromberg. As a Rails/node contractor I'm always polluting my boxes with tonnes of different node/ruby/python versions plus project deps. Would be great to be able to use a combo of skaffold and minikube (along with new VS Code Remote support) to get a dev environment I can run without installing any deps other than minikube and skaffold.

[idcmp]

Reminder that this issue is _closed_, so if it's relevant, then please re-open it and pin it away from fetja-bot.

That said, it's been nearly a year and people are now generally discouraged from using minikube mount since it has basic functionality issues (some examples above, many other examples in other auto-closed fetja-bot issues), so accepting patches just removing mount seems like a reasonable path forward.

[AmaanC]

Still running into issues like the original Invalid argument upon ls, and random i/o errors for npm install / yarn within mounted directories. Using the docker driver on Linux.
Likely this issue mentioned above:

It looks like there's a maximum number of files per directory before some of the moving parts fall over (even with --msize set really high).

[medyagh]

Still running into issues like the original Invalid argument upon ls, and random i/o errors for npm install / yarn within mounted directories. Using the docker driver on Linux.
Likely this issue mentioned above:

It looks like there's a maximum number of files per directory before some of the moving parts fall over (even with --msize set really high).

@AmaanC @idcmp I would be happy to look at the bugs,

do you mind creating new issues with full logs
such as minikube mount --alsologtostderr

and reproducible steps ?

we have integration test for minikube mount command and we never release without passing their tests.
we could add more integration tests to ensure it convers your case

[fabiand]

Note: with virtiofs landing we now have a better opportunity at least for linux hosts

[afbjorklund]

For docker-machine mount we flipped this around, by using sshfs rather than 9p.
https://docs.docker.com/machine/reference/mount/

That is, files are mounted from the VM to the host - instead of from the host to the VM...
So all the large files remain on the VM, and only the ones you want to edit are transferred.