Roslyn: Can no longer git clean solution directory with VS open

[davkean]

Attempting to git clean my repo while Visual Studio is open now complains about files being locked. This looks like a regression in recent minor updates:

E:\project-system2>git clean -dxf
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/db.lock' failed. Should I try again? (y/n) y
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/db.lock' failed. Should I try again? (y/n) y
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/db.lock' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/Server/sqlite3/db.lock: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/storage.ide' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/Server/sqlite3/storage.ide: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/storage.ide-shm' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/Server/sqlite3/storage.ide-shm: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/Server/sqlite3/storage.ide-wal' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/Server/sqlite3/storage.ide-wal: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/sqlite3/db.lock' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/sqlite3/db.lock: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/sqlite3/storage.ide' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/sqlite3/storage.ide: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/sqlite3/storage.ide-shm' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/sqlite3/storage.ide-shm: Invalid argument
Unlink of file '.vs/ProjectSystem/v15/sqlite3/storage.ide-wal' failed. Should I try again? (y/n) n
warning: failed to remove .vs/ProjectSystem/v15/sqlite3/storage.ide-wal: Invalid argument

[CyrusNajmabadi]

@sharwell Perhaps the change to exclusive access in sqlite?

IMO, this is acceptable behavior. This is a folder being used by the IDE while it is running in VS. It is not expected that this would be removed out from under it while running.

--

As a workaround, is it possible to add .vs to your gitignore so that git doesn't try to do anything with it when you're doing cleans and whatnot?

[CyrusNajmabadi]

Note: you are passing -x which is telling git to remove the folder, even if you already told it to ignore it. That's likely always going to be painful as long as roslyn continues to lock this file. I'm not sure if there's any way to get the benefits of the recent sqlite work, while also making the file unlocked unfortunately.

[davkean]

The IDE has always handled that it was removed underneath it. Remember, for this to be added to .gitignore, you basically have to change every single repo in the world.

[CyrusNajmabadi]

The IDE has always handled that it was removed underneath it.

I don't think that's the case. There were certainly folders in VS2005 and whatnot that couldn't be removed while the IDE was running. It's likely at the switch to Roslyn this became possible.

Or, in other words, the ability to remove this folder was never an intentional design point of the IDE. It may have fallen out because of certain design decisions, but it was never a guarantee.

--

Now, what to do about this at this point remains to be seen. There are several options:

1. First, find out if it's possible to get the benefits that we got recently from the sqlite changes, while also removing the locking restriction. I have no idea if this is possible. But @sharwell and @onyxmaster may know of a way.
2. Consider the degradation in git behavior here worse than the gain in Roslyn performance. Roll back the changes made.
3. Consider the degradation in git behavior acceptable given the gain. Keep things as is.

Remember, for this to be added to .gitignore, you basically have to change every single repo in the world.

That honestly doesn't seem like a large problem to me. The users that hit this have to be in the venn diagram of:

A. Use git.
B. Use Roslyn.
C. Want to git clean while Roslyn is running.

Those users will definitely exist. But there will also be many users who won't do any of that. Furthermore, there is a workaround which is fairly easy to apply. Given that, i would lean toward '3' if '1' is not possible. I think the benefit to performance is worth the inconvenience of people doing A, B, and C needing to edit their .gitignore file. Just my 2c on this :)

[davkean]

Which folders did VS2005 create in the source repo?

[tmat]

Perhaps SQLite can be updated to open the files with delete sharing. Depending on what APIs git uses on Windows this might make it possible to delete the files while SQLite is still able to read/writer the content.

[CyrusNajmabadi]

I believe we kept the .projdata file under some directory there. But, as you can imagine, it's been a long time since having to work on that, so i don't remember the exact directory :)

[onyxmaster]

Maybe .vs folder can be moved to temp or appdata? Pollution and matching (identification) would be a problem though.

[sharwell]

:memo: Sounds related to #19768

[sharwell]

:memo: This issue is related to #19768, but differs specifically in that it requests the git clean operation work without closing Visual Studio.

My understanding is implementing this feature would require one of the following:

1. Relocate the SQLite cache so it's no longer part of the repository.
2. Provide the ability to explicitly disable caching.

[CyrusNajmabadi]

Perhaps SQLite can be updated to open the files with delete sharing.

I did some google spelunking, and checked out the parts of the sqlite API that seemed relevant. I couldn't find a way to not have the lock, but also get the benefits of the recent changes.

Maybe .vs folder can be moved to temp or appdata?

It seems like there is an existing solution in the git tooling space already that is intended to address this sort of thing. Specifically, adding .vs to the .gitignore file. I've thought about this more, and even looked at our repo and a bunch of others. .vs is normally already in the .gitignore file as people obviously don't want to check in a huge sqlite DB every time they do work in their repo :) Given that, the issue appears to be the regression solely in the "git clean" scenario but only when using -x. My recommendation would be that in these cases, -x not be used as that will avoid the problem entirely.

Pros: things should just work.
Cons: muscle memory def affected for people who are just used to always writing 'git clean -xdf'

I'm definitely sympathetic to how annoying changes in muscle memory can be. But i also think that needs to be weighed against how painful the workaround is. in this case, the workaround seems to be actually pretty simple. So it doesn't feel like things are htat bad to me.

[sharwell]

@CyrusNajmabadi You can also use this:

git clean -dxf -e .vs/

It ignores the contents of .gitignore, but explicitly adds back .vs/.

[CyrusNajmabadi]

Yup! That way seemed less nice though as it was extra stuff to write :) But i guess it would be hte most correct thing to do in case you really did want git to clean all the other stuff you normally told it to ignore.

[CyrusNajmabadi]

Note: looking at the historical situation, i'm def more sympathetic here. I think i got rid of .projdata back in vs2008 (since, horrifically, having that home-grown database actually caused worse memory usage, worse CPU usage, and all sorts of corruption issues (who would have thunk it)). So it's been nearly a decade without an intentional file lock caused by the C# LS.

There have, of course, been file locks that could occur all the time outside of the .vs folder. For example, on numerous occasions i've had a borked "git pull" happen due to hitting source files that are locked while the IDE was reading them. And cleaning has definitely failed for me in the past due to being unable to remove output dirs due to metadata being locked. But those were more transient issues, where the .vs issue is now an issue for the lifetime of the process. :-/

[Pilchie]

Can we distinguish between "Close VS" and "Close Solution"? If I have VS still running, but have closed the solution, can I clean the .vs directory?

[adamralph]

@Pilchie if the solution is closed then yes, it appears .vs can be cleaned. However, I don't think that alleviates the problem sufficiently. The time to reload a solution can also be significant, even more so than VS start up time. The Roslyn solution is a great example! 😉

[Pilchie]

@adamralph Pretty much my entire job right now is working on improving solution load time. I'm pretty familiar with the problem 😜

Was just checking to see whether the implementation was at least working as designed...

[Pilchie]

BTW - 15.5 Preview includes some significant changes for non-SDK .NET Framework projects, and some improvements for SDK (.NET Core/.NET Standard) projects.

Would love to get peoples impressions.

We've got more underway for 15.6, especially for SDK projects.

[bitbonk]

git clean is a simple and essential tool when switching branches for any kind of project. It allows us to remove all build artifacts and temporary files generated in the current branch before starting to build and work with another branch. It is a bit annoying that this can't be done when VS is open.

[CodingDinosaur]

Seems like this fell off the radar, but this has been coming up for me a lot recently since about 15.7.3.

The issue for me is that this happens _after closing_ the _solution_. I don't feel like VS should be locking files in my repo when I'm no longer working on it. I routinely close one solution, then run a clean on it before I move on to another solution (I like to keep my local repos tidy), but this issue prevents me from doing so unless I close VS entirely and re-open it every time I want to load a different solution.

[jmarolf]

Relocate the SQLite cache so it's no longer part of the repository.

This is my vote for a solution here.

[jinujoseph]

also reported here

[yolpsoftware]

In my case, it seems that VS2017 locks some files in the sqlite3 folder even when the solution is closed. This means I have to close VS to be able to clean the working directory.

[ericsampson]

Is this on the docket to get investigated/fixed @sharwell ? I can confirm that closing the solution is not enough, I have to close VS and that's a real pain. Not everyone can switch to VS2019 although we might like to :) Thanks!

[CyrusNajmabadi]

I think this may have just been fixed by @heejaechang and @jasonmalinowski . Tagging both to confirm. @heejaechang Didn't you just make a change so that if hte solution is changed/closed that the DB lock will be released?

[jasonmalinowski]

The bug @heejaechang fixed was only a recent regression in the last month, and only with VS2019 builds. Whatever this full thread is about isn't that.

[CyrusNajmabadi]

Ah ok. Bummer. This seems to be a similar category though. Are we not listening to solution-closed events and releasing the last lock on the DB? @heejaechang does that seem like something that could be happening? Thanks!

[adamralph]

Would it not be simpler to move the DB outside the repo folder? E.g. somewhere in the temp directory? Then the solution would not even have to be closed. Having to close the solution to do a git clean is also awkward.

[CyrusNajmabadi]

Having to close the solution to do a git clean is also awkward.

Why is git clean trying to delete the database though? That's the part that seems weird to me. Why not just add this folder to .gitignore?

Would it not be simpler to move the DB outside the repo folder?

Adding this folder to .gitignore seems, by far, to be the simplest solution.

[adamralph]

I already have the folder in .gitignore. That doesn't stop git from cleaning it.

[adamralph]

To be clear, what we want is to run git clean -fxd. We _want_ to remove stuff which is in .gitignore. E.g. bin, obj, etc.

.vs comes along for the ride in that scenario.

[CyrusNajmabadi]

To be clear, what we want is to run git clean -fxd

Ah, i see. So you are explicitly overriding .gitignore.

I'm not opposed to hte directly being elsewhere. Though i'm concerned about later issues being reported asking why VS is spewing stuff into different directories instead of the common solution dir.

[ericsampson]

If it matters, I wasn't running git clean manually, just using the VS right-click git 'reset' option.

[adamralph]

Ah, i see. So you are explicitly overriding .gitignore.

That's correct. Running git clean -fxd is a commonly used method of resetting back to a pristine working directory.

[yolpsoftware]

I think it makes sense to leave the .vs folder in the working directory. You just need to clean up the file locks after the solution is closed.

[adamralph]

Once upon a time we could git clean -fxd with the solution open. It's a shame that's regressed.

[CyrusNajmabadi]

with the solution open. It's a shame that's regressed.

I don't believe that was ever an intentionally supported scenario. Indeed, you could definitely run into issues with that even prior to the DB change we made. For example, if the IDE was reading information out of built dlls you'd hit this problem. The main issue is that we have an index that is critical for the performance of many of our operations, and that you are running an operation that is trying to delete while in use.

If the solution is closed, i agree we should release the lock here. But while it is open, it just feels like a bad idea anyways to be ripping out all the live information out from under VS where it is currently being used.

[ericsampson]

@CyrusNajmabadi , I encounter this scenario just using the VS right-click git 'reset > hard' menu item. That seems like it should be supported scenario with the solution open?

[CyrusNajmabadi]

git 'reset > hard' menu item. That seems like it should be supported scenario with the solution open?

That likely is an issue that should go to whatever team owns git integration with VS. I don't think Roslyn is involved with that operation.

Note: that issue affects more than just the .vs dir. I just tried cleaning and i got failures even due to thins like file-locks on normal text files (presumably because some BG process is reading those files).

I get that there is a def ask to be able to rip out everythin from underneath VS while it's running. But it's not something that Roslyn was really designed for. So it's a far larger ask.

--

To be clear, i'm not saying this can't be done. Or that it shuldn't be done. Merely that it isn't nontrivial to do as it's never been part of the design to ensure that locks are not held so that other tasks can destroy files while a solution is opened.

If the solution is closed, i def think we shouldn't hold onto stuff. But making this work reliably with the solution open will likely require some team coordination.

[ericsampson]

@CyrusNajmabadi thanks I'll file another issue with that team, cheers.

[airbreather]

I'm now able to git clean -xdfq after closing my solution in VS2019 RTM, even in at least one case where .vs\SolutionName\v16\Server\sqlite3 folder still had contents (specifically, db.lock and storage.ide were not removed when the solution was closed, though some other files were).

Was this stealth-fixed?

[jasonmalinowski]

@airbreather: it's possible it the clean-while-closed got fixed since we did fix some other lifetime issues. Clean-while-open still will be a problem though.

[airbreather]

Clean-while-open still will be a problem though.

Oh, that was part of this same issue? OK. Personally, I'm satisfied with just being able to clean while the VS application is still open, even if I still need to close the VS solution, but I understand now that there are others on this thread who need more.

Sorry if my comment got anybody else's hopes up 💔.

[jasonmalinowski]

Well, that was what the title of the issue was. It got a bit muddied after a bit...

[frankfuu]

This annoyed me enough so I wrote a one liner to kill the Roslyn process.

Get-Process "*ServiceHub.RoslynCodeAnalysisService32*" | Stop-Process

You might get an error pop up on Visual Studio bitching about Roslyn but at least you can just dismiss it rather than closing, praying and re-opening it again.

[kenlyon]

I had logged a similar bug which was closed as a duplicate of this.

I see from the discussion that there are various levels of "deletability" being requested by different individuals. My main thought on the matter is that Roslyn should release the locks on files within a solution after it is closed. It should have no further need to lock them as there's nothing more happening with regard to that solution.

I think being able to delete files while the solution is still open would be nice, but I can also appreciate the technical challenges associated with that.

[adamralph]

🎉 btw - what is the "P" in 16.5.P2?

[CyrusNajmabadi]

@adamralph "P" is "preview". So this will be available in 16.5-Preview2 and onwards

[adamralph]

@CyrusNajmabadi thanks very much. 👍

Because it's dot-separated, it looks kind of like a patch number and is a bit confusing. Perhaps it would be better to have a dash-separated pre-release identifier, similar to SemVer: 16.5-P2, or for completeness: 16.5.0-P2.

[mkamoski]

FYI, the same thing is (still) happening here and now, Visual Studio 16.4.3 in 2020. Ug. -- Mark Kamoski

[jmarolf]

@mkamoski can you file a new issue on Visual Studio developer community? It may be a component from a different team that is holding on to something.

[CyrusNajmabadi]

@jmarolf Fix is not in 16.4.3. It's only in 16.5 and upwards. Thanks!