Vcpkg: Clear intermediate files after successful packaging
After building every (currently 90, wow!) port for static and dynamic, the combined size of installed, packages, and buildtrees is over 30GB. 23GB of this is because of the buildtrees, which are mostly intermediate files (but also sources).
We should keep sources around for optimal debugging, but we can easily shave off >70% by removing the intermediate directories (typically <triplet>-rel and <triplet>-dbg).
ras0219-msft
All 11 comments
Seems like running this type of command at the end after vcpkg_copy_pdbs.cmake or similiar finalizing would do the trick.
adam4813
on 3 Nov 2016
For the general user none of the buildtree files would be needed. And for port debugging, you'd want to have access to the files generated by Ninja (in the intermediate directories) too.
Suggestion
- By default, after every successful build remove everything, except for the .log-files, from
.\buildtrees\PORT\ - Adding an optional flag to the install command for port development. Resulting in the current default behaviour and keeping all files used deploying the library.
vcpkg install --keep-all <pkg>
- A second flag. To keep the extraction folder after successful installation. This would be usable for frequent rebuilds and while installing multiple configurations.
vcpkg install --keep-source <pkg>
This leaves the packages and download caches. All the rest can be regenerated when needed without external requirements.
Removing the triplet directories could be done in vcpkg_copy_pdbs.cmake.
This would be enough for most projects since header only libraries would not create these directories.
But that is assuming there won't be static only versions of libraries.
And that port implementers always execute this function even when building a static library.
Ideally such operations should not depend on the port-file implementation. The focus there is to get the library working. Clean-up should happen by default at closing.
Farwaykorse
on 14 Mar 2017
Agreed, we need something like this.
genbtc
on 25 Dec 2017
Agreed as well, feel pain when moving whole vcpkg directory ... so many files!
cmpute
on 13 Feb 2018
Agreed, vcpkg is too large after install some package.
lua9520
on 11 Apr 2018
What are the consequences of manually erasing the contents of buildtrees? I guess debugging operations no longer have sources to rely on, but other than that is there anything I should be aware off?
SergioRAgostinho
on 3 Sep 2018
Installing LLVM in just one config (x64-windows) has a 60GB+ buildtrees folder. I had to clear so many things to make the installer finish.
MathiasMagnus
on 13 Sep 2018
@SergioRAgostinho in my experience there are no issues with removing (most of) the content from the downloads and buildtrees folders.
Just don't delete the content of downloads/tools. Although I expect these will be restored at first use.
When updating already installed ports and adding new ports nothing changes.
Installing an additional triplet for a port or reinstalling will result in re-downloading and extracting of the source.
Deleting the archives from vcpkg/downloads, results in re-downloading if retrieved as an archive.
When the content of vcpkg/buildtrees was deleted, then the source will be extracted or downloaded from its repository.
It might be worth it for anyone to at least clean these folders up.
The downloads folder is collecting archives of old source versions.
Similar, buildtrees/port-name/src retains the source from all previous installs.
Farwaykorse
on 21 Sep 2018
Just as a sidenote, due to having so many duplicate copies during the build process, while adding _features_ to the LLVM package, adding just AMDGPU and NVPTX (2 extra targets I need) I cannot finish the install, because 100 GBs is not enough. This is crazy. I'll have to relocate my Vcpkg install to my HDD, because a 256 GB SSD is not enough for Windows and dev tools only (no media, no games).
MathiasMagnus
on 22 Sep 2018
I thought I'd share a workaround for the excessive storage usage. In case you have 2 drives in your system (I have an SSD where my work, including source code resides and an HDD where media resides) you could create symlinks for the intermediate directories. In my case it slowed down builds (HDD vs. SSD) but at least I could build LLVM. For example in PowerShell
Remove-Item -Recurse C:\Users\mnagy\Source\Repos\vcpkg\downloads
New-Item -Type Directory D:\vcpkg-tmp\downloads
New-Item -Type SymbolicLink -Path C:\Users\mnagy\Source\Repos\vcpkg -Name downloads -Value D:\vcpkg-tmp\downloads\
And you should do the same for the buildtrees and packages directories. This way resulting binaries for further builds are fast and only those reside on your primary drive.
MathiasMagnus
on 26 Jan 2019
Any update on this?
MikeGitb
on 2 Mar 2019
Related issues
grandprixgp
路
3Comments
ghost
路
3Comments
madkoala
路
3Comments
oahzuw
路
3Comments
cskrisz
路
3Comments
Most helpful comment
For the general user none of the buildtree files would be needed. And for port debugging, you'd want to have access to the files generated by Ninja (in the intermediate directories) too.
Suggestion
.\buildtrees\PORT\This leaves the packages and download caches. All the rest can be regenerated when needed without external requirements.
Removing the triplet directories could be done in vcpkg_copy_pdbs.cmake.
This would be enough for most projects since header only libraries would not create these directories.
But that is assuming there won't be static only versions of libraries.
And that port implementers always execute this function even when building a static library.
Ideally such operations should not depend on the port-file implementation. The focus there is to get the library working. Clean-up should happen by default at closing.