vcpkg fails to download files using --x-use-aria2 when destination folder/directory does not previously exist

[curoky]

when var vcpkg_download_distfile_FILENAME contains '/', the following path will not exist.

1. download-${vcpkg_download_distfile_FILENAME}-detailed.log
2. download-${vcpkg_download_distfile_FILENAME}-out.log
3. download-${vcpkg_download_distfile_FILENAME}-err.log

https://github.com/microsoft/vcpkg/blob/ef60f1d9b97fab6fcc61651936196866bb6553d0/scripts/cmake/vcpkg_download_distfile.cmake#L126-L135

e.g

https://github.com/microsoft/vcpkg/blob/ef60f1d9b97fab6fcc61651936196866bb6553d0/ports/nlohmann-json/portfile.cmake#L6-L20

Maybe need some string replacements to temporarily circumvent this problem.

[PhoebeHui]

@curoky, thanks for reporting this issue!

I think the the file name should not contain the '/' here, I have updated nlohmann-json to use vcpkg_from_github.

[curoky]

@PhoebeHui Great! That sloved my problem.
But there are still some incorrect file names, maybe need doing some change in vcpkg_download_distfile.

$ cd vcpkg/ports
$ ag 'FILENAME\s*"[\w\.$-{}]*/'
wordnet/portfile.cmake
27:        FILENAME "wordnet-cache/wn3.1.dict.tar.gz"

nlohmann-json/portfile.cmake
9:        FILENAME "nlohmann-json-v${SOURCE_VERSION}/${SUBPATH}"

opencv3/portfile.cmake
72:    FILENAME "opencv-cache/tiny_dnn/adb1c512e09ca2c7a6faef36f9c53e59-v1.0.0a3.tar.gz"
100:    FILENAME "opencv-cache/data/7505c44ca4eb54b4ab1e4777cb96ac05-face_landmark_model.dat"
144:        FILENAME "opencv-cache/ippicv/fe6b2bb75ae0e3f19ad3ae1a31dfa4a2-ippicv_2019_mac_intel64_general_20180723.tgz"
155:        FILENAME "opencv-cache/ippicv/c0bd78adb4156bbf552c1dfe90599607-ippicv_2019_lnx_intel64_general_20180723.tgz"
161:        FILENAME "opencv-cache/ippicv/4f38432c30bfd6423164b7a24bbc98a0-ippicv_2019_lnx_ia32_general_20180723.tgz"
172:        FILENAME "opencv-cache/ippicv/1d222685246896fe089f88b8858e4b2f-ippicv_2019_win_intel64_20180723_general.zip"
178:        FILENAME "opencv-cache/ippicv/0157251a2eb9cd63a3ebc7eed0f3e59e-ippicv_2019_win_ia32_20180723_general.zip"

opencv4/portfile.cmake
76:    FILENAME "opencv-cache/tiny_dnn/adb1c512e09ca2c7a6faef36f9c53e59-v1.0.0a3.tar.gz"
84:      FILENAME "opencv-cache/nvidia_optical_flow/ca5acedee6cb45d0ec610a6732de5c15-79c6cee80a2df9a196f20afd6b598a9810964c32.zip"
112:    FILENAME "opencv-cache/xfeatures2d/boostdesc/0ea90e7a8f3f7876d450e4149c97c74f-boostdesc_bgm.i"
117:    FILENAME "opencv-cache/xfeatures2d/boostdesc/232c966b13651bd0e46a1497b0852191-boostdesc_bgm_bi.i"
122:    FILENAME "opencv-cache/xfeatures2d/boostdesc/324426a24fa56ad9c5b8e3e0b3e5303e-boostdesc_bgm_hd.i"
127:    FILENAME "opencv-cache/xfeatures2d/boostdesc/202e1b3e9fec871b04da31f7f016679f-boostdesc_binboost_064.i"
132:    FILENAME "opencv-cache/xfeatures2d/boostdesc/98ea99d399965c03d555cef3ea502a0b-boostdesc_binboost_128.i"
137:    FILENAME "opencv-cache/xfeatures2d/boostdesc/e6dcfa9f647779eb1ce446a8d759b6ea-boostdesc_binboost_256.i"
142:    FILENAME "opencv-cache/xfeatures2d/boostdesc/0ae0675534aa318d9668f2a179c2a052-boostdesc_lbgm.i"
147:    FILENAME "opencv-cache/xfeatures2d/vgg/e8d0dcd54d1bcfdc29203d011a797179-vgg_generated_48.i"
152:    FILENAME "opencv-cache/xfeatures2d/vgg/7126a5d9a8884ebca5aea5d63d677225-vgg_generated_64.i"
157:    FILENAME "opencv-cache/xfeatures2d/vgg/7cd47228edec52b6d82f46511af325c5-vgg_generated_80.i"
162:    FILENAME "opencv-cache/xfeatures2d/vgg/151805e03568c9f490a5e3a872777b75-vgg_generated_120.i"
167:    FILENAME "opencv-cache/data/7505c44ca4eb54b4ab1e4777cb96ac05-face_landmark_model.dat"
177:        FILENAME "opencv-cache/ippicv/fe6b2bb75ae0e3f19ad3ae1a31dfa4a2-ippicv_2019_mac_intel64_general_20180723.tgz"
188:        FILENAME "opencv-cache/ippicv/c0bd78adb4156bbf552c1dfe90599607-ippicv_2019_lnx_intel64_general_20180723.tgz"
194:        FILENAME "opencv-cache/ippicv/4f38432c30bfd6423164b7a24bbc98a0-ippicv_2019_lnx_ia32_general_20180723.tgz"
205:        FILENAME "opencv-cache/ippicv/1d222685246896fe089f88b8858e4b2f-ippicv_2019_win_intel64_20180723_general.zip"
211:        FILENAME "opencv-cache/ippicv/0157251a2eb9cd63a3ebc7eed0f3e59e-ippicv_2019_win_ia32_20180723_general.zip"

darknet/portfile.cmake
60:    FILENAME "darknet-cache/yolov3.weights"
65:    FILENAME "darknet-cache/yolov2.weights"
70:    FILENAME "darknet-cache/yolov3-tiny.weights"
75:    FILENAME "darknet-cache/yolov2-tiny.weights"
83:    FILENAME "darknet-cache/darknet53.conv.74"
88:    FILENAME "darknet-cache/darknet19_448.conv.23"

[ghost]

• It would be helpful to explain exactly what type of problem being encountered by posting the exact output from the screen and logs from the port giving the problem?

• On a new vcpkg instance vcpkg will create the directories/folders/files within the vcpkg instance provided it has permissions to do so.

• The userid running the vcpkg scripts MUST have FULL control to the directories/folders and files at and below the _vcpkg-root_

NOTE:

• The portfile.cmake assumes the base folder/directory of _vcpkg-root_/downloads/ (unless the default vcpkg downloads location has been changed)
• The script then appends the appropriate directory/folder/filename to the base to put the file in the correct location.
• The forward slash is the correct folder/directory separator to use on windows, linux or macOS from within cmake scripts (unless the default has been changed).
• If the location does not exist, cmake will create it (unless the userid running the script does not have permissions to create new directories and/or files).

A screenshot of my _vcpkg-root_\downloads directory/folder - created by the portfile

[curoky]

@timautry Yeah, in most cases, the parent directory will be created correctly. But the function _execute_process which vcpkg_download_distfile used don't do it on follow args.

• -l
• OUTPUT_FILE
• ERROR_FILE

https://github.com/microsoft/vcpkg/blob/ef60f1d9b97fab6fcc61651936196866bb6553d0/scripts/cmake/vcpkg_download_distfile.cmake#L126-L135

In fact, cmake thinks these args are files and just open it with write mode, not create the parent directory.

https://github.com/Kitware/CMake/blob/facadf6c309a167964d43b7efe481a78ef53cc09/Source/kwsys/ProcessUNIX.c#L868-L874

https://github.com/Kitware/CMake/blob/facadf6c309a167964d43b7efe481a78ef53cc09/Source/kwsys/ProcessUNIX.c#L1916-L1940

[curoky]

Maybe in windows, _execute_process will create parent directory, I am not sure, because they have different implementations.

https://github.com/Kitware/CMake/blob/master/Source/kwsys/ProcessWin32.c#L1804-L1829

[ghost]

@PhoebeHui See additional comments, there is an issue with vcpkg install _PORT_ --x-use-aria2

@curoky

• Are you running vcpkg using the option to download with ARIA2 instead of the default methods?
• I ask as the code snippets being posted appear to be dealing with the option to download using ARIA2 by using the --x-use-aria2 on the vcpkg install command?

If ARIA2 is not being used, then the code starting at line 154 through 172 deals with downloading a file using vcpkg default process ${download_file_path_part}:
https://github.com/microsoft/vcpkg/blob/ef60f1d9b97fab6fcc61651936196866bb6553d0/scripts/cmake/vcpkg_download_distfile.cmake#L154-L172

[curoky]

@timautry Yeah, I use --x-use-aria2 to install package.

[ghost]

@PhoebeHui

To follow-up, I tested the download of wordnet[core,dbfiles] with and without using the --x-use-aria2 vcpkg install options, on a clean, just cloned and fresh compiled vcpkg, here is the result:

1. using --x-use-arias2 and as stated by @curoky, as follows:

   • the 3 log files are created but are subsequently deleted before the install process completes. Maybe the process needs something to prevent it from deleting those files if an error does occur, as I watched the downloads folder and saw the files created then deleted.
   • the download file wn3.1.dict.tar.gz fails because _vcpkg-root_/downloads/wordnet-cache does not exist.
   • Attached is the stdout-x64-windows.log named stdout-x64-windows-using-aria2.log
     stdout-x64-windows-using-aria2.log

2. using vcpkg normal download method as follows:

   • Completes as expected.
   • Attached is stdout-x64-windows.log named stdout-x64-windows-normal-download.log
     stdout-x64-windows-normal-download.log

3. This issue may:

   • need label vcpkg bug
   • a title of vcpkg fails to download files using aria2 when destination folder/directory does not previously exist or something to that effect...
   • modifications as follows:
     
     
     
     • create destination folders/directories as needed
     
     
     • not to delete the log files if an download error occurs
       
       
       
       
       
       • _additional-download-file-name_-out.log
       
       
       
       • _additional-download-file-name_-err.log
       
       
       
       • _additional-download-file-name_-detailed.log
       
       
       
     
     

4. I would say it is highly probable all the ports previously mentioned by @curoky will fail when being downloaded via aria2 when the destination _cache_ folder/directory does not exist in _vcpkg-root_/downloads

5. @curoky the issue could have been identified and work started on a bug fix had the following been included with the original posting (note: very few are using the aria2 download option):

   • the _vcpkg-root_/buildtrees/_port-name_/stdout-_triplet-name_.log
   • the exact ./vcpkg install _port-name_ _options_ entered on the command line

[PhoebeHui]

@timautry, very appreciate for helping diagnose this issue!

Confirmed, it should be a vcpkg bug.
PS E:\vcpkg\clean\vcpkg> ./vcpkg install nlohmann-json --x-use-aria2
Computing installation plan...
The following packages will be built and installed:
nlohmann-json[core]:x86-windows
Starting package 1/1: nlohmann-json:x86-windows
Building package nlohmann-json[core]:x86-windows...
A suitable version of powershell-core was not found (required v6.2.1). Downloading portable powershell-core v6.2.1...
Downloading powershell-core...
https://github.com/PowerShell/PowerShell/releases/download/v6.2.1/PowerShell-6.2.1-win-x86.zip -> E:\vcpkg\clean\vcpkg\downloads\PowerShell-6.2.1-win-x86.zip
Extracting powershell-core...
A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0...
Downloading 7zip...
https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> E:\vcpkg\clean\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg
Extracting 7zip...
A suitable version of nuget was not found (required v4.6.2). Downloading portable nuget v4.6.2...
Downloading nuget...
https://dist.nuget.org/win-x86-commandline/v4.6.2/nuget.exe -> E:\vcpkg\clean\vcpkg\downloads\nuget.exe
A suitable version of git was not found (required v2.21.0). Downloading portable git v2.21.0...
Downloading git...
https://github.com/git-for-windows/git/releases/download/v2.21.0.windows.1/PortableGit-2.21.0-32-bit.7z.exe -> E:\vcpkg\clean\vcpkg\downloads\PortableGit-2.21.0-32-bit.7z.exe
Extracting git...
-- Downloading https://github.com/aria2/aria2/releases/download/release-1.34.0/aria2-1.34.0-win-32bit-build1.zip...
-- Downloading nlohmann-json-v3.7.3/CMakeLists.txt...
-- Downloading nlohmann-json-v3.7.3/CMakeLists.txt... Failed.
Exit Code: The system cannot find the path specified
See logs for more information:
E:/VCPKG/clean/vcpkg/downloads/download-nlohmann-json-v3.7.3/CMakeLists.txt-out.log
E:/VCPKG/clean/vcpkg/downloads/download-nlohmann-json-v3.7.3/CMakeLists.txt-err.log
E:/VCPKG/clean/vcpkg/downloads/download-nlohmann-json-v3.7.3/CMakeLists.txt-detailed.log

CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:175 (message):

  Failed to download file.
  If you use a proxy, please set the HTTPS_PROXY and HTTP_PROXY environment
  variables to "https://user:password@your-proxy-ip-address:port/".
  Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
ports/nlohmann-json/portfile.cmake:7 (vcpkg_download_distfile)
ports/nlohmann-json/portfile.cmake:16 (download_src)
scripts/ports.cmake:90 (include)

Error: Building package nlohmann-json:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then

[ghost]

You are welcome @PhoebeHui. Thanks for all the hard work everyone is doing :)

[JackBoosY]

I think my PR #10287 should fixed this issue.
Any one can confirm this?

[PhoebeHui]

@JackBoosY, the PR doesn't fix this issue, it should be a different issue that due to --x-use-aria2 used.