Vcpkg: Feature Request: allow private repository on vcpkg

[jasjuang]

I have a private repo A that is dependent on some external libraries and private repo B. Currently external libraries are installed in vcpkg and private repo B is installed in the default C:/Program Files (x86). I attempted to write a portfile for private repo B, so everything will be from vcpkg but the problem is when I do vcpkg install repo_B I am expecting a git credential prompt in the terminal, but instead I get the error

Failed. Status: 22;"HTTP response code said error"

Is it possible to not fail directly and prompt for git credentials instead?

[ras0219-msft]

I'm not sure; could you sanitize the portfile for repo_B and post it along with some more details of the situation? (For example, I don't understand why/how you're using git to access C:/Program Files (x86))

[jasjuang]

@ras0219-msft The portfile is shown below,

include(vcpkg_common_functions)

vcpkg_from_github(
    OUT_SOURCE_PATH SOURCE_PATH
    REPO mycompany/repo_B
    REF 1.0
    SHA512 5a7e84ecc5a54320c74776c133bfdbeaf0d4496a7a7fdf2f4ccf89e66b3665a577a370a662ac97a350a2b1f717ce769cb0826057ebb3b13c9c2fee65f20ac7b4
    HEAD_REF master
)

vcpkg_configure_cmake(
    SOURCE_PATH ${SOURCE_PATH}
)

vcpkg_install_cmake()

vcpkg_fixup_cmake_targets(CONFIG_PATH "lib/cmake/repo_B")

vcpkg_copy_pdbs()

file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)
file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/share)

# Put the license file where vcpkg expects it
file(COPY ${SOURCE_PATH}/copying.txt DESTINATION ${CURRENT_PACKAGES_DIR}/share/repo_B/)
file(RENAME ${CURRENT_PACKAGES_DIR}/share/repo_B/copying.txt ${CURRENT_PACKAGES_DIR}/share/repo_B/copyright)

repo_B is private so when vcpkg install repo_B I am expecting a git credentials prompt.

C:/Program Files (x86) is the default install location of repo_B, for example,

git clone https://github.com/mycompany/repo_B
mkdir build && cd build
cmake -A x64 ..
msbuild repo_B.sln
msbuild INSTALL.vcxproj 
# headers, libs, cmake configs of repo_B will be installed to C:/Program Files (x86)

[ras0219-msft]

Ok, I got a "proof of concept" to work:

• Create a PAT on GitHub (under developer settings) with "repo" permissions
• Edit scripts\cmake\vcpkg_download_distfile.cmake and add USERPWD <githubusername>:<PAT> to the parameters for all file(DOWNLOAD) commands

This allows me to download archives from private repos, however it certainly has huge problems (first of which, it will pass your PAT to _all_ websites, not just GitHub).

In general though, I think an interaction-less approach will be better than constantly requiring the user to enter their password.

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub. This should also (hopefully) be very friendly to CI machines, which often have ways to inject credentials through environment variables.

Would that work well for you?

[crackedmind]

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub.

but what about private repos on bitbucket/gitlab ?

[jasjuang]

@ras0219-msft I think it will be great to have both interaction and interaction-less. Interaction-less is great for CI purposes but at the same time there are situations where multiple people are accessing the same computer/server so storing github password through environment variables is not a good idea for those cases. In this case credential prompt would be more suitable.

[ras0219-msft]

In the shared computer case, the idea would be that you could set this environment variable just for the shell you're using:

$env:VCPKG_GITHUB_USERPWD = "user:passwd"
vcpkg stuff

Once the shell closes, your password is no longer stored :) Though, it wouldn't work very well while giving a presentation.

@crackedmind You're definitely right that this only solves GitHub for now! I think a more general solution can be evolved out of this, but I want to make sure we solve @jasjuang's problem in the short term.

Some possible evolutions would be:

• More environment variables for well known services like the ones you mentioned (this has obvious limits, but still does solve real problems for real people in a convenient way)
• A user-wide config file that provides a mapping from hosts to user+password, kind of like how git's credential helpers work. We might even try querying git's credential helper directly, but I think this will take more time to investigate if it is reasonable.

A big problem with the general case is that every service can have completely different ways to expect authentication (Maybe they require OAuth tokens in the URL, maybe they use HTTP basic auth, maybe they want SSL client certs, etc). Therefore, we'll definitely need to collect lots of specific individual use cases!

[jasjuang]

@ras0219-msft Yes that makes sense, so I guess as long as the error message is set your environment variable VCPKG_GITHUB_USERPWD instead of Failed. Status: 22;"HTTP response code said error" that will be a good first version.

[MikeGitb]

@ras0219-msft: Git already has a credential manager, I don't understand why it is necessesary to reinvent it. Can't we just use it? And I'm not sure how I feel about storing my password as plaintext.

[jasjuang]

Just in case anyone is interested. I figured out a hack that will make it work temporarily. I downloaded the release from github manually and place it in the downloads folder, so when I do vcpkg install myprivaterepo it will actually use the "cached" version and proceed to install like usual.

[donny-dont]

@ras0219-msft I'm hitting this as well but with vcpkg_download_distfile. Would it be possible to extend it to include headers? That way we could set the authorization header for a download.

[soroshsabz]

ITNOA

I think a good documentation in wiki for this scenario is a very good choice and needed.

[katusk]

@jasjuang Your hack works for me as well, thanks. Have you tried vcpkg_from_git instead of vcpkg_from_github with an "https://" link? I had good experience with vcpkg_from_git instead of vcpkg_from_bitbucket lately in case of a private Bitbucket repository. I had to provide my credentials upon vcpkg install though (interaction).

Unfortunately, vcpkg_from_git seems to require a "https://" link, see CMake script below. I do not fully get the comment below the check there. @ras0219-msft Can you please provide more detail on this?

• https://github.com/microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_from_git.cmake#L56

Anyway, if I modify that message(FATAL_ERROR ...) to message(WARNING ...), and I pass an "ssh://" link to vcpkg_from_git, then it works with that too for my private Bitbucket repository. If you have your ssh keys properly set up, you do not even need to provide any credentials in this case (interaction-less). We need this for CI purposes.

NOTE: I used 40 characters of an SHA for my purposes, like so (WARNING: potentially bad command invocation; see above):

# WARNING. This one is a modified version of the vcpkg_from_git command.
vcpkg_from_git(
    OUT_SOURCE_PATH SOURCE_PATH
    URL ssh://bitbucket.org/org/proj.git
    REF d6ab0b626ea949c18d3167e6451df3b3961b848a
)

[daydream123]

My thoughts are therefore to add it as an environment variable that you would set (VCPKG_GITHUB_USERPWD?) and it would only apply when pulling from GitHub.

but what about private repos on bitbucket/gitlab ?

I also encountered this problem，seems no place to input my username and password, did you find any other solution? this problem stoped me to use vcpkg in my company projects.

[klalumiere]

It took me a while to figure this one out, even though the information is on the Internet (like always I guess) here. What we want to do is to "inject" our git credentials inside the script.

Here's a TL;DR/particular implementation that I use on Jenkins:
1) Create a script name e.g. .git-askpass.sh that contains

#!/usr/bin/env bash
echo username=$GIT_USERNAME
echo password=$GIT_PASSWORD

2) Run sh "git config --global credential.helper \"${pwd()}/.git-askpass.sh\""
4) Wrap the script that calls e.g. git clone https://github.com/myorg/privaterepo in the block

withCredentials([usernamePassword(credentialsId: 'github',
        usernameVariable: 'GIT_USERNAME',
        passwordVariable: 'GIT_PASSWORD')])
{}

e.g.

withCredentials([usernamePassword(credentialsId: 'github',
    usernameVariable: 'GIT_USERNAME',
    passwordVariable: 'GIT_PASSWORD')])
{
    sh "./scriptThatCallsGit.sh"
}

Of course this can be adapted for Windows with e.g. a PowerShell script.

From what I understand, I'd argue that this approach solves the need raised by this issue. The approach also provides a clear separation of concern between authentication and content retrieval, which makes vcpkg's code easier to maintain.