Home: Installing signed package in offline environment

[pascalberger]

I'm in an offline environment with an in-house package server (using ProGet). If I try to install a signed package it takes 4 minutes until installation succeeds.

How can I install signed packages in an offline environment without access to nuget.org?

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): NuGet.exe

NuGet version (x.x.x.xxx): 4.6.2 (Latest recommended)

dotnet.exe --version (if appropriate): -

VS version (if appropriate): -

OS version (i.e. win10 v1607 (14393.321)):

Worked before? If so, with which NuGet version:

Detailed repro steps so we can see the same problem

1. nuget install cake -version 0.27.2 -source https://mylocalpackageserver

Other suggested things

Verbose Logs

NuGet Version: 4.6.2.5055
Feeds used:
  C:\Users\Administrator\.nuget\packages\
  https://mylocalpackageserver



Attempting to gather dependency information for package 'cake.0.27.2' with respect to project 'C:\temp', targeting 'Any,Version=v0.0'
  CACHE https://mylocalpackageserver/Packages(Id='cake',Version='0.27.2')
Total number of results gathered : 2
Gathering dependency information took 134.67 ms
Summary of time taken to gather dependencies per source :
https://mylocalpackageserver    -   42.87 ms
C:\Users\Administrator\.nuget\packages\ -   16.26 ms
Attempting to resolve dependencies for package 'cake.0.27.2' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'cake.0.27.2'
Resolved actions to install package 'cake.0.27.2'
Retrieving package 'cake 0.27.2' from 'C:\Users\Administrator\.nuget\packages\'.
Adding package 'Cake.0.27.2' to folder 'C:\temp'
Added package 'Cake.0.27.2' to folder 'C:\temp'
Added package 'Cake.0.27.2' to folder 'C:\temp' from source 'C:\Users\Administrator\.nuget\packages\'
Successfully installed 'cake 0.27.2' to C:\temp
Executing nuget actions took 4.01 min

[jainaashish]

@pascalberger I'm not clear about the issue? were you not able to install your signed package without accessing to nuget.org? or taking 4 mins to install this package, is the main issue?

And also, tell us how much time nuget.exe verify command takes to verify signature of this package?

[pascalberger]

@jainaashish The package actually installs sucessfully, but it takes 4 minutes without access to nuget.org.

[pascalberger]

A nuget verify still runs without any output:

C:\temp\nuget verify -all .\Cake.0.27.2.nupkg -verbosity detailed
NuGet Version: 4.6.2.5055

I canceled after 3h

[gep13]

@jainaashish can we get confirmation that the installation of a signed nuget package, in an offline scenario, is something that is going to be supported long term? And also, confirmation that we agree that 4 minutes is far too long for this to take.

At the moment, not that many packages are signed, but as this rolls out, and with the upcoming repository signing, more are more packages are going to be signed, and offline builds are going to become very slow.

[pascalberger]

@jainaashish Additional questions from my side are:

• Is the package signing scenario dependend to nuget.org or also supposed to be supported with other feeds (VSTS, MyGet, ProGet, etc)?
• Do you plan any indication that a package is signed on nuget.org? The current user experience in this case is quite bad, as for example Cake with version 0.27.2 just became slow to restore, and a user has no clue about the differenes and that the package is signed
• I suppose after the 4 minutes trying to validate the package, calls to nuget.org are timeouting and then you proceed to install the package anyway indicating success without any warning. Isn't this a big security issue (by managing to timouting the calls I can trick any user in successfully installing malicious packages he believes are safe)?

[PatoBeltran]

@gep13 offline scenarios are definitely something we are planning to support and we are actively working on them as well.

@pascalberger package signing is independent of the feed. Any feed can host author signed packages and also any feed can decide to implement the required features to support repository signing. We are currently working on adding support for repository signing on nuget.org, once that support is fully added, every package will be signed.

As for your third point, verification for author signed packages is done independently of the source they are gotten from. The only network calls that are needed are to check for revocation, and this are made directly to revocation services. In the case where this information is unavailable it should show a warning (which I'm actually curious about why you didn't see it).

Can you confirm you only see this delay when restoring offline? Also, does nuget verify also hangs with other packages or just with that one? In the case is that one, can you provide a copy of the package so I can try to repro the issue and find the reason?

Thanks!

[PatoBeltran]

@pascalberger I downloaded Cake v0.27.2 from nuget.org (https://www.nuget.org/packages/Cake/0.27.2) and try repro your scenario locally with no luck. I ran nuget.exe verify -signatures on it while running offline (and after cleaning my CRL and OCSP cache) and this is the output I got:

$> NuGet.exe verify -signatures .\cake.0.27.2.nupkg

Verifying Cake.0.27.2
C:\...\cake.0.27.2.nupkg

Signature Hash Algorithm: SHA256
Signature type: Author
Verifying the author primary signature with certificate:
  Subject Name: CN=.NET Foundation, O=.NET Foundation, L=Redmond, S=WA, C=US
  SHA1 hash: 9E6D212CA626AE157139212AE3CC920309AA032A
  SHA256 hash: 3F42B199B1DAF0D0ABB4A0718AE047D749DE8099D098506500D42B7F80A41458
  Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
  Valid from: 9/15/2016 5:00:00 PM to 10/28/2019 5:00:00 AM

WARNING: NU3018: The author primary signature found a chain building issue: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3018: The author primary signature found a chain building issue: The revocation function was unable to check revocation for the certificate.
Timestamp: 5/15/2018 3:06:16 PM

Verifying author primary signature's timestamp with timestamping service certificate:
  Subject Name: CN=DigiCert SHA2 Timestamp Responder, O=DigiCert, C=US
  SHA1 hash: 400191475C98891DEBA104AF47091B5EB6D4CBCB
  SHA256 hash: FC834D5BFFDE31DBA5B79BF95F573F7953BCBF9156E8525163E828EB92EA8A93
  Issued by: CN=DigiCert SHA2 Assured ID Timestamping CA, OU=www.digicert.com, O=DigiCert Inc, C=US
  Valid from: 1/3/2017 4:00:00 PM to 1/17/2028 4:00:00 PM

WARNING: NU3028: The author primary signature's timestamp found a chain building issue: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3028: The author primary signature's timestamp found a chain building issue: The revocation function was unable to check revocation for the certificate.
Finished with 0 errors and 4 warnings.

Successfully verified package 'Cake.0.27.2'.

I also ran Measure-Command with this and here is the time it took:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 2
Milliseconds      : 43
Ticks             : 20438148
TotalDays         : 2.36552638888889E-05
TotalHours        : 0.000567726333333333
TotalMinutes      : 0.03406358
TotalSeconds      : 2.0438148

[clairernovotny]

Did you clear your intermediates too? There could be AIA calls to intermediates that may be having issues.

[pascalberger]

@PatoBeltran Which nuget. exe version did you try with? I used 4.6.2

[pascalberger]

@PatoBeltran

package signing is independent of the feed. Any feed can host author signed packages and also any
feed can decide to implement the required features to support repository signing.

While feeds can host signed packages, how is verification supposed to work there? Does every feed need to implement something so that it's signed packages can be verified?

As for your third point, verification for author signed packages is done independently of the source they are gotten from. The only network calls that are needed are to check for revocation, and this are made directly to revocation services. In the case where this information is unavailable it should show a warning (which I'm actually curious about why you didn't see it).

The warning should also be shown for nuget install or only for nuget verify?

Can you confirm you only see this delay when restoring offline?

The package installs fine with the same NuGet version from another PC / network which has internet access.

Also, does nuget verify also hangs with other packages or just with that one?

Can you advise on other packages which are signed?

In the case is that one, can you provide a copy of the package so I can try to repro the issue and find the reason?

You can download from here

[PatoBeltran]

@pascalberger I tried with 4.6.2 and the latests bits and was not able to repro with any of this. I'm curious to know what is causing it to be slow in your case. When you run nuget verify -signatures do you see the process consuming CPU? This would let us identify if it is an issue with the product or another unrelated issue (maybe with your environment?), also are you familiar with generating a 64 bit dump or using perfview? That might be helpful to get the stack trace of what it is doing in the 3 hours (or the 4 minutes in the restore case) it is running.

While feeds can host signed packages, how is verification supposed to work there? Does every feed need to implement something so that it's signed packages can be verified?

Feeds don't need to implement anything to have signed packages and for clients to verify those packages. If the feeds want to implement repository signing, then they have to implement the necesary features to let the clients know that the packages that come from that feed should have a specific signature. You can read more about package signing here

The warning should also be shown for nuget install or only for nuget verify?

It should be shown on both.

The package installs fine with the same NuGet version from another PC / network which has internet access.

Does it repro in this other machine when it is disconnected from the network? are you running the same nuget version on both? Do both machines have similar environments? Are you running on mono or any platform different than windows?

Can you advise on other packages which are signed?

Try this one

[PatoBeltran]

Hey @pascalberger, is this still an issue? Have you tried the suggestions I wrote in my previous comment? This issue has not been active for two weeks, if this is no longer an issue I will close it soon.

[pascalberger]

@PatoBeltran Yes, sorry this is still an issue, but I hadn't found the time yet to analyze it further. For some of your questions:

Does it repro in this other machine when it is disconnected from the network?

I can reproduce in one of our network were we have a bunch of blades running VMs from different hosts.

are you running the same nuget version on both?

Yes, everything is on 4.6.2

Do both machines have similar environments?

Yes, same versions of tools. But one is the build environment the other a development environment (e.g. Visual Studio Build Tools vs full Visual Studio).

Are you running on mono or any platform different than windows?

No, offline environment is Windows Server 2012 r2, the other where it works is Windows 10

[PatoBeltran]

Good, make sure everything is 4.6.2 or further, we had a perf bug in package verification that was fixed in 4.6.2. Can it be the case that it is an issue with the machine? Please try it with other signed packages or with the same development machine disconnected from the network.

[pascalberger]

Unfortunately I cannot test with the nuget.common package as it uses SemVer and I need to go through our internal ProGet server which still has a V2 feed. Do you know of any other signed package which I can use for testing?

[PatoBeltran]

Any Microsoft owned package should be signed. You can easily search in nuget.org for a package that follows any of your internal server requirements and to be sure it is signed just run nuget.exe verify -signatures on the package.

[pascalberger]

@PatoBeltran I can reproduce it also with other signed packages:

C:\Users\Administrator\temp>nuget install microsoft.aspnet.signalr.client -sourc
e https://mypackageserver
Feeds used:
  https://mypackageserver

Installing package 'microsoft.aspnet.signalr.client' to 'C:\Users\Administrator\
temp'.
  GET https://mypackageserver/FindPackagesById()?id='microsoft.
aspnet.signalr.client'&semVerLevel=2.0.0
  OK https://mypackageserver/FindPackagesById()?id='microsoft.a
spnet.signalr.client'&semVerLevel=2.0.0 377ms


Attempting to gather dependency information for package 'microsoft.aspnet.signal
r.client.2.3.0' with respect to project 'C:\Users\Administrator\temp', targeting
 'Any,Version=v0.0'
Gathering dependency information took 93.5 ms
Attempting to resolve dependencies for package 'microsoft.aspnet.signalr.client.
2.3.0' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'microsoft.aspnet.signalr.client.2.3.0'
Resolved actions to install package 'microsoft.aspnet.signalr.client.2.3.0'
Retrieving package 'Microsoft.AspNet.SignalR.Client 2.3.0' from 'https://mypackageserver'.
Retrieving package 'Newtonsoft.Json 6.0.4' from 'https://mypackageserver'.
  GET https://mypackageserver/package/Newtonsoft.Json/6.0.
4
  GET https://mypackageserver/package/Microsoft.AspNet.Sig
nalR.Client/2.3.0
  OK https://mypackageserver/package/Newtonsoft.Json/6.0.4
 31ms
Installing Newtonsoft.Json 6.0.4.
Adding package 'Newtonsoft.Json.6.0.4' to folder 'C:\Users\Administrator\temp'
Added package 'Newtonsoft.Json.6.0.4' to folder 'C:\Users\Administrator\temp'
Successfully installed 'Newtonsoft.Json 6.0.4' to C:\Users\Administrator\temp
  OK https://mypackageserver/package/Microsoft.AspNet.Sign
alR.Client/2.3.0 561ms
Installing Microsoft.AspNet.SignalR.Client 2.3.0.
Adding package 'Microsoft.AspNet.SignalR.Client.2.3.0' to folder 'C:\Users\Admin
istrator\temp'
Added package 'Microsoft.AspNet.SignalR.Client.2.3.0' to folder 'C:\Users\Admini
strator\temp'
Successfully installed 'Microsoft.AspNet.SignalR.Client 2.3.0' to C:\Users\Admin
istrator\temp
Executing nuget actions took 3.76 min

[PatoBeltran]

Okay, so with it seems like the issue is not on your package. Since I still have not been able to reproduce it, I'm inclined to think it might be something specific about your environment that it is making it slow. To understand what it is, could you try a couple of things?

1. Is it possible to replicate this scenario in another machine? You mentioned there is one online machine where this does not repro. If you set this machine online, does it repro?

2. On this machine, when you run nuget verify -signatures do you see the process consuming CPU? This would let us identify if it is an issue with the product or another unrelated issue, also are you familiar with generating a 64 bit dump or using perfview? That might be helpful to get the stack trace of what it is doing in the 3 hours (or the 4 minutes in the restore case) it is running.

[pascalberger]

@PatoBeltran

@PatoBeltran

Is it possible to replicate this scenario in another machine? You mentioned there is one online machine where this does not repro. If you set this machine online, does it repro?

I can reproduce this behavior on all machines in a certain (offline) network (or better said using different images on different VM hosts).

On this machine, when you run nuget verify -signatures do you see the process consuming CPU? This would let us identify if it is an issue with the product or another unrelated issue, also are you familiar with generating a 64 bit dump or using perfview? That might be helpful to get the stack trace of what it is doing in the 3 hours (or the 4 minutes in the restore case) it is running.

There's no CPU consumed from NuGet.exe while running nuget verify.

I might find some time later to do some deeper analysis.

[pascalberger]

I tried again the verify command and it completes in approximately 4 minutes with the following output (no clue why it didn't succeed before):

PS C:\temp\Cake.0.27.2> nuget verify -all .\Cake.0.27.2.nupkg -verbosity detailed
NuGet Version: 4.6.2.5055

Verifying Cake.0.27.2
C:\temp\Cake.0.27.2\.\Cake.0.27.2.nupkg

Signature Hash Algorithm: SHA256
Signature type: Author
Timestamp: 5/16/2018 12:06:16 AM

Verifying timestamp with timestamping service certificate:
Subject Name: CN=DigiCert SHA2 Timestamp Responder, O=DigiCert, C=US
SHA1 hash: 400191475C98891DEBA104AF47091B5EB6D4CBCB
SHA256 hash: FC834D5BFFDE31DBA5B79BF95F573F7953BCBF9156E8525163E828EB92EA8A93
Issued by: CN=DigiCert SHA2 Assured ID Timestamping CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 1/4/2017 1:00:00 AM to 1/18/2028 1:00:00 AM

    Subject Name: CN=DigiCert SHA2 Assured ID Timestamping CA, OU=www.digicert.com, O=DigiCert Inc, C=US
    SHA1 hash: 3BA63A6E4841355772DEBEF9CDCF4D5AF353A297
    SHA256 hash: CA8D0F4736454AECBEC5DEEC80998C9EBF41D06C728F3C76CCA24151BC62D463
    Issued by: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
    Valid from: 1/7/2016 1:00:00 PM to 1/7/2031 1:00:00 PM

        Subject Name: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
        SHA1 hash: 0563B8630D62D75ABBC8AB1E4BDFB5A899B24D43
        SHA256 hash: 3E9099B5015E8F486C00BCEA9D111EE721FABA355A89BCF1DF69561E3DC6325C
        Issued by: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
        Valid from: 11/10/2006 1:00:00 AM to 11/10/2031 1:00:00 AM


WARNING: NU3028: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3028: The revocation function was unable to check revocation for the certificate.
Verifying signature with author certificate:
Subject Name: CN=.NET Foundation, O=.NET Foundation, L=Redmond, S=WA, C=US
SHA1 hash: 9E6D212CA626AE157139212AE3CC920309AA032A
SHA256 hash: 3F42B199B1DAF0D0ABB4A0718AE047D749DE8099D098506500D42B7F80A41458
Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 9/16/2016 2:00:00 AM to 10/28/2019 1:00:00 PM

    Subject Name: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
    SHA1 hash: 92C1588E85AF2201CE7915E8538B492F605B80C6
    SHA256 hash: 51044706BD237B91B89B781337E6D62656C69F0FCFFBE8E43741367948127862
    Issued by: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
    Valid from: 10/22/2013 2:00:00 PM to 10/22/2028 2:00:00 PM

        Subject Name: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
        SHA1 hash: 0563B8630D62D75ABBC8AB1E4BDFB5A899B24D43
        SHA256 hash: 3E9099B5015E8F486C00BCEA9D111EE721FABA355A89BCF1DF69561E3DC6325C
        Issued by: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
        Valid from: 11/10/2006 1:00:00 AM to 11/10/2031 1:00:00 AM


WARNING: NU3018: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3018: The revocation function was unable to check revocation for the certificate.
Finished with 0 errors and 4 warnings.

Successfully verified package(s).
PS C:\temp\Cake.0.27.2>

[gep13]

The revocation function was unable to check revocation because the revocation server was offline.

This part is very concerning to me, and will no doubt causes lots of problems to people running in an isolated environment.

[pascalberger]

@PatoBeltran Here's a mini dump file taken while it was "hanging" with the following output on the console:

PS C:\temp> nuget install cake -version 0.27.2 -source https://mypackageserver
Feeds used:
  C:\Users\Administrator\.nuget\packages\
  https://mypackageserver

Attempting to gather dependency information for package 'cake.0.27.2' with respect to project 'C:\temp', targeting 'Any,
Version=v0.0'
Gathering dependency information took 139.07 ms
Attempting to resolve dependencies for package 'cake.0.27.2' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'cake.0.27.2'
Resolved actions to install package 'cake.0.27.2'
Retrieving package 'cake 0.27.2' from 'C:\Users\Administrator\.nuget\packages\'.
Adding package 'Cake.0.27.2' to folder 'C:\temp'

https://wetransfer.com/downloads/2ab0eef94f578c86fe957f9483f010e220180716104424/b0f1df9382bad138bb7c65f025adb67a20180716104424/dbf40c

[pascalberger]

I also tried with https://dist.nuget.org/win-x86-commandline/v4.8.0-preview3/nuget.exe, which has the same behavior as 4.6.2 which I'm currently using

[PatoBeltran]

@pascalberger the log messages you saw when you run nuget verify are the expected behavior, so it is good to see that part working now. I still need to investigate why it is taking 4 minutes for it to finish. I will take a look at the mini dump you provided. You said that nuget verify is not consuming any CPU while waiting, this means it is not stuck on doing computation, rather it is only waiting on something. Would you have any idea of something on your network that might be making this process not able to run correctly?

@gep13 the message The revocation function was unable to check revocation because the revocation server was offline. is given by the OS while not being able to verify correctly the revocation status, the reason this is a warning is because there is no way to know if the revocation server is unreachable or if the computer does not have access to a network.

[gep13]

@PatoBeltran said...
@gep13 the message The revocation function was unable to check revocation because the revocation server was offline. is given by the OS while not being able to verify correctly the revocation status, the reason this is a warning is because there is no way to know if the revocation server is unreachable or if the computer does not have access to a network.

Yes, I believe we are on the same page in terms of what the issue is, however, my question is the same...

If I _want_ to do a build on a machine that has _no_ outside network connection, i.e. I have pulled all the nupkg's used as part of my build onto an internal ProGet/Artifactory/Nexus server, there is going to be no access to perform this validation check. I am assuming that there is a way to bypass this check when installing NuGet packages in an offline mode?

Or have I got things completely wrong?

[pascalberger]

@PatoBeltran I think main difference in the network where it doesn't work is that machines are not domain joined and access to internet is blocked by a firewall. Package server (V2 feed on a ProGet server) is the same as in the other network where it works.

[gmcve]

@pascalberger @PatoBeltran I can confirm that this is also happing to me with NuGet 4.6.2, Cake.0.28.0 and using an offline build server.

2018-07-24T08:18:49.3535711Z Adding package 'Cake.0.28.0' to folder ''
2018-07-24T08:22:21.1025539Z Added package 'Cake.0.28.0' to folder ''

[gmcve]

I tested the build server with the followng NuGet.exe versions 3.4.4, 4.1.0, 4.5.1
All work as expected with no 4min time delay on installing the signed package on our offline build server.

[pascalberger]

@PatoBeltran Any news here? Was the dump file helpful?

As more and more packages are signed this will start to become a bigger issue for us, meaning we will have build times increased by 4 minutes (or multiple times 4 minutes) or cannot update to newer version of specific libraries and tools.

Is there any chance to get some kind of flag to skip verification in a hotfix of nuget.exe so that we can have the old behavior back and at least keep updating libraries until this issue is fully solved. Like it is implemented now the feature does not improve security in our case but on the contrary force us to use outdated and probably insecure versions of libraries and tools

[PatoBeltran]

@pascalberger I'm sorry I just recently have time to look at this issue. I cannot download the dump, it seems to be unavailable, can you provide it again?

@gep13 technically speaking, the .NET API we use to build the chain should know if it wasn't possible to build the chain because the machine was offline and will only produce a warning, but you are correct, we should add an option to override the default of making the check online. I have created an issue to add this option to our commands, You can follow up here: https://github.com/NuGet/Home/issues/7173

@gmcve Is it possible for you to provide a dump file when the restore is "hanging" for those 4 minutes so I can analyze if the issue is the same that the one @pascalberger is facing?

[pascalberger]

@PatoBeltran Uploaded again: https://wetransfer.com/downloads/404b44ac2b0b2ae78cd6e01533ff906020180806175922/f9cd2dc21aa8d864cd6526061c8cc0d320180806175922/f89c91

[PatoBeltran]

Hey @pascalberger the dump was not really helpful in giving me the information I needed to understand what was going on. Can you help me out giving me an ETL trace?

Here are some instructions on how to do it:

1. Download and extract PerfView.exe Download here.
2. Run PerfView.exe collect /zip:true /nogui /threadtime:true /AcceptEULA from an elevated command prompt. This will launch another window and begin tracing.
3. Reproduce the issue with the hang
4. Press S on the window that showed up in step 2. This will generate a file called PerfViewData.ETL.ZIP, which then you can send it to me.

Thank you so much for helping me understand this issue!

[adrian-moll]

Hi @PatoBeltran ,
We have the same Issue on our buildserver (Windows Server 2016, no internet access).
Nuget restore is taking about 4 minutes (packages are stored on an in-house ProGet server).
The problem seems to be that on nuget verify the revocation server is not accessible.
(tested with nuget.exe V4.6.2.5055 and V4.7.0.5148: same issue for both)

Is there any possibility to disable the signature check on nuget restore?

I followed your instructions to generate an ETL trace:

Console output

SAC-TFS-Service@BLD01 d:\Test
$ PerfView.exe collect /zip:true /nogui /threadtime:true /AcceptEULA

SAC-TFS-Service@BLD01 d:\Test
$ nuget verify -signature MSTest.TestAdapter.1.3.2.nupkg

Verifying MSTest.TestAdapter.1.3.2
d:\Test\MSTest.TestAdapter.1.3.2.nupkg

Signature Hash Algorithm: SHA256
Signature type: Author
Verifying the author primary signature with certificate:
  Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
  SHA1 hash: F404000FB11E61F446529981C7059A76C061631E
  SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
  Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
  Valid from: 26.02.2018 01:00:00 to 27.01.2021 13:00:00

WARNING: NU3018: The author primary signature found a chain building issue: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3018: The author primary signature found a chain building issue: The revocation function was unable to check revocation for the certificate.
Timestamp: 05.06.2018 15:09:41

Verifying author primary signature's timestamp with timestamping service certificate:
  Subject Name: CN=Symantec SHA256 TimeStamping Signer - G2, OU=Symantec Trust Network, O=Symantec Corporation, C=US
  SHA1 hash: 625AEC3AE4EDA1D169C4EE909E85B3BBC61076D3
  SHA256 hash: CF7AC17AD047ECD5FDC36822031B12D4EF078B6F2B4C5E6BA41F8FF2CF4BAD67
  Issued by: CN=Symantec SHA256 TimeStamping CA, OU=Symantec Trust Network, O=Symantec Corporation, C=US
  Valid from: 02.01.2017 01:00:00 to 02.04.2028 01:59:59

WARNING: NU3028: The author primary signature's timestamp found a chain building issue: The revocation function was unable to check revocation because the revocation server was offline.
WARNING: NU3028: The author primary signature's timestamp found a chain building issue: The revocation function was unable to check revocation for the certificate.
Finished with 0 errors and 4 warnings.

Successfully verified package 'MSTest.TestAdapter.1.3.2'.

Result of tracing

PerfViewData.log.txt
PerfViewData.etl.zip (on Google Drive)

Thank you for your help.

[PatoBeltran]

Hey @adrian-moll, thanks for taking the time to get the ETL trace. I tried analyzing it but the data says that the whole operation for the command took around 93.0 milliseconds. I was not able to find any data showing the 4-minute delay you mention. Could it be that you sent the wrong ETL trace? or that when you captured this it didn't repro?

As one step of the verification, we invoke dotnet APIs to build the certificate chain of the signing certificate and let us know if there was an error while doing this. As part of their verification, this API tries to communicate with CRL and OSCP for all the certificates in the chain.

I suspect that the reason for this delay is because the chain building API is timing out when trying to get a response from the OCSP and the CRL, so I have another idea to get data that might support that claim.

Would it be possible for you to collect another ETL trace where this issue repros? This time can you, at the same time, capture a fiddler trace of the calls made while running nuget verify?

To get all the calls actually be made you will probably have to first clean the OCSP and CRL cache, which you can do by running:

certutil -urlcache * delete

When you run nuget verify if the OCSP and CRL cache are clean, the API will be forced to try to do the calls on the web, so you will see something like this:

Since you are offline, this requests will fail, and if they are taking a long time, fiddler will tell us. I would appreciate if you are able to send me the full summary for all of them to further analyze them.

Also, if any method on the code is waiting for something to finish (e.g. this calls) the ETL trace should show us that the method took a long time to finish and that it was waiting on something.

I appreciate the time you are taking to help me figure out this issue, I hope we can get it sorted out soon to get a fix ready ASAP!

[pascalberger]

@PatoBeltran Here's my ETL trace:

Command

PerfView.exe collect /zip:true /nogui /threadtime:true /AcceptEULA
nuget install cake -version 0.27.2 -source https://mypackageserver

Output

Feeds used:
  C:\Users\TestCloudAgent\.nuget\packages\
  https://packages.bbtlocal.ch/nuget/nuget



Attempting to gather dependency information for package 'cake.0.27.2' with respect to project 'C:\temp', targeting 'Any,
Version=v0.0'
Gathering dependency information took 186.46 ms
Attempting to resolve dependencies for package 'cake.0.27.2' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'cake.0.27.2'
Resolved actions to install package 'cake.0.27.2'
Retrieving package 'Cake 0.27.2' from 'https://packages.bbtlocal.ch/nuget/nuget'.
  GET https://sr-packages01.bbtlocal.ch/nuget/nuget/package/Cake/0.27.2
  OK https://sr-packages01.bbtlocal.ch/nuget/nuget/package/Cake/0.27.2 23ms
Installing Cake 0.27.2.
Adding package 'Cake.0.27.2' to folder 'C:\temp'
Added package 'Cake.0.27.2' to folder 'C:\temp'
Successfully installed 'Cake 0.27.2' to C:\temp
Executing nuget actions took 4.02 min

Trace

WeTransfer

[pascalberger]

@PatoBeltran I also tried the same nuget install command with a ProGet v2 (Quirks Mode) Feed, ProGet v3 Feed and installing a package from the local file system. All three took the same amount of time (4 min)

[pascalberger]

Btw. same behavior for author and repository signed packages. And also with nuget.exe 4.7.1

@PatoBeltran Would you be interested in making a phone call where I can show you the bahvior on our system? I fear if you further rollout the signing stuff with repository signing of all packages without having this issue fixed that our build time will explode from minutes to hours.

[pascalberger]

@PatoBeltran Here is also a Fiddler trace for installing a author signed package from a local file system feed.

In the trace you can see how the requests to digicert.com are made, and after each request it waits 15s before it starts the next one, which leads to the 4 min overall.

[pascalberger]

On another note: My tests were with Windows Server 2012r2 in a fully offline environment. Meaning Windows installation happened offline and machine never had internet access. Windows Updates were installed from a local WSUS server, but no certificate cache or similar. Just pure Windows Server 2012r2.

[clairernovotny]

Thread: https://twitter.com/onovotny/status/1029031004994985986

[vcsjones]

Based on a discussion it, unfortunately, sounds like "everything is working as intended" for now. Nuget.Client explicitly does online revocation checking:

https://github.com/NuGet/NuGet.Client/blob/75dbda58560c7db281816015ad19a6bffdcc0f79/src/NuGet.Core/NuGet.Packaging/Signing/Utility/CertificateChainUtility.cs#L171

So it tries to validate the the certificate chain is not revoked, which appears to include the leaf and intermediate certificates, including any counter signatures. I suspect the timeout is 15 seconds, it gives up, and moves on to the next certificate.

[pascalberger]

@vcsjones This means, that nuget no longer can be used inside networks without internet access? Which is quite a huge restriction, considering that soon all packages will be signed. And which actually forces us to either open our network to the internet (or stay on old package versions as long as they are not repo signed). Both decisions from a security and auditing point of view lead to quite some trouble.

Also, even if the connection is open, it would make builds heavenly dependent on issuer site being available, and in the worst case taking my build pipeline 400 minutes to restore 100 nuget packages from a local and trusted feed.

IMHO there still should be an option to disable the certificate verification in offline scenarios.

[vcsjones]

This means, that nuget no longer can be used inside networks without internet access?

Well... it works... for some definition of "works", but is a terrible experience. You could talk to your network team about getting outbound access to CRL/OCSP servers. Since package authors get to choose their CA, you would have to allow access for any CRL/OCSP that the package signer users, not just digicert. e.g. if a package author signs with Comodo, you'd need to whitelist comodo's CRL/OCSP servers as well.

Both decisions from a security and auditing point of view lead to quite some trouble.

I personally think that synchronous online revocation checking is problematic. At the very least I would think that revocation checking should be configurable by some policy or setting, even if the default is the current behavior.

[pascalberger]

At the very least I would think that revocation checking should be configurable by some policy or setting, even if the default is the current behavior.

+1 for this. Opening the network in some situations is just not an option due to govermental, legal or any other regulations.

[anangaur]

/cc: @rido-min, @karann-msft

[gep13]

@vcsjones said...
Based on a discussion it, unfortunately, sounds like "everything is working as intended" for now. Nuget.Client explicitly does online revocation checking:

Based on this, can we get some clarity on what is meant by this:

@PatoBeltran said...
offline scenarios are definitely something we are planning to support and we are actively working on them as well.

Is there an issue that we can track where this is being worked on? In my opinion, this is a major failing, and it will mean that systems like Artifactory, ProGet, Nexus, and likely many others will start failing, where they had worked perfectly before. How is this, _dare I say it_, breaking change, being communicated?

[anangaur]

We are looking to fix this soon - mostly 4.8.1 (as bits for 4.8.0 is almost locked).

[gep13]

@anangaur can you provide any details on what you mean by...

@anangaur said...
We are looking to fix this soon - mostly 4.8.1 (as bits for 4.8.0 is almost locked).

What is the fix? A flag to turn off checking? Or something else?

[pascalberger]

@anangaur Can you please wait with further rolling out repository signing (for old packages) on nuget.org until this fix has been released, to avoid fully breaking enterprise environments?

[anangaur]

@gep13 We are planning to provide a flag for offline-only revocation checks.

@pascalberger, we are not signing the old packages yet. Only new packages are being repo-signed. We will start repo-signing old packages once we roll out the above mentioned option.

[gep13]

@anangaur thank you very much for confirming. This will certainly help in a number of scenarios.

[adrian-moll]

We are looking forward to be able to disable the revocation checks on nuget restore.

In the meantime our solution is to redirect the revocation-server calls to localhost. This way the revocation checks will fail much faster.

You can catch the list of servers nuget verify tries to reach with Fiddler and make an entry in your host file for each of them (maybe you need to clear your local dns cache : _ipconfig /flushdns_ )

Host file

127.0.0.1 ocsp.digicert.com
127.0.0.1 crl4.digicert.com
127.0.0.1 crl3.digicert.com
127.0.0.1 cacerts.digicert.com
127.0.0.1 s.symcd.com
127.0.0.1 s.symcb.com
127.0.0.1 ts-ocsp.ws.symantec.com
127.0.0.1 ts-crl.ws.symantec.com

Nuget also makes a call to _ctldl.windowsupdate.com_. But this one is hardcoded in Windows (%WINDIR%\system32\dnsapi.dll) and can not be overwritten in the host file.

[PatoBeltran]

hey @pascalberger, @gep13, @adrian-moll I have a PR with a fix for this, would any of you have time Today to help me out test the private bits to make sure this approach fixes correctly the issues you are seeing? Let me know how I can send you a build of nuget.exe to test! Thanks!

[pascalberger]

@PatoBeltran I can do some tests

[PatoBeltran]

@pascalberger awesome! thanks so much.

Here you will find a private build of nuget.exe with those changes, with this build you should be able to set revocationMode in your config file and the 4 minute hang in verify and restore should be gone. You will notice that the warnings in verify will still be there, but the calls in the fiddler trace will not. Your config file should look something like:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key="revocationMode" value="offline" />
  </config>
</configuration>

Please let me know if you have any questions and if this works for you!

[pascalberger]

@PatoBeltran revocationMode=offline fixes the timeout issue. Thanks for your work!

[PatoBeltran]

Awesome to hear that! Thanks for the confirmation, I will work to get this changes merged and published as soon as possible.

[PatoBeltran]

Closing this since it was fixed by https://github.com/NuGet/Home/issues/7173, this code should ship in the next release.

[sehe]

@adrian-moll wouldn't it be better to black-hole instead of causing (hopefully) failing traffic to loopback?

0.0.0.0 ocsp.digicert.com
0.0.0.0 crl4.digicert.com
0.0.0.0 crl3.digicert.com
0.0.0.0 cacerts.digicert.com
0.0.0.0 s.symcd.com
0.0.0.0 s.symcb.com
0.0.0.0 ts-ocsp.ws.symantec.com
0.0.0.0 ts-crl.ws.symantec.com

[PatoBeltran]

nuget.exe 4.8.1 has just shipped with the fix for this issue. You can download it at https://www.nuget.org/downloads

[gep13]

@PatoBeltran can you point at where the documentation for enabling an offline build is? Thanks

[pascalberger]

@PatoBeltran @gep13 I've also created another issue for this: https://github.com/NuGet/Home/issues/7262

[karann-msft]

@gep13 https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu3028#revocation-check-mode

[devlead]

@karann-msft it say's NuGet 4.6.0+ in the article but didn't this fix ship with 4.8.1? So maybe make it more clear that which version env var works in?

[anangaur]

@devlead The issue itself may manifest from 4.6 (clients that support signing). The proposed solution - ‘offline revocation check’ exists in 4.8.1. Just submitted an edit to the docs to state this. Will be live soon.

[devlead]

@anangaur excellent, that will save people time debugging, if they're trying with an older version 👍

[rrelyea]

VS 2017 15.8.4 has this fix as well -- available today.

[pascalberger]

@rrelyea So, you ship a not recommendet version (as per https://www.nuget.org/downloads) to thousands of customers? 😕

Wouldn't it be a wise idea to make it the recommendet version first including proper documentation (as already asked here: https://github.com/NuGet/Home/issues/7262) and also note this change in Visual Studio release notes.