Nservicebus: Fix license management code to work on read-only hosts

[krosenko-at-starbucks]

According to the Licensing documentation page, NServiceBus will always log a FATAL message on start-up when it cannot access C:\Windows\system32\config\systemprofile. This is unacceptable for several reasons.

• Many people are using hosting containers and systems such as cloud providers. On those systems, writing is limited to the application directory by design. That is the correct decision and will not change.
• NServiceBus has been ported to .NET Standard, which runs on non-Windows platforms. There is no such thing as a C: drive and certainly no directory /Windows/system32/config/systemprofile on those platforms.
• Logging a FATAL message on every service start-up is a problem. Enterprise users set up alerts based on log messages, and FATAL messages almost always wake up a person in the middle of the night. Furthermore, many hosting systems can dynamically add or migrate hosts at will and continuously, so service start-up can happen quite frequently, not just at deployments.

Please provide a way for users to get rid of this problematic log message that, quite frankly, is not actually an error.

[bording]

@krosenko-at-starbucks That path is just an example. The point of the note is that you might need file system permissions based on the user account you're using to run your endpoint.

As of NServiceBus 7, we only use the following locations on non-Windows platforms.

• https://docs.particular.net/nservicebus/licensing/#license-management-user-specific-license-location
• https://docs.particular.net/nservicebus/licensing/#license-management-machine-wide-license-location

And when running on .NET Framework we will also check

• https://docs.particular.net/nservicebus/licensing/#license-management-application-configuration-file
• https://docs.particular.net/nservicebus/licensing/#license-management-windows-registry

To get that specific C:\Windows\system32\config\systemprofile path, you'd need to be running your endpoint with a user account that resolves %LOCALAPPDATA% to the system profile.

[krosenko-at-starbucks]

It doesn't really matter what the path is, the point is this section of the linked documentation:

The license management code requires write permissions to store metadata. If the process credentials don't have write permissions the following fatal event log item can be generated:

All the documentation you linked is for _reading_, but I am referring to the one and only section of the documentation that says the library will be _writing_ to the filesystem. There is no mention of the criteria used to choose a write destination, or how to configure it. And when an application using NServiceBus is deployed to (e.g.) Azure Service Fabric, it always triggers this FATAL log message and nobody can figure out how to make it stop. I am asking for a way to make it stop, and hopefully for that way to be documented clearly so others can find it.

[bording]

and when an application using NServiceBus is deployed to (e.g.) Azure Service Fabric, it always triggers this FATAL log message and nobody can figure out how to make it stop.

Have you opened a support case about this? It would be helpful to get more information from you about this, including collecting log files, etc.

[krosenko-at-starbucks]

I'm not sure what more information is required. The host system will not allow writes to arbitrary places in the filesystem. That's not specific to Service Fabric, that is true for many deployment tools and scenarios. The documentation page says that the library is writing metadata to the filesystem but does not say where or provide a way to change the write destination.

I just tried setting the LOCALAPPDATA environment variable to a directory that the application can write to, but the log message did not change and no new files appeared in that directory.

[bording]

Opening a support case will be helpful so we can examine the details of what you're running into more closely, and see if there's a solution for what you're running into. For example being able to see the actual log files that show the errors you're having.

[krosenko-at-starbucks]

This is not a support request. There is no question about what is happening. The log files literally show the same error message as in your documentation, character-for-character. I am not asking for help with my own service.
I am asking that the behavior of the library be changed, or at least be made configurable in some way. This is a feature request, not a support case.

[dvdstelt]

@krosenko-at-starbucks The documentation states that it needs write access, but the doco might not be complete. We're not sure it needs write access on all occassions. @bording and I just discussed it, and we're both not really sure what is happening in your case. And since we have various other customers already running NServiceBus inside Docker and other (cloud based) hosting solutions, we'd really like to verify what is going on. It could be that something is off in our licensing code or something else. Would it be possible for you to send logfiles with stack-traces included so we can verify at which point exactly things go wrong?

I can imagine you rather prefer another way of communication over attaching logfiles to this public GitHub issue, that's why @bording mentioned opening up a support case and sending the logfiles attached to an email. You can send it to [email protected] if you'd like. If you'd rather upload it to GitHub or somewhere else, that will do as well.

Having the logfiles means we can investigate, which might mean we'll have to dig through (licensing related) code and possible try to reproduce your issue. But we'll have a look at the logfiles as soon as possible and try and fix your problem. Let us know if that sounds like a possible solution for the moment. Thanks in advance.

[krosenko-at-starbucks]

In LicenseSourceFilePath.cs:

namespace Particular.Licensing
{
    using System.IO;

    class LicenseSourceFilePath : LicenseSource
    {
        public LicenseSourceFilePath(string path)
            : base(path)
        { }

        public override LicenseSourceResult Find(string applicationName)
        {
            if (File.Exists(location))
            {
                return ValidateLicense(NonBlockingReader.ReadAllTextWithoutLocking(location), applicationName);
            }

            return new LicenseSourceResult
            {
                Location = location,
                Result = $"License not found in {location}"
            };
        }
    }
}

Wrap the if block in a try {} catch.

[krosenko-at-starbucks]

The problem I was trying to point out was that LicenseReminder.cs is logging a FATAL, and that is just not appropriate.

namespace NServiceBus.Features
{
    using System;
    using System.Diagnostics;
    using Logging;

    class LicenseReminder : Feature
    {
        protected internal override void Setup(FeatureConfigurationContext context)
        {
            try
            {
                // ...
            }
            catch (Exception ex)
            {
                //we only log here to prevent licensing issue to abort startup and cause production outages
                Logger.Fatal("Failed to initialize the license", ex);
            }
        }
        // ...
    }
}

[danielmarbach]

@krosenko-at-starbucks Thanks for raising what you call a feature request. My colleagues are asking for more details because on a support request we can get the feature diagnostics startup file as well as the detailed log file which in your case would contain the exception details of the log fatal. By having that we can dig deeper and understand the root cause of the issue.

We want to try to avoid to jumping to conclusions and label something as a feature while effectively it is a bug or vice versa. By being able to diagnose deeper, create reproduction scenarios we can discuss them and find a solution (whether it is a feature or a bugfix) that helps not just you in the log run but potentially also other customers. On the plus side we can preserve the knowledge and if another customer asks us "why did you do this" we can give that context as explanations.

For example which the code that you linked it might indicate that the LicensePath is set by doing

endointConfiguration.LicensePath("path")

plus the path exists but we have no rights to actually read the file. If that is the case the exception logged in the fatal would point us in that direction. Or maybe it is something else that we don't know yet. At this point, all we can do is speculate without more details

[krosenko-at-starbucks]

That was a reasonable and cogent explanation. And as it turns out, the reason we are seeing the log message is that we are deploying the license file incorrectly, much to my chagrin. I am working to fix that now.
I apologize, and you may close this issue. I did uncover another issue related to license expiration dates, but I will report that through the support system.

[danielmarbach]

Thanks @krosenko-at-starbucks for updating us on this matter. I would like to add something for some background information. For our support process we look at github issues, support request over email, discuss groups and other social media channels. It is just if we require more background information that might contain more sensitive information we ask to fall back to support emails. We don't want to give the impression here in this conversation we don't value github conversations ;)

[krosenko-at-starbucks]

You were professional and I never doubted that you took it seriously. I finally realized I was starting to come across as a jerk, and for that I am sorry. I was reassured repeatedly by my colleagues that our license file was valid and up-to-date, and I confirmed it was deployed. But nobody ever validated that it was deployed _correctly_, which is why the FATAL showed up. The error message was about an inability to access a system directory, so I assumed the license check code was looking there to find the license file. The error message is a bit misleading, you could choose to clarify it somehow. But the issue I opened is not valid.