Orleans: SystemTarget not active on this silo

[galvesribeiro]

In 2.0 rc1 grain services are not working as it was before it.

SystemTarget *stg/0/00000000 not active on this silo. Msg=Request S192.168.196.55:10001:257696242*grn/1DFC7212/00000000@28ac39b0->S192.168.196.55:10001:257696242*stg/0/00000000@S00000000 #74: 

Here is the repro case:

internal class MyServiceClient : GrainServiceClient<IMyService>, IMyServiceClient
  {
    public MyServiceClient (IServiceProvider serviceProvider) : base(serviceProvider) { }

    public async Task DoSomething()
    {
      try
      {
        //Here is where it blow when called from a grain method
        await GrainService.DoSomething();
      }
      catch (Exception exc)
      {
        throw;
      }
    }
  }

internal class MyService : GrainService, IMyService
  {

....

    private bool _isRunning;

    public MyService (IOptions<MyOptions> options,
            IGrainFactory grainFactory, IGrainIdentity id,
            Silo silo, ILoggerFactory loggerFactory) : base(id, silo, loggerFactory)
    {
      ...
      this._grainFactory = grainFactory;
      this._logger = loggerFactory.CreateLogger(nameof(MyService));
      ...
    }

    public Task DoSomething()
    {
      return Task.CompletedTask;
    }

    public override async Task Start()
    {
      this._isRunning = true;
      await base.Start();      
    }

    protected override async Task StartInBackground()
    {
      await Task.Factory.StartNew(async () =>
      {
        while (this._isRunning)
        {
         // Do our background work here
      });
      this.Status = GrainServiceStatus.Started;
    }

    public override async Task Stop()
    {
      this._isRunning = false;
      this.Status = GrainServiceStatus.Stopped;
    }
  }

And this is how I'm registering the grain service:

.ConfigureServices(services =>
        {
          services.Configure<MyOptions>(opt =>
          {
            opt.GrainServices.Add(new KeyValuePair<string, short>(/*typeof(MyService).AssemblyQualifiedName*/ "MyAssembly.MyNamespace.MyService, MyAssembly", 1));
          });
          services.Configure<MyOptions>(config.GetSection(MyOptions.SECTION_NAME));
          services.AddSingleton<IMyServiceClient, MyServiceClient>();
        });

Redacted the sample for simplicity.

What is wrong there? I would appreciate anyone jump in as this is a blocker for me.

[galvesribeiro]

cc: @ReubenBond

[galvesribeiro]

Investigating it I tried to run the UnitTests and finally I got the same result as mine:

If I comment the legacy config and use the _new_ 2.0 config, it just fails the same way as on mine. The silo start, the grain service start in a weird state, and when the client calls it, same crash just like on my code.

So, it looks like the legacy methods are doing something else to make the services to work properly that the new config isn't...

Still investigating...

[galvesribeiro]

More one finding...

The system target being registered to the activation catalog. However, messages to it are arriving to a different activation id...

The question is, why?

[sergeybykov]

As we found out while investigating this, the culprit was passing 1 instead of 0 in the following line.

opt.GrainServices.Add(new KeyValuePair<string, short>(..., 1));

As of now, grain services only support a single instance of a service per silo. Hence, it's ID is always 0.

We need to improve the grain service registration API though regardless.

[jason-bragg]

As of now, grain services only support a single instance of a service per silo. Hence, it's ID is always 0.

Where does this constraint come from?

The system target is being registered to the activation catalog. However, messages to it are arriving to a different activation id... The question is, why?

This seems like a good question. Where did the sender get that address? If it got it from the grain service itself, it should be correct. Was it persisted? Was it generated somewhere in code? I think the problem is that the previous version of grain services always used Id 0, but now the id is configurable, which is better, unless there is something I'm missing.

[sergeybykov]

Where does this constraint come from?

It is hardcoded in https://github.com/dotnet/orleans/blob/master/src/Orleans.Runtime/Services/GrainServiceClient.cs#L42.

I think the problem is that the previous version of grain services always used Id 0, but now the id is configurable, which is better, unless there is something I'm missing.

ID isn't really configurable, but the API allows to register a grain service system target with a non-0 ID. We need to fix that for now.

[sergeybykov]

4155 is meant to improve the grain service registration and configuration experience.

[sergeybykov]

Resolved via #4155.