Moleculer: Private services

[icebob]

If a service has a $private: true setting, the service actions can't be called from remote services (only from local services)

module.exports = {
    name: "invoices",
    settings: {
        $private: true
    }   
    actions: {
        create() {}
    }
}

[DeividasJackus]

@icebob, small typo just so it's 100% clear: "can't be called" :)

Suggestions based on the original proposal:

1. Remote nodes shouldn't be able to even list / check for existence of such services (not just call).
2. Private service information should never hit the transporter. Remote nodes should be able to have their own private/local invoices services (i.e. two nodes, each with a different invoices service that can only be accessed locally).
3. Naming the setting $local or $localOnly instead of $private as it describes the scope a bit clearer.

[go4cas]

@deividasjackus, one suggestion from my side on your "Remote nodes shouldn't be able to even list / check for existence of such services (not just call)" - should it not be better if there is a supporting property that will allow discovery of private services, but not being able to call them? One use case I can think of, is a monitoring/metric collection service. What do you think?

[DeividasJackus]

@go4cas perhaps I don't understand the exact use case, but it looks like an edge case to me. That is not to say that my own example is less of an edge case - it probably is, as well :)

For the sake of argument I'd counter that fully hiding them sounds a tiny bit better because of:

1. Better performance: it's less work for the transporter and other nodes (as the service endpoints don't end up in their routing tables).
2. Security: ability to hide implementation details.
3. If you have access to the node you want to collect metrics on, then you have the ability to collect those metrics locally and exposing the results through another, non-private service.

P.S. having said the above I'm no longer sure - perhaps $private would actually be the better setting :)

[pibi]

@deividasjackus private service actions looks like glorified methods. As with local events, it seems we are looking for explicit topologies and deployments, both are quite out of the scope of a "microservice development framework", isn't it?

From what I understood of your use-case and to propose a more "general purpose" solution, what about a "reversed" dependencies property to be used to limit the "visibility" and "callability" of a service? With this you could specify the "invoice" microservice to be visible only to the "shoppingCart" microservice, no matter their location

[DeividasJackus]

@pibi methods are in essence glorified themselves: you are able to take all the methods out of a service schema and put them in a separate package. Yet the feature is still there to allow for more streamlined packaging 😃

Using methods you are declaring the functionality to be only available within a component's (service) scope. Same way with a $private setting you would be declaring the service (a higher level component) as "internal" to the local context (node).

You are proposing a pattern of components controlling who can call them. Translates to service coupling and you now having to also modify a component whenever there's a new dependent.

The whole notion behind the proposal was to continue promoting the bundling of services into single, deployable microsystems made of components designed in the same fashion (as Moleculer microservices), but giving more control over an individual component's exposure.

[brad-decker]

I would be in favor of scoping what brokers are allowed to call a service, or even on a per action basis, as this allows a fair bit of more control over access levels. It is, however, my opinion that if you are creating a service with the intent of it not making its actions public at all to anything outside its local process that it doesn't belong as a moleculer microservice. The service is entirely self contained and should be written as such.

Can one of the proponents of this feature better detail a use case where having the code in the proposed private microservice makes sense to be declared with moleculer?

[pibi]

@deividasjackus I agree that _microsystems_ made of components should be isolated and internal calls should not leak outside its boudaries. The issue is that you are thinking of a _microsystems_ as a single node for the sake of optimization and visibility. I think that @icebob already solved the former by calling local service first, and the latter should be resolved with a location independent solution.

I agree with you that a whitelist of callers introduces an explicit coupling, but this is exactly what you are implicitly doing by "abusing" of the broker location awareness (_invoice_ on a node is coupled with its sibling _ShoppingCart_ . Same for the other, maybe internally different, _invoice_ microservice on a different node with its _MerchantSubscription_ sibling).

My 2 cents working within a large team: explicit is better than implicit.

[pibi]

Thinking about this, I know moleculer has the notion of _event grouping_. Instead of a whitelist, we could extend the group concept to the service layer:

module.exports = {
    name: "invoices",
    group: "shopping",
    actions: {
        create() {}
    }
}  

which limits the visibility and callability (both actions and events) to the services on the same group. (edited: group looks the right fit for a mixed architecture (here I see node#4 and node#5 being in the same group) )

@deividasjackus how does this sound to you?

[pibi]

Working on this: I think it's better to introduce a groups concept, because a service should be able to call or receive requests and events from outer groups. therefore, a ShoppingCart service should be:

module.exports = {
    name: "invoices",
    groups: ["shopping", "api", "*" ],
    actions: {
        create() {}
    }
}

where "*" means the service is able to call, be called and receive events from any other service.

[DeividasJackus]

@brad-decker I hear you, this is an edge case at best. But a component doesn't necessarily have to expose functionality to the entire system in order qualify as a microservice. It might as well just call other services or passively listen to events within the system. I'd like our developers to be able to build components of the same shape regardless of how passive they might be from the start.

As far as I am aware (please correct me if I'm wrong), as of right now a service monitoring the local node's health with 0 non-locally useful actions (but possibly having some health data intake actions for local services) still shows up in the routing tables of every single other node. I'm lobbying for the right to make that behavior configurable. WDYT?

@pibi I now see we had two different features in mind, would you agree? I really like the grouping proposal!

[pibi]

@deividasjackus not at all: your _private_ invoices service become a component of the same group of ShoppingCart. The "local" optimization is achieved by explicitly load the service within the same node. So, your feature is one of the use cases covered by _groups_.

See another point here: in a near future, your ecommerce become hugely successful and you need to scale out your invoices service on multiple instances. With _groups_ you can do that transparently, with _private_ you need to be sure you are not going to clash with other _invoices_ deployed somewhere, or maybe you have to modify your logic.

[brad-decker]

I think if you have the need to call services from a process but don't have anything on that process that is needed by anything else on the network you're better off using a instance of ServiceBroker without any services to call those services instead of implementing a private service just to stick to the same development patterns.

We have a couple of microservices that operate in our ecosystem that do not expose public actions but instead operate on a recurring pattern making updates to tables or syncing data. We don't use moleculer at all for these services but we could by utilizing a broker that can call other services. I still believe that a private true/false flag will be confusing to 99% of the engineers who use this package, and doesn't offer enough real world value to be a part of the public documented api.

I like @pibi 's suggested approach. This appeases a great deal more edge cases then the one provided.

[DeividasJackus]

@pibi I feel we are talking past each other. I am all in for the grouping proposal - separate from $local setting. The case I brought up was a monitoring service meant for the local node and that node only. Grouping would only solve that if all local services are added to a completely separate group unique to that node. Are we on the same page?

@brad-decker that is your choice. We split all logic into services and are committed to following that pattern. We have our own variant of moleculer-runner which is responsible for loading all services and bootstrapping the broker before abstracting it away from the services. I recognize both approaches being valid. Please, let's stick to the original issue - if you want to discus what should constitute as a microservice, feel free to open a new one.

[brad-decker]

The point wasn't go into a debate about what is and what isn't a microservice, my apologies if i wasn't clear. My point being that you can still access the microservice network via a broker without adding services to that broker. The only thing i can think of for a use case to define a service in this way is if you want to subscribe to events from the network and provide no actions on that service but simply react to it.

Even then doing what I just proposed works for that as well, just simply do not declare an action on the moleculer.createService call, add an event listener.

In your original case about having services able to see each other on the private scope the group setting makes the most sense to allow those services to communicate with one another.

[pibi]

Grouping would only solve that if all local services are added to a completely separate group unique to that node.

@deividasjackus exactly, so I don't see why you need $private ( or $local) anymore.

About the @brad-decker points: yep, an event-only service could do the same as $private, but I'm quite sure you can also use an "internal" ServiceBroker for all the "private" actions. It is a bit more complex but it is quite like having "subnodes" which cannot talk externally but through a "bridge" service. Again, this is a matter of deployment and I would rather be explicit about the "grouping" concept

[icebob]

Hi guys,
I read the conversation and I think I leave this issue open to further conversations about $private/$localOnly service property.

On the other hand, @pibi please open a new issue about your proposal (service grouping) and we can discuss it because currently I can't see it completely yet.
Thanks!

[pibi]

Hi @icebob. The service grouping is a location independent alternative for$private/$localOnlyservice property. It should preserve the _limited visibility/callability_ requirement of a group of services but without the need to load the services on the same node.

To open a new issue for discussing this I would rather know from @deividasjackus his thoughts about the differences between $private/$localOnly and the service grouping proposal.

[ColonelBundy]

I'd love to be able to define an action as being private per action basis.
For instance:

actions: {} - Public actions          

actionsLocal: {} - Private actions (can't be called by a request unless it's directly called by broker.call)

And if you'd want a private service you wouldn't use the standard actions
This is how I would love to see it being done.

Another implementation

would be to set a boolean key in the context such as ctx.isRequest and would be true if this was triggered by a user request such as /v1/auth/xxx and be false if it was called from broker.call('v1.auth.xxx')

Working hack for private actions

Name your action with a prefix such as p.XXX
then whitelist your whole service with a regex to exclude p.XXX example : /(service.[^p.+]+)/g
to call a private action then you'd use broker.call('v1.service.p.XXX')

[icebob]

@ColonelBundy currently there is an undocumented action property named publish. If it is false the action is not publish with moleculer-web. Is it good for you?

For isRequest you can set it in the onBeforeCall hook: Example

[ColonelBundy]

@icebob if I set publish: false I will still be able to use broker.call right?

[icebob]

Yes of course. Just moleculer-web handles this property.

[ColonelBundy]

@icebob Thank you! 👍