Loopback: Add support for audit trail

[Amir-61]

Bug or feature request

• [ ] Bug
• [X] Feature request

Description of feature (or steps to reproduce if bug)

Add support for audit-trail middleware.

Can we put this in our agenda and plan it? If yes, I'm willing to work on it...

/to: @bajtos @raymondfeng

[bajtos]

@Amir-61 could you please extend the description with more details? What data should be stored in the audit and for which operations?

[Amir-61]

Hey @bajtos,

could you please extend the description with more details?

tbh this feature requires some internal discussions; but as a high level description I would like to introduce audit-trail middleware where:

• We give the users the option which RESTfull endpoint (or which model) they want to have the audit middleware active for. Or introducing exceptions option which makes the middleware active for all endpoints except the ones mentioned in exceptions
• We give them option which req.methods they want to have this middleware active for... I think it does not make sense to have this middleware active for req.method == 'GET' since this middleware would become a bottleneck of the application and you would not get that useful information either.

What data should be stored in the audit and for which operations?

I'm thinking of this info to be logged:

• timestamp
• method: req.method
• path: req.path
• model
• ip: req.ip
• ips: req.ips
• user: user id of the one who called the endpoint; it can be anonymous depending on ACL for that model
• PK of the model instance
• ...

Please note this is the bare minimum I could think off the top of my head and it definitely needs some internal discussions

[bajtos]

Thank you for the details, @Amir-61, I have a better understanding now. How do you envision to store the audit data? How much flexibility in the configuration do we need?

[bajtos]

I am also wondering, is this something we have to implement in LoopBack (and keep maintaining)? Another options is to provide an example showing users how they can build their own auditing middleware, tailored to their specific needs.

[archenroot]

Why is this closed? I am comming from enterprise SOA and microservices stack and common approach I apply almost everywhere is as following:
in every service pipeline first action is to do Audit (type of ENTRY) and last action Audit (type of EXIT), anything between happen is Audit (ERROR). Of course everyone is interested in different kind of audit information.
In my case it is quite large mixture of pure technical information and those with business context:

• messageId
• correlationId (if not pure CRUD API, but service which is part of complex service chain)
• client id and caller app identification
• domain
• subdomain
• task
• service (static description of service)
• interaction (inbound context - what component issued the call, outbound context - what target is going to be called by exit)
  etc... we usually capture about 50 parameters (including optionally to capture the payload itself - in case of "message tracing" is on) with some kind of key/value model for easy extension in case of special requirements for service.

The architecture I do is usually combination of synchronous REST API (for CRUD operations or as complex process proxy endpoint - for external clients), or fully asynchronous services which doesn't expose any API over HTTP, but only communicate via message broker - here for example previous and future interaction is important, so you can reconstruct complete flow trough number of services from both technical and business perspective easily.

So I am fan of idea just introduce some generic extendable audit system in loopback which can be extended. In my case it has impact on generated REST API services by loopback as I would like to define common header should loopback use and enhance every service REST request schema with.

Thanks for continue with discussion on this item.

Having ENTRY/EXIT/ERROR is also very easy to use for service health check, there always need to exist pair of ENTRY-EXIT or ENTRY-ERROR on any service in audit log, if not you know something is wrong very quickly.

[bajtos]

in every service pipeline first action is to do Audit (type of ENTRY) and last action Audit (type of EXIT), anything between happen is Audit (ERROR).

I think this should be possible to implement using our existing beforeRemote, afterRemote and afterRemoteError hooks.

I would like to define common header should loopback use and enhance every service REST request schema with.

This is a missing feature we are aware of. I am not able to find the specific issue keeping track of it, the closes one I found is this: https://github.com/strongloop/loopback/issues/3116

[archenroot]

During my research on this, I have found following project today. I didn't have yet look much, but maybe:
https://github.com/yantrashala/loopback-audit-trail

Will give it a try.., but looks very basic and not much as what I expect from basic look into code... I raised a question there:
https://github.com/yantrashala/loopback-audit-trail/issues/5

[archenroot]

I have prepared some schemas (converted from older times when on worked for IBM on SOA implementations) just to give you better idea what I am talking about

audit-trail-schemas.zip.zip

[archenroot]

Following my research I found out that actually Loopback already does something similar, but in different place :-)
There is implemented context in product https://github.com/strongloop/microgateway

Microgateway actually is having some of attributes mentioned by me to capture in audittrail and they call it middleware context, the difference is that such context is for example on Oracle Service Bus available directly within the service pipeline queryable at any time, the context also changes during pipeline processing depends if you are in inbound, or outbound stage already. Imagine your input is message queue and output is database table, then in OSB you have transport related information regarding message broker available in inbound part of pipeline and database not, later once you get into outbound half of processing, the message broker context is rewriten dynamically by context data of target, in this example by database.

So in OSB you do (example synchronous sequence flow)

1. request stage
   -- get message header from data object message and dynamic context (depends on transport protocol for instance) and construct ENTRY auditlog - pass to queue
   -- process request (validate it, transform, do external service callout, etc.)

1. response stage
   -- construct response data object message, do additional processing if required
   -- get input message header from request data object message (available in the global pipeline context or stored in global variable) and enhance it with dynamic outbound context (depends again on transport protocol for intance) and construct EXIT auditlog -> pass to queu
   -- reply data object message back to client (REST/SOAP) or pass to reply queue if asynchronous pattern in place

ADD: In case of any failure occured in error tracker you submit ERROR audit log message

So this is one of best case I implemented on multiple projects on OSB 10g,11g and 12c, now how I see the loopback possibility:

1. There could be some framework within loopback implemented (need to be generic), so I generate REST API from Postgres from schema (as I usually do) and I can enhance the generator with Audit functionality described above. I am really not sure if you consider such kind of auditing capabilities or not in the moment at IBM.

2. Maybe microgateway could be the answer
Microgateway is actually having probably some capability to do audittrail because(how I see it):

• it implements from description "middleware context" and this is something I refer to as inbound/outbound context from available in OSB - I am personally missing here inbound and outbound context different, there is simply just one context, so I am not yet 100% sure how generic and robust is such context.
• it can transform data from JSON, XML - so i assume it could be capable of extract from the request the AuditHeader part and pass it into the message queue/topic for all instances like ENTRY, EXIT, ERROR as I am looking for.

Conclusion: By using microgateway in this way I am actually introducing using it as wrapper or container for the real database backend loopback service, so the flow would be like analogically to OSB flow like this:

1. microgateway endpoint (receive request)
   -- extract custom header from message, enhance by middleware context -> pass to audit queue
   -- do request to loopback backend service with pure data message request (no custom header)
2. loopback database backend service endpoint (do data)
   -- received request
   -- do data manipulation
   -- response back to microgateway
3. microgateway endpoint (reply response)
   -- do EXIT or ERROR audit report to the queue
   -- response to client call

This will move audit-trail completely out of loopback, but I am not sure if it is not overkill. Secondly, the timestamp of audit trail ENTRY and EXIT is not related to backend (data service), but to some wrapper only which in this case microservice will be used for. But maybe that is fine... I decouple and do auditing independently of data manipulation processes :-) I am not sure about this in the moment yet.

One drawback is that I am now working on Consul and HAProxy DNSMasq, etc. for dynamic service discovery running on Mesos (Marathon, Chronos, Metronome) and I use it not only for service layer, but databases, other microservices in Java, ERP and other applications, .etc all running in Docker. So I will need to look how and if easily I could integrate microgateway into my stack.

Anyway I think it is worth look at microgateway, as soon as I can keep the "internal" loopback services working in standard way and keep any customizable auditing outside of postgreSQL generated APIs on "external" microgateway layer.

Any hints on this are welcomed.

[bajtos]

Following my research I found out that actually Loopback already does something similar, but in different place :-)
There is implemented context in product https://github.com/strongloop/microgateway
(...)
Microgateway is actually having capability to do audittrail
(...)
This will move audit-trail completely out of loopback.

@raymondfeng could you please chime in and help @archenroot to figure out what is the recommended way for setting up the microgateway to act as and audit logger?

If this proves to be a viable solution, then I think we should describe our recommended setup at http://loopback.io/doc/. cc @ritch @crandmck

[crandmck]

If this proves to be a viable solution, then I think we should describe our recommended setup at http://loopback.io/doc/.

Sure... If so, please open an issue in https://github.com/strongloop/loopback.io. Thanks.

[archenroot]

Thanks community and IBM guys to discuss this, I will publish solution design for this to support in Java based applications. As I use loopback as generated API layer for any SQL database I have here, that is the reason I target it for auditing as well.

I have found https://www.npmjs.com/package/log4js-loopback package. Auditing in Java is also implemented by log4j2 Markers, Custom levels and also Stream Appender (for adding not only text message, but more into log) I will try to align those 2 worlds to support same.

I will publish later on our gitlab and post link here to review (scope for next few weeks).

Thanks again.