Loopback-next: Spike: How to write a LB3-like LB4 application in JavaScript

[dhmlau]

Description / Steps to reproduce / Feature proposal

In the spike,
1. Find out the following, focus on what can be done right now without changing the code too much:
- how to create and boot application class in javascript
- how to register route
- how to customize the sequence without dependency injection
- data access should not depend on LB Repository. Options: 1/ Juggler + connectors. 2/ use db driver directly (e.g. mysql/mongodb modules)

1. Identify pain points

Connect to #560

Out of Scope

• Dependency injection
• use of decorators
• use of Repository and service proxy

_See Reporting Issues for more tips on writing good issues_

[rubys]

Suggestion to make this a bit more concrete: add a --js option or the like to the lb4 command. So, for example, lb4 app --js. The generated code might be less developer friendly, and there may be a minor loss of feature (e.g. dependency injection), but this (a) eliminates the objection, and (b) enables developers to see the benefits of typescript.

[iamjamilspain]

I agree, I would not mind helping out on this one

[hacksparrow]

Created a POC - https://github.com/strongloop/loopback4-example-javascript.

[hacksparrow]

Branch stage-f is the latest optimization in an attempt to create a JavaScript API for LB4.

Findings

1. We can get a LB4 app running using a JS setup (pretty obvious). But It is hacky.
2. Even if we manage to optimize the code further, it would be difficult to come up with an intuitive and easy to use JS codebase, because we are trying to fit in regular JS code to an underlying TS-centric paradigm; they don't fit easily.
3. However, it is technically possible to develop a JS API for the underlying TS LB4 framework.

Notes

1. Users asking for JS version of LB4 are:
   a. LB3 developers who want to upgrade without having to use a different language
   b. JS users who want to use LB4
2. LB3 developers would like the familiarity of the LB3 ways.

Suggestions

1. Although not optimal, show how to use JS with LB4 as shown in https://github.com/strongloop/loopback4-example-javascript, and move on.
2. Create a JS framework with LB4 under the hood.
   a. It would come with its own packages and commandline tool.
   b. The DX would be very close to LB3.

[hacksparrow]

Some notes from the spike meeting. It was decided that we focus on usecase scenarios instead of 100% porting LB4 API to JS.

Plain JS for SOME parts for the app:

1. Compose LB4 app into existing Express app
2. LB4 controller written in JS
3. Front facing LB4 API for a backend API
4. OpenAPI specs for Express routes - request argument parsing with LB4

Technical implementations to figure out:

1. How to define models, additional metadata for model properties?
2. How to define controllers, how to provide OpenAPI spec for controller methods?
3. How to do dependency injection, and how to provide arguments?

[bajtos]

Just a quick summary of a chat I had with @hacksparrow on Slack.

My expectations were not matching the goal of this spike as written in the issue description.

Find out the following, focus on what can be done right now without changing the code too much:

• how to create and boot application class in javascript
• how to register route
• how to customize the sequence without dependency injection
• data access should not depend on LB Repository. Options: 1/ Juggler + connectors. 2/ use db driver directly (e.g. mysql/mongodb modules)

Out of Scope

• Dependency injection
• use of decorators
• use of Repository and service proxy

I thought that we are looking for a way how to expose most of LB4 features to JavaScript developers.

After reading the description, I see that the scope of this spike is much smaller. As I understand that scope, we are looking for a solution to support developers upgrading from JavaScript Express-based application to LB4, one that will allows them to write their Express-style routes in LB4 style and get the benefit of OpenAPI-driven request validation & API docs. A small incremental step on their journey from Express handlers to LB4 controllers (eventually).

An important requirement that's not mentioned in the spike description, but that applies to all our spikes for JavaScript support:

I would LoopBack users to write idiomatic ES6 code. They should use class syntax to define new classes, not class factories as we used for models & datasources in LB3.

To make sure the spike is covering enough complexity of real-world applications, I am proposing to use the following scenario to drive the work:

As a developer using JavaScript to write LB4 application, I want to create a route that will

• accept a filter parameter describing the database query in LoopBack format,
• call find method provided by juggler's DataAcessObject for CRUD operations (e.g. PersistedModel.find),
• return an array of models matching the query.

I want to setup the route in such way that LoopBack will validate the value provided by the caller for the filter parameter.
Ideally, I'd also like to use LoopBack-provided helpers to build filter schema and response schema from model definition.

[bajtos]

how to customize the sequence without dependency injection

Here are few approaches to research:

(1) Access the implementation of individual sequence actions directly by calling require. This may not work if actions are already requiring dependencies to be injected.

const parseParams = require('@loopback/rest').parseParams;

(2) IIRC, if you don't provide any constructor, then your class will inherit the constructor from you base class. I think @inject metadata is inherited too, but I am not sure. So maybe the following can work?

class MySequence extends DefaultSequence {
  handler: async function (context) {
    try {
      const { request, response } = context;
      const route = this.findRoute(request);
      const args = await this.parseParams(request, route);
      const result = await this.invoke(route, args);
      this.send(response, result);
    } catch (err) {
      this.reject(context, err);
    }
  }
}

(3) Use Service locator pattern to resolve dependencies. Service locator is an older pattern that is considered as an anti-pattern in modern object-oriented design, but it may serve us well until we find a JavaScript solution for Dependency Injection.

class MySequence  {
  handler: async function (context) {
    const route = await context.get(RestBindings.SequenceActions.FIND_ROUTE); 
    const parseParams = await context.get(RestBindings.SequenceActions.PARSE_PARAMS);
    // etc.

    try {
      const {request, response} = context;
      const route = findRoute(request);
      const args = await  parseParams(request, route);
      const result = await invoke(route, args);
      send(response, result);
    } catch (err) {
      reject(context, err);
    }
  }
}

[bajtos]

nabdelgadir changed the title Spike: How to write Express-like application in JavaScript Spike: How to write a LB3-like application in JavaScript on Nov 13, 2018

@nabdelgadir @dhmlau do you remember why was the goal of this spike changed from Express-like to LB3-like applications?

I am puzzled about why would we want developers to build LB3-like applications with LB4? It makes sense to me to build Express-like applications, but why LB3-like?

[hacksparrow]

@bajtos thanks a lot for adding more clarity.

[hacksparrow]

@bajtos pasting this from Slack:

Uh oh, I had very different understanding about the goal of this spike. The scope described in 1978 is a valid one, but it means JS users will have access to a very limited subset of LB4 functionality. I agree with you that we should create another spike story (or more) to investigate those out-of-scope items.

As I see it, the scope described in 1978 is useful for people upgrading from JavaScript Express-based application to LB4, it allows them to write their Express-style routes in LB4 style and get the benefit of OpenAPI-driven request validation & API docs.

When I asked if we are looking to implement 100% support for LB4 APIs in JavaScript, you replied:

What I see as important: allow JS developers to write controllers and have access to request metadata that's currently accessed via dependency injection in TypeScript

With that context, can you elaborate on what you are referring to as "request metadata"?

As mentioned by you:

I think support for Controllers is a good scope for the next spike after 1978 is done.

This one's just for capturing information for the subsequent spike.

[hacksparrow]

What's been done WRT the goals in the description:

_Find out the following, focus on what can be done right now without changing the code too much_

a. How to create and boot application class in javascript:
- The Lb4Application class
b. How to register route:
- Just use this.route()
- Miroslav suggests to add support for a more complex scenario, like accepting a request param and sending back a model.
- Follow up task to register routes with DI, which should be preceded by a spike on DI or an alternative to it.
c. How to customize the sequence without dependency injection
- Create custom sequence
- Miroslav suggests to use idiomatic ES6.
- Follow up task to create custom sequence. We need not use DI, as long as the purpose DI serves is achieved using other ways.
- Follow up task to create sequence using idiomatic ES6.
d. Data access should not depend on LB Repository. Options: 1/ Juggler + connectors. 2/ use db driver directly (e.g. mysql/mongodb modules)
- Creating a model
- Using model factory it is very easy to create a model.
- Why make it harder by not using the underlying repository, so let's just use it, users won't have to interact with them directly, anyway.
- Miroslav suggests to use idiomatic ES6.
- Follow up task to create models using idiomatic ES6.

_Identify pain points_
- Porting 100% LB4 APIs to JS may not be easy or will take a lot of work.
- More can be identified from the follow up spike and tasks.

We should probably close this story now, because its scope and goals are ambiguous and confusing; and we have identified more clearer follow up spike and tasks.

[hacksparrow]

@strongloop/loopback-next thoughts?

[dhmlau]

IIUC, the goal for this spike is to find out:

• if it is possible to use JavaScript in order to use LoopBack 4
• what are the things we can possibly do to make the dev experience easier

It was under the assumptions/hypothesis (to be verified in the spike) that:

• someone can use plain JS to do basic stuff in LB4. If that's the case, we should document it and tell users how it can be done.
• for more advanced features (e.g. dependency injection), it is not possible out of the box or unless we add support. But it won't be the scope of this spike and won't be the priority for us in the short term.
• we still want to encourage users to move to TypeScript at some point because of the benefits of using TS and LB4 is a TypeScript-first framework.

Therefore, it seems to me that the spike is complete (without the need to dive into the advanced features).

To be honest, I forgot why we've changed from Express-like to LB3-like.

[bajtos]

a. How to create and boot application class in javascript:

• The Lb4Application class

I love this part, that's how I would like to write LB4 apps in pure JavaScript 👍

I'll review the rest later this week. Sorry for the delays.

[bajtos]

As I see it, @hacksparrow has researched two different approaches for supporting JS in LB4. We have certainly learned more about the problem domain 👍

My problem with the current status is that neither proposal offers a solution with a great user experience, I'd like us to try harder to find an approach that JavaScript developers will love to use.

Also let me remind us that a spike is considered as done when there are follow-up stories created and the acceptance criteria for these stories are clear enough & approved by the team or at least the leads. If we close this spike as done, then we are in a similar position as before this spike started - we don't know how JS support should look like and don't have any clear path to get there either.

I agree that this spike story has pretty uncertain description and it's not clear what is in and out of scope. I like the idea of creating more focused & better defined follow-up spike stories to continue with our research.

So before we call this spike done, I'd like @hacksparrow to:

1. Propose a list of next tasks (spike stories) in enough detail so that they can be later understood and estimated by the team.
2. Discuss this proposal with @bajtos and @raymondfeng and reach consensus on which stories we want to put in our near-term backlog and which we will leave out for now, and also get agreement on the proposed acceptance criteria. Preferably start with a written form that will allow us to do the first few rounds asynchronously.
3. I think it would be great to get the rest of the team to review the proposal too, to ensure we are all on the same board and anybody from the team can pick up the work where we leave it after this spike.
4. When the discussion is over, create new GitHub issues for the follow-up spikes and stories as agreed in the previous steps. Remember to assign the new issues to the appropriate Epic and link them to this spike story/GH issue for future reference.

[bajtos]

What I see as important: allow JS developers to write controllers and have access to request metadata that's currently accessed via dependency injection in TypeScript

With that context, can you elaborate on what you are referring to as "request metadata"?

Sure. By request metadata I am referring to data bound to per-request context, typically current request/response objects and more importantly metadata contributed by extensions like `@loopback/authentication (e.g. the currently logged-in user).

[hacksparrow]

@bajtos @raymondfeng @strongloop/loopback-next here is the proposal for the followup tasks. Anything needs to be clarified or you'd like to be added, do let me know.

1. Spike: Dependency injection or its alternative in JSLB4

_Acceptance criteria:_

a. Should be demonstrated in a class
b. Should be used with JSLB4 Application class
c. Should be able to read properties from the LB4 context
d. Should be able to bind new properties the LB4 context
e. Should be able to unbind properties from the LB4 context

2. Spike: Create Route in JSLB4

_Acceptance criteria:_

a. Should be created as a class
b. Should be useable with JSLB4 Application class
c. Should have access to:
i. The LB4 request object and metadata contributed by LB4 components eg: @loopback/authentication
ii. A Model
iii. The LB4 response object

3. Spike: Create Sequence in JSLB4

_Acceptance criteria:_

a. Should be created as a class
b. Demonstrate usage with JSLB4 Application class

4. Spike: Create Model in JSLB4

_Acceptance criteria:_

a. Should be created as a class
b. Should be automatically loaded in the LB4 app
c. Should be able to describe model properties
d. Should show up on Explorer and should be successfully interactive

5. Spike: Create Repository in JSLB4

_Acceptance criteria:_

a. Should be created as a class
b. Should be automatically loaded in the LB4 app
c. Should have access to a specified Model
d. Should expose CRUD methods to interact with the Model
e. Should allow custom methods to be added, with access to:
i. The LB4 request object
ii. A Model

6. Spike: Create Controller in JSLB4

_Acceptance criteria:_

a. Should be created as a class
b. Should be automatically loaded in the LB4 app
c. Should have access to a specified Repository
d. Should expose CRUD methods to interact with the Repository
e. Should allow custom methods to be added, with access to:
i. The LB4 request object
ii. A Model
iii. A Repository
iv. The LB4 response object

I think, once Spike 1 is sorted out, everything else will come along smoothly.

[dhmlau]

@hacksparrow, thanks for breaking the tasks down. Which one(s) you think we should get it done by Q1? It's part of the release planning/pruning we did yesterday.

cc @bajtos @raymondfeng

[hacksparrow]

@dhmlau for a practical LB4 JS API, all of them.

[dhmlau]

@bajtos @raymondfeng, probably need your inputs here as well. Had a discussion with @hacksparrow today. Here are my questions/thoughts:

• What is the MVP we can deliver? Are the 6 spikes above the MVP? The MVP I have in mind is that JavaScript users can do something meaningful with LoopBack4 using plain JS.
• As @raymondfeng mentioned before, LB4 is a TypeScript-first framework. I think we can have the MVP done in a shorter term and tell our users upfront about what is supported and what's not. We can then test the water to see how popular this plain JS experience, or can also encourage community contributions to help us to reach the full support.

What do you think?

[bajtos]

Good questions, @dhmlau. The 6 spikes above are addressing different technical aspects (road blocks) that are preventing LB4 users from writing their applications in JavaScript. They don't necessarily lead to an MVP solution - some of them may be beyond the scope of MVP, we may discover more missing pieces when we start putting together a cohesive MVP story.

I see the following challenge this research: we need to work both at high-level design to find a solution that's easy to use and attractive to JS developers, and at the same time keep in mind low-level technical limitations imposed by JS.

At high-level, we already have several tools and multiple different ways for designing the applications. Some of them may work better in JS, some will not work at all.

Few examples of different high-level approaches we haven't looked at yet:

To define the REST API, one can use design-first approach where the API is described in openapi.json (or openapi.yaml) file, provided to the application via app.spec, and then the endpoints described in the spec are linked to individual controller methods/route handlers. LB4 defines extension fields for that: x-handler, x-controller-name + x-operation-name. The downside of this approach is that parameter descriptions are far away from the function accepting them (spec file vs. source code file).

Dependency management can be implemented via Service Locator pattern. I am personally not fan of this approach, it makes it too easy to create highly-coupled code that's difficult to test in isolation, but if it makes JavaScript code easier to write & read compared to DI alternatives, then it's probably a trade-off we should accept.

What I am trying to say here is that we haven't explorer all options already offered by the current LB4 code base, not to mention new options that will require only minimal improvements.

From MVP perspective, I am proposing to use our examples/todo application (CRUD only, no Geocoder service integration) as the target MVP application we want to build using pure JavaScript.

If it makes things simpler, I am fine with leaving out customization of the Sequence. AFAICT, the Todo example app is working well will the built-in default sequence.

Now the question I am not able to answer without further research: considering developer experience but also implementation effort required in our runtime, what is a better approach for building the Todo example app: use handler-based routes via app.route(verb, path, spec, handler) or controller classes? While this seems like an easy question, there is more to it.

Take GET /todos?filter=... for an example.

• We need to describe the filter parameter and the return value (response body). We don't want to write the schema by hand, it should be generated from our Todo model. How are we going to wire that up?
• The implementation needs to call todoRepository.find(filter). How do we obtain the repository instance? The repository instance needs a DataSource instance, how is that wired up?
• Are we interested in unit-level tests in JS, similarly to what we have in TS? If yes, then how can we mock the repository instance used by the controller or the handler function?

A comment on a side: I have an idea for an API allowing route handlers to receive dependencies via DI. It's not a solution for DI in general, but may work well for this particular case.

app.route(
  'get',
  '/todos',
  ['repositories.TodoRepository'], // a list of dependencies
  spec, // describe filter arg & return values
  async function findTodos(TodoRepository, filter) {
    // ^^^ dependencies first, spec parameters second
    return TodoRepository.find(filter);
  });

Because the number of arguments is growing out of hand a bit, a new object-style may work better:

app.route(
  verb: 'get',
  path: '/todos',
  inject: ['repositories.TodoRepository'], // a list of dependencies
  spec, // describe filter arg & return values
  handler: async function findTodos(TodoRepository, filter) {
    // ^^^ dependencies first, spec parameters second
    return TodoRepository.find(filter);
  },
});

[bajtos]

here is the proposal for the followup tasks. Anything needs to be clarified or you'd like to be added, do let me know.

@hacksparrow please copy this proposal into a markdown file and commit it into a git. I find it difficult to discuss changes in content that's posted as a comment only.

[hacksparrow]

@bajtos created this PR https://github.com/strongloop/loopback-next/pull/2404 with the file.

[hacksparrow]

Links to the follow up tasks for this spike:

1. https://github.com/strongloop/loopback-next/issues/2473
2. https://github.com/strongloop/loopback-next/issues/2474
3. https://github.com/strongloop/loopback-next/issues/2475
4. https://github.com/strongloop/loopback-next/issues/2476
5. https://github.com/strongloop/loopback-next/issues/2477
6. https://github.com/strongloop/loopback-next/issues/2478

@bajtos can we close this spike now?

[bajtos]

Please create a spike or a story to create a JavaScript version of our Todo example application (minus GeoCoder API), as described in my earlier comment. I think we also need a follow-up task to modify our lb4 app generator to allow users to scaffold JavaScript applications.

I also posted few comments in the follow-up issues you created, please consider to include the information from those comment in issue descriptions/acceptance criteria.

• https://github.com/strongloop/loopback-next/issues/2474#issuecomment-467456485
• https://github.com/strongloop/loopback-next/issues/2476#issuecomment-467460777
• https://github.com/strongloop/loopback-next/issues/2475#issuecomment-467459894
• https://github.com/strongloop/loopback-next/issues/2477#issuecomment-467463138
• https://github.com/strongloop/loopback-next/issues/2478#issuecomment-467464431
• https://github.com/strongloop/loopback-next/issues/2473#issuecomment-467468041

Other than that, I am fine with closing this issue as done.

[hacksparrow]

Added spike for Todo example application- https://github.com/strongloop/loopback-next/issues/2501.

[hacksparrow]

Added issue for supporting the JS app scaffolding in lb4 app - https://github.com/strongloop/loopback-next/issues/2503.

[hacksparrow]

@bajtos good to close this now?