Nest: [Feature Request] Jobs

[roeehershko]

I'm submitting a...

[ ] Regression 
[ ] Bug report
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

Currently if i need a piece of code to run continuously i am creating a component and making the logic in the constructor

Expected behavior

Could be nice if we will have a "Job" type component which run jobs synchronously with options like concurrent, delay etc..

What is the motivation / use case for changing the behavior?

Could be cool and easier to implement instead of everyone writing there own jobs infastracture.

[kamilmysliwiec]

Hi @roeehershko,
Do you have any example + API proposal?

[roeehershko]

Yes, take a look at my current "Job", i removed to logic to make it simple, it takes 100 items from redis list then "Processing" it and finally it removes the item it processed, to give you a bigger picture - this is a tracking system which receive user events and then add it process the data and insert it to the DB in bulk operation 100 at a time, this is the original:

import {Component, Inject} from "@nestjs/common";
import {RedisClient} from "redis";

@Component()
export class AllocateSessionJob {

    private client: RedisClient;

    constructor(@Inject('RedisToken') client: RedisClient) {
        const self = this;
        this.client = client;
        setInterval(function () {
            self.process();
        }, 5000)
    }

    private async process() {
        const self = this;
        this.client.lrange('sessions', 0, 100, function (err, data) {
            // Process
            self.client.ltrim('sessions', 0, 100);
        });
    }
}

a "Job" worker can look like this:

import {Inject} from "@nestjs/common";
import {RedisClient} from "redis";

// Define job worker which has its own options such as concurrent
// or possible a port if we want to make it as separate service maybee
@JobsWorker({ concurrent: 5, port: 3001 })
export class SessionsJob {

    private client: RedisClient;

    // Define injections that will run only after job has started
    // my redisToken inject returning a promise that resolve after
    // redis has established connection
    constructor(@Inject('RedisToken') client: RedisClient) {
        this.client = client;
    }

    // Define a specific job inside the worker, which for example, set the delay after each
    // job is finished
    // In this example i am processing redis list ("sessions") fetching & removing 100 rows in the list
    @Job({ delay: 3 })
    private async processSessionsJob(next) {
        let self = this;
        this.client.lrange('sessions', 0, 100, function (err, data) {
            // Do someting...
            // Calling next will finish the current job process
            // Anther approch is to make the job return Promise, might be cleaner
            next();

            self.client.ltrim('sessions', 0, 100);
        });
    }
}

[fwoelffel]

I'm currently writing a Bull module for Nest; thought it would be good to share and get some feedbacks :)

[fwoelffel]

^ https://github.com/fwoelffel/nest-bull
Here it is. Feel free to share your impressions.

[kamilmysliwiec]

Nest supplies extensible, modular architecture that allows building such great modules as @fwoelffel has created. That's why I think that this kind of features shouldn't be a part of the core framework, but rather we have to provide a set of 3rd party modules that increase developers experience.

[et]

Nest supplies extensible, modular architecture that allows building such great modules as @fwoelffel has created. That's why I think that this kind of features shouldn't be a part of the core framework, but rather we have to provide a set of 3rd party modules that increase developers experience.

@kamilmysliwiec - Just some feedback on this. I have an extensive background in Rails land. What was nice about working in Rails is that the batteries are included and I don't need to plug and play things together. Furthermore, stuff like jobs (which could be an addon) are natively supported so you can be assured that things just work.

About a year ago, I started a job working on node projects with simple express apps that have no convention and take time figuring out how the previous developer set things up.
I've gotten a lot better with it but it took a lot of time.

Stumbling across nest has been a blessing for me. Now my team and I don't have to spend hours discussing where to put things. We've been using nest for about the last month and it's been absolutely splendid. However, we do have the need for jobs and I literally just googled "nest.js jobs" and stumbled across this thread.

It's not difficult for me to build out a job module. However, I'll feel slightly unsure if I did it in the "conventional way". And if I were to work on a different nest.js project set up by a different developer, he/she/they might set up the jobs module differently from me.

Obviously, you know what's best for the framework to keep it from being bloated. There's definitely something to say that "addons need to mature before they are added to the core framework". But once they become more stable, please consider adding them as a standard feature especially since almost all projects will need jobs at some point.

[cgarnier]

@kamilmysliwiec most of the web framework has this feature (Django, laravel, symphony, loopback... Even WordPress can). I think it should be a core feature. It s something required by most of the projects.

[marcus-sa]

If you're so lazy that you can't even bother to include a module manually instead of it being part of the core, then you shouldn't even use a "modular architecture framework" in the first place.
As simple as that.
Plus if Nest were to separate everything server related (going to do a PR once I've reflected the current behavior), then the proposal would have to be not part of the core features anyway.
Instead there should be a https://github.com/nestjs/some-name repository containing a lerna monorepo with all available modules, so that they can use the @nestjs scope on NPM aswell by simply doing a PR, then @kamilmysliwiec can merge it and publish it to the NPM registry.
Exactly the same way Microsoft does it with their @types scope.

[et]

If you're so lazy that you can't even bother to include a module manually instead of it being part of the core, then you shouldn't even use a "modular architecture framework" in the first place.

No need to be mean about it. The reality of the situation is that it's better if it's supported by core because it provides confidence that one can write a production app and that when major releases happen to nest, that everything will likely still just work.

While I believe that @fwoelffel will do a great job supporting their job module, it's hard to have confidence that it'll still work when v6 comes out. Even if there is an upgrade plan for 3rd party modules when v6 comes out, we don't know if @fwoelffel will have time to update their job module.

[BrunnerLivio]

Lets keep this discussion civil.

@et I think what you’re asking for is an implementation which is backed by multiple people / a trustable organization. I fully support and agree on this. Coming from a highly regulated pharma industry I wouldn’t want to rely on any package from “some” guy on Github either (not meant to be rude 😁).

What would you think about, if the jobs feature would be an additional module of the Nest org itself? That would mean it will have better maintainance and backed by a larger community but the Nest core would still stay as minimal as possible, so you can install the jobs package only if you actually need it.

I am not the person who will decide whats going to happen but maybe you, @kamilmysliwiec and @fwoelffel can come to a solution for your concern.

[cgarnier]

Just need an "official" way to do it. And a recipe in the doc.

[marcus-sa]

Wasn't meant to be harsh, my bad. Sometimes being honest might come out in a "mean" way @et

[kamilmysliwiec]

I don't think that we should put jobs module directly into core packages. However, a dedicated module under @nestjs/* organization may help in certain cases to get more confidence while using it in the production (and the guarantee that the API will always be compatible with the latest releases).

[fwoelffel]

I feel the same way as everyone about having a dedicated "official" module. That would probably help with this "jobs" module maintainability as it will give it more visibility and make it more legitimate.

[bashleigh]

I have some experience in creating a job packages. I made a package for my previous job (in PHP though :fearful:) that used serialised objects in a queued system with execute from datetime stamps etc. I even added middlewares so that the objects could be hashed, base64 encoded with options drivers etc.

I can imagine this being a class/object that we can store and rehydrate and call an execute method that requires a payload on a class that implements JobInterface with a method of execute. The problem is the transport and or storage. Laravel can use the database, the package I made used both AMQP and the database. I wouldn't want to make the decision of "We're all going to use these methods of transport" so I think I may need some help implementing different delivery method and scheduling (if scheduling is desired).

import {JobInterface, Job, BASE64, JSONENCODE} from '@nestjs/jobs';
import {ExampleService} from './example.service';

@Job()
export class ExampleJob implements JobInterface {
  public readonly name: string = "example";
  public readonly onQueue: string = "example_queue";
  public readonly middlewares = [JSONENCODE, BASE64]; //in desired order 

  constructor(private readonly exampleService: ExampleService) {}

  async execute(payload: RehydratedObject): Promise<bool> {
    await this.exampleService.doSomething(payload);
    return true;//for error handling or redelivering 
  }
}

Could even add the properties set in the above into the classDecorator and add reflect for each?
If you want I can start experimenting with this when I find some free time?
Job

export const Job = (): ClassDecorator => {
  return target => {
    Reflect.defineMetadata(SOME_JOB_CONST, true, target);
  };
};

import {Controller, Get} from '@nestjs/common';
import {JobService} from '@nestjs/jobs';
import {ExampleJob} from './example.job';

@Controller('job')
export class JobController {
  constructor(private readonly jobService: JobService) {}

  @get('example')
  index() {
    this.jobService.dispatch(ExampleJob, {key: "somevalue"});// dispatch(provider: Provider|ComponentMetadata, payload: any): void
     //ExampleJob is passed in to get the name of the job to store against. This could also be an extended class 
    //so we could do `ExampleJob.dispatch(payload)` instead using a static method
  }
}

Requirements

• middlewares for serialising/modifying
• singular handler for receiving jobs and distributing to @Job classes
• 'Rehydrate' object using their property type using the payload
• dispatch jobs using a JobService
• schedule jobs (if possible)
• drivers for sending/storing serialised messages
  
  
  
  • AMQP driver
  
  
  • Database driver
  
  
  • NATS driver
  
  
  • Redis driver
  
  

Another question would be how would the drivers be installed in this situations as it would mean installing different required packages. npm i --save @nestjs/jobs @nestjs/jobs-amqp-driver?

[bhoudu]

Maybe integrating other libraries can be considered for this feature instead of starting from scratch? I'm thinking about Bull which has many features (probably long to implement again, but has fixed requirements such as Redis only) : https://github.com/OptimalBits/bull

[RDeluxe]

We are using Bull a lot in production, and this is a solid piece of software. Having a small @nest/nest-bull module could be nice, even if - tbh - I don't really see the point.

[fwoelffel]

I'd move the nest-bull repo to the nest organisation with pleasure :slightly_smiling_face: I'm convinced this would help with its visibility and its amount of maintainers.

[fwoelffel]

I've spend a few hours lately on nest-bull and I believe I'll soon be releasing 1.0.0 :crossed_fingers:
Feedback/help would be greatly appreciated, especially with the tests and documentation.

[udanpe]

@fwoelffel After update to 1.0.0 nest-bull will be move to nest repository or it stay be unofficial?

[fwoelffel]

I guess it'll stay on my unofficial repo for a while, just to make sure everything is ok for everyone currently using the module. Then it will be @kamilmysliwiec's call to make this official or not.

[ps2goat]

bump

[cgarnier]

I m using nest-bull. It does the job ;)

[ps2goat]

Why aren't these responses showing up on the issue thread?

Anyway, I just wanted a standard way to create a daemon app using NestJS,
minus the Express functionality. I like how NestJS is similar to Angular,
so that it's a similar experience whether coding the server or the front
end. I wanted the same experience in a daemon, using the modules and
dependency injection from NestJS.

I wasn't looking to queue up several sub-jobs/tasks this way, though this
is also interesting.

On Tue, Sep 3, 2019 at 9:19 AM Khoa Hoàng [e-mail] wrote:

[image: image]
https://user-images.githubusercontent.com/49465776/64180077-5f3caa80-ce8e-11e9-8cab-e18b39c9b196.png
How about this?

I've written task-managements library based on NestJS + BullJS

• Just add @task https://github.com/task(options) for any functions
  in service which you want to execute it through bullJS (distributed, queue,
  tasks)
• Trigger task by:
  this.injectedService.functionHaveTaskDecorator(...args)
• Supported scoped injection
• Supported auto triggers scheduling based on cron string
  [image: image]
  https://user-images.githubusercontent.com/49465776/64180783-91024100-ce8f-11e9-88e8-59546bd1aee8.png

[image: image]
https://user-images.githubusercontent.com/49465776/64181167-34ebec80-ce90-11e9-97fc-ab829b944dc7.png
[image: image]
https://user-images.githubusercontent.com/49465776/64181298-682e7b80-ce90-11e9-8ddc-9b7e7f7e8619.png

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nestjs/nest/issues/404?email_source=notifications&email_token=ABJCUHA2A72MOVGD4JD3ZPLQHZW6XA5CNFSM4EPT3VAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5YLE6A#issuecomment-527479416,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABJCUHCTWM3VTKNGWVJFZJTQHZW6XANCNFSM4EPT3VAA
.

[kamilmysliwiec]

hey @fwoelffel, could you send me a message on Discord? If you're still interested, we should consider moving this package under the official org shortly :)

[kamilmysliwiec]

Bull package has been move under the official organization https://github.com/nestjs/bull

We are working on the docs here https://github.com/nestjs/docs.nestjs.com/pull/887

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.