Angular-cli: Support ng generate webWorker for libraries

[klausj]

🚀 Feature request

Command (mark with an x)

- [ ] new
- [ ] build
- [ ] serve
- [ ] test
- [ ] e2e
- [x] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Description

I would like to use Web Workers in an Angular 8 library project.
In version 8.1.1 this seems to be supported for project type 'application' only:

ng generate webWorker my-module/myWorker --project my-lib
Web Worker requires a project type of "application".

Describe the solution you'd like

Generating a Web Worker should work for project type 'library' as well.

Describe alternatives you've considered

Rewrite TypeScript worker code to JavaScript file.

[emps]

hey do any have any workaround ?

[SansDK]

Hey @klausj,

It's been a while since you opened this issue, do you have any kind of solution / workaround on how to use a web worker in an Angular library?

Thanks

[klausj]

Hi @SansDK

As a workaround I create an object URL which contains the code of a TypeScript method for the worker. The worker code is wrapped by a self executing function to be available to the worker.

// Helper class to build Worker Object URL
export class WorkerHelper {

 static buildWorkerBlobURL(workerFct: Function): string {

      // Update 2020-07-16: Try to mitigate XSS attacks
      // Should ensure a type check at runtime and reject an injected XSS string.
      if(! (workerFct instanceof Function)) {
         throw new Error(
            'Parameter workerFct is not a function! (XSS attack?).'
         )
      }
      let woFctNm = workerFct.name;
      let woFctStr = workerFct.toString();

      // Make sure code starts with "function()"
      // Chrome, Firefox: "[wofctNm](){...}", Safari: "function [wofctNm](){...}"
      // we need an anonymous function: "function() {...}"
      let piWoFctStr = woFctStr.replace(/^function +/, '');

      // Convert to anonymous function
      let anonWoFctStr = piWoFctStr.replace(woFctNm + '()', 'function()')

      // Self executing
      let ws = '(' + anonWoFctStr + ')();'

      // Build the worker blob
      let wb = new Blob([ws], {type: 'text/javascript'});

      let workerBlobUrl=window.URL.createObjectURL(wb);
      return workerBlobUrl;
    }
}

A TypeScript class which requires a worker looks like this:

export class ClassWithWorker{

 private workerURL: string;
 private worker: Worker | null;

  constructor() {
        this.worker = null;
        this.workerURL = WorkerHelper.buildWorkerBlobURL(this.workerFunction)
    }

    /*
     *  Method used as worker code.
     */
    workerFunction() {
       self.onmessage = function (msg) {
        let someInputData = msg.data.someInput;
        // Work on someInput
        ...
        // Post result
        postMessage({someOutput:someData });
       }
    }

    start(){
       this.worker = new Worker(this.workerURL);
       this.worker.onmessage = (me) => {
           let result = me.data.someOutput;
           ...
       }
       this.worker.postmessage({someInput: someInputData});
     }

    stop() {
      this.worker.terminate();
     }
}

The approach works but has some drawbacks:

• The string returned by Function.toString() seems not be well defined. It differs on different browsers and depends on the transpiler version used ( for example I had to change the code migrating from Angular 7 to 8).
  So my approach might break in future versions of browsers or build environments!

• If the worker code requires other classes they must be put inside the workerFunction. Duplicate code might be necessary.

[SansDK]

Thanks for the explanation and the example, very useful!

[Magn3s1um]

Just gonna drop my 2 cents, but allowing libraries to encapsulate web worker functionality away from consuming applications would be an absolute game changer.

With this, a future where we'll only be chaining awaits vs handling call backs and subscriptions isn't far off. I suspect performance would skyrocket as library maintainers would be able to offload computations completely from the UI thread.

Code quality would increase in the ecosystem as a whole.

[dgp1130]

Talked about this a bit in amongst the team. Workers have a lot of use cases for libraries. The challenge here is that workers must be loaded from a URL, and a library typically can't assume anything about the URL layout of a particular application.

Fortunately this is pretty easy to work around. Libraries can just export a function that does what they want and applications can easily import and call this function from their own worker. For example:

// my-lib.ts

export function work() {
  console.log('Does work');
}

Then the application can define a worker file which calls this library:

// my-worker.worker.ts

import { work } from 'my-lib';
work();

The worker can be started like any other by the application:

// app.component.ts

const worker = new Worker('./my-worker.worker');

The work() function can add event listeners to provide a proper service or support whatever API is useful for the library. This adds a little more boilerplate to the application, but keeps URL layout limited to the application and out of the library.

We're curious to hear if there are other use cases which might not be well-supported by this kind of design. The only one I can think of would be a library which wants to manage its workers directly (like a ThreadPool in Java), but that could be worked around and I'm not aware of any web worker use cases which fit that kind of model. You can't run arbitrary functions in a worker, so the comparison to ThreadPool isn't entirely accurate here. If there are other use cases that don't work well here, please post them so we can re-evaluate if necessary.

The workaround previously mentioned is quite clever but not usable for production.

• This introduces potential XSS security vulnerabilities (what if a malicious user tricked application code into calling WorkerHelper.buildWorkerBlobUrl() with their own user-input string?)
• Closures and global data are not packaged in the string format, so you really have no guarantee that this function will work.
  
  
  
  • WorkerHelper.buildWorkerBlobUrl(() => console.log(window.foo)); // window is not defined
  
  
  • WorkerHelper.buildWorkerBlobUrl(function() { console.log(this.foo); }.bind({ foo: 'bar' })); // this is undefined
  
  

[Magn3s1um]

That sounds nice, but ideally, libraries would automatically wire up in a consuming application.

If there were some kind of standard that we developed for workers inside a library, the compiler would be able to automatically wire up service workers correctly, would they not?

This same issue somewhat exists for assets, styles, and scripts. In order to consume some libraries, you have to modify the consuming applications angular.json sections manually.

To circumnavigate this, I've seen libraries create a service that during App Initialization will essentially wire up scripts by appending to document body some CDN paths.

In our case, we end up deploying npm-install scripts in the packages that has to search, then modify, these sections of the angular application. While scaffolding exists, its extremely heavy duty for what is essentially a json merge of these sections from library and application.

[s4m0r4m4]

Here's a use case to mention: I have an angular library for websocket comm. It contains an Angular Service, MyWebsocketService, which injects MyMessageHandlerService (from a different lib) to handle displaying/logging info and errors. MyWebsocketService is used across multiple applications and has deserialization code that creates MyMessage classes from the websocket buffer (rather than using JSON.stringify).

Sometimes, messages get very big (~300MB) and this causes MyWebsocketService to lock up the main thread while processing. I started looking to move all of the message handling (deserialization and MyMessage creation) to a webworker, which would then send the MyMessage classes to MyWebsocketService, which could broadcast the messages via an observable.

@dgp1130 thank you for posting such detailed instructions. I followed your approach:
1) start with an application-specific service for handling the websocket communication, App1WebsocketService
2) make a web worker for a specific application, App1Webworker
3) import the general deserialization code from the library into App1Webworker
4) when a new websocket communication comes in, App1Webworker deserializes and creates a MyMessage and uses postMessage to send it to App1WebsocketService. It can also throw an error.
5) App1WebsocketService then injects MyMessageHandlerService (from a different lib) and has to respond to completed messages or errors.

Here are the drawbacks:
A) Each application then has to replicate the same App1Webworker code, which is unfortunate, but manageable since most of the heavy lifting can be done in the general deserialization code from the lib.
B) Each application also has to implement the way in which it responds to errors from the websocket, which I would ideally be able to enforce the same response across all apps.
C) I can't import code from barrels that includes angular components. I've noticed that if I import code from barrel that includes an angular component, ng serve throws the following error where it appears to try and parse the HTML template for the angular component:

ERROR in ..../app1.worker.ts (C:/.../node_modules/worker-plugin/dist/loader.js?name=0!./src/app/core/services/rh-ui-router.worker.ts)
Module build failed (from C:/.../node_modules/worker-plugin/dist/loader.js):
ModuleParseError: Module parse failed: Unexpected token (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
> <span *ngIf="this.m_cAsyncTask">
|     <strong>Task Status</strong> (Id: {{ this.m_cAsyncTask.Id }})
|
    at handleParseError (C:\AIMDev\CI-Web\node_modules\webpack\lib\NormalModule.js:469:19)
    at C:\...\node_modules\webpack\lib\NormalModule.js:503:5
    at C:\...\node_modules\webpack\lib\NormalModule.js:358:12
    at C:\...\node_modules\loader-runner\lib\LoaderRunner.js:373:3
    at iterateNormalLoaders (C:\...\node_modules\loader-runner\lib\LoaderRunner.js:214:10)
    at C:\AIMDev\CI-Web\node_modules\loader-runner\lib\LoaderRunner.js:205:4
    at VirtualFileSystemDecorator.readFile (C:\...\node_modules\@ngtools\webpack\src\virtual_file_system_decorator.js:42:13)
    at processResource (C:\...\node_modules\loader-runner\lib\LoaderRunner.js:202:11)
    at iteratePitchingLoaders (C:\...\node_modules\loader-runner\lib\LoaderRunner.js:158:10)
    at runLoaders (C:\...\node_modules\loader-runner\lib\LoaderRunner.js:365:2)
    at NormalModule.doBuild (C:\...\node_modules\webpack\lib\NormalModule.js:295:3)
    at NormalModule.build (C:\...b\node_modules\webpack\lib\NormalModule.js:446:15)
    at Compilation.buildModule (C:\...\node_modules\webpack\lib\Compilation.js:739:10)
    at C:\...\node_modules\webpack\lib\Compilation.js:981:14
    at C:\...\node_modules\webpack\lib\NormalModuleFactory.js:409:6
    at C:\...\node_modules\webpack\lib\NormalModuleFactory.js:155:13

If I simply import the code directly from its file (rather than the barrel), the error goes away.
D) I can't use postMessage in the shared library code (since it doesn't use the worker postMessage). This means that if I want to code something like "any time you receive a websocket message, send it to the client via postMessage()", I need to implement it in each specific worker rather than the shared code.
E) Another use case - an angular image component that renders the image on canvas using a web worker. I have multiple applications that would love to utilize this functionality, but I can't put same angular component in a library to share because the angular component utilizes a web worker, which can't go in a library.

TLDR - it does indeed seem like there are ways of work around this limitation, but it results in duplicate code across applications, and there are all sorts of nuiances that I'm still discovering. I somewhat understand the challenges you describe here, but it would certainly be nice if I could simply package my web worker into a library that any application could use.

[klausj]

The workaround previously mentioned is quite clever but not usable for production.

• This introduces potential XSS security vulnerabilities (what if a malicious user tricked application code into calling WorkerHelper.buildWorkerBlobUrl() with their own user-input string?)

@dgp1130 thanks for your comment.
Could you please explain in more detail how an XSS attack could occur.
The argument to WorkerHelper.buildWorkerBlobURL() is the worker method/function of my own class. There is no user input involved.

[dgp1130]

@klausj, I don't immediately see anything exploitable in your ClassWithWorker example, however the general design relies on application code doing the right thing in a way that can't be guaranteed. I made a quick example of a well-meaning but flawed application which attempts to log all query parameters in a worker (and does so in a vulnerable way).

https://stackblitz.com/edit/typescript-b4g24s

// Vulnerable to XSS, do NOT use this example!
class VulnerableWorkerWrapper {
  static worker?: Worker;

  work() {
    self.onmessage = ({ data }) => {
      console.log(`${data.key}: ${data.value}`);
    };
  }

  start() {
    VulnerableWorkerWrapper.worker = new Worker(WorkerHelper.buildWorkerBlobURL(this.work));
    for (const [ key, value ] of Object.entries(this)) {
      VulnerableWorkerWrapper.worker.postMessage({ key, value });
    }
  }
};

const vulnerableWorker = new VulnerableWorkerWrapper();

const params = new URL(window.location.href).searchParams;
for (const [ key, value ] of Array.from(params.entries())) {
  vulnerableWorker[key] = value; // BAD!!! What happens if `key === 'work'`?
}

vulnerableWorker.start();

This example can be trivially exploited by setting JavaScript code in the work query parameter: https://typescript-b4g24s.stackblitz.io/?work=function(){console.log(%22Imma%20mine%20some%20bitcoin%20in%20here!%22)}.

Ultimately buildWorkerBlobURL() is converting a string into a function and relies on that input being sanitized and trusted. It is incredibly difficult to prove that a given piece of data can't be corrupted by a clever attacker. Maybe trusted types could help work around this (not sure of the current status), but there is always a fundamental XSS risk when you interpret a string like this.

[klausj]

Thanks for your interesting demo!
I program mainly with statically typed programming languages and sometimes forget that a function property can be assigned with any value in JavaScript.

And I updated my code snippet with a type check which should ensure that the builder method only accepts values of type 'Function'. Injected code strings should be rejected with an error at runtime.

[Magn3s1um]

@dgp1130 Sanitizing input is always a must, even in in Angular today. If Angular allows interpolation without requiring the user to sanitize, I don't see why this case should be any different.

The power that could be unleashed by allowing libraries to encapsulate and provide worker support is just too tempting.

Like i mentioned before, we already hook into npm install in order to auto wire up asset, style, and dependent package support. I'm beginning to do the same with this just for investigation.

[Magn3s1um]

Using some of @dgp1130 thoughts, I was able to kludge together a solution that worked. Not ideal. Won't be putting this into production in any way shape or form until there's less glue required to get this to work.

Getting the thing to compile required absolute paths in any of the webworker code. Got a bunch of typing errors when any library was references by relative or module name.

It seems when writing the code in a library, there can be no javascript imports or else it won't compile correctly. On top of that, in the parent application, I had to reference the js file directly in the web worker (after compiling the worker using tsc manually in the library and deploying it with the package).

import { SomeClassWithTrueImplementation } from '@mylib';

forget about doing that. You'll be met with typescript typings mismatches between the dom library and the worker library.

It also seems that any functionality in the lib must be restricted to exporting functions. I originally wrote my library code in a class. In the parent worker, using new MyClass(). Instantly would break.

The component then requires an input of type Worker that the parent passes to it (since calling new Worker inside the library doesn't work at all).

That being said, the performance improvement, even without using Transferable between the worker and parent, was astounding.

Better support/encapsulation of this technology in angular would be revolutionary.

[Ocean-Blue]

I am also trying to do the same thing. So far the only way I can make it work is by:

• having the worker script (e.g. app.work.ts) in the main application level
• put the logic of the worker script in the library as an exported function
• initialize the worker object from the main application level
• then pass the worker object to the library
  My example here: https://github.com/Ocean-Blue/angular-web-worker/tree/master/web-worker-solution-2

I have also tried to keep the logic for the web worker in the library as much as possible, by keeping the initialization of the worker object in the library. But to do this, I have to pass the hard-coded file path to the worker script. Passing the path by using Input or InjectionToken does not work.
Example here: https://github.com/Ocean-Blue/angular-web-worker/tree/master/web-worker-solution-1

Another workaround I have tried is to write the worker script as a JS script, then import it through angular.json. But not able to make this work. It gives me a mimeType (text/html) not matched.
Example here: https://github.com/Ocean-Blue/angular-web-worker/tree/master/web-worker-solution-js

Has anyone tried the 3rd workaround?