Node: Functions within worker thread lose global variable access

[Ethan-Arrowood]

• Version: v11.2.0
• Platform: arwin Ethans-MacBook-Pro.local 18.0.0 Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64 x86_64
• Subsystem: worker-threads

The buffer I'm passing to my worker threads is losing its standard method .write when called within a function. I believe the scoping is causing the issue but I cannot get my VSCode debugger to work on my thread instances.

const {
  Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

if (isMainThread) {
  const bf = Buffer.from('xyzqwerty')
  const worker = new Worker(__filename, {
    workerData: {
      dataBuffer: bf
    }
  });
  worker.on('message', res => {
    console.log(res.toString())
  });
  worker.on('error', err => {
    console.error(err)
  });
  worker.on('exit', (code) => {
    if (code !== 0)
      throw new Error(`Worker stopped with exit code ${code}`)
  })
} else {
  const { dataBuffer } = workerData
  console.log('foo')
  console.log(Buffer.isBuffer(dataBuffer))
  function setItem(buff, item, i) {
    console.log('bar')
    console.log(Buffer.isBuffer(dataBuffer))
    buff.write(item, i)
  }

  setItem(dataBuffer, 'abc', 2)

  parentPort.postMessage(dataBuffer)
}

Error out:

TypeError: buff.write is not a function
    at setItem (/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/trial.js:26:10)
    at Object.<anonymous> (/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/trial.js:29:3)
    at Module._compile (internal/modules/cjs/loader.js:722:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:733:10)
    at Module.load (internal/modules/cjs/loader.js:620:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
    at Function.Module._load (internal/modules/cjs/loader.js:552:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:775:12)
    at MessagePort.port.on (internal/worker.js:460:27)
    at MessagePort.emit (events.js:182:13)
/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/trial.js:20
      throw new Error(`Worker stopped with exit code ${code}`)
      ^

Error: Worker stopped with exit code 1
    at Worker.worker.on (/Users/ethanarrowood/Documents/Wentworth/Semesters/Fall2018/OperatingSystems/Project/trial.js:20:13)
    at Worker.emit (events.js:182:13)
    at Worker.[kOnExit] (internal/worker.js:321:10)
    at Worker.(anonymous function).onexit (internal/worker.js:277:51)

[apapirovski]

When you pass along a buffer like that, what comes through on the other side is a Uint8Array, as far as I understand it. That will not have a write method available.

[addaleax]

I believe the scoping is causing the issue but I cannot get my VSCode debugger to work on my thread instances.

So: The issue is that for message passing, we do what the browsers do (because they have a spec for it), and in particular we use the HTML structured clone algorithm.

As part of that, objects lose their prototype information; for instances of user-defined classes, this means that they look like “plain” objects on the receiving side. For Buffers, which is a subclass of Uint8Array, this means that on the receiving side they look like “plain” Uint8Arrays, without all the extra methods of Buffers.

I can think of at least two solutions for this:

• Use the v8.serialize API to perform serialization and deserialization yourself. It’s the same algorithm, but we’ve modified it so that it handles the Buffer case differently by default. This is going to come with some overhead, though.
• Convert to Buffer manually after receiving; i.e. buffer = Buffer.from(uint8array.buffer, uint8array.byteOffset, uint8array.length).