Node-rdkafka: TypeError: Unicode-objects must be encoded before hashing on node-gyp (Python 3.6.7)

[felipebritor]

Environment Information

• OS [Ubuntu 16.04.5 LTS]: [edited]
• Node Version [10.18.1]:
• NPM Version [6.13.4]:
• C++ Toolchain [gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609]: [edited]
• node-rdkafka version [2.7.4]:

Steps to Reproduce

Try to build the following Dockefile:

FROM python:3.6.7

RUN apt-get update \
    && apt-get install -y build-essential \
    ca-certificates \
    libsasl2-dev \
    libsasl2-modules \
    libssl-dev

RUN curl -sL https://deb.nodesource.com/setup_10.x | bash && apt install -y nodejs

RUN node -v && npm -v

RUN npm install node-rdkafka

node-rdkafka Configuration Settings

Not needed

Additional context

Hello.

last week, I had a surprise when deploying an app, our build is throwing:

> [email protected] install /home/travis/es/src/.../node_modules/node-rdkafka
> node-gyp rebuild

Traceback (most recent call last):
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py", line 50, in <module>
    sys.exit(gyp.script_main())
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2215, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 794, in Write
    extra_mac_bundle_resources, part_of_all)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 978, in WriteActions
    part_of_all=part_of_all, command=name)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1724, in WriteDoCmd
    force = True)
  File "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1779, in WriteMakeRule
    cmddigest = hashlib.sha1(command if command else self.target).hexdigest()
TypeError: Unicode-objects must be encoded before hashing

gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:198:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
gyp ERR! System Linux 4.19.52
gyp ERR! command "/home/travis/.nvm/versions/node/v10.18.0/bin/node" "/home/travis/.nvm/versions/node/v10.18.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/travis/es/src/github.ibm.com/CIO-SETS/engage-support/foe-fom/services/node_modules/node-rdkafka
gyp ERR! node -v v10.18.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok 
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2020-01-09T18_26_58_064Z-debug.log

We use travis and it's python version is 3.6.7, I've tested other version and it worked fine on all of them. Which is strange, as 3.6.* is technically supported by node-gyp.

May someone give some light on what can I do? Forcing the use of another python version is something I'll have to explain to our ops team, so I would rather leave it to a last resort

[felipebritor]

I ended up setting another version on travis' before-install script, which worked. But the bug remains for python v3.6.7. Is it an issue for node-gyp itself?

[karousn]

the version that use it here is for legacy python version, i guest 2.7.* . so i think will be better to update node-gyp version use it here to be able for as to install recent python version.

[sadvazha]

Had the same issue, i have solved it by setting a PYTHON env variable to PYTHON=/path/to/python2.7 for npm install. There was a problem with node-gyp

[redduck3]

Had the same issue, I found that the node-gyp version in package.json is locked to 3.X. Please upgrade node-gyp to support python3.

[sam-github]

3.x is pretty old!

(from memory, but pretty sure this is the case)

node-gyp 5.x supports py2 or py3 (but prefers py2), 6.x prefers py3.

npm has 5.x

I'll take a shot at PRing an update, but no promises on whether it gets merged+released, that is out of my control.

[sam-github]

Looked, this isn't related to the node-gyp version in the package.json, which is a dev dep (not install dep), so the fact that it is v3.x isn't related.

gyp ERR! node-gyp -v v5.0.5

I help maintain node-gyp, I'll see if I can track down if this is an issue with upstream.

[sam-github]

Looks like this is a node-gyp bug, fixed in https://github.com/nodejs/node-gyp/pull/1925, released in v5.0.6. Use a version of npm that includes that version. I just did an npm install -g npm, got npm 6.14.2:

% npm ls -g node-gyp
/home/sam/.nvm/versions/node/v12.16.1/lib
└─┬ [email protected]
  ├── [email protected] 
  └─┬ [email protected]
    └── [email protected]  deduped

Did an npm install of node-rdkafka, and it fixed the error above.

I believe that should close this issue, its not an issue with this project, and its fixed upstream, and released in npm.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.