Pnpm: Error: Unable to update lock within the stale threshold

Created on 1 Feb 2017 · 28Comments · Source: pnpm/pnpm

Code to reproduce

pnpm install

Actual behavior:

pnpm starts to install packages in node_modules/.resolutions then fails with only 300 of them installed (1000 to be installed). node.exe process memory goes up to 800MB and exits with the following error.
Running the command again installs a few more packages each time (around 5). Please note that i have 11GB RAM free before running the command :)

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

Additional information:

pnpm-debug.log.txt

pnpm 0.50.0
node 6.9.4
Windows 7

breaking change bug

Source

aecz

👍4

All 28 comments

Just found this explanation on the proper-lockfile repo.
https://github.com/IndigoUnited/node-proper-lockfile/issues/11
Looks like we need to set a bigger delay or at least add an option for it.

aecz on 1 Feb 2017

Error still happening unfortunately even after using next version 0.52.0.
You can get the package.json on the linked PR.

Here is some more info of my last test:

1st run: 700 packages installed, max memory used (didnt check the first run :( )

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

2st run: 14 more packages installed, max memory used 500 MB

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

3st run: 70 more packages installed, max memory used 620 MB

ERROR unexpected end of file
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

4th run: 237 more packages installed, max memory used 710 MB

Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

5th run: 29 more packages installed, max memory used 730MB

WARN Skipping failed optional dependency [email protected]
ERROR unexpected end of file
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

6th run: 15 more packages installed, max memory used 642MB

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

7th run: 16 more packages installed, max memory used 628MB

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

8th run: 3 more packages installed (stopped measuring memory, doesnt seem an issue)
(core-js then karma then webdriverio)

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

9th run: 1 more packages installed (pm2)

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

10th run: 2 more packages installed (rxjs, ng2-google-place-autocomplete)

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

11th run: 2 more packages installed (rxjs, ng2-google-place-autocomplete)

C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:180
    compromised = compromised || function (err) { throw err; };
                                                  ^
Error: Unable to update lock within the stale threshold
    at options.fs.utimes (C:\Users\aecz\AppData\Roaming\nvm\v6.9.4\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:123:15)

12th run: 3 more packages installed (libxmljs-mt, libxslt, gulp-xslt)
=> No errors and binaries compiled successfully!

So it looks like only some packages are problematic. Once installed, no issues so far.
Another issue has appeared maybe unrelated about zlib.

aecz on 3 Feb 2017

I think the solution for #600 will solve your issues as well

zkochan on 3 Feb 2017

I believe this is fixed in 0.52.1. I cannot reproduce this anymore

zkochan on 5 Feb 2017

Still seeing this (intermittently) with 0.52.1. Had to put a retry loop around it.

00:00:29.814 + pnpm install
00:00:56.492 /node4/lib/node_modules/pnpm/lib/node_modules/proper-lockfile/index.js:180
00:00:56.492 compromised = compromised || function (err) { throw err; };
00:00:56.492 ^
00:00:56.492
00:00:56.492 Error: Unable to update lock within the stale threshold
00:00:56.492 at /node4/lib/node_modules/pnpm/lib/node_modules/proper-lockfile/index.js:108:66
00:00:56.492 at FSReqWrap.oncomplete (fs.js:82:15)
00:00:56.493
00:00:56.523 ERROR: script returned exit code 1
00:00:56.523 Retrying
00:00:56.601 Running shell script
00:00:56.870 + pnpm install

(worked the second time)

akomakom on 6 Feb 2017

As a temporary fix, we can

make the timeout configurable
add a --no-lock option

zkochan on 6 Feb 2017

I'm still seeing this error in 0.66.2 on Ubuntu 16.04.

--lock-stale-duration=0 seems to prevent it.

I think instructions to enable this flag should printed when this error occurs. But it would be better to try avoid it...

vjpr on 3 May 2017

Seeing it on macOS Sierra too.

vjpr on 3 May 2017

OK, let me add a --no-lock option for now, and then we'll try to solve it better.

Plus an error message as you suggested

zkochan on 7 May 2017

--no-lock released with v0.67.0

zkochan on 7 May 2017

🎉1

I am also experiencing this when installing truffle and ethereumjs-testrpc on Alpine Linux during linking.

louy2 on 11 Jun 2017

I'm still seeing this with pnpm i --no-lock --lock-stale-duration=0...

/usr/local/lib/node_modules/pnpm/lib/node_modules/proper-lockfile/index.js:186
    compromised = compromised || function (err) { throw err; };
                                                  ^

Error: Unable to update lock within the stale threshold
    at options.fs.utimes (/usr/local/lib/node_modules/pnpm/lib/node_modules/proper-lockfile/index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:135:15)

vjpr on 29 Jun 2017

@vjpr published a fix in 1.0.1. As next for now because I did a lot of refactoring

zkochan on 29 Jun 2017

@zkochan I just got the error again on Ubuntu 16 with a completely warm store.

Resolving: total 2040, reused 1672, downloaded 0
/usr/local/lib/node_modules/pnpm/node_modules/proper-lockfile/index.js:186
    compromised = compromised || function (err) { throw err; };
                                                  ^

Error: Unable to update lock within the stale threshold
    at options.fs.utimes (/usr/local/lib/node_modules/pnpm/node_modules/proper-lockfile/index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:135:15)

Happens at random package installs.

Resolving: total 2040, reused 1672, downloaded 0
pnpm i --lock-stale-duration=0
Resolving: total 2040, reused 1972, downloaded 0
Resolving: total 2040, reused 1991, downloaded 0

At then end the reused counter starts to slow down and stall, and then the error comes.

--no-lock seems to have allowed it to finish. It becomes really slow at the end though. Maybe there is a memory leak or some inefficient code...

vjpr on 25 Jan 2018

👍1

I don't understand why proper-lockfile has so many issues. Maybe we just need to save the process.id in a file, in the root of the store and check whether the process still runs.

zkochan on 25 Jan 2018

Is this the only place proper-lockfile is used:

https://github.com/pnpm/fs-locker/blob/ddbf32e276b10e4e02f1289801874f9632b67d22/src/index.ts#L21

Maybe some inefficiency is causing the lock to timeout...

vjpr on 25 Jan 2018

yes, but two locks are created. One for the store and one for the project

zkochan on 25 Jan 2018

@zkochan , we recently implemented a lockfile that we believe does not have any race conditions and runs on Windows, Mac, and Linux. https://github.com/Microsoft/web-build-tools/blob/master/libraries/node-core-library/src/LockFile.ts . For OSX and Linux it relies on a combination of pid & starttime to determine if the process is still holding the lock. We didn't like lockfile or proper-lockfile packages since they relied on weird designs like having the process holding the lock constantly update the modified time.

We haven't yet implemented an API for waiting for the LockFile to become available. It would likely be as easy as putting tryAcquire() in a retry loop. (Of course that simple approach would not address issues like starvation).

nickpape on 8 Feb 2018

@nickpape-msft sounds great, maybe you could move that logic to a separate package and then we'll be able to substitute proper-lockfile in @pnpm/fs-locker

zkochan on 9 Feb 2018

Still not able to lock within stale threshold. On setting the --no-lock flag it works shouldn't the default timeout increased further?
OS: Windows 10
Node: 9.4.0
Yarn: 1.4.0
npm: 4.6.1
Watchman: 4.9.1

HarshulSharma000 on 9 Feb 2018

I'm seeing this with 1.35.1 (Windows 10, Node 8.9.0)

D:\dev\src\abc>pnpm i
Resolving: total 766, reused 0, downloaded 0
D:\dev\nodejs\pnpm-global\1\node_modules\.registry.npmjs.org\pnpm\1.35.1\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:186
    compromised = compromised || function (err) { throw err; };
                                                  ^

Error: Unable to update lock within the stale threshold
    at options.fs.utimes (D:\dev\nodejs\pnpm-global\1\node_modules\.registry.npmjs.org\pnpm\1.35.1\node_modules\pnpm\lib\node_modules\proper-lockfile\index.js:108:66)
    at FSReqWrap.oncomplete (fs.js:135:15)

msssk on 1 Mar 2018

I also saw this recently in Ubuntu 14.
On Thu 1. Mar 2018 at 08:00, Mangala Sadhu Sangeet Singh Khalsa <
[email protected]> wrote:

I'm seeing this with 1.35.1 (Windows 10, Node 8.9.0)

D:\dev\src\abc>pnpm i
Resolving: total 766, reused 0, downloaded 0
D:\dev\nodejs\pnpm-global\1\node_modules.registry.npmjs.org\pnpm\1.35.1\node_modules\pnpm\lib\node_modulesproper-lockfile\index.js:186
compromised = compromised || function (err) { throw err; };
^

Error: Unable to update lock within the stale threshold
at options.fs.utimes (D:\dev\nodejs\pnpm-global\1\node_modules.registry.npmjs.org\pnpm\1.35.1\node_modules\pnpm\lib\node_modulesproper-lockfile\index.js:108:66)
at FSReqWrap.oncomplete (fs.js:135:15)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pnpm/pnpm/issues/594#issuecomment-369495528, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AARLRREZ9lcyyx1U2MioS19MgbVvrNHmks5tZ5yrgaJpZM4Lzv2y
.

vjpr on 1 Mar 2018

I'm having this issue on macOS with node 6.8.1 and 8.9.4.

imeredith on 4 Apr 2018

To my knowledge, all operations that are done by pnpm, are done atomically, so it should be safe to run pnpm without locks.

I suggest to turn off locks by default.

zkochan on 9 Apr 2018

@nickpape-msft sounds great, maybe you could move that logic to a separate package and then we'll be able to substitute proper-lockfile in @pnpm/fs-locker

(FWIW we generally avoid creating tiny NodeJS packages with only 300 lines of code in them. It's bad for performance. Also, the more individual dependencies you have, the harder it is to avoid side-by-side versions.)

octogonz on 29 Apr 2018

👍1

This still happens quite frequently in my CI pipelines with pnpm version 4.14, on Windows 10 and Windows Server machines.

xbtsw on 28 Apr 2020

OK, let's remove the lock in v5

Wei Chen notifications@github.com ezt írta (időpont: 2020. ápr. 28., K
18:44):

This still happens quite frequently in my CI pipelines with pnpm version
4.14, on Windows 10 and Windows Server machines.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pnpm/pnpm/issues/594#issuecomment-620687342, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOWTGYE3YFWARYRX4INOYDRO32T7ANCNFSM4C6O7WZA
.

zkochan on 28 Apr 2020

I created a PR to remove locking #2553

zkochan on 13 May 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

macOS install error EACCES

quinton-ashley · 27Comments

integrity checksum failed when using sha1

nickpape · 40Comments

The global store should always be on the same disk on which installation is done

zkochan · 33Comments

Deduping algorithm not working properly

nickpape · 31Comments

How to handle "bad" packages that require() things that aren't in their package.json?

nickpape · 37Comments