Yarn: @yarnpkg/lockfile can't parse current lockfiles

[Silic0nS0ldier]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
@yarnpkg/lockfile fails to parse yarn.lock, raising the error SyntaxError: Unknown token 3:1 in lockfile. The last release for this package was from before the 1.0.0 release, and manually building from the yarn source produced a package that worked, so its stands to reason that the major version may have introduced breaking changes.

If the current behavior is a bug, please provide the steps to reproduce.
Simply take any yarn.lock file, and parse it as described in the README.

What is the expected behavior?
Lockfile is successfully parsed, and an object is returned.

Please mention your node.js, yarn and operating system version.
Issue was replicated on WSL and Windows, with yarn uninstalled when using WSL to stop conflicts.
WSL (Ubuntu 16.04.3 LTS) and Windows 10 1709 16299.192:

• Node: 9.4.0
• Yarn: 1.3.2

[rally25rs]

Seems to work OK for a simple lockfile:

~/Projects/yarn-test 🐒   cat yarn.lock
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


"@yarnpkg/lockfile@^1.0.0":
  version "1.0.0"
  resolved "https://registry.yarnpkg.com/@yarnpkg/lockfile/-/lockfile-1.0.0.tgz#33d1dbb659a23b81f87f048762b35a446172add3"

"intro.js-react@git+https://github.com/HiDeoo/intro.js-react.git#a475b79":
  version "0.1.5"
  resolved "git+https://github.com/HiDeoo/intro.js-react.git#a475b7968067e5c5778e431d6448ef46aa576761"

~/Projects/yarn-test 🐒   node
> const fs = require('fs');
undefined
> const lockfile = require('@yarnpkg/lockfile');
undefined
> let file = fs.readFileSync('yarn.lock', 'utf8');
undefined
> let json = lockfile.parse(file);
undefined
> console.log(json);
{ type: 'success',
  object:
   { '@yarnpkg/lockfile@^1.0.0':
      { version: '1.0.0',
        resolved: 'https://registry.yarnpkg.com/@yarnpkg/lockfile/-/lockfile-1.0.0.tgz#33d1dbb659a23b81f87f048762b35a446172add3' },
     'intro.js-react@git+https://github.com/HiDeoo/intro.js-react.git#a475b79':
      { version: '0.1.5',
        resolved: 'git+https://github.com/HiDeoo/intro.js-react.git#a475b7968067e5c5778e431d6448ef46aa576761' } } }

Could you post your lockfile?

[Silic0nS0ldier]

Sure thing. I'll throw in the package.json that it came from too.
yarn.lock.txt
package.json.txt

Just to be sure, I ran your debug code as well.

D:\Users\jorda\Desktop\Sources\UserFrosting\build [develop ≡ +0 ~4 -0 !]> node
> const lockfile = require('@yarnpkg/lockfile')
undefined
> const fs = require('fs')
undefined
> let file = fs.readFileSync('../app/assets/yarn.lock', 'utf8')
undefined
> let json = lockfile.parse(file)
SyntaxError: Unknown token 3:1 in lockfile

> console.log(json)
ReferenceError: json is not defined

[rally25rs]

It seems your lock file has Windows CRLF line endings. If you convert them to Unix style endings then it works.

The lockfile parser does seem to have some code to handle CRLF:

    if (input[0] === '\n' || input[0] === '\r') {
      chop++;
      // If this is a \r\n line, ignore both chars but only add one new line
      if (input[1] === '\n') {
        chop++;
      }

and the unit tests have a couple tests where windows style line endings are used

  expect(parse(`foo:\r\n  bar:\r\n    foo "bar"`).object).toEqual(nullify({foo: {bar: {foo: 'bar'}}}));

  expect(parse('foo:\r\n  bar:\r\n    yes no\r\nbar:\r\n  yes no').object).toEqual(
    nullify({
      foo: {
        bar: {
          yes: 'no',
        },
      },
      bar: {
        yes: 'no',
      },
    }),
  );

so not sure why the syntax error. 😕

I'll try to dig into it a bit more...

[Silic0nS0ldier]

Line endings... Should have known. https://github.com/yarnpkg/yarn/commit/05bf97728cbe92f034f7d4596830f481a65fe8fa looks to have done this with https://github.com/yarnpkg/yarn/pull/4495 fixing the lockfile parser.

The gap between the changes is massive, but that could easily be attributed to factors like the change being slow to make it into a release, people not actively updating their copies of yarn, and how it can only be encountered on Windows systems.

Issue here seems to be just that @yarnpkg/lockfile hasn't be updated since the EOL fix was rolled out.

• August 22, 2017 - lockfile package released
• September 19, 2017 - EOL fixes for lockfile merged

[Silic0nS0ldier]

Any progress on this? I can always bundle a newly built copy with the fixes in my package, but I'd rather not resort to that.

[laurence-myers]

My workaround for now is to manually replace all "r" characters with an empty string, "".

let unparsedLockfile = readFileSync(appRootPath.resolve('yarn.lock'), 'utf8');
unparsedLockfile = unparsedLockfile.replace(/\r/g, '');
const parsedLockfile = lockfile.parse(unparsedLockfile);

[Silic0nS0ldier]

Just to "poke the bear", is there anyone around who could push an updated release of this dependency out? Even better would be to integrate this into the yarn release process, so that they are always in sync.

[flegall]

Same issue here, it would be nice to have updated releases of this dependency !

[Silic0nS0ldier]

1.1.0 has been released and has the line ending bits.