Keystone: windows compatibility: monorepo does not symlink

[bholloway]

From @1337cookie https://github.com/keystonejs/keystone-5/pull/1463#issuecomment-519957463

@MadeByMike and I saw the same issue. It is not seen when using admin privillages but that is definitely not advised and not a solution.

C:\Users\user\Documents\web\keystone-5>bolt
⚡️   bolt v0.23.6 (node v10.16.1)
�   [1/4] Validating project...
�   [2/4] Installing project dependencies...
$ C:\Users\user\AppData\Local\Yarn\Data\global\node_modules\bolt\node_modules\.bin\yarn config get user-agent yarn/1.17.3 npm/? node/v10.16.1 win32 x64
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "apollo-server-express > apollo-server-core > [email protected]" has incorrect peer dependency "[email protected] - 14.1.x".
warning "graphql-playground-react > [email protected]" has incorrect peer dependency "react@>=15.5 <16".
warning "graphql-playground-react > [email protected]" has incorrect peer dependency "react-dom@>=15.5 <16".
[4/4] Building fresh packages...
$ yarn build:build-field-types && preconstruct dev && node ./packages/build-field-types/bin.js dev  && yarn run --silent contributing-guide
yarn run v1.17.3
$ rimraf packages/build-field-types/lib && babel packages/build-field-types/src --out-dir packages/build-field-types/dist --ignore '**/__tests__/**/*','**/__mocks__/**/*'
Successfully compiled 42 files with Babel.
Done in 4.41s.
� info project is valid!
� error { [Error: EPERM: operation not permitted, symlink 'C:\Users\user\Documents\web\keystone-5\packages\arch\packages\alert\src\index.js' -> 'C:\Users\user\Documents\web\keystone-5\packages\arch\packages\alert\dist\alert.cjs.js.flow']
� error   errno: -4048,
� error   code: 'EPERM',
� error   syscall: 'symlink',
� error   path:
� error    'C:\\Users\\user\\Documents\\web\\keystone-5\\packages\\arch\\packages\\alert\\src\\index.js',
� error   dest:
� error    'C:\\Users\\user\\Documents\\web\\keystone-5\\packages\\arch\\packages\\alert\\dist\\alert.cjs.js.flow' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error Error: spawn C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/node_modules/.bin/yarn ENOENT
error     at notFoundError (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:11:11)
error     at verifyENOENT (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:46:16)
error     at ChildProcess.cp.emit (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:33:19)

[bholloway]

I will install windows VM and try to investigate this.

[1337cookie]

Here's the output from my machine with a fresh clone of the repo. It seems you're right about the privileges but I then encounter a different error.
Windows 10 version 1809.

Running with non-elevated privileges: click to show

```
C:UsersuserDocumentstestkeystone-5>bolt
⚡️ bolt v0.23.6 (node v10.16.1)
🔎 [1/4] Validating project...
📦 [2/4] Installing project dependencies...
$ C:UsersuserAppDataLocalYarnDataglobalnode_modulesboltnode_modules.binyarn config get user-agent yarn/1.17.3 npm/? node/v10.16.1 win32 x64
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "win32" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "apollo-server-express > apollo-server-core > [email protected]" has incorrect peer dependency "[email protected] - 14.1.x".
warning "graphql-playground-react > [email protected]" has incorrect peer dependency "react@>=15.5 <16".
warning "graphql-playground-react > [email protected]" has incorrect peer dependency "react-dom@>=15.5 <16".
[4/4] Building fresh packages...
$ yarn build:build-field-types && preconstruct dev && node ./packages/build-field-types/bin.js dev && yarn run --silent contributing-guide
yarn run v1.17.3
$ rimraf packages/build-field-types/lib && babel packages/build-field-types/src --out-dir packages/build-field-types/dist --ignore '**/__tests__/**/*','**/__mocks__/**/*'
Successfully compiled 42 files with Babel.
Done in 4.16s.
🎁 info project is valid!
🎁 error { [Error: EPERM: operation not permitted, symlink 'C:\Users\user\Documents\test\keystone-5\packages\arch\packages\alert\src\index.js' -> 'C:UsersuserDocumentstestkeystone-5packagesarchpackagesalertdistalert.cjs.js.flow']
🎁 error errno: -4048,
🎁 error code: 'EPERM',
🎁 error syscall: 'symlink',
🎁 error path:
🎁 error 'C:\Users\user\Documents\test\keystone-5\packages\arch\packages\alert\src\index.js',
🎁 error dest:
🎁 error 'C:\Users\user\Documents\test\keystone-5\packages\arch\packages\alert\dist\alert.cjs.js.flow' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error Error: spawn C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/node_modules/.bin/yarn ENOENT
error at notFoundError (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:11:11)
error at verifyENOENT (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:46:16)
error at ChildProcess.cp.emit (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/cross-spawn/lib/enoent.js:33:19)

</details>

<details>
  <summary>Running with elevated privileges (Administrator): click to show </summary>

  ```
C:\Users\user\Documents\test\keystone-5>bolt
⚡️   bolt v0.23.6 (node v10.16.1)
🔎   [1/4] Validating project...
📦   [2/4] Installing project dependencies...
$ C:\Users\user\AppData\Local\Yarn\Data\global\node_modules\bolt\node_modules\.bin\yarn config get user-agent yarn/1.17.3 npm/? node/v10.16.1 win32 x64
yarn install v1.17.3
[1/4] Resolving packages...
success Already up-to-date.
$ yarn build:build-field-types && preconstruct dev && node ./packages/build-field-types/bin.js dev  && yarn run --silent contributing-guide
yarn run v1.17.3
$ rimraf packages/build-field-types/lib && babel packages/build-field-types/src --out-dir packages/build-field-types/dist --ignore '**/__tests__/**/*','**/__mocks__/**/*'
Successfully compiled 42 files with Babel.
Done in 1.36s.
🎁 info project is valid!
🎁 success created links!
🎁 info project is valid!
🎁 success created links!
📝 Contributing to KeystoneJS?
🔗 Read the full Contributing Guide (https://github.com/keystonejs/keystone-5/blob/master/CONTRIBUTING.md)
Done in 4.52s.
🔗   [3/4] Linking workspace dependencies...
error Error: Can't read shim path from 'C:/Users/user/Documents/test/keystone-5/packages/keystone/bin/cli.js', it doesn't appear to be a cmd-shim
error     at cb (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/dist/modern/utils/fs.js:154:75)
error     at AnyPromise (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/typeable-promisify/index.js:13:3)
error     at new Promise (<anonymous>)
error     at promisify (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/typeable-promisify/index.js:12:40)
error     at readCmdShim (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/dist/modern/utils/fs.js:154:42)
error     at cmdShim (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/dist/modern/utils/fs.js:96:88)
error     at createWindowsSymlink (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/dist/modern/utils/fs.js:122:18)
error     at Object.symlink (C:/Users/user/AppData/Local/Yarn/Data/global/node_modules/bolt/dist/modern/utils/fs.js:134:18)

[bholloway]

When installing Git in windows.

I'll check the box and see how it goes.

[bholloway]

Stuck in hell. Had to install python, windows build tools. Build tools install failed. 😒

[gautamsi]

I do want this to work on Windows. for now I used WSL (Windows Subsystem for Linux) for Windows 10 so that I can get through any issue.

[1337cookie]

Seems like a windows/symlink limitation put in place as some sort of security measure.
The short answer is Microsoft would like you to enable "developer mode" in windows if you want to create a symlink from a non-elevated command shell.

How to enable developer mode CLICK HERE TO SHOW

1. Open settings (Win + I) and click Update and Security.

1. Click "For Developers" on the left menu.

1. Select the "Developer mode" radio box and wait for it to install.

I personally didn't require a restart or to sign out/sign in.

More related stuff @ ember.js docs including how to use "Local Security Policy" to change this setting, https://ember-cli.com/user-guide/#enabling-symlinks :
To create symlinks the account running Ember CLI must have the SeCreateSymbolicLinkPrivilege. Users in the Administrators group have this permission already. However, if UAC (User Access Control) is enabled, users in the Administrators group must run their shell using Run As Administrator because UAC strips away certain permissions from the Administrators +group, including SeCreateSymbolicLinkPrivilege.

Windows developer blog info https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/:
For users who have Developer Mode enabled, the mklink command will now successfully create a symlink if the user is not running as an administrator.

Mystery solved?

[1337cookie]

Stuck in hell. Had to install python, windows build tools. Build tools install failed. 😒

npm install -g --production [email protected]

[bholloway]

Restarted installation with a fresh VM snapshot.

• Enable windows Settings > For developers > developer mode
• Install git with symlink option
• Install nvm-windows and nvm install latest and nvm use ... that version
• [admin] npm install -g --production [email protected]
• [admin] npm install -g node-gyp (per this issue with node-gyp)
• clone the repo
• bolt install

Now that I go through the motions I realise that I _have_ used "developer mode" previously.
I'm not sure the git symlink option is helpful, possibly a detail to determine later.

I'm encountering can't read shim path too now.

[bholloway]

Possibly https://github.com/boltpkg/bolt/issues/207

[1337cookie]

Possibly boltpkg/bolt#207

Yeah I also came across that one.

So I built the bolt package including the changes in this PR @ https://github.com/boltpkg/bolt/pull/215 I had to install and use the WSL with ubuntu, node and yarn, to build it. (Thanks for reminding me the WSL exists @gautamsi )

Using "yarn link" on the updated bolt directory in windows and "yarn link bolt" in the keystone directory, running "bolt", "bolt dev" and "bolt demo meetup dev" seems to work as expected, with the todo and meetup demo working properly in the browser.

Click here to see output from "bolt", "bolt dev" and "bolt demo meetup dev"

C:\Users\usernameremoved\Documents\web\keystone-5>bolt
⚡️   bolt v0.23.6 (node v10.16.1)
�   [1/4] Validating project...
�   [2/4] Installing project dependencies...
$ C:\Users\usernameremoved\Documents\web\bolt\node_modules\.bin\yarn config get user-agent yarn/1.12.3 npm/? node/v10.16.1 win32 x64
yarn install v1.12.3
[1/4] Resolving packages...
success Already up-to-date.
$ yarn build:build-field-types && preconstruct dev && node ./packages/build-field-types/bin.js dev  && yarn run --silent contributing-guide
yarn run v1.12.3
$ rimraf packages/build-field-types/lib && babel packages/build-field-types/src --out-dir packages/build-field-types/dist --ignore '**/__tests__/**/*','**/__mocks__/**/*'
Successfully compiled 42 files with Babel.
Done in 1.42s.
� info project is valid!
� success created links!
� info project is valid!
� success created links!
� Contributing to KeystoneJS?
� Read the full Contributing Guide (https://github.com/keystonejs/keystone-5/blob/master/CONTRIBUTING.md)
Done in 4.53s.
�   [3/4] Linking workspace dependencies...
�   [4/4] Linking workspace binaries...
�  success Installed and linked workspaces.
�   Done in 24.51s.

C:\Users\usernameremoved\Documents\web\keystone-5>bolt dev
⚡️   bolt v0.23.6 (node v10.16.1)
$ yarn demo todo dev
$ yarn --cwd ./demo-projects todo dev
$ yarn --cwd todo dev
$ cross-env NODE_ENV=development DISABLE_LOGGING=true keystone dev
i Command: keystone dev
√ Validated project entry file ./index.js
√ Keystone server listening on port 3000
√ Initialised Keystone instance
√ Connected to database
√ Keystone instance is ready �
� Keystone Admin UI:   http://localhost:3000/admin
� GraphQL Playground:  http://localhost:3000/admin/graphiql
� GraphQL API:         http://localhost:3000/admin/api

C:\Users\usernameremoved\Documents\web\keystone-5>bolt demo meetup dev
⚡️   bolt v0.23.6 (node v10.16.1)
$ yarn --cwd ./demo-projects meetup dev
$ yarn --cwd meetup dev
$ cross-env NODE_ENV=development DISABLE_LOGGING=true keystone dev
i Command: keystone dev
√ Validated project entry file ./index.js
√ Keystone server listening on port 3000
√ Initialised Keystone instance
- Connecting to database[ ready ] compiled successfully
\ Connecting to database[ wait ]  compiling ...
- Connecting to database[ ready ] compiled successfully
√ Connected to database
√ Keystone instance is ready �
� Keystone Admin UI:   http://localhost:3000/admin
� GraphQL Playground:  http://localhost:3000/admin/graphiql
� GraphQL API:         http://localhost:3000/admin/api
[ event ] build page: /
[ wait ]  compiling ...
[ ready ] compiled successfully

Running "bolt demo blog dev" however shows a new error which might not be related.

Click here to see "bolt demo blog dev" output

``C:\Users\usernameremoved\Documents\web\keystone-5>bolt demo blog dev ⚡️ bolt v0.23.6 (node v10.16.1) $ yarn --cwd ./demo-projects blog dev $ yarn --cwd blog dev $ cross-env NODE_ENV=development DISABLE_LOGGING=true node -r dotenv-safe/configyarn bin`/keystone dev
internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'C:UsersusernameremovedDocumentswebkeystone-5demo-projectsblog`yarn'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
error Error: spawn ../bolt/node_modules/.bin/yarn ENOENT
error at notFoundError (C:/Users/usernameremoved/Documents/web/bolt/node_modules/cross-spawn/lib/enoent.js:11:11)
error at verifyENOENT (C:/Users/usernameremoved/Documents/web/bolt/node_modules/cross-spawn/lib/enoent.js:46:16)
error at ChildProcess.cp.emit (C:/Users/usernameremoved/Documents/web/bolt/node_modules/cross-spawn/lib/enoent.js:33:19)
```

Regardless I think the symlink issue can be solved by documenting the "developer mode" option in the docs for the demo projects and pushing for https://github.com/boltpkg/bolt/pull/215 to be merged/reassessed.

[bholloway]

That branch from the bolt PR did work for me in windows.

I spoke to one of the bolt maintainers in person today - They will need me to show why it works and determine whether it is the best change.

So I’ll working on that now.

[MadeByMike]

Seems like this could be "fixed" by documenting the issue?

[bholloway]

Yes @MadeByMike docs for the "Developer Mode".

We also need this bolt PR to merge boltpkg/bolt#241. So a PR to keystone to update the dependency.

[1337cookie]

PR 241 has been merged in bolt 0.24.1 (funny how the PR # and version match 🤔simulation theory confirmed?)

[gautamsi]

after #1727 I see this error

🎁 error @arch-ui/controls the dist directory isn't included in the published files for this package, please add it to the files field in the package's package.json
error Command failed with exit code 1.                                                                                  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command. 

[gautamsi]

PR #1727 deprecated use of bolt and now insist on yarn, I have latest yarn installed (v1.19.1).

[Vultraz]

@gautamsi getting that error too. No idea how to fix :( The specific package it complains about does seem to vary, though.

EDIT: the error seems to be coming from preconstruct dev. Even running in an elevated shell or with Developer Mode on doesn't help.

[Vultraz]

Ok, after hours and hours of looking I've tracked down the dist directory isn't included in the published files problem! It's a bug in preconstruct Opened a PR.

Pinging @mitchellhamilton