Theia: bug: failure to build using 'next' packages

[vince-fugnitto]

Bug Description:

The application fails to build when using next packages (as also viewed from theia-apps). When attempting to 'compose an application', the build results in the following failure:

$ yarn run clean && yarn build && yarn run download:plugins
yarn run v1.22.4
$ theia clean
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module '../lib/theia'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/home/evinfug/Desktop/test-app/node_modules/@theia/cli/bin/theia:2:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Steps to Reproduce:

If one wants to reproduce, you can simply follow the steps provided in composing applications.

1. create a directory, and a package.json inside of it

2. update the package.json to include the following:

   {
   "private": true,
   "dependencies": {
       "@theia/callhierarchy": "next",
       "@theia/file-search": "next",
       "@theia/git": "next",
       "@theia/json": "next",
       "@theia/markers": "next",
       "@theia/messages": "next",
       "@theia/mini-browser": "next",
       "@theia/navigator": "next",
       "@theia/outline-view": "next",
       "@theia/plugin-ext-vscode": "next",
       "@theia/preferences": "next",
       "@theia/preview": "next",
       "@theia/search-in-workspace": "next",
       "@theia/terminal": "next"
   },
   "devDependencies": {
       "@theia/cli": "next"
   },
   "scripts": {
       "prepare": "yarn run clean && yarn build && yarn run download:plugins",
       "clean": "theia clean",
       "build": "theia build --mode development",
       "start": "theia start --plugins=local-dir:plugins",
       "download:plugins": "theia download:plugins"
   },
   "theiaPluginsDir": "plugins",
   "theiaPlugins": {
       "vscode-builtin-css": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/css-1.39.1-prel.vsix",
       "vscode-builtin-html": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/html-1.39.1-prel.vsix",
       "vscode-builtin-javascript": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/javascript-1.39.1-prel.vsix",
       "vscode-builtin-json": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/json-1.39.1-prel.vsix",
       "vscode-builtin-markdown": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/markdown-1.39.1-prel.vsix",
       "vscode-builtin-npm": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/npm-1.39.1-prel.vsix",
       "vscode-builtin-scss": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/scss-1.39.1-prel.vsix",
       "vscode-builtin-typescript": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/typescript-1.39.1-prel.vsix",
       "vscode-builtin-typescript-language-features": "https://github.com/theia-ide/vscode-builtin-extensions/releases/download/v1.39.1-prel/typescript-language-features-1.39.1-prel.vsix"
   }
   }
   

3. perform yarn and notice the error

Additional Information

_yarn_: v1.22.4

[thegecko]

This is due to the lib folder missing on @next releases since https://github.com/eclipse-theia/theia/commit/d9a0973039c571d53dba1d6c18ec5759ef071403.

It doesn't look like that PR caused an issue, that's just when it started happening.
I've confirmed the @theia/cli and @theia/core packages (at least) are missing the lib folders since this version.

[thegecko]

Looking at the build job from then, I can't see how theia is even built before deploy, the step which does it is skipped.

I assume therefore, we load it from cache somewhere/somehow. I'll stop my investigation here as I'm sure there are others with a better understanding of the CI setup :)

[akosyakov]

It could be that something went wrong with npm/yarn during last publishing and next publishing will fix it.

[thegecko]

It could be that something went wrong with npm/yarn during last publishing and next publishing will fix it.

There has been a publish since then which didn't fix it :(

[marcdumais-work]

I just tried building a next theia app, that it built. I also confirm I see the lib folder for the core and cli extensions, so it looks like the publish is normal again? Can someone confirm?

[vince-fugnitto]

I just tried building a next theia app, that it built. I also confirm I see the lib folder for the core and cli extensions, so it looks like the publish is normal again? Can someone confirm?

I'll restart the cron on theia-apps and hopefully it passes :)

[marcdumais-work]

I think we have at least one new problem, seemingly related to ripgrep:
```[4/4] Building fresh packages...
598error /home/theia/node_modules/vscode-ripgrep: Command failed.
599Exit code: 1
600Command: node ./lib/postinstall.js
601Arguments:
602Directory: /home/theia/node_modules/vscode-ripgrep
603Output:
604internal/modules/cjs/loader.js:638
605 throw err;
606 ^
607
608Error: Cannot find module 'proxy-from-env'
609 at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
610 at Function.Module._load (internal/modules/cjs/loader.js:562:25)
611 at Module.require (internal/modules/cjs/loader.js:692:17)
612 at require (internal/modules/cjs/helpers.js:25:18)
613 at Object. (/home/theia/node_modules/vscode-ripgrep/lib/download.js:11:24)
614 at Module._compile (internal/modules/cjs/loader.js:778:30)
615 at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
616 at Module.load (internal/modules/cjs/loader.js:653:32)
617 at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
618 at Function.Module._load (internal/modules/cjs/loader.js:585:3)
619info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
620The command '/bin/sh -c yarn --pure-lockfile && NODE_OPTIONS="--max_old_space_size=4096" yarn theia build && yarn theia download:plugins && yarn --production && yarn autoclean --init && echo .ts >> .yarnclean && echo *.ts.map >> .yarnclean && echo *.spec. >> .yarnclean && yarn autoclean --force && yarn cache clean' returned a non-zero code: 1
621The command "./build_container.sh $NPM_TAG $IMAGE_NAME $NODE_VERSION $PORT $ENV_VARS" failed and exited with 1 during .

It happens that a new version was released yesterday, and yet another new one today:

'1.5.9': '2020-06-07T19:20:36.052Z'
'1.6.0': '2020-06-08T14:12:13.919Z'
```

[marcdumais-work]

I assume the yarn.lock is currently insulating the main repo from the issue above, pinning ripgrep to an earlier version vs what we pull in theia-apps.

[marcdumais-work]

New issue opened on vscode-ripgrep: https://github.com/microsoft/vscode-ripgrep/issues/13

[marcdumais-work]

As a temporary work-around for affected apps, we could use a resolutions block that pins vscode-ripgrep to 1.5.8.

[marcdumais-work]

Looks to be fixed in vscode-ripgrep v1.6.1

[marcdumais-work]

Closing this issue for now - I think we do not know exactly what caused it, but it seems gone. Please re-open if not.

[cdietrich]

this seems to happen again

[cdietrich]

yarn run v1.22.4
$ /Users/cdietrich/git/xtext-languageserver-example/theia-vscode/node_modules/.bin/theia build
internal/modules/cjs/loader.js:984
  throw err;
  ^

Error: Cannot find module '../lib/theia'
Require stack:
- /Users/cdietrich/git/xtext-languageserver-example/theia-vscode/node_modules/@theia/cli/bin/theia
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:981:15)
    at Function.Module._load (internal/modules/cjs/loader.js:863:27)
    at Module.require (internal/modules/cjs/loader.js:1043:19)
    at require (internal/modules/cjs/helpers.js:77:18)
    at Object.<anonymous> (/Users/cdietrich/git/xtext-languageserver-example/theia-vscode/node_modules/@theia/cli/bin/theia:2:1)
    at Module._compile (internal/modules/cjs/loader.js:1157:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1177:10)
    at Module.load (internal/modules/cjs/loader.js:1001:32)
    at Function.Module._load (internal/modules/cjs/loader.js:900:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/cdietrich/git/xtext-languageserver-example/theia-vscode/node_modules/@theia/cli/bin/theia'
  ]
}

[cdietrich]

seems to have selfrepaired