Theia: [ci] Ensure that `yarn.lock` is in sync with the state of the `package.json`s

[kittaakos]

Recently, a bunch of PRs introduced new dependencies, but people forgot to commit the yarn.lock changes. Every time I check out a branch and run yarn --force I have an outgoing change. Just check out the master (aa7754c9a5a17a35e3b103514817f784dbbb648d) and run yarn.

Does anyone have an idea how we can avoid this in the future? On the CIs we could run a git status and that should be empty after running yarn and before running the tests. Otherwise, we could bail the build. Are there any other approaches?

[marcdumais-work]

Would using this yarn switch for CI do what we want?
yarn install --frozen-lockfile

https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-frozen-lockfile

[marcdumais-work]

I'll give it a try with the commit referenced above and report what happens

[marcdumais-work]

Hummm... It did not work.

On its own, --frozen-lockfile does not update yarn.lock, even if I think it should, like for commit master (aa7754c9a5a17a35e3b103514817f784dbbb648d) above.

I get the feeling that we would need to also use --force at the same time, but that still does not work. Related issue, I think:
https://github.com/yarnpkg/yarn/issues/5215

[kittaakos]

The following prints 0 if you do not have any Git changes. Otherwise, it prints a positive integer.
Unfortunately, my command line skills are limited to be able to integrate it into the build (Travis-CI would be sufficient) and fail before the yarn run test phase. Could you help me with this @marcdumais-work?

git status -s | wc -c | awk '{$1=$1};1'

[marcdumais-work]

@kittaakos yes, let me have a look

[kittaakos]

We incorrectly check the git status. With this commit, we messed up the Git command.

[kittaakos]

I will take care of this.

[marechal-p]

@kittaakos hi, what is wrong with it ?

[kittaakos]

It should be git status -s and not git status --porcelain.

[marechal-p]

       -s, --short
           Give the output in the short-format.

       --porcelain
           Give the output in an easy-to-parse format for scripts. This is
           similar to the short output, but will remain stable across Git
           versions and regardless of user configuration. See below for
           details.

Did you notice any issues with --porcelain that -s would fix ?

[kittaakos]

I have just checked it, you're right. Still, it does not work. Just clone the repository and run yarn.

[kittaakos]

Here is the full output:

lerna success - @theia/example-electron
✨  Done in 118.57s.
✨  Done in 182.05s.
Akoss-MacBook-Pro:theia akos.kitta$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   yarn.lock

no changes added to commit (use "git add" and/or "git commit -a")
Akoss-MacBook-Pro:theia akos.kitta$ git status --porcelain
 M yarn.lock
Akoss-MacBook-Pro:theia akos.kitta$ git status -s
 M yarn.lock
Akoss-MacBook-Pro:theia akos.kitta$ git diff yarn.lock 
diff --git a/yarn.lock b/yarn.lock
index 70a39014..8744866d 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -1880,6 +1880,14 @@ cache-base@^1.0.1:
 cacheable-request@^2.1.1:
   version "2.1.4"
   resolved "https://registry.yarnpkg.com/cacheable-request/-/cacheable-request-2.1.4.tgz#0d808801b6342ad33c91df9d0b44dc09b91e5c3d"
+  dependencies:
+    clone-response "1.0.2"
+    get-stream "3.0.0"
+    http-cache-semantics "3.8.1"
+    keyv "3.0.0"
+    lowercase-keys "1.0.0"
+    normalize-url "2.0.1"
+    responselike "1.0.2"

 caching-transform@^1.0.0:
   version "1.0.1"

[kittaakos]

I am a bit puzzled, it happens when you call yarn --force not yarn. How is that possible?

[marechal-p]

Isn't yarn --force overwriting the yarn.lock if new versions are available ?

https://github.com/yarnpkg/yarn/issues/4379#issuecomment-331836680 ?

[kittaakos]

Isn't yarn --force overwriting the yarn.lock if new versions are available ?

yarnpkg/yarn#4379 (comment) ?

Where did you see this? I must be blind.

yarn --force should not do anything else but refetch the dependencies even if they're already installed.

yarn upgrade should downloading the latest version of all dependencies and yarn upgrade package-name should update the version of the package-name dependency. What do you think?

[marechal-p]

It was a genuine question from me, as I don't really know all the options for yarn, but it make sense that it should only re-fetch dependencies. On the other hand, I just tried to use yarn --force instead of the regular yarn, and I end up with the same diff as you :/

When I just use yarn I have no diff. This behavior is weird indeed. Maybe worth an issue on the yarn repo?

Maybe this is also only a one time thing and we should do it once and commit the new yarn.lock.

Else would git clean -xfd and then yarn work?

[kittaakos]

I do not see this problem anymore. Perhaps, one has already pushed the yarn.lock change. Please reopen if the same happens in the future.