Flow: Projects get broken for no good reason

[pleku]

_When updating the platform version in pom.xml file, I expect the project to keep on working without any additional changes, instead of causing errors in the client side and forcing me to remove files I have no idea about_

_When some frontend dependency has updated, I don't want it to automatically break my project, but instead keep things working as is and maybe provide me the information that I might want to update things_

Related to issues like vaadin/beverage-starter-flow#383, vaadin/bookstore-starter-flow#152 and vaadin/skeleton-starter-flow#192.

Related slack threads:

• https://vaadin.slack.com/archives/C6RAXJATF/p1561451299172400
• https://vaadin.slack.com/archives/C6RAXJATF/p1561440435160000
• https://vaadin.slack.com/archives/C6RAXJATF/p1561450281166600

[pleku]

For me it would also be fine if I need to do some extra when I add an external 3rd party frontend dependency to the project to get everything up-to-date, in comparison to having to wait for more time on
1) starting up the project without any frontend dependency changes
2) waiting for automatic refresh tools (Java/frontend) to resolve things

Though I do recall someone saying that Webpack should be able to do npm install for requested new modules on the fly.

[caalador]

https://docs.npmjs.com/cli/ci.html

[manolo]

For the investigation:
1.- Revert the starters to the status where the package-lock file was breaking the app
2.- Check that project is broken
3.- run npm ci and see if project fixed
4.- if project continues broken, run rm package-lock.json && npm ci and see if project is fixed.

If project was fixed in 3, we just need to replace npm install with npm clean-install in java executor, and update the documentation about the required npm version.

If project was fixed in 4, we replace npm install with npm clean-install but in addition remove package-lock.json from java when executor updates target/frontend/package.json because a version change (not in the case of a new component addition)

[caalador]

Note on investigation step 4. npm ci states ci can only install packages with an existing package-lock.json or npm-shrinkwrap.json
This means that in the case where package-lock.json doesn't exist we should run npm install for the first installation round to get the package-lock.json file. and onlu it the pacage-lock exists should we run npm ci

[mehdi-vaadin]

Acceptance criteria

• When no dependency has been changed the build should work without any problem. So, if a new version of a dependency is released it shouldn't affect the build.
• When the user bumps to a new version of the platform that should be enough for the build to work and, e.g. they don't need to manually remove any file/folder.

Technical requirements

• When platform's shrinkwrap version changes, the project needs to be cleaned:
  
  
  
  • Remove node_modules, package-lock.json, and /target/frontend/node_modules
  
  
  • Since SNAPSHOT version do not contain shrinkwrap information, the user needs to do this cleaning manually - inform user about this issue.
  
  
  • Suggestion: provide a cleaning scrip/maven task for the user so they do not need to suffer
  
  
• This does not fix the situation where user does the following:
  
  
  
  • clone vaadin-skeleton-starter-flow
  
  
  • test it by running mvn jetty:run
  
  
  • adds new component (such as the Tabs) to MainView
  
  
  • runs again with jetty:run -> BAM! No styles for Tabs
  
  

[ujoni]

Something related to consider:
https://vaadin.slack.com/archives/C6X43FE8M/p1561555544080700

It appears that introducing new components to skeleton started might not work out of the box, if the project has been built once. It will skip npm install and does not download the necessary dependencies.

[denis-anisimov]

Preliminary notes:

• That's not that strait-forward: neither approach works for me (even removal everything doesn't help).
• At the moment the only way which reliably fix the beverage project is: remove node_modules, all config files, do mvn clean. Only after this I may run the application and it works as expected.
• If I have npm ci doesn't fix if I have node_modules. But I can't find a way when it works at all at the moment (so I don't know whether the node_modules is a reason or not)
• as I said : if I remove everything manually: target folder, all package* file, all webpack* files it doesn't help, the project is still broken. As mentioned above: I have to run mvn clean and remove everything.

So I need to see what exactly influence on the behavior when the project is broken. There are too many artifacts.

[denis-anisimov]

So the final results are:

• npm ci doesn't work without node_modules.
• npm ci doesn't help if you don't delete node_modules.
• so npm ci is useless
• to be able to make project working you have to delete: node_modules, package-lock.json and target ( do mvn clean ) or only target/frontend/node_modules inside target .

[caalador]

npm ci works fine without the node_modules directory, but it requires that package-lock exists and all the package.json files should be available e.g. ./package.json and ./target/frontend/package.json
And running npm ci will not help if the package-lock.json is already corrupted (e.g. there exists a /target/frontend/node_modules for the package-lock , this is an error by the updater also as it was not understood that this may happen).

[manolo]

Technical details about implementation decided.

Considerations

• There is the file target/frontend/package.json which is updated each time a new dependency is added or removed or the version changes
• In the file target/frontend/package.json there is an entry for the "@vaadin/vaadin-core-shrinkwrap": "14.0.0-rc1" whose version meets the platform one
• In the node_modules/@vaadin/flow-deps/package.json we have the same content thatn in the target/frontend/package.json
• The shrinkwrap version should be also in the package-lock.json file

Implementation

• When we detect a change in the shrink-wrap version we should delete rm -rf package-lock.json node_modules target/frontend/node_modules
• How to detect the shrink-wrap version would need to visit package-lock.json, node_modules/@vaadin/flow-deps/package.json and target/frontend/package.json

References

• How to read a *.json and get its JsonObject representation https://github.com/vaadin/flow/blob/master/flow-server/src/main/java/com/vaadin/flow/server/frontend/NodeUpdater.java#L182 it should work for parsing the package-lock.json file, for the others there should be already a method using this.
• How to know that a dependency needs to be added because of missing or changed https://github.com/vaadin/flow/blob/master/flow-server/src/main/java/com/vaadin/flow/server/frontend/NodeUpdater.java#L219 it can be used as reference about something has changed
• The point where shrink-wrap dependency is added is in https://github.com/vaadin/flow/blob/master/flow-server/src/main/java/com/vaadin/flow/server/frontend/TaskUpdatePackages.java#L71 here we should check for the new version of the platform and take the decision to remove