30-seconds-of-code: [SCRIPTS] Major changes coming to all scripts

[Chalarangelo]

As discussed in #762, updates have been made to the packager process to streamline builds. However, the discussion around that change raised a few interesting points, namely that we could use the daily package builds for tests. To further expand on that idea, I will write a short list of the changes that I want to make to the build pipelines, as they have been mostly untouched for a while. Along with these changes, we should update Travis to improve our builds. Here's my proposed plan:

Scripts

• [x] Change the structure of test directory. Move the .test.js files to the top-level directory and remove all the subdirectories. This will help us search things more easily. - #827
• [x] Change rollup.js and module.js. I believe we could combine those into one script that does both things at the same time. The whole packager process should also generate a _30s.js file in the test directory, which will be used on a per-commit basis to test snippets. - #827
• [x] Update tdd.js to do what is stated above (generate tests in top-level test directory) by default. The only other tasks of tdd.js should be to actually run the tests and nothing else. - #827
• [x] Remove the localize.js script. That idea has been discontinued for a while. We can salvage some data from it, if needed, but let's bury that script and move on. - Dealt with in 034d4c5.
• [x] Update extract.js to generate a few more things, like a list.json file for the whole list of snippets with tags (as used in the website search). The long-term plan is to convert the website to a React app with asynchronous calls to JSON files hosted right from GitHub Pages, so let's get the script started on that idea and we will see if and when we can move forward with that. - Dealt with in ccdf861.
• [x] Integrate analyze.js into extract.js, they are essentially part of the same procedure, only we don't run analyzer anywhere just yet. - Dealt with in ccdf861.
• [x] Fix web.js's static page generation. about.html and contributing.html still have no menu. This is also tracked in #788. - Dealt with in b306242.
• [x] Actually create a .eslintrc file from the Codacy rules and the contribution guidelines, plug that into lint.js if possible. - #853
• [x] Remove testlog from test to clean up commits and avoid merge conflicts, set up coverage via jest and configure it to work with Codacy. This will greatly help us figure out what is going on with our code.
• [x] Add a vscode.js script (name not final) to generate the snippets for VSCode as discussed in #722. This will output nightly like packager.js, but we will publish monthly or so like we normally publish the package. - Dealt with in 7d0b8d3.

Travis CI + Other integrations

• [x] As stated above, update the ESLint ruleset from Codacy to work with a file from the repo and set up coverage.
• [x] Update Travis CI order as follows:
  
  
  
  • tagger - Tags are the easiest and first task, as usual
  
  
  • linter - Linting the code should be done early on to avoid issues
  
  
  • packager - We should generate the _30s.js in test in this step and any package files in dist if it's a CRON job
  
  
  • tester - Test everything at this point, generate coverage, kill the build if this fails (fail = coverage below a percentage, let's be less strict about this, a 95% coverage is pretty good and will help us fix issues while the repo moves forward with new additions)
  
  
  • extractor - While not important right now, this will be important to be above the webber script in the future
  
  
  • vscoder - Might use data from the extractor, so this should be below it
  
  
  • glossary:keymaker - Generate the key file for the glossary, this might be important to keep at this point, so its output can be used directly in webber
  
  
  • webber - Build the website, this requires a decent amount of time and some parts run asynchronously, so I think this runs in parallel to the next few tasks
  
  
  • builder - Actually build the README, at this point everything should be ok
  
  
  • glossary:librarian - Generate the glossary library
  
  

[Chalarangelo]

@fejes713 @atomiks @flxwu @skatcat31 Opinions on this?

[fejes713]

I * love this idea ❤️ I see whats your vision on packages - this is definitely doable! Great @Chalarangelo

[flxwu]

Sounds like a good plan! 👍

[skatcat31]

As discussed in #762

That addressed most problems, but I'm curious what you mean about this sections:

The whole packager process should also generate a _30s.js file in the test directory, which will be used on a per-commit basis to test snippets.

I thought you wanted to use the files generated and put in the dist folder? If so why not just navigate to them(either through that file, or through direct navigation)?

[Chalarangelo]

@skatcat31 dist builds are going to be nightly and we should check those anyways. test builds should run after each commit or PR merge to test new code that is being added so we can iron out any issues before the nightly builds.

However, test builds will only generate the ES6 version, not all files and could be clean on the nightly build process if they exist.

[skatcat31]

as long as they aren't on the final push then i'm sold. It's just the duplication between deploying tests and the dist folder that gets to me

[dohboyjoke]

As discussed in #762, updates have been made to the packager process to streamline builds. However, the discussion around that change raised a few interesting points, namely that we could use the daily package builds for tests. To further expand on that idea, I will write a short list of the changes that I want to make to the build pipelines, as they have been mostly untouched for a while. Along with these changes, we should update Travis to improve our builds. Here's my proposed plan:

Scripts

• [ ] Change the structure of test directory. Move the .test.js files to the top-level directory and remove all the subdirectories. This will help us search things more easily.
• [ ] Change rollup.js and module.js. I believe we could combine those into one script that does both things at the same time. The whole packager process should also generate a _30s.js file in the test directory, which will be used on a per-commit basis to test snippets.
• [ ] Update tdd.js to do what is stated above (generate tests in top-level test directory) by default. The only other tasks of tdd.js should be to actually run the tests and nothing else.
• [x] Remove the localize.js script. That idea has been discontinued for a while. We can salvage some data from it, if needed, but let's bury that script and move on. - Dealt with in 034d4c5.
• [ ] Update extract.js to generate a few more things, like a list.json file for the whole list of snippets with tags (as used in the website search). The long-term plan is to convert the website to a React app with asynchronous calls to JSON files hosted right from GitHub Pages, so let's get the script started on that idea and we will see if and when we can move forward with that.
• [ ] Integrate analyze.js into extract.js, they are essentially part of the same procedure, only we don't run analyzer anywhere just yet.
• [ ] Fix web.js's static page generation. about.html and contributing.html still have no menu. This is also tracked in #788.
• [ ] Actually create a .eslintrc file from the Codacy rules and the contribution guidelines, plug that into lint.js if possible.
• [ ] Remove testlog from test to clean up commits and avoid merge conflicts, actually set up coverage via jest and configure it to work with Codacy. This will greatly help us figure out what is going on with our code.
• [ ] Add a vscode.js script (name not final) to generate the snippets for VSCode as discussed in #722. This will output nightly like packager.js, but we will publish monthly or so like we normally publish the package.

Travis CI + Other integrations

• [ ] As stated above, update the ESLint ruleset from Codacy to work with a file from the repo and set up coverage.

• [ ] Update Travis CI order as follows:

  • tagger - Tags are the easiest and first task, as usual
  • linter - Linting the code should be done early on to avoid issues
  • packager - We should generate the _30s.js in test in this step and any package files in dist if it's a CRON job
  • tester - Test everything at this point, generate coverage, kill the build if this fails (fail = coverage below a percentage, let's be less strict about this, a 95% coverage is pretty good and will help us fix issues while the repo moves forward with new additions)
  • extractor - While not important right now, this will be important to be above the webber script in the future
  • vscoder - Might use data from the extractor, so this should be below it
  • glossary:keymaker - Generate the key file for the glossary, this might be important to keep at this point, so its output can be used directly in webber
  • webber - Build the website, this requires a decent amount of time and some parts run asynchronously, so I think this runs in parallel to the next few tasks
  • builder - Actually build the README, at this point everything should be ok
  • glossary:librarian - Generate the glossary library

[Chalarangelo]

Update

I am currently working on the packaging and testing scripts, I'll update this comment as I go:

• Merged module.js and rollup.js into one script. Minor performance increase (8.3s -> 7.8s).
• Updated module.js to cut down the number of I/O operations. Reasonable performance increase (7.8s -> 5.1s).
• Updated module.js to output a full (snippets + archive) _30s.js file in test. Negligible performance impact.
• Made module.js skip bundles when running on non-CRON/API builds. Performance for the test package seems to be ~52ms on my machine, which should be pretty good for running tests on commits.
• c7400ae: Updated tdd.js to follow the structure discussed above, updated all .test.js files, fixed a few failing tests, cleaned up orphaned tests etc. Overall performance is about the same, maybe we managed to earn ourselves a second or two.

Note: Performance is a concern here, if we make tests depend on the packager building a package for the test directory, code needs to be bundled as quickly as possible.

[Chalarangelo]

As far as the tagger is concerned, should we move towards the model of 30interviews (and 30react now) where tags are inlined as comments in the snippets? That could be worth a shot.

[fejes713]

@Chalarangelo I agree with you. Having the same logic everywhere is good both for us and contributors. Less difference = More productive for everyone. Let's give it a shot, shall we?

[Chalarangelo]

@fejes713 Absolutely, we kinda have to change all of the scripts that rely upon this and build a little tool that will migrate all the tags to their individual snippet files then. Doesn't sound too complicated but might take a little bit.

On a side note, if we do this, we can kind of skip on the tagger script and make it all run as part of the linter or some other script that runs early on. After all, we just want to make sure we know which snippets are uncategorized, which would be far easier if we had the tags inlined.

I'm also voting to move expertise into the snippets like in other repos, so we have an expertise comment at the bottom with 0,1,2 - that sort of thing. Sounds a lot easier to work with on the website as well.

This whole lot of changes will probably simplifiy a lot of our processing and speed it up considerably, as we will have one less script to run and we will not have to read and map from the tag_database. Sounds pretty good to me!

[Chalarangelo]

• #788 has been resolved, static pages now have navigation.
• analyzer is gone, extractor is now generating proper data, along with ES5-transpiled versions of the snippets and a snippetList.json file with the minimum data required for snippets (which will eventually be used to populate a React-based website and navigation).
• VSCode snippets are ready, along with instructions. Plugin might come later down the line, but it seems like Azure Devops' bureocracy and some other issues might make this take a little while.

Last thing left to do is to finish up the testing systems. I will give it a shot and report back.

[Chalarangelo]

Coverage is now set up. Check Codacy to see what's missing. 92% is not too bad either. Closing issue, as the changes are finally finalized.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for any follow-up tasks.