Image-sequencer: Discuss: workflow for managing Dist files (i.e. on a 'stable' branch)

[dojutsu-user]

A new demo folder should be created, containg the deployment code, so that each time dist folder does not raises Merge Conflict errors.

• A demo folder or a separate branch can be created from where the code can be deployed. This will ensure that if any wrong code commited to main branch does not get deployed immediately.
• Also this will solve the problem of having Merge Conflicts with the dist files.
• dist folder should be added to .gitignore.

What's your thought on this? @jywarren @tech4GT

[welcome[bot]]

Thanks for opening your first issue here! Please follow the issue template to help us help you 👍🎉😄
If you have screenshots to share demonstrating the issue, that's really helpful! 📸 You can make a gif too!

[Divy123]

@dojutsu-user great thoughts.
Can you give some example where such pattern has been followed?
I would be really helpful.
Thanks for your help. :smile:

[harshkhandeparkar]

@dojutsu-user there has been a lot of discussion about managing dist files. I think the decision was to create a pre-commit hook to ensure that nothing wrong is committed.

611

[dojutsu-user]

@Divy123
Yes.

1. You can see a similar pattern in this repo: https://github.com/mozilla/kuma. If you see the list of branches there (https://github.com/mozilla/kuma/branches), a separate branch prod-push is used for the deployment purpose.
2. Also, you can see a similar pattern in Read the Docs as mentioned in their docs: https://docs.readthedocs.io/en/stable/faq.html#what-commit-of-read-the-docs-is-in-production

Many github repositories (like elasticsearch: https://github.com/elastic/elasticsearch/) also use separate branches for maintaining their different releases.

[harshkhandeparkar]

@dojutsu-user your idea of separate branches is pretty cool. I had suggested doing that before but @jywarren said that cloning the repo and having the code ready is much better than building it manually or pulling from somewhere else. What do you think?

[dojutsu-user]

@HarshKhandeparkar
Contributors need not to clone another branch except the main for the development purpose. Only the deployment will be done through another branch.

deployment branch can be merged with the main branch later.

[harshkhandeparkar]

@dojutsu-user that is not what @jywarren meant(i think). What he meant was that adding dist to gitignore would create problem. Since the code version is not bumped very often, the code is not linked on github releases. jywarren wants the users yo be able to clone/zip the repo and have the latest dist readily available.

[dojutsu-user]

@HarshKhandeparkar Let's see his thoughts on this.

[harshkhandeparkar]

Sure 😊

[Divy123]

Cool examples dojutsu-user.
Harsh one thing I recommend that tagging someone one time will send an email to the concerned people.
And thanks for your reviews, they mean a lot.

[harshkhandeparkar]

Divy123 sometimes people prioritize their emails by whether they are tagged or not. If they are tagged earlier, they will receive the email even if they wre not concerned so it is a good way to sort/prioritize. Also I don't like referencing people by name, rather by username. Thanks.

[Divy123]

Sorry closed by mistake.

[jywarren]

Hi, all, there are a few different threads here, and I do want to bring the discussion back to #601 where we talk about how to manage dist files. One suggestion there was to not ask for dist files on each commit, and to just generate them periodically. If we want to /also/ have a discussion about an alternative branch, the plots2 project (https://github.com/publiclab/plots2/) uses a stable branch to deploy from, so there is one extra layer between merged code and publication. But while this is cool, we mainly make use of it by publishing to a stable build server so that people can test things out one last time before they go live. We'd want to do the same here or it might not really add much to just have a separate branch (unless it's part of the dist files discussion).

We could consider the stable branch idea here, and break off the dist files discussion back to #601 maybe?

[jywarren]

And -- thank you!!!!!

[Divy123]

That would be great to have a stable branch to deploy and I think that could make the maintenance work divided and cleaner.
Thanks.
I think we can implement it right away.
I would like to help in this.

[dojutsu-user]

@jywarren
Aren't the problem of dist files and stable branch related?
Making a stable branch will resolve the problem of dist folder and then dist folder can be added to .gitignore.

[harshkhandeparkar]

We can discuss on gitignoring dist files. Moved the pre-commit hook discussion to this issue.

[jywarren]

OK, I've opened a documentation issue to clarify the policy on submitting dist files to PRs (i.e. you don't have to anymore, and I'll compile them in periodically). I think we can do this without stable for starters, but a bigger issue for me is that I'd like some help regularly (every week? 2 weeks?) building everything and publishing a new release. This could be done via a stable branch, maybe.

But is it something we could share responsibility for, and is it something we could manage via PRs?

What if anyone in @publiclab/is-reviewers could open a PR with built files from main to stable branch once per week, and we had a chance to look it over, and if it worked, we'd try to automate the process of:

1. merging to stable
2. publishing a new release on npm
3. updating gh-pages demo

Anything else?

Thank you all!

[jywarren]

OK, I've opened a documentation issue to clarify the policy on submitting dist files to PRs (i.e. you don't have to anymore, and I'll compile them in periodically). I think we can do this without stable for starters, but a bigger issue for me is that I'd like some help regularly (every week? 2 weeks?) building everything and publishing a new release. This could be done via a stable branch, maybe.

But is it something we could share responsibility for, and is it something we could manage via PRs?

What if anyone in @publiclab/is-reviewers could open a PR with built files from main to stable branch once per week, and we had a chance to look it over, and if it worked, we'd try to automate the process of:

1. merging to stable
2. publishing a new release on npm
3. updating gh-pages demo

Anything else?

Thank you all!

[harshkhandeparkar]

If the tasks listed above are done by running commands, I can create scripts for those like the one created in #857

[jywarren]

Yes, i think you got them except for publishing to npm -- it'd be npm publish but it's only one line, so don't need a script. Thanks!

[harshkhandeparkar]

@jywarren shall we start by creating the stable branch atleast?

[harshithpabbati]

What's the issue??

[harshkhandeparkar]

We want to gitignore dist files and force add them to a stable branch periodically.

[harshithpabbati]

If you need any help I would really love to do it

[harshkhandeparkar]

Only @jywarren and other maintainers can create and push to a protected branch so both of us can't do anything.

[harshithpabbati]

Oh

[jywarren]

Sorry Harshith! Yes, i'll put some thought into this, and like the idea --
but if you could write up a procedure -- documentation -- for how to do
this, we could post that so there is more transparency about how it works,
too. We could brainstorm/refine the docs in this issue. Thanks! Very busy
today, apologies for replying slowly!

On Tue, Mar 19, 2019 at 3:04 PM Harshith pabbati [e-mail]
wrote:

Oh

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/image-sequencer/issues/773#issuecomment-474528114,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ0asdto82TAPdtdyFJOoeDaH792kks5vYTSGgaJpZM4a8C6W
.

[jywarren]

OK, reopening with an inquiry -

Now, when I build dist files in stable, i can no longer cleanly merge from main. It has to be rebased, or it has to be a merge commit. Is this ok with folks?

[harshkhandeparkar]

@jywarren I have messaged you on gitter about the same. Please have a look at it.

[jywarren]

Just copying in Harsh's response here:

Hello! I just wanted to tell that the normal script you use to update the gh-pages branch and maybe stable branch will not work because of gitignored dist. There is a nice script(created by me) on Harshith's repo https://github.com/Harshithpabbati/Update-is . This script only adds the required node_modules, only the required dist and also removes unnecessary src files, test/ and spec/ etc. Maybe you can take some inspiration from that script. It even has an option to download its latest version(update itself)!

[jywarren]

I think we still need to decide if stable's git history should be kept linear or if we should do merge commits after building and adding dist files...

My feeling is that we should treat main as canonical and never add merge commits there. Perhaps either stable can have merge commits, or we make another branch called "dist" which is always just one commit ahead of main?

[harshkhandeparkar]

@jywarren if we want to regard npm versions as stable versions, we can create releases for each of those.

[jywarren]

Sure -- but unfortunately, if we aren't building or tracking dist files on the master branch, then any npm version is a branch off the trunk, because it includes commits not included in the master branch.

I can imagine two ways forward:

1. we track dist files in the main branch, but continue to gitignore them, and have maintainers build them periodically; new PRs would not include changes to dist files. And, when we build the dist files, we merge to stable and publish to NPM.
2. we create a new branch with built dist files each time we publish to NPM and gh-pages, but that is not in the trunk; we abandon that branch and keep adding to the main branch only.

I guess I prefer scenario 1 -- because it seems simpler and more linear, but there are pros and cons to each. @publiclab/is-reviewers what do you think?

[jywarren]

Some thoughts on this here, but i'd love to read what best practices are in another project; please share links if you can! https://gist.github.com/nzakas/7633640

We could also just look at how some other repositories are set up, like https://github.com/bevacqua/woofmark or a Leaflet library.

[jywarren]

I just published to gh-pages, by the way!

[jywarren]

@tech4GT sorry this is holding up publishing the new gl work to npm... love to hear your input too!

[harshkhandeparkar]

• If maintainers build dist files periodically on the main branch (as in scenario 1) it will be almost the same as pushing to stable periodically.

• Secondly, creating a new branch/tag each time we push to npm will help others get a specific version of the library in the browser. They can install a specific version in node but if they want to do the same in the browser, they can just include the dist files from a tag/ref/branch for the release.

• I prefer the second scenario as it is more systematic even though it is a bit difficult.

• Keeping the commit history clean in all branches may be important devs but not the end users(i think). Adding just an extra dist commit wouldn't be much of a change.

What are your thoughts? @jywarren @publiclab/is-reviewers

[tech4GT]

@jywarren One thing we do need to make sure is that dist files are published to npm, since gl-work relies on that.

[tech4GT]

Also, I think approach 1 is much more manageable!

[harshkhandeparkar]

Why are the dist files reqd in npm?

[tech4GT]

Because I used them inside puppeteer to set up Image Sequencer.

[harshkhandeparkar]

Can't it be made dynamic? Can't the ImageSequencer.js file be included using a require() statement?

[harshkhandeparkar]

Another problem with approach no. 1:

If someone clones the repo, the code will(most of the time) work differently in the browser than in node as the dist files will(can) be outdated. Some things can work in node whereas they may not work in browser.

@jywarren @tech4GT

[tech4GT]

It would be okay Harsh since before running on localhost the files built locally anyway.
Also you can’t use require in browser context, which is why we need to have the dist script in the first place.

[harshkhandeparkar]

Ok. The dist isn't a problem. I was just asking.

I also don't mean that the dist will cause problems for the devs as they will obviously build them. If they build them, there isn't a neee to add them in the first place but that is a different case.

I am talking about users. Suppose they want to use the lib. They clone the repo and use the dist in the browser. Now it they use the same repo to use or maybe do some quick tests in node itself, it a may work differently.

Maybe we can take a mixed approach. We can periodically push builds as well as create tags/refs/releases for each npm version. We can actually do that even using curl or anything that can interact with the github API. Creating tags will help people in getting the version of their choice and pulling main or stable can give them the latest ones as well.

I would still insist on managing files on stable though. Maybe we can give the latest files on unstable? This will be more organized but also a bit difficult.

How about a poll? We can ask all IS contributors/reviewers/users anonymously.

We can also ask for suggestions from other contributors of other projects of PL as well as other orgs

@tech4GT

[jywarren]

If someone clones the repo, the code will(most of the time) work differently in the browser than in node as the dist files will(can) be outdated. Some things can work in node whereas they may not work in browser.

I agree this is an issue, but we should be able to add a section to the README about it. For the version on NPM, we can (if we are rigorous) guarantee that the dist files will match the src files, and we can recommend that anyone doing a stable usage should point at NPM, not GH... what do you think?

I would still insist on managing files on stable though. Maybe we can give the latest files on unstable? This will be more organized but also a bit difficult.

Sorry can you elaborate on this?

Regarding "having the latest dist files in main branch" i think ultimately it's OK to have the dist files in main branch, but have them not be the 100% most up to date. If people want the latest greatest code, they can accept the compromise that they won't be precompiled. And if their use case requires stability and stricter matching of dist/src, they can point at NPM. I think this sounds reasonable, no?

So the policy could then be that dist files are in main, but we update them only when we merge to stable. Then stable becomes the source for NPM, and comes with more stability and guarantees. main is more like our development branch, but will always be able to FF-merge into stable after being built.

What do you say?

[VibhorCodecianGupta]

@jywarren let's get this solved before SoC begins. This is essential to avoid inconvenience. I'll get on it tomorrow, read up the discussion here and come up with something :D

[harshkhandeparkar]

Sorry can you elaborate on this?

I mean that we can push the dist files of each commit into an unstable branch(somehow, can this be automated?). Only stable npm version files on stable. This will be difficult but organised. @jywarren

[jywarren]

@HarshKhandeparkar you mean stable and NPM wouldn't have dist files? I think we can't do that, as people really rely on dist files for downstream production work. Or do you mean we'd branch off a release branch like v3.2.0 and publish to NPM from there each time we did that?

Like right now, because we don't track dist files at all, i would need to do a merge commit (or worse, rebase the entire main branch), in order to merge to stable and rebuild the dist files. So there's a pretty messy tangle in our git history each time that happens. I think tracking dist files but also .gitignoring them makes sense as a result. I'm not sure how to publish to NPM right now, is the issue.

Thanks all, for your thoughts!

[jywarren]

So summary, i think our options are:

1. branch off a release branch like v3.2.0 and publish to NPM from there each time after merging to stable
2. track dist files in main but only build and track built versions just before merging to stable; stable and NPM versions will be same and include dist files

[harshkhandeparkar]

@HarshKhandeparkar you mean stable and NPM wouldn't have dist files? I think we can't do that, as people really rely on dist files for downstream production work. Or do you mean we'd branch off a release branch like v3.2.0 and publish to NPM from there each time we did that?

Oops not at all. I mean that we will always push dist files to npm and branch tags. What will be the latest on npm will be on stable. What will be on main will be on a new unstable branch but will also have dist files. This will be difficult but can be managed(maybe it can be automated?)

[jywarren]

Oh ok thanks, so that is pretty close to (2) above, right?

[harshkhandeparkar]

Like right now, because we don't track dist files at all, i would need to do a merge commit (or worse, rebase the entire main branch), in order to merge to stable and rebuild the dist files. So there's a pretty messy tangle in our git history each time that happens. I think tracking dist files but also .gitignoring them makes sense as a result. I'm not sure how to publish to NPM right now, is the issue.

I don't think clean git history is a requirement (in my opinion it isn't) because people(users) will only pull the stable branch, they won't care about the history, for a history they can have a look at the main branch commit history.

For managing files and npm publishes, I can create a script for you so that you can open your terminal, tyoe one command, type your username and password. Done. What OS do you use?

[harshkhandeparkar]

Oh ok thanks, so that is pretty close to (2) above, right?

Not really, it is a mix of both. So I mean that we never push dist files to main.

[jywarren]

Oh, so where do we build them? Maybe I don't exactly follow What will be on main will be on a new unstable branch but will also have dist files.. Thank you for explaining!

[jywarren]

Whoa... just noting: Downloading Chromium r641577 - 109.8 Mb [================= ] 85% 31.4s

since we started including puppeteer... but yeah, cool! @tech4GT so did you say this actually uses the dist files locally in main? I think that may be a good reason to have them in main as well.

[harshkhandeparkar]

We have to install pupeteer and other deps anyways. For that we use the setup script. The setup script already builds dist files.

[jywarren]

Well, some people may use npm install but in any case, can you describe the disadvantage to having dist files in main, so we can weigh the pros and cons? Thanks!

[harshkhandeparkar]

Maybe I don't exactly follow What will be on main will be on a new unstable branch but will also have dist files.

Unstable branch(which we need to create) can hold the latest beta. We will build dist files here. After each PR, we can push to this branch. Or at least once every 2 to 3 PRs. I can semi-automate this using a script. Only barrier between fully automated and semi-automated(you will have to run the command) is authentication.

Ok. I can create a highly specialised script with different options for npm pushes, unstable pushing, etc. Each time you push to npm, it will also push to stable. This will only be done when bumping npm version. I love writing scripts(I had mentioned harshith's update-is before).

So, what OS do you use?

[jywarren]

I use Linux but sorry @HarshKhandeparkar i just don't want to make work for you on script writing before we've come to a consensus on this. Thanks!

[jywarren]

Also, shouldn't we try to write an NPM script that would be cross-platform? But i think we may be getting ahead of the goal here, which is to decide a workflow. Can you tell a bit about the disadvantages you see to including dist files in main? Thank you!!!

[harshkhandeparkar]

Well, some people may use npm install but in any case, can you describe the disadvantage to having dist files in main, so we can weigh the pros and cons? Thanks!

Ok. Pros and cons of not having dist files:(in my opinion)

Pros

• No conflicts
• There will be no version conflicts(difference in browser and node as mentioned above

Cons

• end users need to take the extra step of cloning or pulling from stable
• devs and users have to be extra careful

We can actually manage the dist files building on main(when it is freshly pulled) by documenting it properly. We can directly link to stable branch in the docs wherever it links to dist files.

[harshkhandeparkar]

Also, shouldn't we try to write an NPM script that would be cross-platform?

Yes that can be done.

[harshkhandeparkar]

Pros and cons of having dist files:

Pros

• Clone it, use it
• No need to manage extra branches

Cons

• There can be differences in browser and node
• Dist files can confuse newcomers

[jywarren]

This is really helpful. Let's hear from some others now. Thanks, Harsh!

[harshkhandeparkar]

Sure.

[harshkhandeparkar]

Majority opinion should be full and final.

[jywarren]

Just found this guide to making gh-pages build from Travis!

“Automate deployment of projects to GitHub-pages using Travis CI” by Dhruv Barochiya https://link.medium.com/9wOQZZr4JX

[tech4GT]

@jywarren I would then suggest that, we keep a beta branch for all the dev work and only push to master on releases. We don't want buggy code to get deployed.

[harshkhandeparkar]

@tech4GT what do you mean by dev-work?

[tech4GT]

I mean the any new code should be pushed to the beta branch first! It should actually get deployed with the next release.
This will make sure that even if a PR with buggy code get's merged, it's not immediately deployed.

[harshkhandeparkar]

We have a beta branch on Harshith's repo. The code is pushed to it either by me or him.
https://beta.sequencer.publiclab.org. I had also suggested the creation of unstable branch.

[harshkhandeparkar]

@publiclab/is-maintainers Should we close this now?

[jywarren]

sounds good!