All (including @metasoarous, @colinmegill, @urakagi, @patcon, and @david-nadaraia):
TL;DR: what are your thoughts about a more formal git branching structure for pol-is that would include tagged releases, explicit naming of feature, bugfix, and hotfix branches, and potentially a dev branch? I'm very willing to help to develop and implement this structure, but I would like to ensure that potential benefits are worth any costs.
It would provide a more structured way allow development across multiple deployments to be integrated and synchronized. It should also decrease the difficulty and consequences of accepting pull requests from forks, as they could first be accepted into branches that are functional and self-consistent, but not necessarily fully compatible with the master branch.
On our list of goals are to 1) get a dev deployment running in containers (done), 2) get a production ec2 + s3 deployment running in containers, 3) get a production deployment running on aws fargate, 4) update deprecated packages, 5) refactor, & 6) add new functionality. These tasks should not be done in a single branch.
Some short-term steps could be to tag the current master branches as 2.0.0, to create new dev branches off of these, and to break development and merging tasks into smaller and more manageable chunks.
I've been chipping away at pol-is since 2018. I know that there are at least 3 deployments in use: the mothership pol.is, one that @urakai has been working on, and the one that I'm working on, and I'm worried that the current branching system is not flexible enough to support both active deployments as well as new users who are interested in building their own deployments.
I'm quite new to the open source world, but I have been a HPC code user for decades, and I am familiar with a code release cycle that involves dev, pre-prod, and prod releases, and tags for version numbers (Major.Minor.Patch). In our code release world, even minor versions are prod versions, and odd minor versions are pre-prod or dev.
A relatively simple example:
More references:
Respectfully and sincerely,
-Chris Krenn
PS. I know that the branches in my fork are currently badly organized. I can and will work on improving this, most notably by increasing tags and decreasing branches.
The original polis is quite bound to it's developing and deploying environment, and as-is my modification. There are no "core elements" extracted. It's not a matter of versioning yet, in my opinion.
I think you need to step in deeply, separate codes that are environment-dependent and those are not, modules them, then you can try to do what you want to do.
Hi Chris! Hi Urakagi! :wave: Thanks for this initiation and summary!
Related: My understanding is that both @colinmegill and @metasoarous are busy with timely bioinformatics projects, specifically the open-source project https://nextstrain.org/ (which you may have noticed in the news lately). So I suspect they might not have much attention to offer. I've been thinking on how to contribute with that in mind, and so my aspiration was to:
what are your thoughts about a more formal git branching structure for pol-is that would include tagged releases, explicit naming of feature, bugfix, and hotfix branches, and potentially a dev branch?
Obviously, I'm very excited to try something new here! But as someone who's in the past been very excited about the pattern you mention, I'm a soft -1 on using git-flow. Not blocking if other feel that's the solution though! My experience has been that git-flow is more complex than required, and doesn't support the faster pace that it sometimes hints at.
On our list of goals are
Is it alright to ask who "we" is? :)
- get a dev deployment running in containers (done),
- get a production ec2 + s3 deployment running in containers,
- get a production deployment running on aws fargate,
- update deprecated packages,
- refactor, &
- add new functionality.
Agree generally with order 4-6, but feel testing is missing from very beginning, and that infra tasks maybe shouldn't be considered "done" until a solution is found that can ideally be merged upstream, or at least working on a non-divergent fork. But that last bit is just a loose personal preference :)
I would alternatively propose re-committing to mainline development on master (features branches only) but with initiating test coverage and continuous integration. Specifically [uncertain of order]:
The hope was to not actually try to make _any_ real changes to codebase (neither in project organization, nor code style, nor new features) until test coverage is achieved on the area in question. And even then, code style changes should strive to only follow established airbnb conventions, not our personal opinions (just for simplicity).
I feel like I've seen (or heard of) so many people arriving at this codebase, but then wandering away, so I'd love to figure out the smallest possible changes we can comfortably get in to make future edits easier <3
tags for version numbers (Major.Minor.Patch)
Versioning and releases feel interesting, but graciously: to me, tags feel more like something to discuss when the pace is fast and messy, not when we're still figuring out how to take our first cautious steps with outside contributors :)
Anyhow, I hope it's coming across that I'm really grateful for your message @crkrenn! And @urakagi's quick reply! Sorry for all the words -- I'll try to be more terse.
Alternatively, I'd love to arrange a time for any of those interested to jump into an open video call, and maybe discuss some of our hopes and aspirations face-to-face
I also need to mention that I am also busy against COVID-19 since I work for the Taiwan government.
I can glimpse at the process and answering some questions, but not able for deeper works.
@patcon and @urakagi , Thanks very much for your responses and thoughts, and for the reminder of active work on covid.
@patcon, I agree with everything you've written. "We" currently include me and @david-nadaraia, who've I've hired part-time from toptal.
To minimize the pressure on the maintainers and to get our own polis conversations up and running again, I'm leaning towards the following:
Guidance/advice/suggestions for testing would be welcome.
https://jestjs.io/docs/en/getting-started
https://jestjs.io/docs/en/puppeteer
Hi @david-nadaraia 🖖
That sounds like a great split. I'm totally supportive of (1), but will prob work on (2) myself, just since I believe we'll need that before upstream merges will be possible. To be candid, I've only dabbled in testing practices til now, but excited to learn and put it more into practice! 🎉
Interested to do a quick call in a few days? I was thinking we (@urakagi, @patcon, @crkrenn, @david-nadaraia, ...?) might be able to create a shared community fork in a new _temporary_ organization, and try to put into practice ways this repo could be collectively maintained. If so, we should probably sync up on expectations, and so it seems like a call might allow that to happen quickly?
(To clarify, I wouldn't personally want this fork to be used perpetually, but just so we can unblock ourselves and _act_ exactly as if someone here had just chosen to trust us with push access on the main repo 🙂)
All,
I’ve asked David to look at puppeteer testing of the most essential functions over the next week.
I will try to make some nights and weekend progress on the deployment.
I’d be happy to host branches in my github account and could give @patcon and @urakagi access if they have already signed the pol-is contribution form and are willing to go through the pull request process a few times.
I also have some scripts that will get master pol-is up and running on a bare ec2 ubuntu instance. I can commit these to a new directory in polis-deploy with a little time.
I’ve been talking to David about once a week at ~8AM Pacific/8PM Georgia. Can @patcon meet virtually early next week via skype? @urakagihttps://github.com/urakagi, would you like to participate, at least briefly? @urakagi, what hours in Tawain could work? @patcon, what time zone are you in, and what hours could work for you?
-Chris
From: Patrick Connolly notifications@github.com
To: pol-is/polis-issues polis-issues@noreply.github.com
Subject: Re: [pol-is/polis-issues] Changes in pol-is branching model? (#127)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/127#issuecomment-606137410, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V7YMU3F5IDLTH3EWOTRKDJUZANCNFSM4LWC7G7Q.
Great! Are you opposed to using a shared org account as neutral territory, instead of using someone's existing personal or org account? That way, we can all be admins, using shared understandings instead of access control.
Also, definitely strongly in favour of only making changes via PR with +1's, but eager to discuss on call.
Can @patcon meet virtually early next week via skype
:+1:
@urakagi, would you like to participate, at least briefly?
:pray:
@patcon, what time zone are you in, and what hours could work for you?
I'm Atlantic Time, and I'm flexible, so noon AT (8am PT / 7pm GET) would be fine. (Since Taiwan is +12h, I'm prepared for early morning or late night to accommodate if needed)
If you're open to an org account, I can just add you to pol-is-2, which I'd previously created to demonstrate an improvement in how to organize the upstream pol-is org https://github.com/pol-is/polis-issues/issues/99
No objections to the shared org. Particularly as you already have created it. C.
Sent from Workspace ONE Boxerhttps://whatisworkspaceone.com/boxer
On March 30, 2020 at 12:42:04 PM PDT, Patrick Connolly notifications@github.com wrote:
Great! Are you opposed to using a shared org account as neutral territory, instead of using someone's personal or org account? That way, we can all be admins, using shared understandings instead of access control.
Also, definitely strongly in favour of only making changes via PR with +1's, but eager to discuss on call.
Can @patconhttps://github.com/patcon meet virtually early next week via skype
👍
@urakagihttps://github.com/urakagi, would you like to participate, at least briefly?
🙏
@patconhttps://github.com/patcon, what time zone are you in, and what hours could work for you?
I'm Atlantic Time, and I'm flexible, so noon AT (8am PT / 7pm GET) would be fine. (Since Taiwan is +12h, I'm prepared for early morning or late night to accommodate if needed)
If you're open to an org account, I can just add you to pol-is-2https://github.com/pol-is-2, which I'd previously created to demonstrate an improvement in how to organize the upstream pol-is org pol-is/polisServer#189https://github.com/pol-is/polis-issues/issues/99
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/127#issuecomment-606205016, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5VYH2X27KBLWOW7QKHDRKDYX7ANCNFSM4LWC7G7Q.
K, renamed to https://github.com/pol-is-trial-balloon and sent invites! Woooo
I won't make any pushes until we have a chance to speak. (For now, I basically treat it as if Colin and Chris have given us push access to the official repos.)
FY, I had some difficulty because you changed the name after sending the invite. You might want to resend the invite. And, you and I might be the only active users for a little while…
-C
From: Patrick Connolly notifications@github.com
Reply-To: pol-is/polis-issues reply@reply.github.com
Date: Monday, March 30, 2020 at 5:58 PM
To: pol-is/polis-issues polis-issues@noreply.github.com
Cc: Ckl krenn1@llnl.gov, Mention mention@noreply.github.com
Subject: Re: [pol-is/polis-issues] Changes in pol-is branching model? (#127)
K, renamed to https://github.com/pol-is-trial-balloon and sent invites! Woooo
I won't make any pushes until we have a chance to speak. (For now, I basically treat it as if Colin and Chris have given us push access to the official repos.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/127#issuecomment-606332907, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V3HB5WSNOWDYPN2VH3RKE53LANCNFSM4LWC7G7Q.
Ah good call -- for anyone else, visiting this page should allow acceptance of invite:
https://github.com/orgs/pol-is-trial-balloon/invitation
Hey folks! Thanks SO MUCH for kicking off this conversation. This is super timely, actually. (Threading in @ballPointPenguin as well).
The last couple of years we've really been in org building mode, and what little time we've had outside of that has been going towards building more advanced analytics (see https://github.com/pol-is/notebooks). We realize that we have not been the most responsive OS maintainers, and we apologize for the friction that has contributed to. Right now we're getting to a point where strategically we need to start streamlining this process and would like to involve all of you in improving the deployability and contribution process.
@colinmegill and I just finished talking about this, and we think one of the next big steps to improving things is to merge all of the client repositories with the main server repo. A lot of the challenges with devability, deployment and contribution process I think come down to this separation, and can be simplified if we go this route.
Overall, we're warm towards adopting a master/dev/feature-branch strategy and would love to have more testing and CI support in place. However, I do _not_ want agressive linting as part of this process. In my experience, a lot of linting strategies end up being too agressive, slow down the PR process and make it hard for folks to contribute. More testing however would be amazing!
I would love to hop on a call sometime to discuss the path forward. Indeed, both @colinmegill and I are beginning to be wrapped up with covid related work right, so time may be tight, but as I said, unblocking community contribution is quickly rising to the top our priority list.
I should have a bit of time on Friday for a call, and am on Pacific Time. What time works for folks that day?
Thanks again to all of you for your contributions to this project, and for pushing to smooth out the process on this front!
I’m largely free on Friday. I would like @david-nadaraia on the call if possible. He is currently Pacific Time +11 hours. Tawain is Pacific time +15 hours.
+1 on the single repo. Can this include polisMath too? It would deployment/testing/collaboration as well.
+1 on minimal linting, at least until tests are in place. If done, it should be done, tested, and committed to the master quickly. We still haven’t recovered from a branch, lint, develop sequence on one of the projects I am working on.
Sent from Workspace ONE Boxerhttps://whatisworkspaceone.com/boxer
On April 1, 2020 at 7:22:44 AM PDT, Krenn, Chris krenn1@llnl.gov wrote:
I’m largely free on Friday. I would like @david-nadaraia on the call if possible. He is currently Pacific Time +11 hours. Tawain is Pacific time +15 hours.
From: Christopher Small notifications@github.com
Reply-To: pol-is/polis-issues reply@reply.github.com
Date: Tuesday, March 31, 2020 at 4:26 PM
To: pol-is/polis-issues polis-issues@noreply.github.com
Cc: Ckl krenn1@llnl.gov, Mention mention@noreply.github.com
Subject: Re: [pol-is/polis-issues] Changes in pol-is branching model? (#127)
Hey folks! Thanks SO MUCH for kicking off this conversation. This is super timely, actually. (Threading in @ballPointPenguinhttps://github.com/ballPointPenguin as well).
The last couple of years we've really been in org building mode, and what little time we've had outside of that has been going towards building more advanced analytics (see https://github.com/pol-is/notebooks). We realize that we have not been the most responsive OS maintainers, and we apologize for the friction that has contributed to. Right now we're getting to a point where strategically we need to start streamlining this process and would like to involve all of you in improving the deployability and contribution process.
@colinmegillhttps://github.com/colinmegill and I just finished talking about this, and we think one of the next big steps to improving things is to merge all of the client repositories with the main server repo. A lot of the challenges with devability, deployment and contribution process I think come down to this separation, and can be simplified if we go this route.
Overall, we're warm towards adopting a master/dev/feature-branch strategy and would love to have more testing and CI support in place. However, I do not want agressive linting as part of this process. In my experience, a lot of linting strategies end up being too agressive, slow down the PR process and make it hard for folks to contribute. More testing however would be amazing!
I would love to hop on a call sometime to discuss the path forward. Indeed, both @colinmegillhttps://github.com/colinmegill and I are beginning to be wrapped up with covid related work right, so time may be tight, but as I said, unblocking community contribution is quickly rising to the top our priority list.
I should have a bit of time on Friday for a call, and am on Pacific Time. What time works for folks that day?
Thanks again to all of you for your contributions to this project, and for pushing to smooth out the process on this front!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/127#issuecomment-606938164, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V3TD6QGZFNI5D4CO33RKJ32ZANCNFSM4LWC7G7Q.
yesssssss. super psyched. 🎉 ohai @ballPointPenguin! Also, def no need for apologies. Y'all are amazing :)
merge all of the client repositories with the main server repo
💯! I can experiment with a subtree merge approach to preserve valuable git history. (I'll push any experiments asap, and don't let that block others from taking a stab at it.)
Can this include polisMath too?
enthusiastic 👍 from me on this. full monorepo makes me giddy in anticipation ;)
However, I do _not_ want agressive linting as part of this process.
+1 minimal linting. Admittedly, Golang had converted me to the merit of a strong opinion -- _any_ opinion -- and I'm willing at any future point to accept airbnb's rules wholesale, just to have _some_ consensus without discussion :)
so time may be tight, but as I said, unblocking community contribution is quickly rising to the top our priority list.
Alternatively, to keep pressure low on your attention, we could _still_ opt to work in a new org, but knowing there's an explicit desire to upstream the code in short-term. You could chime in on PRs, and we'll try to hash things out without being blocked on you. Upstreaming changes (or moving totally to the pol-is org) could happen on a well-understood timeframe, so we're all clear on details. Happy to queue this for call convo.
What time works for folks that day?
Timezones offer slim overlap, so proposing this:
https://www.worldtimebuddy.com/?qm=1&lid=8,1668341,611717,5037649&h=8&date=2020-4-3&sln=20-21
| who | time | place |
|---|---|---|
| @metasoarous @crkrenn | 8pm Friday | West coast, USA
| @ballPointPenguin | 10pm Friday | Minneapolis, USA
| @patcon | midnight Friday | New Brunswick, Canada
| @david-nadaraia | 7am Saturday | Georgia, Eastern Europe
|@urakagi | 11am Saturday | Taipei, Taiwan
late-nights on Friday and early saturday seems like bad work-life balance, but seems to be what we have if everyone's to be there.
Thanks all!
Morning PST is the other option. I don’t have a strong preference. In my interactions with David, he seems to be more of a night owl than an early bird.
@metasoarous, do you have scheduling constraints or a strong preference on AM vs. PM?
https://www.worldtimebuddy.com/?qm=1&lid=8,6138517,1668341,611717&h=8&date=2020-4-1&sln=7.5-8
PDT/PST, Pacific Daylight Time (US) 7:30a Wed, Apr 1 2020
SAINT JOHN, Canada, New Brunswick 11:30a Wed, Apr 1 2020
TAIPEI, Taiwan 10:30p Wed, Apr 1 2020
TBILISI, Georgia 6:30p Wed, Apr 1 2020
If there's strong support for moving towards a full monorepo (polismath included), I think we'd be open to that. We can use src/{clj,cljs,js,py}/ directories as appropriate to organize source code, and have separate build tools doing their thing side by side (to the best of their ability).
Alternatively, to keep pressure low on your attention, we could still opt to work in a new org, but knowing there's an explicit desire to upstream the code in short-term.
I'd actually rather we keep things centralized here so that it's easier for issues to reference each other and such. To unblock you though, I'll suggest that if you make your PRs to some new nonmaster branch, for now I'm happy to rubber stamp all of your PRs to that branch till we can get the merged into master with a more thorough review. Please let me know how that sounds to you.
I can experiment with a subtree merge approach to preserve valuable git history.
A-MAZE-ING !!! We were thinking about loosing git history as being a big cost to carrying this out, but if you are game to sort this out, that would be phenomenal.
My... this is a hard clock to schedule.
I can't do Friday evening night at 8pm Pacific; I could potentially do around 10pm Pac, but that will be pretty late for some folks. I could do a 7am, 7:30am or 8am Pac as well, either tomorrow or the next day, as long as I know the night before.
Doodle poll: https://doodle.com/poll/mzhaddbs6tnazirr
Another idea here; If it's difficult for us to all meet at the same time, we could always try doing a couple of meetings.
Thanks so much everyone!
@metasoarous Looks like somewhere between 7-9am pacific on friday (tomorrow) works best. wanna say 8-9am PT?
@crkrenn @ballPointPenguin @patcon @david-nadaraia Hey folks; Looks like 7:30am Pacific tomorrow is the ticket. I look forward to speaking with you all then!
I noticed zoom listed in the doodle poll, but I didn't see a link or id. Will it be posted here or sent privately?
We could use https://whereby.com/patcon in a pinch?
Ok, we decided to move to Google Hangouts so we don't have a cap:
https://hangouts.google.com/call/Tbet1nPm1B82yWwmUCbXAEEE?no_rd
Notes: https://hackmd.io/1U58Q-KOSpeDUagCMCDwzQ?view
already signed the pol-is contribution form
Is this still relevant, @metasoarous? If so, any docs or context to share?
The hope was to not actually try to make any real changes to codebase (neither in project organization, nor code style, nor new features) until test coverage is achieved on the area in question. And even then, code style changes should strive to only follow established airbnb conventions, not our personal opinions (just for simplicity).
Strong +1 from me on this
@crkrenn @david-nadaraia how invested in Puppeteer are you at this point? The reason I ask is that I'm a big fan of Cypress and would like that to be considered. I realize I should have chimed in earlier, so if the Puppeteer ship is already sailing, I will join the crew. I have set up Cypress testing for end-to-end javascript heavy web applications at a few companies now and as a veteran web app tester, I am extremely impressed. Here's a poignant medium article if you want to read more https://medium.com/tech-quizlet/cypress-the-future-of-end-to-end-testing-for-web-applications-8ee108c5b255 which makes comparison to Puppeteer and others.
David wrote a few tests with Puppeteer
https://github.com/crkrenn/polis-deploy/tree/feature/docker_compose_localhost_no_math/e2e-tests
We are also talking about the pros and cons of rewriting the javascript code from scratch.
-Chris
Q: Why is this email [hopefully] five sentences or less?
A: http://five.sentenc.es
From: Benjamin Rosas notifications@github.com
Reply-To: pol-is/polis-issues reply@reply.github.com
Date: Wednesday, April 29, 2020 at 11:41 AM
To: pol-is/polis-issues polis-issues@noreply.github.com
Cc: Ckl krenn1@llnl.gov, Mention mention@noreply.github.com
Subject: Re: [pol-is/polis-issues] Changes in pol-is branching model? (#127)
The hope was to not actually try to make any real changes to codebase (neither in project organization, nor code style, nor new features) until test coverage is achieved on the area in question. And even then, code style changes should strive to only follow established airbnb conventions, not our personal opinions (just for simplicity).
Strong +1 from me on this
@crkrennhttps://github.com/crkrenn @david-nadaraiahttps://github.com/david-nadaraia how invested in Puppeteer are you at this point? The reason I ask is that I'm a big fan of Cypress and would like that to be considered. I realize I should have chimed in earlier, so if the Puppeteer ship is already sailing, I will join the crew. I have set up Cypress testing for end-to-end javascript heavy web applications at a few companies now and as a veteran web app tester, I am extremely impressed. Here's a poignant medium article if you want to read more https://medium.com/tech-quizlet/cypress-the-future-of-end-to-end-testing-for-web-applications-8ee108c5b255 which makes comparison to Puppeteer and others.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/127#issuecomment-621391419, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V5HVFHMDICYLTHAPCLRPBYFDANCNFSM4LWC7G7Q.
Nice one with the tests, @david-nadaraia!
We are also talking about the pros and cons of rewriting the javascript code from scratch.
I'm not sure on the rewrite, so I'll wait to hear what others have to say :) @crkrenn Sorry -- the server/client code, or your test code?
EDIT
[tests. code style.]
Strong +1 from me on this
Perhaps we could all discuss on the friday call? Added to agenda: https://hackmd.io/xJ2XZa6mSWeCp4sd_rQepg?view (anyone can add what they want)
how invested in Puppeteer are you at this point?
not at all invested for me, but strongly prefer not to discard work contributed by someone before it's had a fair shake. Would be +1 on using multiple tools -- no loss to run various tools in separate CI runs imho, with different coverage. Would prob help us all agree on the better tool later :)
EDIT: Some of these convos seem bigger than a single-issue convo. Hoping we can spawn new threads soon and migrate pieces to their own places :)
Is it cool if we migrate e2e testing convo here: https://github.com/pol-is/polis-issues/issues/142
(would like to work toward closing out this giant thread :) )
@crkrenn what would you like to see from this thread before we can consider it closed?
Will add here as well: strongly prefer not to add two testing frameworks.
Puppeteer is a great choice.
On Sat, May 2, 2020 at 11:22 PM Patrick Connolly notifications@github.com
wrote:
@crkrenn https://github.com/crkrenn what would you like to see from
this thread before we can consider it closed?—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/127#issuecomment-623047985,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANQGGK2VB6BV5AHEES43O3RPTPRDANCNFSM4LWC7G7Q
.
I think these topics are all represented in other threads. I am closing this. Please let me know if you want to reopen.
Most helpful comment
@crkrenn @ballPointPenguin @patcon @david-nadaraia Hey folks; Looks like 7:30am Pacific tomorrow is the ticket. I look forward to speaking with you all then!