Server: Clarification on branches

[Julusian]

Today there were some force pushes against the 2.1.0 branch so that git lfs could be enabled for the branch including the history. As the history was rewritten this killed all the open pull requests and caused confusion for some of us in the community.
This has been reverted but the PRs are still a bit broken and are all closed. Am I right in thinking that github will not let those be reopened? If so it would be good to make a list somewhere of what was closed and may be worth merging, so that they dont get lost and forgotten, or perhaps just to reopen them as new PRs.

Additionally, there is now a 2.2.0 branch. Does this mean that 2.1.0 is being skipped and work will continue on 2.2.0? And so should any future PRs be against 2.1.0 or 2.2.0?

@ronag

[ronag]

GitHub auto closes prs when the history is re-written. It then doesn’t allow re opening even if the history is restored.

I’ve labeled the force closed prs with “cherry-pick”.

2.2.0 is what’s we are going onwards with. There are some breaking changes relative to 2.1.0 in the works. Mostly in the form of removing rarely used and/or unstable features. The priority is to get a stable version out.

Sorry for the confusion. I’ve got limited amount of hours budgeted on this.

[dotarmin]

Me and Robert had a quick chat yesterday about this and came across that it would be best to branch off 2.1.0 and continue with a 2.2.0 branch instead. As Robert says, the primary focus now is to get a stable version out and from that continue everything else.

Sorry for the confusion! It was quick decisions about this one.

Best regards,
Armin

[Julusian]

I’ve labeled the force closed prs with “cherry-pick”.

OK, if they get merged before 2.2.0 then that should be fine. If they are still there after that then it would be worth making them more visible.

There are some breaking changes relative to 2.1.0 in the works. Mostly in the form of removing rarely used and/or unstable features.

Are either of you able to share what the features being removed are? It would be useful to know what features to avoid using if they are going to be removed or pushed to a future release.

Is 2.2.0 likely to get another force push, or are you happy with it now? I expect I will have some more PRs to raise soon, but I can hold off for a while if 2.2.0 is likely to have its history changed again.

[jesperstarkar]

Are either of you able to share what the features being removed are? It would be useful to know what features to avoid using if they are going to be removed or pushed to a future release.

+1 really, it would be good to know what's being planned.

[ronag]

Are either of you able to share what the features being removed are? It would be useful to know what features to avoid using if they are going to be removed or pushed to a future release.

Breaking stuff so far:

• [ ] Re-do cmake
• [ ] Refactor dependencies
• [ ] Remove test
• [ ] Remove auto generated docs
• [ ] Remove deploy & CI stuff
• [ ] Remove framerate_producer
• [ ] Remove channel_layout in favour of ffmpeg filters (producer + consumer) and a fixed internal 8 channel audio pipeline

Additional breaking stuff might come up in the future... @dotarmin please correct if something is incorrect.

[ronag]

Is 2.2.0 likely to get another force push, or are you happy with it now? I expect I will have some more PRs to raise soon, but I can hold off for a while if 2.2.0 is likely to have its history changed again.

It should be ok now. We will do a force push across the board on all branches in the future to truly get rid of the huge repo but not right now.

[Julusian]

Thanks, that mostly sounds fine to me, just a couple of queries.

Remove deploy & CI stuff

Do you mean the deploy and build-scripts folders? Will the scripts be merged into part of cmake, or will it end up being more steps to follow from the README when compiling?

Remove framerate_producer

This feels like it might still have some value if using reroute across channels with different framerates. However this is probably not a very likely scenario, so I dont mind it going.

Downgrade psd_producer, stage_producer, reroute_producer to experimental

I assume you mean scene_producer not stage_producer.
As long as they are still available in the default compile, perhaps disabled by default with a config option to enable them, then I have no problem with that.

[jesperstarkar]

It would be a shame to se reroute_producer gone, without any opt-in options.

[Julusian]

It should be ok now. We will do a force push across the board on all branches in the future to truly get rid of the huge repo but not right now.

Will you have to remove the tags too? It might be cleaner to move the old tags and branches to a CasparCG-Legacy repository instead so that they can be preserved and to save yourself the effort of having to enable lfs on them.
I'm not sure if the old branches and tags will be of use to anyone, but its not hard to move them there just in case they are.

[dimitry-ishenko]

• [ ] Re-do cmake
• [ ] Refactor dependencies

@ronag would it be possible to add an option to build with external (vs bundled) dependencies? Similar to what I did in #394 awhile ago?

[Julusian]

@ronag I was getting the following error when attempting to clone 2.2.0. Looks like something went wrong when pushing a file to lfs.

Downloading dependencies64/large_files_linux.tar.xz (25 MB)
Error downloading object: dependencies64/large_files_linux.tar.xz (4b7114f): Smudge error: Error downloading dependencies64/large_files_linux.tar.xz (4b7114fc0a784784b0645b396b98e7817be997ab8911d157c748a8f17e00b977): [4b7114fc0a784784b0645b396b98e7817be997ab8911d157c748a8f17e00b977] Object does not exist on the server: [404] Object does not exist on the server

I also noticed that dependencies64/large_files_windows.7z has some versions still not in lfs.

In my googling to see if I could push the missing file (which I was able to) I have found that git-lfs ship a command to help migrate a repo. After a bit of playing, I used the following command to get the repo down to 55mb (vs the 345mb that 2.2.0 currently is)

git lfs migrate import --include="*.mp4,*.mov,*.pdf,*.psd,*.dll,*.lib,*.exe,*.7z,*.zip,*.bmp,*.tga,*.gz,*.tar,*.xz,*.so*,*.sdf,*.pak,*.ttf,*.lib,*.100,*.101,*.102,*.a,*.55,*.2,*.0,*.pdb,*.dat,*.obj" --include-ref=2.1.0

There are a few more file types included at the end of the list that weren't in your .gitattributes file, as they were showing up as being a reasonable size with this command: git lfs migrate info --include-ref=2.2.0-test --top=25

You can see the result of this in my branch https://github.com/Julusian/CasparCG-Server/tree/2.2.0-test
I tested the branch size by cloning with the following git clone --single-branch --branch=2.2.0 https://github.com/CasparCG/Server.git

I'll hold off on the PR I was about to raise against 2.2.0 until you've had a chance to look at this.

[ronag]

@Julusian thanks! I force pushed your branch to 2.2.0

[dotarmin]

Breaking stuff so far:

Re-do cmake
Refactor dependencies
Remove test
Remove auto generated docs
Remove deploy & CI stuff
Remove framerate_producer
Remove channel_layout in favour of ffmpeg filters (producer + consumer) and a fixed internal 8 channel audio pipeline

Additional breaking stuff might come up in the future... @dotarmin please correct if something is incorrect.

@ronag That's correct for now!

[dotarmin]

Are either of you able to share what the features being removed are? It would be useful to know what features to avoid using if they are going to be removed or pushed to a future release.

We understand this and I couldn't agree more that it should be clear what's happening! The project is going through some sort of transformation, at least it feels like that and after this release that Robert is working on things will be more clearer. Stay with us! 😄

[Julusian]

@dimitry-ishenko

would it be possible to add an option to build with external (vs bundled) dependencies? Similar to what I did in #394 awhile ago?

https://github.com/nxtedition/caspar/issues/7#issuecomment-359471619

[k0d]

There's a lot of planned changes to the build system, but of course this takes time so we're doing it in stages.

The goal in my view is to enable a one line command to run a build and create a package with the required dependencies for that platform. For example "brew install casparcg-server" for mac (if/when we make a mac version), or "apt-get install casparcg-server"....
...and to also allow easy building with custom dependencies.

I have to read all the past discussions on this, to understand other people's views, I will do it...as quick as I can!

It'd be really useful to have some people testing the Linux builds, anyone able to do this and provide feedback?

[Julusian]

I am happy to test out some linux builds. I only have ubuntu 14.04 installed currently as that is all that was supported, but am happy to upgrade that to 16.04/18.04.

I do have a couple of thoughts on the new build scripts.
I like that it is now moving to using docker, that will make automated builds a lot easier to do.
I liked how that before there was one script to run to produce the whole binary zip, it would be good to have that again to streamline the process.
I don't see a list of what runtime dependencies are needed. That would be useful to include in a readme in the zip and and for it to be referenced in the build instructions file.

[dimitry-ishenko]

I've got Ubuntu 16.04 on our production server and can test Linux builds. I've also got a few Gentoo boxen, which I can use to test builds with custom deps.