Vega-strike-engine-source: Git Submodules?

[BenjamenMeyer]

There's an existing submodule (data) that maps the Asset Productions repository to the data directory.

Question: Do we want to continue using submodules? Or do we want to encourage folks to use separate clones that they manage?

Reason: Subversion did a great job with external stuff and mapping things from one section of the SVN repo to another with svn:externals; however, Git Submodules doesn't provide the same capabilities as easy to work with. Instead one has to do:

$ git submodules init
$ git submodules foreach "git submodule update --init"

or something like that; it requires additional steps and becomes complicated to keep up-to-date and in-sync.

Recommendation: Do one of the following:

• Provide a script to work with the git submodules so it's easier for our users/developers to work with.
• Drop using Git Submodules and provide direction on how to get a set of assets to work with.

[BenjamenMeyer]

Personally I lean towards dropping git sub-modules:

1. They're hard to work with.
2. We can use it as a means to help with development and testing of multiple assets so more games get developed on top of the Vega Strike base.
3. If we provide support for #97 (and similar on Mac and Windows) then we the asset location is simply the directory working directory where the vegastrike binary is launched from and doesn't need to be in the repository itself.
4. It helps split the engine from the assets that make the UtCS game.

If folks don't want to do drop sub-modules, then I'll be happy with a script to make it easy for folks to do likewise with updated directions in the README to load the assets.

[nabaco]

The work with git submodules is quite simple
git clone --recurse-submodules
git pull --recurse-submodules

and that's all there is to it...

For development, and especially CI, it's nicer to have a submodule I think. For release, maybe not...

[BenjamenMeyer]

The work with git submodules is quite simple
git clone --recurse-submodules
git pull --recurse-submodules

That works if you're aware they are there during the initial download. Get the second works after, though it's generally advised to do fetch/merge separately instead of a straight pull (which does both).

and that's all there is to it...

For development, and especially CI, it's nicer to have a submodule I think. For release, maybe not...

Release will depend on the packaging methods; but likely we'll want to separate them.

[stephengtuggy]

I would like to drop the git submodule(s). I haven't been able to use them myself. So for me, they're just taking up space (on GitHub, at least) that could be used for other purposes.

Plus I like the idea of separating out the asset set, like @BenjamenMeyer was saying.

[nabaco]

In terms of Github, adding a submodule doesn't take any space - it's like a symlink. If you guys preffer and insist, we can remove it, no problem.
I personally prefer to keep, and just add instructions in the README how to work with it.

[stephengtuggy]

Here's another consideration: There are multiple copies of the asset set now, and I have 3 of them checked out on my local machine. Vega Strike (specifically vssetup) is getting mixed up about which copy to use. The algorithm for determining the asset folder location, is not that smart.

I think it would be a lot cleaner if we kept the asset sets in their own repos, if for no other reason than this. So you don't end up using the wrong asset copy, and wondering why the changes you made aren't there.

[BenjamenMeyer]

@stephengtuggy would you please file that as a bug? We probably should have something very deterministic via the config file.

[BenjamenMeyer]

@stephengtuggy nevermind...filed #111 to track that aspect.

[BenjamenMeyer]

So as experienced by a user in #117, once one has a checkout, then the only resolution is:

$ git submodule  update --init

At the minimum we need to ensure the documentation is in place if we're going to continue with submodules, then both this and #117 can be closed out.

[BenjamenMeyer]

Vote:

:+1: Keep Submodules and update documentation

• provide documentation on fresh clone with submodules
• provide documentation on existing clone with needing to retrieve submodules

:-1: Drop submodules and update documentation

• provide documentation on where to find an asset set to use and how to use it

[Loki1950]

Either way we need to update the documentation Dropping the sub-modules may allow keeping data sets separate in userland for testers who use muti data sets

[ermo]

I'm not really that keen on sub-modules if I'm honest -- for instance, I found it simpler to just fork and clone the assets into a separate folder on my box and work on a branch there when I did the vssetup -> vegasettings PRs for code and data respectively.

If there's an easy way to duplicate that workflow with sub-modules, feel free to point me to the relevant documentation.

What is the traditional upsides associated with using sub-modules?

[royfalk]

To be honest, I only have a vague understanding of know what we're talking about here. I'm not familiar with sub-modules, so I'll abstain.

[stephengtuggy]

I just noticed that the data submodule in Vega-Strike-Engine-Source is pointing to a commit 3 months old. Yet the Assets-Production repo was last updated 7 days ago.

Apparently it doesn't update automatically?

[BenjamenMeyer]

No submodules don't auto-update. They're locked to specific git hashes and have to be specifically updated

[nabaco]

I think that's good actually, at least in terms of releases and version tracking

[ermo]

I just noticed that the data submodule in Vega-Strike-Engine-Source is pointing to a commit 3 months old. Yet the Assets-Production repo was last updated 7 days ago.

Apparently it doesn't update automatically?

No submodules don't auto-update. They're locked to specific git hashes and have to be specifically updated

I think that's good actually, at least in terms of releases and version tracking

IME, for maintainers, this is a good thing as it makes for very little hassle.

For VS developers and contributors, perhaps not so much, since you will sometimes need to reference latest master in both code and data (like I did with the vegasettings PR).

Therefore, I propose a dual approach, where:

• Users and maintainers wanting to compile a known working version of VS are encouraged to use the submodule approach with tagged releases and shallow clones
• Developers are encouraged to use separate checkouts (with a depth suitable to their purposes)

Thoughts?

[nabaco]

Sounds good to me

[BenjamenMeyer]

@ermo my only concern is for folks that are packaging multiple projects using Vega Strike as the engine - e.g packaging UtCS and PWCU:PPU. UtCS packagers would be find with it; but PWCU:PPU packagers would not want to download UtCS.

So a slight modification:

• Users and UtCS Maintainers wanting to compile a known working version of VS are encouraged to use the submodule approach with tagged releases and shallow clones
• non-UtCS Maintainers and Developers are encouraged to use separate checkouts (with a depth suitable to their purposes)

[ermo]

Thinking some more about it, why use submodules at all then?

Aren't we just as well off with shipping an empty data directory with the VS repo and then instructing users and maintainers to check out shallow clones of tagged releases of VS and the data-sets in question and then dumping sub-modules entirely?

For a maintainer, being able to package a known-working version of VS separately via a tag and then being able to package UtCS and PWCU as separate data-sets (also via shallow clones) is the obvious solution?

And if it's good enough for a maintainer, it's good enough for a user? All he has to do is check out shallow clones of tagged releases and symlinking the relevant VS binaries into his data dir?

And for the developer, the only difference is not using shallow clones and using master as their base for topic branches?

EDIT: Not even sure we need the data/ directory in VS then.

[BenjamenMeyer]

Thinking about this more...I think we need to rethink how we're using submodules.

The engine is not tied to any asset set. Yes, it needs assets for testing, playing, etc; but it's not tied to a specific asset set.

OTOH, the Game Assets might be tied to a version of the engine.

So instead of having the engine use submodules to pull in a copy of any game assets for game play; perhaps it should be the other way around - the assets pull in a version of the engine via a submodule.

This also clarifies the various relationships and how they interact with the various parts of the system, and should make it clear to maintainers too which version of the engine is known to work with the game assets.

If we want to do this, then yes the data directory in the engine repo would go away, and we'd need to think about how to change up the asset repositories to easily match this for their layout. Any such related work should fall under the refactoring stories.

[ermo]

I like how turning around the relationship between engine and data-sets makes the mental model clearer:

The interesting bit to the end user is the data-set. The engine is really just an implementation detail to the user in terms of the experience.

This also mirrors what Privateer: Gemini Gold ended up doing with their own engine fork.

[stephengtuggy]

I like how turning around the relationship between engine and data-sets makes the mental model clearer:

The interesting bit to the end user is the data-set. The engine is really just an implementation detail to the user in terms of the experience.

This also mirrors what Privateer: Gemini Gold ended up doing with their own engine fork.

Yes, I like this too.

[BenjamenMeyer]

the one thing mentioned on Gitter was the PR builds... two notes there:

1. I'd like to create a smaller set of assets for testing purposes, meant only for the CI/CD testing, etc
2. if we have a dedicated set of assets for testing, then we could lock that in with submodules or update the CI/CD scripts to pull it

But yes, we'll need to solve that before we can remove it entirely.

[nabaco]

If it's dedicated for testing we could just include inside the engine's main repo

[stephengtuggy]

After further reflection, I'm thinking that the engine and the data-sets should just be peers, for conceptual and source control purposes, but released together, as VS UtCS etc. That's how I keep them on my system: Vega-Strike-Engine-Source and Assets-Production are children of the same parent folder. This seems to work the best in my past experience.

[BenjamenMeyer]

@stephengtuggy they can be peers; but whether or not they get released on the same schedule...

So technically the assets are dependent on the engine. That can't be avoided.

Also, the engine and assets may progress at different paces. In the end we shouldn't necessarily hold up releasing one over waiting on the other; but I don't see us allowing them to be on separate schedules until after 0.8.x is released.

[bmorel]

Hello. Not from the project, but maybe this idea will interest you:

Make assets and code git repositories without submodules. Create a repo which contains CI, and which uses submodules toward code and assets.
This way, people can easily not use submodules by mkdir VS; cd VS; git clone --depth 1 code;git clone --depth 1 assets; while submodule users (and no doubt it can be nice for CI, for example) can git clone VS --recurse-submodules=all.

This way, code have only code-related stuff, assets have only assets stuff, and the full project have the automatic build and tests stuff. Just saying that because in various projects, I see a lot of commits related to CI, which usually includes a lot of try&fail commits, which, imo, pollute the code's history with things not related to it at all.

[BenjamenMeyer]

@bmorel an interesting proposal; though I think some of the future changes in 0.9.x may negate that; namely...

0.9.x will see the Vega Strike Engine become more of a standalone project that can be re-used; so dropping the sub-modules there won't be an issue. This will occur b/c the changes in the relationship in the packaging. Historically they VS:UtCS game and VS engine were packaged together as a single deliverable - on tarball, one RPM, one Deb. With 0.6.x we moved to separate packaging - one for each the VS:UtCS game and VS Engine independent of each other - however, the relationship remained that the engine still depended on the VS:UtCS (at least in source). Once we get to 0.9.x I do intend on seeing that relationship reversed where the VS:UtCS game may, if desired, have the git sub-module for the engine as it will be dependent on the engine; however the engine will then no longer be dependent on VS:UtCS.

We may end up creating a substitute for testing purposes - I'd very much like to see some explicit test games assets for unit/functional/integration testing capabilities; but this will make it far easier for downstream projects (PWCU - https://github.com/pwcu/privateer_wcu) being able to integrate with the VS engine more directly and simplistically; even enable us to make several VS games ourselves as well (long term).

In summary: I think the current sub-module usage holds back the VS Engine from being as re-usable as it should be. Reversing the relationship should help encourage others to use it.