Cura: Make Uranium into a submodule

[kakaroto]

The Uranium repo seems to be very tied to the Cura repo, so if we update Cura but forget to update Uranium, it will often lead to bugs. Also, if you ever have to roll back to an old version (or trying to bisect to find which commit introduced a regression), you'd need to roll back Uranium at the same time and it would be impossible to know which commits to roll it back on to match Cura.
I would therefore suggest making Uranium into a submodule of Cura. With the submodule, it will be checked out together and updated together, and most importantly, each Cura commit will be associated to a specific commit of Uranium, so if we check out an old version of Cura, the Uranium submodule will be updated to the exact 'known working' Uranium commit for that Cura commit.

Is there any reason preventing Uranium from being made into a submodule of Cura ? If not, can we please make do it ?
Thanks.

[thopiekar]

In my opinion: Absolute veto!! No! Nada!
If you properly setup an editor, like Eclipse(+PyDev+EGit), you can easily see which branch is being used.
Additionally using my master PPA you won't even need to set any PYTHONPATHs. You can begin to work directly.
Finally, using submodules will break the conversation from Git to Bzr (needed for PPA builds) via fastimport/-export.

In short:
:skull: :bomb:

[sedwards2009]

I personally wouldn't mind if we just merged Uranium directly into the Cura repo. As kakaroto said, the two are so closely related and coupled they really belong together.

[thopiekar]

@sedwards2009 So what is the reason they were separated in the past?
Also there is software out there using the UM framework toolkit...

[sedwards2009]

@thopiekar From what I am told, yes, once upon a time there was some other software which was in development at Ultimaker and used Uranium. It is no longer being developed by the company, so the reason to keep Cura and Uranium separate disappeared.

[thopiekar]

@sedwards2009 .. but might appear in the future, if the market changes?

[thopiekar]

If there is a strong need to have something like this, we can create a "cura-bundle" repository with everything submoduled into this repository. Maybe a kickstart script to setup PYTHONPATH for example...

[fieldOfView]

There are at least two other current projects that are based on Uranium that I know of (with the great sounding names of Argus and NinjaKittens2). We _could_ fold a fork of Uranium into Cura, but I am pretty sure @awhiemstra and @nallath would be less than thrilled with the idea.

[kakaroto]

@thopiekar

If you properly setup an editor, like Eclipse(+PyDev+EGit), you can easily see which branch is being used.

I use emacs + a terminal, as I've always done for years and I don't think that's relevant to the issue here. Also, it's not a matter of "which branch you're using", but rather of "which specific commit in the branch you are". Like I said, especially for debugging bugs that require doing a "git bisect" and/or checking out of old revisions/branches.

Additionally using my master PPA you won't even need to set any PYTHONPATHs. You can begin to work directly.

I have no issue with PYTHONPATH, I have everything set up correctly, but doing a git pull on one directory does not do a git pull on another directory automatically, regardless of PYTHONPATH. I could also create a script that does that and do an alias for 'git' in my bashrc, but that's just messy and only solves the 'update both at the same time' issue. It still doesn't solve the issue of "which exact revision of Uranium were we using 2 months ago when this bug was introduced.. and is the bug because of this specific commit changes in Cura or because Uranium is not the exact revision that Cura expects it to be", you get the idea..
Also, I don't use a PPA, I run from source and use git when doing development (and I'm on Fedora). If your PPA has daily builds, and an archive that goes back for months/years and has the commit revision from each repository included in it, then it could be used to map which Uranium commit id is to be used with a specific Cura commit id... but that kind of cura hash->uranium hash mapping is exactly what a submodule is for...

Finally, using submodules will break the conversation from Git to Bzr (needed for PPA builds) via fastimport/-export.

Ok, now that's a different issue, and I have no knowledge of how git-to-bzr or PPAs work, so I cannot comment on that. I hope there is an alternative to that though, I'm going to research that and will let you know if I find anything.

@fieldOfView forking Uranium would not be a good idea, as it would just cause more confusion in my opinion and would make it hard to merge commits back and forth between the forks.

[thopiekar]

Well, using Eclipse with PyDev and EGit makes a difference here, because if you set your branch up to follow the master branch, then there is a indicator behind the project name, which indicates how far you are behind master and how much commits are new on your copy. That's how I know whether my copy is up to date or not.

There is no alternative. BZR does not support submodules. So the only alternative is to build packages from git directly. I already have to do the syncing manually, because there is a bug on launchpad.net. Don't need more work here.
If someone really decides to use submodules here I definitely turn off my PPAs. Cura will work for me then without my PPAs - no need to care about that anymore. Having the stable PPAs is only there because I have master PPA already set up.

But ok, it is not only my decision... It is the decision of the whole team 😉

@kakaroto: agree with you. The same crap would begin like in WebKit or Wayland/Mir.. I don't like forking at all.

[nallath]

There is software that uses Uranium; Scanner software (which i'm still developing) and NinjaKittens (Laser cutter software developed by Daid). On that account i'd say no.

The other point is that it would take quite a bit of development time to do this, with the dubious "gain" of not being able to forget to check out uranium. I feel that that is far to little gain for to much work.

[fieldOfView]

@kakaroto correct me if I am wrong or missing a point, but you could create a repository with Cura, Uranium, CuraEngine and libArcus as submodules (because the latter two have the same "problem" of exact versions/commits)? That way you can pull (recursively) once for all repos, and if you do that regularly you get your archive of exact commits combinations.

[awhiemstra]

I am with @thopiekar and @nallath on this one. While the primary reason for separating them was to be able to develop two separate applications with a shared based, during development the additional advantage came out that it enforces a certain amount of separation of concerns. I know not everyone agrees with me on that, but for me it is valuable.

[kakaroto]

@thopiekar By the way, while bzr doesn't support submodules, why would this affect you ? I'd guess it would only affect you if your script was expecting Uranium to be pulled in the submodule, but your script (or recipe or whatever, I'm not sure how that launchpad stuff works) will already check out Cura and Uranium separately, no? So in your case, the worst case is that you'll have an empty Uranium directory in your Cura bzr clone, but it shouldn't affect anything else as far as I know. I may (probably) be wrong, but I'm not seeing the problem (other than you wouldn't have the cura->uranium commit id mapping, but you already don't and it doesn't seem to bother you since you check out the HEAD anyway to create your packages)

@nallath Other software using Uranium are irrelevant to the discussion. Of course, not if the question was to fork or merge Uranium into Cura, but the suggestion is for a submodule, that wouldn't affect any of the other projects. It wouldn't affect Uranium at all either, since the git submodule is just a directory inside Cura that can point to any other git repo.
As for the "too much work", I'm not sure what work you see in there, when you commit to Cura, it commits the currently checked in revision of Uranium at the same time, nothing else is required as far as I know. But I haven't used submodules in a while, so maybe I'm wrong here again.

@fieldOfView Yes, I could create a separate repo and add them as submodules there, but it wouldn't have the mapping unless I pull on each repo every day and commit the new HEADs into that mega repository. I guess it could be done every few hours with a script on some server and that would partly mitigate the issue. It's clearly not the best solution and i'm not a fan but it could be a good alternative if the final decision here is no. Thanks for the suggestion!

@awhiemstra @nallath I agree that the separation is a good idea, and I somehow feel like there's a misunderstanding here about merging Uranium into Cura, this is not what this issue is about. All I'm asking is to make a git submodule of Uranium inside the Cura repo. A submodule can point to any git repo (even to one you don't own, like the QT or python repositories if you wanted to), it doesn't affect the target repository, nor should it. The git submodule is the equivalent of having a single file called 'Uranium' that has this content "https://github.com/Ultimaker/Uranium.git ad7cb983ed81687b51b19cef51342e74ec25d29f" (the current master HEAD) and basically says that when the committer pushed that commit, he was using this specific revision of Uranium and his code is "known to work" with that specific revision. Git just automates the process of checking the content of that file and checks out that specific revision for you whenever you change branches, roll back to old commits or pull newer commits.

I have seen repositories that have submodules of their dependencies (low level libraries it uses, such as zlib, libpng, etc..) just to make sure the correct version of the library is used since in some cases, the newer versions introduced regressions or something like that, or simply to make sure the user that checks out their repo has all the dependencies available when he builds. That does not affect the target repo (zlib/libpng) in any way, and it does not remove any abstraction/separation of code between the two projects either. It's just a convenience. And like I said before, a very useful one if you ever need to do a "git bisect" (which I did need to do recently by the way).

I hope that clears out any confusion. If the answer here is still no, then I'll have to work around it, as @fieldOfView suggested. Thanks for considering this proposal!

[fieldOfView]

Unfortunately submodules are not without issues:
http://slopjong.de/2013/06/04/git-why-submodules-are-evil/
https://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/
I have used git submodules myself on very small projects, and indeed ran into some issues. I could see them getting out of hand for a project such as Cura/Uranium.

The "too much work" that @nallath is talking about relates to our build process among other things. Our build server is "a fickle bitch" and we currently cannot risk another big change due to deadlines.

[thopiekar]

@kakaroto : The reason I have to make the conversation because some people are using signed commits. And using submodules will truly kill the whole "git --fastexport | bzr --fastimport" process. Had this work in the past with "Darling" (project which tries to make OSX executables working on Linux - like Wine but for OSX) and I always had to make the sync by hand: "git pull" and "bzr commit" and had to care about new files and deleted files. After a time I just dropped my PPA on that project because of that waste of time. I wish there was a better build system than launchpad.net (the only reason I use Ubuntu and not Debian by the way).
Just imagine: A submodule is not truly a directory for the VCS system.. It something like a "virtual symbolic link". Finally this is what bzr doesn't get at all.
Currently there is no way to build packages via recipes from Git repositories directly. So you always have to sync (if Launchpad can't) the code manually, so the BZR repository gets updated. If it gets updated, the recipe build system recognizes it and starts a daily build.

[kakaroto]

@fieldOfView ok, thanks for the links, very interesting reads. I can see some of those issues as deal breakers (especially from the second link).
@thopiekar thanks for the explanation. Yeah, that makes it hard to maintain in that case.

Considering the points made, I'm convinced, submodules are not the way to go so I'll close this issue now. Thanks everyone for the discussion.