Syndesis: Big Bang Move to Monorepo

[rhuss]

Before doing the bing bang, the following point needs to be checked:

• Can we keep the Git History ? Especially useful for git annotate to find the author of some code.
• What about issues ? Can the easily moved in bulk or needs this to be done manually isssue-by-issue ? How to label them ? (see also #9)
• PR: Each PR author should rebase and resolve conflicts and then move the PR over via a patch.
• What are the consequences for productisation ? // @cunningt

[igarashitm]

Yep author info is sometimes useful, you can ask about the intention and/or background history if they are still around. But IMO the most important thing is that the commit history itself, it's usually corresponding to a single feature/bug, so it makes so much easier to understand why/how it's implemented. Commit logs are story to read, and make debugger's life far easier. @chirino , your effort to preserve commit history in SwitchYard did help us! Thank you so much!

[rhuss]

@cunningt We are planning to move soon to a monorepo. This means instead of many separate repos we will have a single repo with several subdirs. Any idea how this would affect the pipeline builds and productisation ?

[chirino]

We can keep history. I created a https://github.com/chirino/syndesis/blob/merge-modules branch which does not have the git submodules thing going and has a fresh history. It contains the following script:

https://github.com/chirino/syndesis/blob/merge-modules/merge-modules.sh

Which when run, will merge all the project modules master branches into the merge-modules branch with history.

When we are ready for the big bang, run the script then push rename the merge-modules branch to master and push to origin with --force.

[cunningt]

@rhuss For the dev builds, I think we're going to have to rework the scripts to deal with the monorepo, and I'm not sure what other issues will pop up. Not sure how it'll affect prod.

[chirino]

For folks that have open PRs, you can re-apply them as a patch. For example, say you have PR https://github.com/syndesisio/syndesis-rest/pull/773 open. You could re-apply it in the monorepo by running:

cd rest
curl https://patch-diff.githubusercontent.com/raw/syndesisio/syndesis-rest/pull/773.patch | patch -p 1

[igarashitm]

Tested against atlasmap projects, and it just worked perfectly :)
Note that as the path info no longer match, you can't get git log with specifying specific file, but merged logs are actually there. GitHub Blame still wisely shows those history for individual files.
One trivial thing, you might want to be on the merge-modules branch when run it, or change git checkout merge-modules to git checkout -b merge-modules in the script.

[rhuss]

Looks good for me, too.

@chirino do you know why the e.g. git log pom.xml only gives the move comment ? But I think we can live with it ...

I'm now trying to apply a PR via patch ...

[chirino]

@rhuss no not sure.. likely due to the --allow-unrelated-histories option. Log probably only goes down one of the history paths of the merge.

Using 'git log --full-history pom.xml' likely shows you what you want.

[rhuss]

@chirino @igarashitm I fixed that issues --> https://github.com/syndesisio/syndesis-project/pull/180

You just need to use git mv for putting the files in the proper place.

I tested the script from the PR on an empty directory (better to start a fresh from a freshly cloned repo I think).

Works now very smoothely, looks good for Wednesday ;-)

[igarashitm]

@rhuss cool, it now preserves log for top level files in the submodule. But I still don't see git log for the files in subdirectories of submodule. Do you see same behavior? GitHub Blame still works fine for all of them.

[igarashitm]

Ah git log --follow just works perfectly for that. It sounds like git log basically doesn't follow the file rename without this option.

... from Git Manual ...
       --follow
           Continue listing the history of a file beyond renames (works only for a single file).

[rhuss]

Yes, I see the same.

I could update the script to recursively add all files individually instead of adding whole directories, but I wonder whether its worth the hassle.

wdyt ?

[rhuss]

I tried to fix this, but its not trivial (especially not with shell only).

So lets keep it as it is. I think we can live with it.

[igarashitm]

yeah, git log --follow would be fine for us, let's move forward :-)

[deeleman]

How are we going to proceed with 3rd party dependencies (and most particularly, the frontend ones)?

1. Will we move npm binaries to control-version too (as the theory of trunk-based dev suggests)?
2. Or will we still fetch npm dependencies through yarn|npm install?

BTW, regarding npm dependencies, do we want to manage those on a centralized fashion, or on a per (frontend-related) project basis?

[rhuss]

Going to close this issue as we already performed the move

@deeleman if there is something left todo for the 3rd part UI deps, could you please open a new issue ?