Orbit-db: Monorepo approach

[thiagodelgado111]

I've been exploring the codebase the last few days and I constantly find myself jumping between different repos, that's not a problem, not at all. But maybe the monorepo approach could be interesting for orbit? (https://github.com/babel/babel/blob/master/doc/design/monorepo.md)

[thiagodelgado111]

Thoughts on this one?

cc @haadcode

[tyleryasaka]

I think that a monorepo structure could make things easier to follow. The lack of release tags and the fact that master branches across inter-dependent repos are not compatible with one another makes for a very challenging dev experience. For example, I wanted to make a change to orbit-db-store. But when I cloned the repo and linked it locally, the code didn't compile. Turns out, the master branches of orbit-db-store and orbit-db are currently not compatible. (I ended up having to go several commits back to the latest "release", which wasn't an actual git release but just a titled commit).

I've just started using lerna and overall I've been happy with it. Makes it pretty easy to manage multiple npm packages in one repo.

[aphelionz]

Rekindling and pinning this issue as some other repos, js-ipfs particularly seem to be using lerna sucessfully. Please vote thumbs up, thumbs down on the OP to vote on this.

While we're not guaranteeing we'll follow the votes exactly, it will help us gauge community interest in such an approach

[phillmac]

Thumbs down. I found it very confusing, and I can't see the argument put forward that having everything in one place simplifies the process for contributers. I was looking for something in the js-ipfs-http-client and gave up eventually because I was unable to find it. If we do go ahead with this we need to drasticaly improve the structure so we don't scare off new contributers with a monsterous collection of branches. Also it breaks npm install from git as the master branch contains the monorepo config, and you cannot install multiple branches of the same repo

[aphelionz]

Well this is good at least we're having a structural conversation. Done open questions:

1. What ways can the structure be improved?
2. How can we introduce newcomers to all the myriad packages?

The most cogent point you made is the npm install from git, which we do all the time... But we would check out one repo and include everything from the packages folder. So I'm not sure if that concern would go away or not.

To your point about js-ipfs-http-client, that's completely fair. We would have to make it crystal clear where docs are and how to find them

[phillmac]

Ok quite possibly this is my total lack of understanding, is the monorepo approach everything on a single branch, or different packages on different branches? That's what js-ipfs felt like for me, all the various packages had a seperate master branch, then all the in-progress features were each on an individual branch, and the whole thing felt like an incomprehensible fuster cluck. I'm most likely totally misunderstanding this however. What's the actual proposal here , a single directory structure with all the repos integrated on a single branch?

[phillmac]

Ok in my defense js-ipfs has been improved a metric ton since I last looked, there wasn't a packages dir last time I checked, I may have caught them midway thru the transition.

[aphelionz]

Sounds like I need to learn more about what lerna does because I certainly don't want every package to have its own branch. I was thinking just one repo with the packages inside and then each _feature_ or _bug fix_ can have its own single branch which I think would indeed make development and releases simpler.

[aphelionz]

We could also theoretically consolidate benchmarking and testing

[vaultec81]

I am not entirely against a monorepo. But having NPM packages and releases for each independent module separate is an absolute must for me. Especially when considering a lot of the pieces of orbit-db can be used else where.
Response to to @aphelionz's questions

1. I think the best thing is to streamline and create a solid release schedule for orbit-db. Each sub package will have a separate release schedule. Dependent packages such as orbit-db-store should not be concerned with the release schedule of orbit-db-kvstore for example. Each store type can implement different a version of orbit-db-store theoretically without creating conflict.
2. One of the ways we can help new comers would be having a list of packages with descriptions of what each package does, and what dependents each package has. orbit-db-kvstore relies on orbit-db-store etc.

[aphelionz]

having NPM packages and releases for each independent module separate is an absolute must for me

This wouldn't change. In js-ipfs this is the case. Each npm module is just under the /packages directory.

With regards to releases, I believe this will streamline the releases even more since they can be come even more scriptable.

Check out the dependency graph in https://github.com/orbitdb/welcome to see how it fits together. The number of packages is non-trivial and it's certainly not easy to get your head around how it all fits together as a newcomer.

[phillmac]

Ok one issue I just ran into: Is lerna compatible with windows? My windows docker image failed to build with this error from js-ipfs:

> [email protected] postinstall C:\orbit-db-http-api\node_modules\ipfs
> lerna bootstrap

'lerna' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `lerna bootstrap`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\ContainerUser\AppData\Roaming\npm-cache\_logs\2020-03-31T00_26_56_964Z-debug.log
The command 'cmd /S /C npm install' returned a non-zero code: 1
##[error]The process 'C:\Program Files\Docker\docker.exe' failed with exit code 1
Finishing: Build an image

This is something I'll have to take up with the js-ipfs guys but its good to know what potential issues we might be facing

[aphelionz]

That just looks misconfigured to me. Just can't find the lerna executable... Hmm..

[aphelionz]

Looks like we have mostly thumbs downs, including one verbal one from @haadcode 😆

@haadcode did come up with a better idea which may be the best of both worlds - move all the dependencies to the orbit-db repo itself, thus eliminating the release dance but still letting the packages be independent of one another.

[tabcat]

the only thing i dont like is that the issues and prs are all located in the same repo, at the same all the orbitdb packages being contained in the same repo couples them together more closely.

i thought lerna could be set up to release everything in /packages/ as their own package, i want to look at it more.
i also thought ipfs had a monorepo built like this but they dont, they seperate all their ipfs- prefixed repos like orbitdb currently does.

[wallynm]

That's a interesting topic, recently i've been working into a PR to create a better examples folder with a structure to grow with other examples and use-cases to help understand better how to implement things

https://github.com/orbitdb/orbit-db/pull/776

Today we have some repos that aren't exposed into base readme from orbit-db, this way several users don't get to know these examples - about that, i understand that examples should be kept all together to help others to test and see different configs.

[aphelionz]

I'm leaning more and more towards just rolling up dependencies to orbit-db. Having this discussion allowed me to drill down to the real problem, which is that updating all the dependencies up the chain makes it difficult to do releases.

For example, if you update ipfs-log you have to update orbit-db-store, which means you have to update kvstore, docstore, feedstore, eventstore, and counterstore, then finally update orbit-db.

If we can reduce the transience, or at least come up with a plan to do so, then we'll likely not need to move to a monorepo and can close this issue.

[phillmac]

Ive got at leat one pull request stalled at draft stage here because it requires simultaneous updates to orbit-db-store and orbit-db proper, and it feels prohibitively hard to cordinate the releases

[aphelionz]

Yep. A lot of people are in that scenario and have been similarly stalled, sometimes on the order of a year or more. It's totally a bummer. Suggestions are welcome!

[vaultec81]

That's why we need to have very clear release schedules for orbit-db, orbit-db-store and all the other store types. At the very least create a list of PRs to be included into a release. It would also be helpful if semantic versioning is respected. I think creating a release issue template for large well planned out releases is a start. I am sure multiple PRs can be easily included into a single version assuming there is coordination on merging, and only finalizing the version after the PRs have been merged and tests passed. If there is major backlog on a particular PR, it can always be moved out of the flow of importance to allow the release to go through.

Additionally it may be helpful to have a person in charge of managing releases/PRs in their assigned/repo of choice.

[aphelionz]

I agree this will help, and I also don't think the problem is _merely_ process-based. I think there are structural things that are preventing people from keeping the architecture and everything in their head at one time, which increases cognitive load.

To your point, what about org-wide GitHub Projects? Perhaps having a central place where planning can occur across all of the repos can help. I've had success with this in other projects.

[vaultec81]

I think there are structural things that are preventing people from keeping the architecture and everything in their head at one time, which increases cognitive load.

This is why having an individual focusing on a single repo will help with it. They only need to focus on what the repo does and how it fits in with a global model (planning area involved here). I know IPFS has a repo dedicated to planning and coordination. I am sure we can do something similar. I am in full agreement on creating a GitHub project page.

[aphelionz]

@vaultec81 You've been managing the v0.24.0 release in the ticket, so why not have a go at tackling the project. I added you to the new v0.24.0 project I just created

[vaultec81]

@aphelionz Definitely I would be willing to work on it.

[aphelionz]

Great! I saw you started working in there. It looks really good

[aphelionz]

Looks like we're going to opt to streamline the dependencies as opposed to moving to a monorepo for now. The rationale here is that we can get a lot of mileage out of simply structuring things better before we introduce a new mission-critical dependency.