Node-postgres: monorepo

[brianc]

@charmander @bendrucker @sehrope @vitaly-t @abenhamdine

Sorry for the pings, but y'all are probably the most active contributors & I wanted to get your opinions on something I've been thinking about for a while....

Long, long ago pg was one big repo with lots of functionality in it (native bindings, the pool, some connection string parsing, type parsing). Then later as the trend of smaller modules became a thing I spun some stuff out into their own repos. I then built some extensions that are pretty dependent on the version of pg. So now we have this:

https://github.com/brianc/node-postgres
https://github.com/brianc/node-postgres-docs
https://github.com/brianc/node-pg-native
https://github.com/brianc/node-libpq
https://github.com/brianc/node-pg-types
https://github.com/brianc/node-pg-pool
https://github.com/brianc/node-pg-query-stream
https://github.com/brianc/node-pg-cursor

This ends up making it a lot harder for me to track issues across all the issues and coordinate semver changes between the various dependencies. Particularly w/ things like pool error handling changing. How would y'all feel about combining some of these into a single repo and managing all the packages with lerna? I don't need to combine them all - particularly pg-types & its friends are more managed by @bendrucker (I'm happy to though if you want).

At the least I'd like to bring in the docs, pool, query stream, and cursor into this repo as often times working on the pool makes me want to change behaviors in the client or vise-versa. And query-stream and cursor are really dependent on the behavior of errors and types and so on from the core lib. I might leave the native bindings external...not sure about that yet....those really could use a full blown new maintainer as working with C/C++ is so difficult for me....I don't ever do it at my day job anymore and so I get lost easier. Anyways....what do you think? As a first step I was thinking of collpasing pg-pool pg pg-query-stream and pg-cursor (and the docs) into this repo and managing it all w/ lerna. That should make it easier to coordinate an 8.0 release of this library as well as everything can be major versioned in lockstep. I'd like to think it'd make issue management both a bit easier for me (one place for issues instead of 10) and hopefully for users as well as it's just 1 repo to open an issue on instead of trying to find where it goes.

Any strong feelings either direction on this?

[vitaly-t]

If this should happen, you should definitely go for TypeScript. The conversion would be quite simple, and make it much easier to maintain.

I've done it myself a number of times, taking small independent modules and converting them into TypeScript. It simplifies the integration as well.

But the Native Binding should remain external, because bringing in a gyp dependency will cause trouble to many novice developers.

[brianc]

Yeah I love typescript, that's definitely something I want to do. I think having a monorepo might make incrementally moving towards that easier too.

[brianc]

Other things i'd like to do which the monorepo would likely help w/ would be normalizing lint rules (and prettier) across the repos since they're all slightly different due to me starting the repos at different times and my editor always gets weird when I move between them.

[abenhamdine]

In my experience, monorepos are indeed easier for versioning and simply navigate through the codebase.
However, the number of issues is of course higher so it would probably be a good idea to enable issue templates and/or automatic issues labelling/assignement (depending on code owners too).

[sehrope]

Overall +1

If there's no clear separation or potential for re-use, then monorepo has a lot of convenience. I particularly like having docs inline with source as it allows for updating them in tandem rather than as a ~forgotten~ separate docs PR.

Big +1 to having a single place for issues. There's a huge overlap and most of the time issues on the downstream repos start off as an issue in core pg.

RE: query stream, cursor, copy, pool, and friends, I can't imagine anyone using them standalone. They're deeply tied to this library and it'd be nice if it was "batteries included". If there's a breaking change you'd be better able to update them in tandem and ensure they're properly tested as well. This would better allow updating the internal query submit API as well.

pg-types is a bit special and I have used it outside core pg. I'm not opposed to it being inlined as long as it's also available as it's own standalone module published to npm. Having development tied to core pg would be nice as there's plenty of lock-step upgrading involved, e.g. the recent numeric and BigInt discussion.

Besides testing to make sure I haven't broken anything, I've never used pg-native and I always recommend using the pure-JS version. Having a common interface between pure-JS and native is nice but it has issues as not everything is supported (ex: copy and cursors). Pretty sure the parameter handling is also different (IIRC native expects everything as a string: https://github.com/brianc/node-postgres/issues/1680#issuecomment-401631727). Even the "sync" usage isn't as useful anymore as async/await makes modern JS pleasant enough to not need it.

I think pg-native should be cut out entirely into it's own separate project. If anybody is using it then hopefully they can step up to help maintain the C++ bindings. If not then why are we maintaining it?

+1 to either code in TypeScript (preferable!) or at least inline type definitions. Could be a step two after combining things to prevent too much moving at once.

[brianc]

@sehrope I was gonna quote and inline respond w/ agreement to the things you said but I agree with all of it! Also 100% down to leave pg-types outside. Thanks so much for your thoughts as always!

[bendrucker]

Seems worthwhile to bring the "client" modules that deal with networking, querying, etc. into a monorepo if they're tightly coupled. Dubious on TypeScript. Publishing transpiled code to npm introduces issues tracing lines back to the source code. To date I've asked anyone looking to contribute types to do so via a .d.ts file and provide some tests that do basic type validation via https://github.com/SamVerschueren/tsd.

Most of the type packages are not very tightly coupled. 👍 to deferring that for now. Might make sense to be its own separate monorepo, but I'd like to see how this goes in core before committing a bunch of time to migrations.

Probably also worth adopting scoping, i.e. @postgres/*. That username is abandoned so if you'd like to do that I can ask npm about it and create an org.

[charmander]

A single repo sounds good to me! It’s definitely easier to keep information together that way.

[brianc]

closing this out as its underway now. thanks for your help & thoughts!