I was thinking a lot how to solve all problems with new differing drivers like mongodb one.
Solution is to separate packages:
@typeorm/mysql
@typeorm/mariadb
@typeorm/sqlite
@typeorm/sql-server
@typeorm/oracle
@typeorm/websql
@typeorm/mongodb
and possible future drivers like:
@typeorm/redis
@typeorm/elasticsearch
@typeorm/cassandra
@typeorm/rethinkdb
@typeorm/neo4j
etc.
This requires lot of refactoring efforts and increases orm complexity, but that seems the only way to go further with different driver types.
We may also have general packages like @typeorm/core
, @typeorm/sql
, @typeorm/nosql
however I'm not sure about them. Alternatively we can simply pack everything needed into their packages using different build scripts and tsconfigs.
Also another solution is to have @typeorm/typeorm
and everything go from there, other packages just use declaration merging ability of TypeScript and add additional functionality, and users just use single @typeorm/typeorm
namespace everywhere.
Need help - suggestions, proposals or coding help if possible.
@19majkel94 maybe its time to appear in list of typeorm contributors 馃槈
As a personal preference, I would prefer different packages for each driver. It would help me as a developer keep things clear as to what driver is being used in the project. A single namespace would be nice... but I can see declaration merging becoming increasing complicated as time went on and more drivers were added.
Thoughts?
I have been following this project for about a month and find it very interesting. Great work so far!
Hi @pleerock
you can use something a monorepo like lerna, I'm using it for my projects.
But it comes with restrictions:
-Publishing you cannot use an output folder for publishing, it publishes the project folder. you can overcome this by adding a .npmignore file to ignore unwanted files in the tarball
-Linking : on bootstrap lerna links dependents packages, and in case of peerDependenies, it May cause some problems.
perhaps we can start by refactoring so we have clear packages first, and then add a monorepo tool.
thanks guys for the feedback!
@madCoder24 as for me I see declaration merging is the best approach, it will help to easy migrations between different drivers, it will be more consistent, and documentation maybe easier to write, and overall users experience will be better I think. However its less flexible and requires good design of single namespace.
@Salimlou hi, publishing of multiple packages isn't a deal, I already have a build script which makes build and publishing specific way.
I can't argue with a better user experience as a single namespace is always nice. Would you have general packages at all if you went with declaration merging? I would imagine you wouldn't, correct? Were you thinking different interfaces for each driver?
Just curious on what thoughts you had for the multi-declaration design.
@typeorm/typeorm
namespace (others do declaration merging)Other packages will be used just to install the package to load some additional declaration merging. This requires perfect design of interfaces in @typeorm/typeorm
namespace which can be then appended (using declaration merging) by other packages.
some of proposed changes:
decorators
ColumnType
, ColumnOptions
, IndexOptions
and other options can be customized by drivers using declaration merging. Main package will contain only options all drivers have.ObjectIdColumn
instead PrimaryColumn
can be usedPrimaryGeneratedColumn
instead PrimaryColumn
+ new decorator, lets say ColumnGeneration(type: ColumnGenerationType)
can be used where ColumnGenerationType
is "auto"|"sequence"|"uid"
depend of driver.Column
decorator Column
even for mongodb? Or shall we name it different way? Alternative can be to have both Column
and Field
decorators which will do same thing, e.g. field will be synonym decorator for column.CreateDateColumn
, UpdateDateColumn
, VersionColumn
for all drivers including future drivers? It seems answer is "yes" for me, let me know if I am wrong.DiscriminatorColumn
, ClassEntityChild
, ClosureEntity
, SingleEntityChild
, TableInheritance
, JoinTable
, JoinColumn
, all relational decorators, transaction decorators, tree decorators, DiscriminatorValue
, Index
. Is it okay to have extra decorators in main package? connection, connection options and connection manager
drivers
DriverOptions
can b customized by drivers using declaration merging. Thats awesome!entity manager and repository
query
, transaction
, createQueryBuilder
methods can be customized. In main package they should not present because are missing in some drivers (mongodb and others )find options
query builder
cli commands
Thats just quick overview of main public interfaces of typeorm. Please feel free to comment, and suggestions and complete this proposal.
Lets clarify something. By declaration merging I mean, for example,
We have a Repository
class in @typeorm/typeorm
export class Repository<Entity> {
find(): Entity[];
}
Now user install @typeorm/mongodb
which has such declaration merging code:
declare module "@typeorm/typeorm/repository/Repository" {
interface Repository<Entity extends ObjectLiteral> {
findUsingCursor(): Cursor<Entity>;
}
}
Now when user uses orm, he can do:
import {getConnection} from "@typeorm/typeorm";
// note: no need to importing something from other packages!
const postRepository = await getConnection().getRepository(Post);
postRepository.find(); // compiler okay, type hinting is okay
postRepository.findUsingCursor(); // compiler okay, type hinting is okay
If he uninstall @typeorm/mongodb
package and install @typeorm/postgres
for example it will have new declaration merging with postgres-specific functions, and postRepository.findUsingCursor();
will complain now.
Version with declaration merging will be very awfull in developing. How to add methods to existing class in TypeScript? There's no extending without extending, so you have to go back to require and patching prototype? There will be a lot of work to keep code and declarations in sync.
IMO the packages should be like class inheritance:
@typeorm/core
contain all basic common stuffs (metadata, etc.) and basic logicBaseDriver
with their XXXDriverSo user would do:
import {getConnection} from "@typeorm/mongodb";
// note2: no need to have imports from /core, all is in dbspecific package
const postRepository = await getConnection().getRepository(Post);
// note: connection will be always MongoConnection type with no sync-schema methods, etc.
postRepository.find(); // compiler okay, type hinting is okay
postRepository.findUsingCursor(); // compiler okay, type hinting is okay
// note: MongoRepository would have specific methods
So the deps graph would look like this:
So user could choose if he want to use @typeorm/sql
to be able to switch between database engines or use @typeorm/mysql
if he need some specific features like json types or sth.
And thanks to inheritance it should be easy to extend or overwrite methods and other stuffs.
not using "require", but yes, patching prototype. Yeah, its not that pleasant, but is that such a big problem? I think its not. Having multiple packages is already a pain in developing, patching prototype in some places doesn't deal too much there... As I already mentioned, declaration merging will help to easy migrations between different drivers, it will be more consistent, and documentation maybe easier to write, and overall users experience. Advantages bring by declaration merging worth having extra pain patching prototype I think. Maybe there are other more important disadvantages of this approach?
"declaration merging will help to easy migrations between different drivers"
Renaming import { x, y, z } from "@typeorm/mysql"
to import { x, y, z } from "@typeorm/postgres"
takes 15s in every editor or IDE. And you explicitly see that you getConnection from mysql not mongo, etc., no need to look in config what db is it.
By patching prototype can you use two drivers in one app? Like when you have existing sql database and want to migrate to mongo or have old postgres database add you add new functionality to the project using mysql db? If you use separate packages you still have typesafe with separate imports, declarations merging would then introduce type-hell and patching prototypes with mysql and postgres in the same app can result in total fail :wink:
Having multiple packages is already a pain in developing
There is npm link
, so instant developing of packages from separate repos is not a problem at all
.
I would like this kind of code instead of patching prototypes and declarations merging:
// @typeorm/mongo/repository
import { Repository } from "@typeorm/core";
export class MongoRepository<T> extends Repository<T> {
findUsingCursor(): Cursor<Entity> {
//
}
}
// @typeorm/mongo/index
export function getRepository() {
return new MongoRepository();
}
// I know, getConnection(), etc. - it's just an sample of idea
It's like old good inheritance, just in one level up. Simple in understand, clean code, no dirty patching, pure amazing! 馃槃
So you have common API, switching is just a import rename, you have typesafe, better code, I don't see any advantage of patching prototypes and declarations.
I am watching the issues here for a little now, but I am far away from having an overview over the project but I am curios cannot we have a single package and lets webpack, rollup and other tools to make the tree-shaking?
@NoNameProvided we are discussing a bit different problem. Problem with different drivers having different methods, implementations and interfaces. We don't need webpack, rollup btw because we are on the backend where minification, tree-shaking and similar processes are pointless.
I would like this kind of code instead of patching prototypes
I understood that you like classic inheritance style instead of javascript-style patching prototypes, I like this approach either. However I don't see a big deal in this, and can scarify a bit of code purity in favour of better user experience.
declaration merging is actually your idea I took from our discussion in mongodb PR.
By patching prototype can you use two drivers in one app
@19majkel94 thats very good point actually. Nope, patching prototype wont work then. It means that declaration merging approach may not work, if there is no fix for this problem.
in favour of better user experience.
User experience will be nearly the same - there would be still one source of imports, which only might require 15s findAndReplace on db switch by user comparing to @typeorm/typeorm
version.
And for me it's more intuitive when I explicitly import from @typeorm/mysql
rather than @typeorm/typeorm
and the used driver is defined implicitly by installing @typeorm/mysql
package in node_modules.
I understood that you like classic inheritance style instead of javascript-style patching prototypes, I like this approach either
For me the biggest problem with patching prototype is that TypeScript with doesn't allow you to do it so you have to go back to require
to have any
type on imported class. So you loose all good stuffs from TS like type-safe and intellisense. And you have to always keep declarations in sync with JS code which is problematic.
So if patching prototype approach can result in two driver packages patching the same Repository
from core
which will end really bad, I think we should go the way of db specific packages as a modules for user to import which depends on core
and extend classes or overwrite default functions, exporting common stuffs from core
.
maybe its time to appear in list of typeorm contributors 馃槈
I would love to, but I would need some tips witch parts should stay in core as it's used both for mongo and sql (like metadata args storage). Then I could start to do some magic to split files to packages and refactor things like drivermanager (switch-case over driver type from connection options, which would be then unnecessary).
And for me it's more intuitive when I explicitly import from @typeorm/mysql rather than @typeorm/typeorm and the used driver is defined implicitly by installing @typeorm/mysql package in node_modules.
@19majkel94 I agree with this point. I touched on this earlier in this thread. It allows a developer to know just by looking at the imports which database they are working on for the project. Which just makes things much cleaner from my perspective. The inheritance idea that you proposed seems like a cleaner solution than using declaration merging. Inheritance is kind of built for this case... build a base class and just extend it as seen fit for each case.
I will admit though that I don't know enough about declaration merging to provide a reasonable argument in favor of it or against it. This is my first GitHub project that I will try to contribute towards... so if I seem a bit ignorant on some these issues. I apologize in advance.
Those are my thoughts on the conversation.
For me the biggest problem with patching prototype is that TypeScript with doesn't allow you to do it so you have to go back to require to have any type on imported class.
I didn't get why you would need require?
@madCoder24 thanks for your feedback 馃憤
@19majkel94 actually I came up today to one problem with having multiple imports from multiple namespaces. Right now you define multiple connections this way:
[{
type: "mysql",
entities: [A, B]
},
{
type: "postgres",
entities: [B, C]
},
{
type: "sqlite",
entities: [D, E]
}]
And everything will work as your expect. See how entity B can be reused in different databases. However with different namespaces user will need to define B
columns and tables twice:
import {PrimaryColumn as MysqlPrimaryColumn} from "@typeorm/mysql";
import {PrimaryColumn as PostgresPrimaryColumn} from "@typeorm/mysql";
@MysqlPrimaryColumn()
@PostgresPrimaryColumn()
id: number;
What do you think about this small but unpleasant problem?
I see the problem but it's definitely not solvable with patching prototypes approach.
If you want to use your entity both with mysql and postgres, you should import decorators from @typeorm/sql
package which is common for mysql
and postgres
.
Metadata args storage is attached to global scope so it's shared between all typorm
db specific packages. Also shared things can be stored in @typeorm/core
but it might not work as a singleton if we use old mysql package and new mongodb which depends on different versions of core - each package will have their own different copy of core
. It can be solved by peerDependecies
(drawback: need for manual install @typeorm/core
) but it might prevent you to use old mysql and new mongo packages in one project.
But shouldn't we take care of the case with shared entity with different (incompatible) versions of typeorm? Who does that? 馃槈
If you want to use your entity both with mysql and postgres, you should import decorators from @typeorm/sql package which is common for mysql and postgres.
decorators should be exported from a single place, otherwise it wont be a good UX...
Also I'm not sure if core
package should really be... Because if we do @typeorm/mysql
, @typeorm/mongodb
etc then everything users uses from his driver should go from his package, thus @typeorm/core
. If we say that only mysql
and mongodb
will use them, then... um I don't think we need that because we can simply make a build system and pack everything into destination packages, without having to duplicate code, but duplicate in destination packages.
But shouldn't we take care of the case with shared entity with different (incompatible) versions of typeorm? Who does that?
right thats not super important because its unlikely scenario.
decorators should be exported from a single place, otherwise it wont be a good UX...
It's not possible if you want to have two TypeORM drivers/versions in one project and type-safe decorators (column types specific for database like jsonb
or sth).
And it's ok that when you want db specific things, you use specific imports for each db (postgres, mysql) and if you don't need it, you should use portable sql
version of typeorm for easy switch between mysql/postgres 馃槈
But with approach that mysql/postgres depends and extends @typeorm/sql
which is a common package and metadatas storage is shared, you can just use import from mysql even if the entity is used both for mysql and postgres side of project until you do sth db specific like column type that is not supported by both drivers. So prefer imports from sql
for common entities for type-safe.
Also I'm not sure if core package should really be...
I'm not a fan of attaching everything to global scope, I prefer package as singleton. If only metadata args storage has to be shared to be able to use mysql/postgres and common entities in one project, then core might be embedded with build system.
decorators should be exported from a single place, otherwise it wont be a good UX...
I meant exported from a single place in context of one driver. e.g. if I use only mysql driver I want to export everything only from @typeorm/mysql
, adding @typeorm/core
and @typeorm/sql
may look messy.
for easy switch between mysql/postgres
you already told that its 15s to replace "@typeorm/mysql" to "@typeorm/postgres" in any IDE 馃槈
btw do you know how to create @typeorm
on npm? Looks like I need to buy paid account on npm?
you already told that its 15s to replace "@typeorm/mysql" to "@typeorm/postgres" in any IDE 馃槈
But using postgres
imports allows you to use db specific things, custom column types, etc., so renaming import will cause type-error because mysql
is not fully compatible with postgres
.
btw do you know how to create @typeorm on npm? Looks like I need to buy paid account on npm?
It shouldn't be necessary, you just name you package with @typeorm/
prefix 馃槈
https://docs.npmjs.com/misc/scope
cool, thx for the info
how shall we call @Entity
decorator? I renamed it from @Table
to @Entity
because it served for both table
and collection
. But now, with separation both drivers can have their own decorator. So, shall we use @Table
for rdbms and @Collection
for mongodb, or left it as @Entity
?
Note, some orms also use @Entity
naming.
Also keep in mind how other decorators from "future databases" will be named.
Also keep in mind that we probably will have separate decorator @Column
and @Field
for rdbms/mongodb drivers
Y U always have hard questions? 馃槅
Going back to @Table
after breaking change isn't a good idea I think. But if we stick to @Entity
, we should standardize naming, so @Field
should be introduced.
And BTW I'm not sure about @Collection
name - there would be then @EmbeddableCollection
or it's not used by Mongo driver at all? The collection
prefix might be confusing because we don't nest collection but an entity (document in mongo), so maybe @Document
decorator name would be better?
Going back to @Table after breaking change isn't a good idea I think.
yes I agree its not pleasant, however if its a right choice I can do that including fact that there will be much more breaking changes with this issue resolved. Here, better to choice the right way to prevent further changes again.
But if we stick to @Entity, we should standardize naming, so @Field should be introduced.
I won't remove @Column
. Every orm uses decorator with such name, and I think its a good name. But for mongodb @Field
is a better name. So it seams we'll have at least two different decorators. Also need to consider other nosql databases.
@EmbeddedableEntity
Right now its used in mongo to represent a nested object. And yes, @EmbeddedableCollection
is not a correct name, it should be @EmbeddedableDocument
or simply @Embeddedable
. Or alternatively - @SubDocument
But for the entity itself I see @Collection
name is really good - because its a collection and in mongodb you call getCollection(T)
and lot of other places where collection term is used. I like it more then @Document
@19majkel94 checkout this repo to see first experiments over new structure
Sorry for late response.
Your approach on type-safe decorators looks nice, you could even simplify the function declarations with lookup type like in the multi-driver sample. You can change second parameter type depend on first parameter string.
How do you see Connection
and Repository
implementations for different drivers?
BTW, is this repo related to this experiment?
simplify the function declarations with lookup type like in the multi-driver sample.
are you suggesting to pass driver name to decorators? 馃槰
How do you see Connection and Repository implementations for different drivers?
probably same way as now, they all will have their own classes + inheritance from base class.
BTW, is this repo related to this experiment?
it was an experiment which bring me to typescript issue which I reported on their github. I don't think thats possible what I did in that example, so we can treat that experiment as failed.
are you suggesting to pass driver name to decorators? 馃槰
No! 馃槅 Just an lookup type like there:
interface ColumnTypeLookup {
SimpleColumnType: ColumnCommonOptions & ColumnPostgresOptions
EnumColumnType: ColumnCommonOptions & ColumnPostgresOptions & ColumnEnumOptions
}
export function Column<K extends keyof ColumnTypeLookup>(type: K, options?: ColumnTypeLookup[K]): Function;
I haven't test it in typeorm-experiment
code, it's a draft but should work.
ah I got you, its more complex but looks more pretty
btw what about question regarding table / entity / collection naming?
I think that RDBMS should use @Table
and @Column
and Mongo should use @Collection
(no 's', like no Tables) and @Field
. Does typeorm/mongo have all the @ClassEntityChild
and others decorators?
Does typeorm/mongo have all the @ClassEntityChild and others decorators?
no, but mongodb will have @Embeddedable
decorator
yes I agree its not pleasant, however if its a right choice I can do that including fact that there will be much more breaking changes with this issue resolved. Here, better to choice the right way to prevent further changes again.
I agree with that...
I have one big question: Where Redis will fit into this?
I mean it is not a rdbms, it looks like a nosql, but it is not exactly as mongodb or other nosql db...
The hashes
type looks like a json object, despite it doesn't support nested hashes.
But hey, what about the other data types?
We have lists
, sets
, sorted sets
and they act like key/pair values not as an object.
Also, what about the operations? LPOP, LPUSH, ZSCORE, etc.
They are really weird in this context. They will be supported at all?
And don't forget, we have a really cool feature from redis that is the Pub/Sub
.
I know you want to abstract some concepts and try to make a "good base" for all the other drivers, but I think from your list, redis is by far the most different of all.
Now, making a small suggestion not directly related to the design, but how about using a tslint perhaps to maintain coding style consistency?
It would be interesting since the project is right at the beginning.
how about using a tslint perhaps to maintain coding style consistency?
we do use tslint, however its not completely configured. We simply need to configure it better.
What about redis... You know, elasticsearch is also quite different, especially its complicated query language and column definitions can be huge. Also mongodb was quite different from rdbms and I implemented it (right now its there you can checkout master and take a look on examples how mongodb is working). Im not redis expert and familiar only with basic its operations, however I think we can fix most of them?
For example:
@RedisModel() // temporary decorator names
class Post {
@Key()
id: string;
@Field()
name: string;
@Field()
description: string;
}
const post = new Post();
post.id = "first";
post.name = "First post";
post.description = "About first post";
await connection.getRepository(Post).lpush(post); // will do: LPUSH post.id JSON.stringify(post)
You know, elasticsearch is also quite different
No, I didn't know. I just skipped him馃槅
Thanks for the information. I will take a quick overview later.
I am assuming you are going to create two major packages (sql and nosql) like you have suggested on the first post.
So, Redis and/or Elasticsearch will fit into which category? If they fit in, at all...
Yes, sure, we can try to map LPUSH
as an insert, LPOP
as a delete and so on...
Nothing too complicated here, it is possible and kinda make sense...
My real question is: will you try to convert the "redis world" into a "sql" or "nosql" package?
That was my original question, I'm not sure if it was clear before.
If so, about the other commands like ZSCORE
. Do you have plans to expose them on the specific redis package (@typeorm/redis) in the future?
About mongodb:
I saw it, it's looking good and seems promising 馃憤
About your example:
You will need to append the model name on the key name, something like this:
await connection.getRepository(Post).lpush(post); // will do: LPUSH "Post:"+post.id JSON.stringify(post)
Otherwise it will be overwritten by other models/entities (or whatever you are going to call it), but I understood what you mean...
I'm going to sleep now, cya 馃槈
I am assuming you are going to create two major packages (sql and nosql) like you have suggested on the first post.
no, as many packages as many databases, e.g. @typeorm/mysql
, @typeorm/postgres
, @typeorm/mssql
, @typeorm/mongodb
, @typeorm/redis
etc
each package will only have what it can do, e.g. there will not be a QueryBuilder
and transactions
in redis, but will be in mysql/postgres/etc. Or Cursor
will be only in mongodb package. But there are also common things: connection, repository, entity managers etc. But their methods may differ depend on driver type.
I decided to postpone this issue because of value/complexity it brings. Mongodb is part of typeorm main package at the moment.
I understand that separating existing codebase to separate packages with separate interfaces and the whole inheritance chain is a complex thing but It's only one time job. Then you can take profit from this feature not only as a TypeORM user but also as TypeORM developer (param/options intellisense in db-specific drivers, etc.). Having one interface and decorators for multiple db means that there's a lot of any or false-positive options which is bad in TypeScript world.
I'm not sure whether you understand my vision of packages and inheritance as you think about big "value/complexity it brings". Maybe should I create PR for this when 0.1.0 will have stable API or be released?
It brings relatively small value for a complexity it brings. Its not simply one time job. Its highly increases maintenance costs because working with such huge codebase with hundred more files and duplicate (or long) names will be harder. Including fact that we wont bring extra databases in a near future, including fact that bringing each new database will still bring lot of problems and costs with multiple packages as well and including fact that right now everything is working very well already I decided not to do this task. And this is a right decision.
It looks like your assumption about separate packages are different than what I was thinking about. It's not only about e.g. getRepository()
which return normal Repository
for rdbms and MongoRepository
for mongo. It also about dev experience when he set in decorator not supported option or column type, etc. - you have to do manual checking in driver to print error on console on runtime with the info (or now it fails silently?). With typesafe you don't have to do it because you can assume that lib-user is doing it correctly. So it can reduce the maintenance costs as well as improve developers experience.
It brings relatively small value for a complexity it brings.
If I got $1 everytime JS-dev tell this about TypeScript... 馃槅 But you use TS here so you see that sometimes it might looks like complicated and unnecessary work but it give you much more profit later 馃槈
If I got $1 everytime JS-dev tell this about TypeScript...
yeah thats right
it might looks like complicated and unnecessary work but it give you much more profit later
this decision cannot be compared with "use typescript over javascript". Everything has pros and cons and in this case I measured and there are more cons of this approach at the moment.
he set in decorator not supported option or column type
again as I told value of this is too low. Yeah he may have extra properties for mongodb column decorator which are not available. And so? Is it a problem? No. Its completely not. For people its more important to have lets say migrations generator then having such almost useless in this case typesafety. Right? And it means I better spend my resources on implementing migrations generator then do this thing and then keep maintaining this sht. Again this is a right decision at this moment.
Right, it's not a critical feature and now there are more important things to do as you said, so it might be not a right time. I just thought that you completely abandon this idea ("I decided not to do this task. And this is a right decision.") not just postpone and put it in "nice-to-have" backlog 馃槣
Hey all, just ran into a big bundle and was investigating, saw that the various query runners are taking up a lot of space. This is a long thread, is there a up to date TL;DR? Is this implemented or planned?
no, its not planned. If you talk about bundle size then split into bundles may save only 10% no more then that. Why do you care about bundle size? Where do you use typeorm?
Elasticsearch with TypeORM would be amazing :D
After it might be lot of work, cause it might introduce lot of new decorator for creating the indexes and so on...
We have started development in NestJS and we were excited to use TypeORM but unfortunately, we are using ElasticSearch as a database.
if TypeORM supports ElasticSearch then it sounds really awesome, though it looks trickier to implement but so far one of most complex MongoDB is already covered, So probably not as trickier as I think.
@pleerock Is it fit in near future plan or roadmap?
an ElasticSearch driver would be an awesome community contribution!
Most helpful comment
Y U always have hard questions? 馃槅
Going back to
@Table
after breaking change isn't a good idea I think. But if we stick to@Entity
, we should standardize naming, so@Field
should be introduced.And BTW I'm not sure about
@Collection
name - there would be then@EmbeddableCollection
or it's not used by Mongo driver at all? Thecollection
prefix might be confusing because we don't nest collection but an entity (document in mongo), so maybe@Document
decorator name would be better?