Typeorm: [serverless] Connection Reuse is broken in a Lambda environment

[Wintereise]

Issue type:

[ ] question
[x] bug report
[ ] feature request
[ ] documentation issue

Database system/driver:

[ ] cordova
[ ] mongodb
[ ] mssql
[ ] mysql / mariadb
[ ] oracle
[x] postgres
[ ] sqlite
[ ] sqljs
[ ] react-native
[ ] expo

TypeORM version:

[x] latest
[ ] @next
[ ] 0.x.x (or put your version here)

Steps to reproduce or a small repository showing the problem:

I'm primarily expanding further on https://github.com/typeorm/typeorm/issues/2598#issue-345445322, the fixes described there are catastrophic in nature, because:

1. They either involve closing connections without understanding that the lambda may not just be serving that one request.
2. Full bootstrap of the connection a second time anyway (defeating the purpose of caching it in the first place)) and I believe this is something that should be addressed in core.
3. Even the somewhat saner metadata rebuild causes problems (since it cannot be awaited and runs in parallel). This results in metadata lookups while queries are running (for another controller, perhaps) randomly start failing.

The issue is exactly as described, if we attempt to reuse connections in a lambda environment, the entity manager no longer seems to know anything about our entities.

The first request made completes successfully (and primes our connection manager to reuse the same connection). Subsequent requests quit with RepositoryNotFoundError: No repository for "TData" was found. Looks like this entity is not registered in current "default" connection?

Here's a complete test case (with no external dependencies other than typeorm and uuid): https://gist.github.com/Wintereise/3d59a0414419b4ecca5137c20fc29622

When the else block (line #22 onwards) is hit, all is well, things work fine. When the cached connection from manager.get("default") is hit however, we can no longer run queries.

If it's needed, I can setup a project with serverless-offline for click-and-go testing.

[Wintereise]

The problem effectively originates here: https://github.com/typeorm/typeorm/blob/ad04ec9207483cc12dc5fb9ed1c3fa3ecae9d8f1/src/entity-manager/EntityManager.ts#L730

For all subsequent attempts using a cached connection, Connection#hasMetaData inexplicably seems to return false.

The bizzare thing is, we can actually see that the entitiyMetaDatas array is actually NOT empty even when this is actively happening. Here's a dump from one such case, showing that it does exist: https://puu.sh/CxqaS/3cfb02e653.png + https://puu.sh/Cxqd3/9eb3f5eb05.png

I noticed that internally, it all calls the findMetadata function. Somehow, between the two invocations, it seems to fail to find the EntityMetadata object. Could the class equality comparison not be working as expected somehow?

protected findMetadata(target: Function|EntitySchema<any>|string): EntityMetadata|undefined {
        return this.entityMetadatas.find(metadata => {
            if (metadata.target === target)
                return true;
            if (target instanceof EntitySchema) {
                return metadata.name === target.options.name;
            }
            if (typeof target === "string") {
                if (target.indexOf(".") !== -1) {
                    return metadata.tablePath === target;
                } else {
                    return metadata.name === target || metadata.tableName === target;
                }
            }

            return false;
        });
    }

@dstrekelj Did you ever end up figuring out what was happening?

[pleerock]

Could the class equality comparison not be working as expected somehow?

I guess yes. In your case probably it does TData === TData and it fails even if both are TData. I don't have experience working with severless, so I don't know exactly why it has such behaviour. Can you explain me how serverless "caches" things, things like "connection"?

[Wintereise]

Could the class equality comparison not be working as expected somehow?

I guess yes. In your case probably it does TData === TData and it fails even if both are TData. I don't have experience working with severless, so I don't know exactly why it has such behaviour. Can you explain me how serverless "caches" things, things like "connection"?

SLS itself doesn't really cache anything (it's basically a glorified deploy framework), the caching is all handled by AWS directly.

The whole idea is basically "loading the script once" (which fires all code outside of the handler function itself. Any object(s) created here will stick around until this container is destroyed) and then repeatedly calling the handler function (handler in my example) until it's time to go back to sleep (or be destroyed altogether) again.

Unfortunately, AWS doesn't have a handy explanation of how this "going back to sleep" thing is actually implemented. However, we're actually seeing this on offline-testing, the implementation of which is VERY available.

Handler caching can be seen here (https://github.com/dherault/serverless-offline/blob/2e0a41402469e2ee5f9e24297061feeeea621297/src/functionHelper.js#L86)

Actual usage is here (https://github.com/dherault/serverless-offline/blob/2e0a41402469e2ee5f9e24297061feeeea621297/src/index.js#L892)

I'm not really sure why it's just the metadata that has an issue. Calling these two after retrieving connection from the manager "fixes" it (but is an API violation, since both are protected).

            // @ts-ignore
            connection.options = config;

            // @ts-ignore
            conn.buildMetadatas();

@pleerock

[pleerock]

@Wintereise can you please figure out why in such environment there are different instances of "TData" ? I know a hack-solution for this problem, but I would like to get to a source of the problem.

[Wintereise]

@Wintereise can you please figure out why in such environment there are different instances of "TData" ? I know a hack-solution for this problem, but I would like to get to a source of the problem.

I'll fork typeorm and see what I can figure out, will get back to you.

[Wintereise]

I did confirm that it's the class equality comparison that's the problem @pleerock

Replacing it (with the very hacky) if (metadata.target.name === target.name) makes this issue go away. I however am not really sure on the /why/ of the problem.

Debugger shows the two objects as identical as far as I can tell, yet the equality comparison fails. Do you possibly have better ideas to establish "sameness" here?

[Wintereise]

~A match on the constructor (metadata.target.constructor === target.constructor) also works (and is somewhat safer, I suppose), verified~. Constructor match seems to return true for every entity!

Could we avoid the need for such filtering altogether by internally using a map instead of an array? This should also yield better performance due to not needing to run the find every single time.

Common sense says that it should work, but if the map keys themselves now have a matching problem...

[dstrekelj]

@Wintereise I'll have to go through the discussion when I get off work, but I did end up using the fix / workaround suggested in #2598 by the OP.

Here's what it boiled down to (formatting slightly adjusted for readability):

import {
    createConnection,
    Connection,
    getConnection,
    getConnectionManager,
    ConnectionOptions,
    DefaultNamingStrategy,
} from "typeorm";
import { User } from "./entity/User";
import { Listing } from "./entity/Listing";
import { RelationLoader } from "typeorm/query-builder/RelationLoader";
import { RelationIdLoader } from "typeorm/query-builder/RelationIdLoader";

const CONNECTION_OPTIONS : ConnectionOptions = {
    type: process.env.TYPEORM_CONNECTION,
    host: process.env.TYPEORM_HOST,
    port: process.env.TYPEORM_PORT,
    username: process.env.TYPEORM_USERNAME,
    password: process.env.TYPEORM_PASSWORD,
    database: process.env.TYPEORM_DATABASE,
    dropSchema: false,
    synchronize: false,
    logging: false,
    entities: [ User, Listing ],
}

/**
 * Establishes and returns a connection to the database server. If an existing
 * connection is found, the connection is reused.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @export
 * @returns {Promise<Connection>}
 */
export async function getDatabaseConnection() : Promise<Connection> {
    try {
        console.log("Establishing connection...");
        const connectionManager = getConnectionManager();
        let connection: Connection;

        if (connectionManager.has("default")) {
            console.log("Reusing existion connection...");
            connection = injectConnectionOptions(
                connectionManager.get(),
                CONNECTION_OPTIONS,
            );
        } else {
            connection = connectionManager.create(CONNECTION_OPTIONS);
            await connection.connect();
        }

        console.log("Connection established");
        return connection;
    } catch (e) {
        console.error(e);
        throw e;
    }
};

/**
 * Injects missing / outdated connection options into an existing database
 * connection.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @param {Connection} connection
 * @param {ConnectionOptions} CONNECTION_OPTIONS
 * @returns {Connection}
 */
function injectConnectionOptions(
    connection: Connection,
    CONNECTION_OPTIONS: ConnectionOptions,
) : Connection {
    // @ts-ignore
    connection.options = CONNECTION_OPTIONS
    // @ts-ignore
    connection.manager = connection.createEntityManager();
    // @ts-ignore
    connection.namingStrategy = connection.options.namingStrategy ||
        new DefaultNamingStrategy();
    // @ts-ignore
    connection.relationLoader = new RelationLoader(connection);
    // @ts-ignore
    connection.relationIdLoader = new RelationIdLoader(connection);
    // @ts-ignore
    connection.buildMetadatas();

    return connection;
}

The above code is located in a Database.ts file in the root of the project source directory. It can then be used in the following fashion:

import { getDatabaseConnection } from "./Database";

export async function getStuff(event: APIGatewayEvent, context: Context, callback: Callback) {
    const connection = await getDatabaseConnection();
    // Do stuff with the connection
}

Needed it for a proof of concept project and it did its job.

I have the development environment set up in Vagrant and the project itself with Serverless so I can make something reproducible out of it, I just need to find the time.

[Wintereise]

@Wintereise I'll have to go through the discussion when I get off work, but I did end up using the fix / workaround suggested in #2598 by the OP.

Here's what it boiled down to (formatting slightly adjusted for readability):

import {
    createConnection,
    Connection,
    getConnection,
    getConnectionManager,
    ConnectionOptions,
    DefaultNamingStrategy,
} from "typeorm";
import { User } from "./entity/User";
import { Listing } from "./entity/Listing";
import { RelationLoader } from "typeorm/query-builder/RelationLoader";
import { RelationIdLoader } from "typeorm/query-builder/RelationIdLoader";

const CONNECTION_OPTIONS : ConnectionOptions = {
    type: process.env.TYPEORM_CONNECTION,
    host: process.env.TYPEORM_HOST,
    port: process.env.TYPEORM_PORT,
    username: process.env.TYPEORM_USERNAME,
    password: process.env.TYPEORM_PASSWORD,
    database: process.env.TYPEORM_DATABASE,
    dropSchema: false,
    synchronize: false,
    logging: false,
    entities: [ User, Listing ],
}

/**
 * Establishes and returns a connection to the database server. If an existing
 * connection is found, the connection is reused.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @export
 * @returns {Promise<Connection>}
 */
export async function getDatabaseConnection() : Promise<Connection> {
    try {
        console.log("Establishing connection...");
        const connectionManager = getConnectionManager();
        let connection: Connection;

        if (connectionManager.has("default")) {
            console.log("Reusing existion connection...");
            connection = injectConnectionOptions(
                connectionManager.get(),
                CONNECTION_OPTIONS,
            );
        } else {
            connection = connectionManager.create(CONNECTION_OPTIONS);
            await connection.connect();
        }

        console.log("Connection established");
        return connection;
    } catch (e) {
        console.error(e);
        throw e;
    }
};

/**
 * Injects missing / outdated connection options into an existing database
 * connection.
 *
 * @see https://github.com/typeorm/typeorm/issues/2598#issue-345445322
 * @param {Connection} connection
 * @param {ConnectionOptions} CONNECTION_OPTIONS
 * @returns {Connection}
 */
function injectConnectionOptions(
    connection: Connection,
    CONNECTION_OPTIONS: ConnectionOptions,
) : Connection {
    // @ts-ignore
    connection.options = CONNECTION_OPTIONS
    // @ts-ignore
    connection.manager = connection.createEntityManager();
    // @ts-ignore
    connection.namingStrategy = connection.options.namingStrategy ||
        new DefaultNamingStrategy();
    // @ts-ignore
    connection.relationLoader = new RelationLoader(connection);
    // @ts-ignore
    connection.relationIdLoader = new RelationIdLoader(connection);
    // @ts-ignore
    connection.buildMetadatas();

    return connection;
}

The above code is located in a Database.ts file in the root of the project source directory. It can then be used in the following fashion:

import { getDatabaseConnection } from "./Database";

export async function getStuff(event: APIGatewayEvent, context: Context, callback: Callback) {
    const connection = await getDatabaseConnection();
    // Do stuff with the connection
}

Needed it for a proof of concept project and it did its job.

I have the development environment set up in Vagrant and the project itself with Serverless so I can make something reproducible out of it, I just need to find the time.

I see. Please be aware that this fix has a really big issue with the async nature of lambda. Every single time you "rebuild" the metadata for a connection (which may be servicing other queries concurrently), those in-flight queries can no longer be reconciled and will fail (in my case, with TypeError: this.SubQuery is not a function) if the rebuild does not complete by the time it's time to parse the DB results.

You can probably reproduce it with some trivial stress testing.

[dstrekelj]

@Wintereise Thanks for the heads up! I stumbled upon that particular issue by spamming endpoints with curl GET requests via a shell script. For the purposes of my proof of concept project it didn't matter much, but it's undoubtedly an issue for any serious work.

[Wintereise]

I looked into it more, the constructor match won't work (returns true for all instances of schemas). Pretty much the name match is the only thing that seems to reliably work so far.

[didac-wh]

Any news on this?

Using @Wintereise solution (btw thanks for that):

        // @ts-ignore
        connection.options = config;

        // @ts-ignore
        conn.buildMetadatas();

Seems to improve the situation, though after some load testing it doesn't seem to be 100% reliable.

@pleerock Do you think the other hack provided by him
if (metadata.target.name === target.name) in findMetadata()
is a better "solution"?

If not, can you provide the hack you mentioned?

Let me know if there's something I can do to help sort this out.

Thank you.

[Wintereise]

@didac-wh I'm using the name hack, works well. The buildMetadatas() one is a terrible idea (for reasons I explained in this issue already).

[didac-wh]

@Wintereise I tried the name hack, but it doesn't seem to work with ManyToMany relations.

Example of error I get when a ManyToMany relation is involved:

{ QueryFailedError: ER_BAD_FIELD_ERROR: Unknown column 'distinctAlias.PurchaseUserRoute_id' in 'field list' at new QueryFailedError (MyProject\Backend\src\error\QueryFailedError.ts:9:9) at Query.<anonymous> (MyProject\Backend\src\driver\mysql\MysqlQueryRunner.ts:163:37) at bound (domain.js:301:14) at Query.runBound [as _callback] (domain.js:314:12) at Query.Sequence.end (MyProject\Backend\node_modules\mysql\lib\protocol\sequences\Sequence.js:83:24) at Query.ErrorPacket (MyProject\Backend\node_modules\mysql\lib\protocol\sequences\Query.js:90:8) at Protocol._parsePacket (MyProject\Backend\node_modules\mysql\lib\protocol\Protocol.js:278:23) at Parser.write (MyProject\Backend\node_modules\mysql\lib\protocol\Parser.js:76:12) at Protocol.write (MyProject\Backend\node_modules\mysql\lib\protocol\Protocol.js:38:16) at Socket.<anonymous> (MyProject\Backend\node_modules\mysql\lib\Connection.js:91:28) at Socket.<anonymous> (MyProject\Backend\node_modules\mysql\lib\Connection.js:499:10) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10) at TCP.onread (net.js:607:20) message: 'ER_BAD_FIELD_ERROR: Unknown column \'distinctAlias.PurchaseUserRoute_id\' in \'field list\'', code: 'ER_BAD_FIELD_ERROR', errno: 1054, sqlMessage: 'Unknown column \'distinctAlias.PurchaseUserRoute_id\' in \'field list\'', sqlState: '42S22', index: 0, sql: 'SELECT DISTINCTdistinctAlias.PurchaseUserRoute_idas "ids_PurchaseUserRoute_id" FROMroute_join_route_group_typedistinctAliasORDER BYPurchaseUserRoute_idASC LIMIT 10', name: 'QueryFailedError', query: 'SELECT DISTINCTdistinctAlias.PurchaseUserRoute_idas "ids_PurchaseUserRoute_id" FROMroute_join_route_group_typedistinctAliasORDER BYPurchaseUserRoute_idASC LIMIT 10', parameters: [ 'ecc4a53d-0c29-4ced-b64b-8ff9084fc4d1', true ] }

Doesn't matter if I query the repository directly (.find()) or via queryBuilder (.createQueryBuilder()).

[Wintereise]

@didac-wh Confirmed, I saw the same with ManyToMany, though, I'm not sure if it's due to the name matching hack.

Sounds like there's another problem. @pleerock Any ideas?

[didac-wh]

@Wintereise Without using the name hack, the ManyToMany relation works. But of course, with concurrent queries I get the "no metadata found" issue.

[didac-wh]

I want to provide a little bit more info I found out by running some load tests.

Having the buildMetadata() trick seems to work well with concurrent connections as long as your query doesn't involve any relations (i.e. querying a simple table)
Tests performed:

• Loading 10 concurrent connections every second for 10 seconds
• Results: 100 success responses

If your query has relations, then the buildMetadatas() won't help at all. Here comes in handy the name hack provided by @Wintereise but (as discussed earlier) just as long as your query doesn't involve any ManyToMany relations.
Tests performed
(without name hack and with oneToMany relations):

• Loading 1 connection every second for 10 seconds:

- Result: 10 success responses, 0 error responses

• Loading 2 connection every second for 10 seconds:
• Result: 1 success response, 19 error responses

(with name hack and with oneToMany relations):

• Loading 1 connection every second for 10 seconds:

- Result: 10 success responses, 0 error responses

• Loading 2 connection every second for 10 seconds:
• Result: 20 success responses, 0 error responses

With ManyToMany relations, name hack won't work at all. Without name hack some requests may work, but concurrency problem will still be there.

I hope this info helps somehow.

[sdebaun]

watching this issue with bated breath, trying some of the fixes suggested here.

same use case: connection pooling shits the bed when we use it in serverless offline. TY @Wintereise for all the research and troubleshooting!

[sdebaun]

well that was painless. ugly as all hell but for the other people running into this, a temp fix (so far) seems to be:

lib/typeorm-monkeypatch

import { Connection, EntitySchema, EntityMetadata } from 'typeorm'

// this is a copypasta of the existing typeorm Connection method
// with one line changed
// @ts-ignore
Connection.prototype.findMetadata = function (target: Function | EntitySchema<any> | string): EntityMetadata | undefined {
  console.log('monkeypatched function')
  return this.entityMetadatas.find(metadata => {
    // @ts-ignore
    if (metadata.target.name === target.name) { // in latest typeorm it is metadata.target === target
      console.log('found target===target')
      return true;
    }
    if (target instanceof EntitySchema) {
      console.log('found name===name')
      return metadata.name === target.options.name;
    }
    if (typeof target === "string") {
      if (target.indexOf(".") !== -1) {
        return metadata.tablePath === target;
      } else {
        return metadata.name === target || metadata.tableName === target;
      }
    }

    return false;
  });
}

and then i require that file before i do anything with typeorm.
thank you again @Wintereise, this was killing our velocity

[sdebaun]

for those others who fall down this hellhole of spite and loathing:

that was not the glorious fix i thought it was.

running into other issues. using the metadata.target.name === target.name seems to be resulting in false positive matches. e..g im trying to save an instance of EntityA and its using the definition for EntityB and shitting the bed. i don't understand enough about how typeorm is managing this, will dig deeper.

[sdebaun]

ok i isolated my problem to some typeorm + webpack interaction hell, it may only be peripherally related to the original serverless connection reuse. i have something that looks like its working in my environment. i looks like the root of some of these issues lies in how typeorm stores and searches for metadata for a given entity instance.

when you tell typeorm about entities via the decorator and base class
its registering metadata in a list

when you later try to do things with instances of one of those entities
typeorm tries to find the matching metadata that its tracking

problem:
no matter how you use @Entity or BaseEntity (even if you supply a name property)
when typeorm tries to match the target instance
it tries to match on target.name
which is the name of the function, in this case the class name

in most use cases, that would be 'Profile' (for my Profile entity)
which would then let typeorm find the Profile metadata
and work its magic

with webpack: the minification does shenanigans like:

let d = class extends BaseEntity

so the 'name' property for every. single. fucking. entity.
is 'd'

fixes:

1. turn off minification in webpack with

module.exports = {
  ...
  optimization: {
    minimize: false,
  },
  ...
};

2. if we really need minification

2.1 add a static property to all our @Entity classes

static entityName = 'Profile'

and monkeypatch typeorm findMetadata to use that field

if (metadata.target.entityName === target.entityName)
```

[Wintereise]

Backend minification makes little sense, my recommendation is to just turn it off. I have it off too. @sdebaun

[pontusab]

Is there a more official way to handle this connection pooling issue on lambda yet? Or is it still the monkeypatch from @sdebaun 🥇 ?

[gl2748]

@Wintereise could you make your TypeOrm fork with the hack public?

[Wintereise]

@gl2748 Here's a patch instead, to be applied against [email protected]

diff --git a/node_modules/typeorm/connection/Connection.js b/node_modules/typeorm/connection/Connection.js
index 66dad98..4d885fc 100644
--- a/node_modules/typeorm/connection/Connection.js
+++ b/node_modules/typeorm/connection/Connection.js
@@ -443,7 +443,7 @@ var Connection = /** @class */ (function () {
      */
     Connection.prototype.findMetadata = function (target) {
         return this.entityMetadatas.find(function (metadata) {
-            if (metadata.target === target)
+            if (metadata.target.name === target.name)
                 return true;
             if (target instanceof __1.EntitySchema) {
                 return metadata.name === target.options.name;

Obvious caveats are that you now need to guard against reusing the same class name (and turn off any and all post compilation transforms (a la webpack minify / what not) that might cause that.)

I auto-apply this using https://github.com/ds300/patch-package using npm's postinstall hook.

[gl2748]

@Wintereise bit of a side question but nevertheless ont that relates to this issue - why does one want to put a DB connection, which relies on global state in a FaaS architecture? The notion of a singleton that persists beyond the lifecycle of the container seems peculiar?

[Wintereise]

@gl2748 Relational DBs have a very finite maximum connections at a time limit that is a particularly poor fit for Lambda's "scale until demand is met" execution model.

This is a bandaid to get more out of less. That and since RDBMS connections in fact have state associated with them, you don't want to waste time opening one every single time you want to interact with the DB. Ergo, persistent connections are needed.

Lambda simply caches everything defined out of the local function invocation block and reuses it across container invocations. This model can be (ab)used to get what we want out of it, that's more or less it.

[pontusab]

But this fix isn't something that Typeorm should have build in?

[pontusab]

@Wintereise My connection never gets reused, am I missing something obvious here https://gist.github.com/pontusab/17669973d098b072e7d1ef4fe3a3ed2c

[claridgicus]

@pleerock
Is there confirmation on including this fix / a more elegant approach for this one.
I'm starting to move large amounts of my nest/typeorm work over to serverless and I can't even contemplate life without typeorm ahaha

[apalumbo]

@Wintereise Lambda should not serve multiple request per time https://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
@gl2748 FAAS is not stateless, containers can be kept warm, so it is safe to use the stateful behaviour for caching or keep the connection pool, but you should never rely on a state related to a previous request (like to persist something)

[hasifroslee]

Here's my patch against 0.2.18. For ManyToMany, metadata.target.name is undefined. This should work.
diff --git a/node_modules/typeorm/connection/Connection.js b/node_modules/typeorm/connection/Connection.js index 851ffba..0f61d12 100644 --- a/node_modules/typeorm/connection/Connection.js +++ b/node_modules/typeorm/connection/Connection.js @@ -465,8 +465,12 @@ var Connection = /** @class */ (function () { */ Connection.prototype.findMetadata = function (target) { return this.entityMetadatas.find(function (metadata) { - if (metadata.target === target) + if (metadata.target.name && metadata.target.name === target.name) { return true; + } + if (metadata.target === target) { + return true; + } if (target instanceof __1.EntitySchema) { return metadata.name === target.options.name; }

[rlehrke]

This solution below from @pueue (from https://github.com/typeorm/typeorm/issues/2598) fixed this for me.

I'm now able to successfully re-use open connections in Lambda functions (both serverless-offline and "live").

I no longer get the RepositoryNotFoundError due to entities not reloading properly from cached connections.

export const connectDatabase = async (callback: Function = () => {}) => {
  try {
    const connectionManager = getConnectionManager()
    let connection: Connection;

    if (connectionManager.has('default')) {
      console.log('use a connection already connected')
      connection = injectConnectionOptions(connectionManager.get(), connectionOptions)
    } else {
      connection = connectionManager.create(connectionOptions)
      await connection.connect()
    }

    await callback();
    return connection;
  } catch (e) {
    console.log(e);
    throw e;
  }
};

export const injectConnectionOptions = (connection: Connection, connectionOptions: ConnectionOptions) => {
  /**
   * from Connection constructor()
   */

  // @ts-ignore
  connection.options = connectionOptions
  // @ts-ignore
  connection.manager = connection.createEntityManager();
  // @ts-ignore
  connection.namingStrategy = connection.options.namingStrategy || new DefaultNamingStrategy();
  // @ts-ignore
  connection.relationLoader = new RelationLoader(connection);
  // @ts-ignore
  connection.relationIdLoader = new RelationIdLoader(connection);

  /**
   * from Connection connect()
   */
  // @ts-ignore
  connection.buildMetadatas();

  return connection;
}

[leohajder]

Any news on this? This is what I did to bypass it...

// src/app.module.ts
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { AdsModule } from './ads/ads.module';
import { ConfigModule } from 'nestjs-config';
import * as path from 'path';
import { TypeOrmConfigService } from './config/database';

@Module({
  imports: [
    ConfigModule.load(path.resolve(__dirname, 'config', '**', '!(*.d).{ts,js}')),
    TypeOrmModule.forRootAsync({
      useClass: TypeOrmConfigService,
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

// src/config/database.ts
import { Injectable } from '@nestjs/common';
import { TypeOrmModuleOptions, TypeOrmOptionsFactory } from '@nestjs/typeorm';
import { ConnectionManager, getConnectionManager } from 'typeorm';
import * as dotenv from 'dotenv';
import * as fs from 'fs';

@Injectable()
export class TypeOrmConfigService implements TypeOrmOptionsFactory {
  async createTypeOrmOptions(): Promise<TypeOrmModuleOptions> {
    const connectionManager: ConnectionManager = getConnectionManager();
    let options: any;

    if (connectionManager.has('default')) {
      options = connectionManager.get('default').options;
      await connectionManager.get('default').close();
    } else {
      const env: any = dotenv.parse(fs.readFileSync(`.env`));
      options = {
        type: env.TYPEORM_CONNECTION,
        host: env.TYPEORM_HOST,
        username: env.TYPEORM_USERNAME,
        password: env.TYPEORM_PASSWORD,
        database: env.TYPEORM_DATABASE,
        schema: env.TYPEORM_SCHEMA,
        port: env.TYPEORM_PORT,
        logging: true,
        entities: [__dirname + '/../entities/**.entity{.ts,.js}'],
        synchronize: false,
      };
    }

    return options;
  }
}

[jiangell]

I too would like to see this issue addressed. This also seems to be happening on Firebase.

[mlg87]

the solution from @hasifroslee did the trick for us. this should be addressed in a release. the fix is quite easy

[stefanedberg]

We're also having the same problem with serverless offline and typeorm 0.2.18.

[NozzerFX]

We're also having the same problem with serverless offline and Lambda with typeorm 0.2.18. We've had to patch typeORM for now with the fix suggested by @hasifroslee , is there a pull request for this, or is more work needed to come up with a more reliable solution?

[saachinsiva]

We are running our application in Docker container and when we stress test the app, we noticed this error appearing for some of our requests while other request complete successfully. Also I see that most of you are using connection manager to create connection and use a wrapper class to get the connection but I am not doing it, I just use createConnection method at the start of our server and then use getManager or getRepository method . I am not sure if we are doing it wrong and start using connection manager and use our own wrapper to get the connection.

For now we are planning to use @hasifroslee fix and see if we can get rid of this problem.

Note : we are using 0.3.0-alpha.23 and postgres as our db

[n8sabes]

Hi All -- I'm using AWS Lambda functions and AWS RDS Aurora Postgres-Serverless which sleeps the database after n-minutes of inactivity. The DB takes 20-40 seconds to come back online. Despite all the various workarounds people have posted across different issues/threads, none work reliably. TypeORM's connection model fails in various ways without an easy avenue to a workable solution.

I love TypeORM and truly, truly, truly want to use it with the new Serverless stacks, but this connection issue is preventing it.

1. Has anybody else found a reliable workaround to reuse connections via Lambda functions that have not been purged by the node Execution Environment, and Aurora which sleeps and warm-starts the database?

2. If so, can we get a PR into a standard release (soon) as serverless projects are already quite mainstream.

Any help is greatly appreciated.

[Wintereise]

Hi All -- I'm using AWS Lambda functions and AWS RDS Aurora Postgres-Serverless which sleeps the database after n-minutes of inactivity. The DB takes 20-40 seconds to come back online. Despite all the various workarounds people have posted across different issues/threads, none work reliably. TypeORM's connection model fails in various ways without an easy avenue to a workable solution.

I love TypeORM and truly, truly, truly want to use it with the new Serverless stacks, but this connection issue is preventing it.

1. Has anybody else found a _reliable_ workaround to reuse connections via Lambda functions that have not been purged by the node Execution Environment, and Aurora which sleeps and warm-starts the database?
2. If so, can we get a PR into a standard release (soon) as serverless projects are already quite mainstream.

Any help is greatly appreciated.

Aurora's sleep -> rewake model is problematic for many issues, but that isn't the issue being discussed here.

You'll have more luck using the Aurora data-api driver TypeORM has now, but pre-warming and keepalives (of the aurora sls instance) are still a requirement.

[n8sabes]

Thank you @Wintereise, I'll check out the typeorm-aurora-data-api-driver which hopefully works well for deployed lambda functions. I'm using TypeORM with Postgres in both deployed and serverless-offline mode (via ssh tunnel to Aurora Postgres) for development debugging, so I need both to function. To your point, this may expand beyond the context of this issue despite the similarities -- we just need a simple/clear workable endgame. If I find a solution, I'll post back (or new thread). Thank you again, and I welcome any additional insights of course.

[Wintereise]

A long time has passed, and I finally had some time to actually look into this. It appears that this isn't really a problem in the AWS runtime at all, merely when using sls-offline to locally debug.

The reason (and fix) for that are mentioned in #2623 (which this issue is a duplicate of).

tl;dr: If you're using sls-offline, please invoke it with --skipCacheInvalidation and none of the patches mentioned in this issue will be required. It's always been fine in the actual runtime (AWS) itself.

What basically happens is that the singleton instance of TypeORM gets messed with when using the shared code runner from sls-offline.

@pleerock If more people confirm this is effective, I think you can close this issue.

[Wintereise]

For those of us using the monkey patch (a la https://github.com/typeorm/typeorm/issues/3427#issuecomment-504134702 and MAYBE the PR at https://github.com/typeorm/typeorm/pull/4804 assuming it's a verbatim copy), be advised that cascade with ManyToMany does NOT function past the first load (in sls-offline).

Reverting and using the option above will fix this.

[Shubhanshu1201]

We are also facing the same use where i am using typeorm 0.2.18 and postgres aws Aurora RDS with pg version 7.11.0. During connection create for 1st time the query is executing successfully but later at random generating the following error :

{ "errorType": "Error", "errorMessage": "Connection terminated unexpectedly", "stack": [ "Error: Connection terminated unexpectedly", " at Connection.con.once (/var/task/node_modules/pg/lib/client.js:252:9)", " at Object.onceWrapper (events.js:286:20)", " at Connection.emit (events.js:198:13)", " at Connection.EventEmitter.emit (domain.js:448:20)", " at Socket.<anonymous> (/var/task/node_modules/pg/lib/connection.js:77:10)", " at Socket.emit (events.js:198:13)", " at Socket.EventEmitter.emit (domain.js:448:20)", " at TCP._handle.close (net.js:607:12)" ] } or Client has encountered a connection error and is not queryable.

The issue start appearing suddenly after running successfully for a while.

Any help is greatly appreciated.

[abdullah2993]

Having the same issue, even on the first request(The entities were not registered properly)

offline: GET /dev/items (λ: items)
Unable to get items: RepositoryNotFoundError: No repository for "Items" was found. Looks like this entity is not registered in current "default" connection?
    at new RepositoryNotFoundError (C:\Users\user\src\accounting-server\src\error\RepositoryNotFoundError.ts:10:9)
    at EntityManager.getRepository (C:\Users\user\src\accounting-server\src\entity-manager\EntityManager.ts:1008:19)
    at Connection.getRepository (C:\Users\user\src\accounting-server\src\connection\Connection.ts:341:29)
    at Object.getRepository (C:\Users\user\src\accounting-server\src\index.ts:285:55)
    at C:\Users\user\src\accounting-server\.build\src\controllers\items.js:52:52
    at step (C:\Users\user\src\accounting-server\.build\src\controllers\items.js:33:23)
    at Object.next (C:\Users\user\src\accounting-server\.build\src\controllers\items.js:14:53)
    at fulfilled (C:\Users\user\src\accounting-server\.build\src\controllers\items.js:5:58)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  name: 'RepositoryNotFoundError',
  message: 'No repository for "Items" was found. Looks like this entity is not registered in current "default" connection?'

Note: --skipCacheInvalidation fixed the issue for subsequent requests

[rifflock]

--skipCacheInvalidation fixed this issue for us as well running version 0.2.24

@Wintereise, I think you're right that this can be closed.

[Wintereise]

Update to sls offline 6 and you won't even need that. Just use worker
threads or child processes. That deals with the shared singleton scope
problem quite well.

On Thu, Apr 30, 2020, 11:51 PM rifflock [e-mail] wrote:

--skipCacheInvalidation fixed this issue for us as well running version
0.2.24

@Wintereise https://github.com/Wintereise, I think you're right that
this can be closed.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/typeorm/typeorm/issues/3427#issuecomment-622006492,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABHGGEQIC3GTGRSFT6TR3DDRPG3D3ANCNFSM4GQJF5YA
.

[acuthbert]

I just downgraded from sls offline 6 to 5 because it doesn't have a hot reload facility. I then ran into this issue. --skipCacheInvalidation fixes this issue, but no I'm on v5 of the plugin and hot reload no longer works again!

Has anyone got a solution where hot reload works AND connections can be reused?

[rifflock]

@acuthbert could you use something like nodemon to run hot reloads with the local invocation?

[Wintereise]

@acuthbert could you use something like nodemon to run hot reloads with the local invocation?

Unfortunately, due to the compile step (where sls.yml has to be resolved, and if you run something like webpack/ts -- then even more of that) kills process-restart HMR =/

I mean, sure, it works -- but you might as well be manually restarting at that stage. Would be nice to figure out a fix for this, dev experience isn't very optimal currently.

[rifflock]

It actually seems like the --skipCacheInvalidation might be causing issues with the hot reload. I've noticed since adding it that the reload isn't running the changed code. Perhaps I spoke too soon, because in order to run the changed code I have to kill the process and run it again.

[Wintereise]

It actually seems like the --skipCacheInvalidation might be causing issues with the hot reload. I've noticed since adding it that the reload isn't running the changed code. Perhaps I spoke too soon, because in order to run the changed code I have to kill the process and run it again.

Well known, yes. It doesn't cause issues, it flat out kills it. Between that and v6 lacking the feature completely, we don't have a working HMR solution at the moment.

[imnotjames]

Looks like much of this has been solved by using v6 of the serverless-offline plugin, separate from typeorm. Closing.