Sentry-javascript: Errors are not mapped back to the original TypeScript file

[mark-veenstra]

• [x] Review the documentation: https://docs.sentry.io/
• [x] Search for existing issues: https://github.com/getsentry/sentry-javascript/issues
• [x] Use the latest release: https://github.com/getsentry/sentry-javascript/releases
• [x] Provide a link to the affected event from your Sentry account

Package + Version

• [ ] @sentry/browser
• [x] @sentry/node#5.1.0
• [x] @sentry/integrations#5.2.0
• [ ] raven-js
• [ ] raven-node _(raven for node)_
• [ ] other:

Sentry Version:

Using version 9.1.1 within a own Docker deployment.

Description

I am trying to get source mappings to work with compiled TypeScript sources. I have read the documentation on the following points:

• https://docs.sentry.io/platforms/javascript/sourcemaps/#verify-your-source-maps-are-built-correctly
• https://docs.sentry.io/platforms/node/typescript/

And also had a look at the bug reports:

• https://github.com/getsentry/sentry-javascript/issues/1857
• https://github.com/getsentry/sentry-javascript/issues/1785
• https://github.com/getsentry/sentry-javascript/issues/1873

But still I have no success. So I try to describe and give as much information as possible. If necessary I could give access to a bug report on our own instance.

Project file layout from IDE:

• dist
  
  
  
  • src
  
  
  • controllers/CallController.js
  
  
  • controllers/CallController.js.map
  
  
  • index.js
  
  
  • index.js.map
  
  
• src
  
  
  
  • controllers/CallController.ts
  
  
• index.ts
• tsconfig.json

My tsconfig.json looks like:

{
  "compileOnSave": true,
  "compilerOptions": {
    "target": "es2018",
    "module": "commonjs",
    "declaration": true,
    "noImplicitAny": true,
    "inlineSourceMap": false,
    "inlineSources": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "alwaysStrict": true,
    "resolveJsonModule": true,
    "lib": [
      "es2018",
      "dom"
    ],
    "sourceMap": true,
    "sourceRoot": "/",
    "typeRoots": [
      "node_modules/@types"
    ],
    "outDir": "./dist"
  },
  "include": [
    "./src/**/*",
    "./src/**/*.json",
    "./index.ts"
  ]
}

Then my index.ts file looks like this:

declare global {
    namespace NodeJS {
        interface Global {
            __rootdir__: string;
        }
    }
}

global.__rootdir__ = __dirname || process.cwd();

import * as Sentry     from '@sentry/node';
import {RewriteFrames} from '@sentry/integrations';
/* Some more imports code specific */

Sentry.init({
    dsn: 'https://<key>@<custom_domain>/<project>',
    release: '[email protected]',
    integrations: [
        new RewriteFrames({
            root: global.__rootdir__
        })
    ]
});

/* rest of code */

Then the file I throw an error in looks in typescript like this:

   /* START OF CODE WHICH IS LEFT OUT */

    /**
     * Returns the requested Call.
     *
     * @param {IRequest} req
     * @param {IResponse} res
     * @return {Promise<IResponse>}
     */
    @Route.get('/:id')
    @Validate.json({
        request: {
            params: './src/schema/params.id.json'
        },
        response: './src/schema/call.document.json'
    })
    public async get(req: IRequest, res: IResponse): Promise<IResponse> {
        throw new Error('Error in [email protected]');
        try {
            return this.sendJsonOr404(res, await Call.get(req.params));
        } catch (err) {
            this.log.error(err);
            return res.sendStatus(500);
        }
    }

   /* END OF CODE WHICH IS LEFT OUT */

I use sentry-cli to upload the source maps as following:

sentry-cli releases files [email protected] upload-sourcemaps dist

Now when I startup the node process and visit the URL to trigger the error. I receive the error in Sentry in the project, see next screen shot:

So the error occurs in app:///src/controllers/CallController.js which should be correctly rewritten using the global parameter as stated in the docs.

When I take a look at the uploaded artifacts of this release it shows me this:

From the docs I have read that the ~ should be okay and match any protocol.

When I open the ~/src/controllers/CallController.js file it shows me that the source mapping url is in place:

//# sourceMappingURL=CallController.js.map

From the docs I found that Sentry would look for this file in the same dir as the JS file errored from. So the JS error comes from app:///src/controllers/CallController.js so it will look for app:///src/controllers/CallController.js.map. Above in the artifacts screenshot this file is also uploaded as ~/src/controllers/CallController.js.map, which should match.

The content of ~/src/controllers/CallController.js.map looks like:

{"version":3,"file":"CallController.js","sourceRoot":"/","sources":["src/controllers/CallController.ts"],"names":[],"mappings":";;;;;;;;;;;<MAPPINGS>","sourcesContent":["<CONTENT>"]}

This all seems okay to me. But as you can see the errors are not mapped to the TS file. Sentry does display the error:

But I do not see what I am missing here. Any help would be appreciated.

[matomesc]

I got my sourcemaps working with help from @kamilogorek (https://github.com/getsentry/sentry-javascript/issues/1857).

[mark-veenstra]

@matomesc When I read your issue it seems you got some problems with Windows and the RewriteFrames and afterwards you have added the JS files beside the map files.

As you can see in this issue all those things seem right to me in this issue. Any idea what I am missing? Are you using sentry.io or an own installation with Docker?

[mark-veenstra]

It seems that the sentry worker has problems getting the sourcemaps:
IOError: [Errno 2] No such file or directory: u'/var/lib/sentry/files/ce/f6af/4cd63f4851b8a1bf877d32b1ea'

The directory /var/lib/sentry/files/ is completely empty on the worker. And when I upload the files the sentry containers do have the files in /var/lib/sentry/files. So it seems to me that there is no shared volume mount for all the Docker images for /var/lib/sentry.

We have created a docker-compose.yml file to deploy sentry into a stack. This docker-compose.yml looks like this:

version: '3.1'

networks:
  net:
    external:
      name: loadbalancer_net
  sentry_net:
    driver: overlay

services:
  sentry:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    labels:
      - "traefik.enable=true"
      - "traefik.port=9000"
      - "traefik.frontend.rule=Host:sentry.example.org"
    networks:
      - net
      - sentry_net

  celery-beat:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    command: sentry run cron
    networks:
      - sentry_net
    deploy:
      replicas: 0

  celery-worker:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    command: sentry run worker
    networks:
      - sentry_net
    deploy:
      replicas: 0
  redis:
    image: redis
    networks:
      - sentry_net

  postgres:
    image: postgres:latest
    environment:
      - POSTGRES_USER=${SENTRY_DB_USER}
      - POSTGRES_PASSWORD=${SENTRY_DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    networks:
      - sentry_net

volumes:
  postgres_data:

And we deploy this stack using the following deploy.sh script:

# SENTRY POSTGRESS
export SENTRY_POSTGRES_HOST="postgres"
export SENTRY_DB_USER="sentry"
export SENTRY_DB_PASSWORD="<PASS>"

# SENTRY REDIS
export SENTRY_REDIS_HOST="redis"

# SENTRY SECRET
export SENTRY_SECRET_KEY="<SECRET>"

# SENTRY MAIL
export SENTRY_EMAIL_HOST="smtp.gmail.com"
export SENTRY_EMAIL_PORT="587"
export SENTRY_EMAIL_USER="<USER>"
export SENTRY_EMAIL_PASSWORD="<PASS>"
export SENTRY_EMAIL_USE_TLS=true
export SENTRY_SERVER_EMAIL="<EMAIL>"

docker stack deploy -c docker-compose.yml sentry

Above is the configuration that throws the error that the files DO NOT exist on the worker. On the sentry container they do exist and therefore the UI seems okay, but the worker can not find the files.

With the above docker-compose.yml the /var/lib/sentry/files is not shared. So we tried to share this volume as following:

version: '3.1'

networks:
  net:
    external:
      name: loadbalancer_net
  sentry_net:
    driver: overlay

services:
  sentry:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    volumes:
      - sentry_lib_files:/var/lib/sentry/files
    labels:
      - "traefik.enable=true"
      - "traefik.port=9000"
      - "traefik.frontend.rule=Host:sentry.example.org"
    networks:
      - net
      - sentry_net

  celery-beat:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    command: sentry run cron
    volumes:
      - sentry_lib_files:/var/lib/sentry/files
    networks:
      - sentry_net
    deploy:
      replicas: 0

  celery-worker:
    image: sentry
    environment:
      - SENTRY_POSTGRES_HOST=${SENTRY_POSTGRES_HOST}
      - SENTRY_DB_USER=${SENTRY_DB_USER}
      - SENTRY_DB_PASSWORD=${SENTRY_DB_PASSWORD}
      - SENTRY_REDIS_HOST=${SENTRY_REDIS_HOST}
      - SENTRY_SECRET_KEY=${SENTRY_SECRET_KEY}
      - SENTRY_EMAIL_HOST=${SENTRY_EMAIL_HOST}
      - SENTRY_EMAIL_PORT=${SENTRY_EMAIL_PORT}
      - SENTRY_EMAIL_USER=${SENTRY_EMAIL_USER}
      - SENTRY_EMAIL_PASSWORD=${SENTRY_EMAIL_PASSWORD}
      - SENTRY_EMAIL_USE_TLS=${SENTRY_EMAIL_USE_TLS}
      - SENTRY_SERVER_EMAIL=${SENTRY_SERVER_EMAIL}
    command: sentry run worker
    volumes:
      - sentry_lib_files:/var/lib/sentry/files
    networks:
      - sentry_net
    deploy:
      replicas: 0
  redis:
    image: redis
    networks:
      - sentry_net

  postgres:
    image: postgres:latest
    environment:
      - POSTGRES_USER=${SENTRY_DB_USER}
      - POSTGRES_PASSWORD=${SENTRY_DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    networks:
      - sentry_net

volumes:
  postgres_data:
  sentry_lib_files:

The volume should be shared now. But when I upload the artifacts. The sentry container does not throw any error, but the files are not saved into that directory on both containers. And therefore the files are not accessible from the UI.

Any idea's how we can share the volume with this docker-compose.yml setup and get everything up and running?

Could it be related to this issue: https://github.com/getsentry/sentry-cli/issues/141#issuecomment-486268346. I would like to try and clean all related file tables. But how do I do that on the postgresql container?

[mark-veenstra]

After deleting the files from postgres database and adding the volume mounts in the docker-compose as described in this post. All works and sources are mapped.

Deleting files from postgres can be done with the following commands. Make sure to login to the shell of the sentry-postgres container:

psql -U sentry -d postgres
delete from sentry_releasefile; delete from sentry_fileblobindex; delete from sentry_fileblob; delete from sentry_file where type = 'release.file';