Sentry-python: Using multiple DSNs, choosing based on the logger

[saifelse]

I'm looking to upgrade from raven-python[flask] to sentry-sdk[flask]. We previous had two DSNs for our backend: one for errors and one to log performance issues (e.g. slow requests / high DB query count).

We were previously able to configure this via:

from raven.contrib.flask import Sentry
from raven.handlers.logging import SentryHandler

performance_logger = logging.getLogger("benchling.performance")
performance_logger.setLevel(logging.WARNING)
sentry = Sentry(logging=True, level=logging.WARNING)

def init_sentry(app):
    sentry.init_app(app, dsn=app.config["SENTRY_DSN_SERVER"])
    performance_handler = SentryHandler(dsn=app.config["SENTRY_DSN_BACKEND_PERFORMANCE"])
    performance_logger.addHandler(performance_handler)

With the new architecture, this seems hard to do since the DSN is configured once via sentry_sdk.init where the LoggingIntegration simply listens to the root logger. I was able to hack around this by monkey-patching the logging_integration's handler as follows:

import logging

import sentry_sdk
from sentry_sdk.client import Client
from sentry_sdk.hub import Hub
from sentry_sdk.integrations.celery import CeleryIntegration
from sentry_sdk.integrations.flask import FlaskIntegration
from sentry_sdk.integrations.logging import LoggingIntegration


def register_clients_for_loggers(logger_name_to_client):
    """Monkeypatch LoggingIntegration's EventHandler to override the Client based on the record's logger"""
    hub = Hub.current
    logging_integration = hub.get_integration(LoggingIntegration)
    if not logging_integration:
        return
    handler = logging_integration._handler
    old_emit = handler.emit

    def new_emit(record):
        new_client = logger_name_to_client.get(record.name)
        previous_client = hub.client
        should_bind = new_client is not None
        try:
            if should_bind:
                hub.bind_client(new_client)
            old_emit(record)
        finally:
            if should_bind:
                hub.bind_client(previous_client)

    handler.emit = new_emit

def init_sentry(app):
    sentry_sdk.init(
        dsn=app.config["SENTRY_DSN_SERVER"],
        release=app.config["SENTRY_RELEASE"],
        environment=app.config["SENTRY_ENVIRONMENT"],
        integrations=[
            LoggingIntegration(
                level=logging.WARNING,  # Capture info and above as breadcrumbs
                event_level=logging.WARNING,  # Send warnings and errors as events
            ),
            CeleryIntegration(),
            FlaskIntegration(),
        ],
    )

    performance_logger = logging.getLogger("benchling.performance")
    performance_logger.setLevel(logging.WARNING)

    perf_client = Client(
        dsn=app.config["SENTRY_DSN_BACKEND_PERFORMANCE"],
        release=app.config["SENTRY_RELEASE"],
        environment=app.config["SENTRY_ENVIRONMENT"],
    )
    register_clients_for_loggers({performance_logger.name: perf_client})

Two questions:

1. Does this approach seem reasonable, or is there a better way to handle this?
2. If there isn't a better way, would it be possible to have sentry_sdk.init let you specify this mapping?

[untitaker]

What you can do is:

perf_client = Client(..)
regular_client = Client(..)

def send_event(event):
    if  event.get("logger") == "performance.logger":
        perf_client.capture_event(event)
    else:
        regular_client.capture_event(event)

init(transport=send_event)

We don't have a good story right now for using multiple clients transparently, although explicitly rebinding clients is something that we do aim to support:

with Hub(Hub.current) as hub:
    hub.bind_client(client)
    ...

# old hub + client is active here

[saifelse]

@untitaker : Finally picking this back up 😅, and thanks for the suggestion.

I wanted to double-check my understanding of your workaround. The gist of it is that there will be _three_ clients configured: two of which are configured just to have their DSN set properly so that we indirectly instantiate two HTTP transports, and then one more that is automatically configured via sentry_sdk.init which is fully configured correctly and uses a function as the transport that just switches between the two explicitly instantiated clients.

However, a few things that seem off about this approach:

1. Client.capture_event actually does more than necessary, and it seems like I'd actually just want to call client.transport.capture_event(event), which seems easy enough.

2. Using a function for the transport, means that flush will be _not_ be called for these nested transports, which seems to be called via hub.client.flush() for a few integrations (e.g. celery, lambda). Noting that my original approach would not have handled flushes correctly either.

It seems like the thing I _truly_ want looks closer to:

from collections import defaultdict

import sentry_sdk

def make_transport_proxy_cls(make_transport=sentry_sdk.transport.make_transport, dsn_by_logger={}):
    """Return a custom Transport subclasss that routes to a child transport based on an event's logger.

    :param make_transport: a function that returns a Transport instance given sentry_sdk.init `options`
    :param dsn_by_logger: a dict mapping a logger to a specific DSN.
    """

    class TransportProxy(sentry_sdk.transport.Transport):
        def __init__(self, options):
            super(TransportProxy, self).__init__(options)
            default_dsn = options["dsn"]
            self._dsn_by_logger_name = defaultdict(lambda: default_dsn)
            self._transport_by_dsn = {default_dsn: self._make_transport(default_dsn)}
            for logger, dsn in dsn_by_logger.iteritems():
                if dsn:
                    self._dsn_by_logger_name[logger.name] = dsn
                    self._transport_by_dsn[dsn] = self._make_transport(dsn)

        def _make_transport(self, dsn):
            return make_transport(dict(self.options, dsn=dsn))

        def flush(self, timeout, callback=None):
            for transport in self._transport_by_dsn.itervalues():
                transport.flush(timeout, callback)

        def kill(self):
            for transport in self._transport_by_dsn.itervalues():
                transport.kill()

        def capture_event(self, event):
            logger_name = event.get("logger")
            dsn = self._dsn_by_logger_name[logger_name]
            transport = self._transport_by_dsn[dsn]
            transport.capture_event(event)

    return TransportProxy

with usage like:

sentry_sdk.init(
    dsn=app.config["SENTRY_DSN_SERVER"],
        transport=make_transport_proxy_cls(
            dsn_by_logger={performance_logger: app.config["SENTRY_DSN_BACKEND_PERFORMANCE"]}
        ),
    ...
)

Getting a bit more in the weeds here, but does this approach seem sound?

[wss-chadical]

@untitaker

We have the same scenario - multiple projects for different types of issues/messages. Currently, we are just switching between two projects on the fly using sentry_sdk.init().

This seems to work just fine - but I'm pretty sure we are heavily leaking memory somewhere by doing this...

Help please. :-)

+1

[untitaker]

@wss-chadical Calling init multiple times definetly leaves a lot for GC and creates a lot of threads. If you do that, a good and simple improvement is to call Hub.current.client.close() before calling init() again, but I have no idea if that's going to fix all of your problems.

@saifelse's approach is good, but whether it's suitable for you depends on whether you want to have separate client config besides DSN for each sentry project.

[untitaker]

btw sorry @saifelse for the lack of response. Passing a function as a transport is only used for testing purposes. I think your approach is fine. Something we could improve is to put a TransportMultiplexer class into the SDK which forwards the event to all transports, and then each transport decides by itself if it wants to send.

[wss-chadical]

@untitaker I'll start with the close() - Thank you 👍

[untitaker]

to be clear I still strongly recommend against this approach because it will likely impact perf. If you need multiple clients and can't use transports, I will post a solution but it takes a while to write.

[wss-chadical]

This is a data integration server (zato) - the transactional volume is not high - a little extra overhead is not a problem. If the hub.close reduces the thread/mem leaks it will work for us.

[wss-chadical]

Hub.current.client.close() has slowed the bleeding, but not stopped it.

I decided to better understand the thread above... and it looks like I have it figured out. :-)

def send_event(event):
    if 'extra' in event and 'project' in event['extra'] and 'data-issues' in event['extra']['project'].lower():
        sio_data_issues.capture_event(event)
    else:
        sio_zato_esb_production.capture_event(event)

def sentry_init():
    global sio_zato_esb_production
    global sio_data_issues

    if sentry_sdk.Hub.current.client is None:
        sio_zato_esb_production = sentry_sdk.client.Client("https://[email protected]/34563456")
        sio_data_issues = sentry_sdk.client.Client("https://[email protected]/12341234")

        sentry_sdk.init(transport=send_event)

def test():
    init_test()
    with sentry_sdk.push_scope() as scope:
        # flag this to go to the data-issues project
        scope.set_extra( 'project', 'data-issues' )
        sentry_sdk.capture_message('test')
   # raise below goes to zato-esb-production
   raise ValueError

[untitaker]

Yes, this can work as well. You might encounter odd issues since this will trim too large strings twice (because an event goes through two clients) but I am not sure if it even matters. It's certainly a flexible solution.

[wss-chadical]

that's good to know - I'll keep an eye on it. I do pass quite a bit of extra.

It's very nice to have support here guys.

@saifelse Thanks for getting this started!

[untitaker]

I'll close this since the original question has been answered by now.

[kiddick]

What is the easiest way to use multiple dsn nowadays? I do not need to configure somehow, I just need 2 dns supported simultaneously.

[wss-chadical]

@kiddick I'm still using the method from 4/19

[yossivainshtein]

We have the same use case, is there yet a supported way to implement this?
Is @saifelse 's method with make_transport_proxy_cls still the best option?