Onpremise: Support for Redis unix domain socket connections

[eandersons]

Hi,

It would be really nice to have an option to use Redis unix domain socket connection out of the box if desired. The only reference I found to this regard is getsentry/sentry issue #1549 which therefore after a little web search leads to Redis Python package descricption's section "Connections".

The problem is that using unix_socket_path in config.yml is not enough. If Redis is installed on host and SENTRY_REDIS_HOST is not defined in .env then the following error is displayed when creating the Sentry web Docker container:

!! Configuration error: Exception: Error: REDIS_PORT_6379_TCP_ADDR (or SENTRY_REDIS_HOST) is undefined, did you forget to `--link` a redis container?

If containerized Redis server is used and SENTRY_REDIS_HOST is not specified then the default configuration values are applied and it seems that sentry.conf.py overrides config.yml. Not to mention that in Redis server TCP loopback is used by default.

The main reason I wanted to use unix domain socket is to to get better (at least in theory) performance.

It is important to mention that unix domain sockets can be used if everything runs on one server (Redis server and Sentry containers or Redis and Sentry containers) and that TCP connections are scalable and more portable so unix domain socket connections to Redis are not suitable for all use cases.
References:

• How fast is Redis? > Factors impacting Redis performance
• How to Connect to Redis with Unix Sockets in Docker (this is a bit outdated regarding docker-compose.yml, now named volumes should be used instead of volumes_from)
• Benchmarking IP and Unix domain sockets (for real)
• unix domain sockets vs. internet sockets

---

After many hours of trial and error I managed to get Sentry to use unix domain socket connections to Redis running on host. If anyone is eager to use unix domain socket connections to Redis, below is description of what I did.

Three dots (...) in configuration samples means that lines were omitted.

Edit Redis configuration to use unix domain socket connections and to allow containers to access socket file. Adjust other settings as necessary.

...
# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 0
...
# Unix socket.
#
# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
unixsocket /var/run/redis/redis-server.sock
unixsocketperm 776
...
# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /data
...

Some may consider permissions 776 too permissive and it probably is possible to use more restrictive permissions (post Use Unix sockets with Docker may provide some ideas).

Restart Redis server after necessary configuration changes have been made.

Modify docker-compose.yml to mount Redis (installed on host) unix domain socket file in sentry containers:

x-defaults: &defaults
  ...
  volumes:
    ...
    - /var/run/redis/redis-server.sock:/var/run/redis/redis-server.sock
...

If containerized Redis server is used, named volume is the way to share the Redis socket file between Docker containers.
In this case copy Redis configuration to the directory where docker-compose.yml is located and make the necessary adjustments mentioned above.

docker-compose.yml for containerized Redis:

...
x-defaults: &defaults
  ...
  volumes:
    ...
    - redis-run:/var/run/redis/
...
services:
  ...
  redis:
    ...
    command: [
      "/bin/sh", "-c",
      "chown -R redis /var/run/redis/ && redis-server /etc/redis/redis.conf"
    ]
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
      - redis-run:/var/run/redis/
... 
volumes:
  ...
  redis-run:

Remove (if present) SENTRY_REDIS_HOST and SENTRY_REDIS_PORT from .env and add a new environment variable SENTRY_REDIS_UNIX_SOCKET_PATH:

...
SENTRY_REDIS_UNIX_SOCKET_PATH=/var/run/redis/redis-server.sock
#SENTRY_REDIS_PASWORD=password
#SENTRY_REDIS_DB=1
...

SENTRY_REDIS_PASWORD must be specified if the Redis installation is set up with a password.
SENTRY_REDIS_DB may be needed if Redis is installed on host and other instances are already using it. In this case set an appropriate database number.

Some tweaks are needed in sentry.config.py as well to actually utilise the new environment variable SENTRY_REDIS_UNIX_SOCKET_PATH that just was set in .env to configure Sentry to use unix domain socket connections to Redis:

...
#########
# Redis #
#########

# Generic Redis configuration used as defaults for various things including:
# Buffers, Quotas, TSDB
# https://github.com/getsentry/sentry/issues/1549

redis = env('SENTRY_REDIS_HOST') or env('SENTRY_REDIS_UNIX_SOCKET_PATH') \
    or (env('REDIS_PORT_6379_TCP_ADDR') and 'redis')

if not redis:
    raise Exception(
        'Error: REDIS_PORT_6379_TCP_ADDR (or SENTRY_REDIS_HOST or '
        + 'SENTRY_REDIS_UNIX_SOCKET_PATH) is undefined.'
    )

redis_connection_type = 'unix_socket_path' \
    if env('SENTRY_REDIS_UNIX_SOCKET_PATH') else 'host'
redis_password = env('SENTRY_REDIS_PASSWORD') or ''
redis_port = env('SENTRY_REDIS_PORT') or '6379'
redis_db = env('SENTRY_REDIS_DB') or '0'

sentry_redis_options = {
    'redis.clusters': {
        'default': {
            'hosts': {
                0: {
                    redis_connection_type: redis,
                    'password': redis_password,
                    'db': redis_db
                }
            }
        }
    }
}

if redis_connection_type == 'host':
    sentry_redis_options['redis.clusters']['default']['hosts'][0]['port'] = redis_port

SENTRY_OPTIONS.update(sentry_redis_options)
...
#########
# Queue #
#########

# See https://docs.getsentry.com/on-premise/server/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

rabbitmq = env('SENTRY_RABBITMQ_HOST') or (env('RABBITMQ_PORT_5672_TCP_ADDR') and 'rabbitmq')

if rabbitmq:
    ...
else:
    # https://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#configuration
    if redis_connection_type == 'host':
        BROKER_URL = 'redis://:' + redis_password + '@' + redis + ':' \
            + redis_port + '/' + redis_db
    else:
        BROKER_URL = 'redis+socket://:' + redis_password + '@' + redis \
            + '?virtual_host=' + redis_db

Before docker-compose build --pull && docker-compose up --detach is run, one line must be added in requirements.txt otherwise in Sentry cron and worker containers the error TypeError: __init__() got an unexpected keyword argument 'socket_connect_timeout' will be thrown (solution was taken from a comment in a Celery GitHub issue):

kombu == 3.0.36

There will be error ERROR: sentry 9.1.2 has requirement kombu==3.0.35, but you'll have kombu 3.0.36 which is incompatible. while building an image, but it seems that the version 3.0.36 of kombu does not break anything.

Finally build the image, run migrations and create containers: docker-compose build --pull && docker-compose run --rm web upgrade && docker-compose up --detach (sudo before each docker-compose should be used if the user executing these commands have not been added to the group docker).

[BYK]

Hi @eandersons,

Thanks a lot for sharing your journey on getting Redis to run over a unix socket here. That said I'm not sure what's your expectation from us on the on-premise repo or on Sentry side.

Also, I noticed you are using the files from 9.1.2 which are now changed quite a bit for v10 beta so your guide may need some updating. I'm assuming things will be easier though.

[eandersons]

In version 10 things are easier indeed.

Here is what I would try:

• config.yml:

  ...
  # The redis.clusters setting is used, unsurprisingly, to configure Redis
  # clusters. These clusters can be then referred to by name when configuring
  # backends such as the cache, digests, or TSDB backend.
  redis.clusters:
  default:
  hosts:
  0:
  # https://github.com/andymccurdy/redis-py/tree/master#getting-started
  #host: redis
  #port: 6379
  # https://github.com/andymccurdy/redis-py/tree/master#connections
  unix_socket_path: /var/run/redis/redis-server.sock
  ...

• sentry.conf.py:

  ...
  #########
  # Redis #
  #########

  # Generic Redis configuration used as defaults for various things including:
  # Buffers, Quotas, TSDB

  SENTRY_OPTIONS["redis.clusters"] = {
  "default": {
  "hosts": {
  0: {
  "password": "",
  "port": 0, # Default: 6379
  "host": "/var/run/redis/redis-server.sock", # Default: redis
  "db": 2, # Default: 0
  "scheme": "redis+socket" # Default redis
  }
  }
  }
  }

  #########
  # Queue #
  #########

  # See https://docs.getsentry.com/on-premise/server/queue/ for more
  # information on configuring your queue broker and workers. Sentry relies
  # on a Python framework called Celery to manage queues.

  rabbitmq_host = None

  if rabbitmq_host:
  BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
  username="guest", password="guest", host=rabbitmq_host, vhost="/"
  )
  else:
  # https://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#configuration
  BROKER_URL = "{scheme}//:{password}@{host}:{port}/{db}".format(
  **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
  )
  ...

• change docker-compose.yml to mount Redis socket in containers and to remve service redis.

"would" because I could not test this as I am using an old laptop as a home server therefore I could not get to run the ClickHouse container/server because of missing features in laptop's processor and therefore some other tweaks might be necessary as well.

To sum all up, I created this issue because in the beginning I had a feeling that setting Sentry up to use unix domain socket connection to Redis server should be easier. Like it is with PostgreSQL.
Now that in version 10 things are easier, the least you could do is to document (in files or in documentation) possible connection options for redis.clusters in config.yml and for Redis BROKER_URL in sentry.conf.py. Or at least provide links to relevant sections in original documentation (like I did above):

• for redis.clusters in configy.yml: https://pypi.org/project/redis/#getting-started and https://pypi.org/project/redis/#connections;
• for Redis BROKER_URL in sentry.conf.py: https://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#configuration.

That in my opinion actually should be enough.

[BYK]

I've thought about this again and I don't think this applies to on-premise as sharing Unix sockets across containers is not well-supported, especially on platforms other than Linux. Although this has the performance benefit, its use case is quite narrow and not very applicable to this repo.

Would you be content this information living in an issue @eandersons or do you still think this should be somewhere in the official docs as an edge case?

[eandersons]

@BYK As using Unix domain sockets is quite specific, I think that this issue as a source of information would be enough.

[BYK]

Thanks a lot again for taking the extra effort of sharing this with us and the community @eandersons, very much appreciated.