Onpremise: sslmode=require in docker image

[scottrobertson]

Important Details

How are you running Sentry?

• [x] On-Premise docker [Version 9]
• [ ] Saas (sentry.io)
• [ ] Other [briefly describe your environment]

Description

Cannot connect to PGDB when starting the docker image due to it requiring sslmode=require. I see no way to set this as a env var etc.

Steps to Reproduce

1. Set the DB env vars
2. Try and start the docker image
3. It will fail to connect to the DB due to requiring SSL.

Good items to include here include:

OperationalError: could not create SSL context: library has no ciphers
FATAL:  no pg_hba.conf entry for host "*****", user "doadmin", database "sentry", SSL off

08:13:02 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

What you expected to happen

Should be able to set an SSLMODE env var

Possible Solution

Allow the config to be set by env var. Example of config: https://github.com/BCCVL/sentry/blob/master/sentry.conf.py#L60

[scottrobertson]

I patched this in, which seems to have taken, but now i get:

    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 236, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/web/frontend/error_500.py", line 57, in dispatch
    return HttpResponseServerError(t.render(Context(context)))
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/test/utils.py", line 85, in instrumented_test_render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/test/utils.py", line 85, in instrumented_test_render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/usr/local/lib/python2.7/site-packages/django/test/utils.py", line 85, in instrumented_test_render
    return self.nodelist.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)
  File "/usr/local/lib/python2.7/site-packages/django/template/debug.py", line 78, in render_node
    return node.render(context)
  File "/usr/local/lib/python2.7/site-packages/django/template/base.py", line 1125, in render
    return func(*resolved_args, **resolved_kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/templatetags/sentry_react.py", line 74, in get_react_config
    user = getattr(request, 'user', None) or AnonymousUser()
  File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 213, in inner
    self._setup()
  File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 298, in _setup
    self._wrapped = self._setupfunc()
  File "/usr/local/lib/python2.7/site-packages/sentry/middleware/auth.py", line 54, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/usr/local/lib/python2.7/site-packages/sentry/middleware/auth.py", line 14, in get_user
    user = auth_get_user(request)
  File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 144, in get_user
    user = backend.get_user(user_id) or AnonymousUser()
  File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/backends.py", line 69, in get_user
    return UserModel._default_manager.get(pk=user_id)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
    return self.get_queryset().get(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 304, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 785, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 162, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 95, in _cursor
    cursor = super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 132, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 115, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 115, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not create SSL context: library has no ciphers

Is no on really using this with SSL? Surely other people are?

[UserNotFound]

Seeing this same error, as well. I was able to run the database migration (sentry upgrade), but the web service cannot connect to PG w/ SSL.

[nhnicwaller]

I'm getting the same error after upgrading to Sentry 9.1 on-prem. We're running the docker containers for Sentry in AWS and using an RDS-managed Postgres database with the rds.force_ssl=1 option enabled.

Unlike @UserNotFound above, I was unable to run the database migration (sentry upgrade). Luckily, that made it easy for me to downgrade back to Sentry 9.0 which still works as expected.

[scottrobertson]

Note: hard coding the docker image version to 9.0.0 specifically fixes this.

[insertjokehere]

I'm getting this as well in my Kubernetes cluster using the Zalando Postgres Operator to provision databases. Has anyone figured out a work around? Or at least what changed from the 9.0 image to break things like this?

[insertjokehere]

This issue looks suspiciously like unbit/uwsgi#1569 - apparently this is solved by uninstalling psycopg2 and reinstalling it but forcing source install. I will test this later today

[insertjokehere]

Yep, forcing a source install of psycopg2 fixes this. I use getsentry/onpremise to build my image, and adding:

RUN PSYCOPG=$(pip freeze | grep psycopg2) \
    && pip uninstall -y $PSYCOPG \
    && pip install --no-binary :all: $PSYCOPG

to my Dockerfile results in an image that will talk to my DB with SSL enabled

[BYK]

@insertjokehere - thanks a lot for sharing your solution. So the permanent solution for us is to upgrade psycopg2? I'd love to get this fixed in the repo for 9.1.2 if there is a good fix we can incorporate into sentry or onpremise repos.

[mattrobenolt]

I don't think we need to upgrade psycopg2, at least we shouldn't. But it seems that maybe we don't have the correct headers around when installing it to get the right SSL contexts? I'm not sure, but I'd say if the answer is to upgrade psycopg2, we're probably gonna have a bad time.

It seems the answers in here support not needing to do that. It's probably more that we need libssl or whatever the development headers are when installing, if we're not doing that already.

Alternatively, I'd like to investigate if we switch to psycopg2-binary if that solves the issue too. This package is just a re-packaging of psycopg2 that is more recommended since it's a compiled whl and should contain what's needed already. I'd like to switch to using this regardless at some point in the future since it's just a drop in replacement.

[BYK]

Can you all please try this again with 9.1.2 (which is what the latest master points to now). If this is fixed, I'll close the issue, if not we'll investigate more.

[davidchua]

I have a similar issue with OP. I have a postgres database that requires sslmode.

Using the official sentry docker image of 9.1.2, I get a

django.db.utils.OperationalError: could not create SSL context: library has no ciphers
FATAL:  SSL connection is required. Please specify SSL options and retry.

9.0.0 works fine though.

[BYK]

@davidchua - yours look like a different issue to me, related to the move from psycopg2 to psycopg2-binary in the 9.1.2 release. I'd recommend using 9.1.1 for now or try modifying the base Docker image here to install the latest version of psycopg2-binary (2.8.3) to see if it fixes the problem (it may as it is using a newer version of OpenSSL). I'll also try if we can get this to getsentry/sentry but I wouldn't be keen on releasing a 9.1.3 unless this affects a wide range of people.

[davidchua]

@BYK Sorry, my github notification is a bit wonky, didn't see your reply till now.

sentry:9.1.1 also has the same issue for me

[scottrobertson]

Just tried 9.1.2. Still fails when trying to connect to DB that requires SSL.

Doing this did work though. https://github.com/getsentry/onpremise/issues/201#issuecomment-493843675

[anitabee]

Had the same issue and pinning the version to psycopg2-binary>=2.7.0,<2.9.0 in my req.txt file solves the issue for me. I use getsentry/onbuild as my base image and run RUN pip install -r /tmp/req.txt from it.

[CH-anhngo]

we hitting the same issue. Running 9.1.2 also. What can we do to resolve it ?

[BYK]

@CH-anhngo have you tried this: https://github.com/getsentry/onpremise/issues/201#issuecomment-542891780

[CH-anhngo]

@BYK I didn't try it yet. We just using what out of helm/sentry. So I guest we need to build a customize image for sentry and run it in our k8s ?

[CH-anhngo]

still hitting this after rebuild follow the @anitabee comment above :(

sentry-web-bb56c776c-hnrcl sentry-web OperationalError: could not create SSL context: library has no ciphers
sentry-web-bb56c776c-hnrcl sentry-web FATAL:  no pg_hba.conf entry for host "10.16.73.89", user "sentry", database "sentry", SSL off
sentry-web-bb56c776c-hnrcl sentry-web 00:58:30 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

[anitabee]

Also could be that your DB has sslmode=on but that is not happening from Django's side? If so in sentry.conf.py OPTIONS you would need to add something like:
'sslmode': 'require',

[anitabee]

Also other solution that worked: https://github.com/getsentry/onpremise/issues/201#issuecomment-493843675

[CH-anhngo]

Also could be that your DB has sslmode=on but that is not happening from Django's side? If so in sentry.conf.py OPTIONS you would need to add something like:
'sslmode': 'require',

I used the require mode also with custom build image but still hitting error :(

[CH-anhngo]

Also other solution that worked: #201 (comment)

The build keep failing on me. Can you share the Dockerfile ?

Step 9/9 : RUN PSYCOPG=$(pip freeze | grep psycopg2)    && pip uninstall -y $PSYCOPG    && pip install --no-binary :all: $PSYCOPG
 ---> Running in 53e54f9d91f6
Collecting psycopg2-binary==2.8.4
  Downloading https://files.pythonhosted.org/packages/00/7b/a623f49b3248957e7eaaac52eba1117209775d54e7a8501c460473a7ba4f/psycopg2-binary-2.8.4.tar.gz (378kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-Ca_ZNf/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-Ca_ZNf/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info
         cwd: /tmp/pip-install-Ca_ZNf/psycopg2-binary/
    Complete output (23 lines):
    running egg_info
    creating /tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info
    writing /tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
    writing top-level names to /tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/top_level.txt
    writing dependency_links to /tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
    writing manifest file '/tmp/pip-install-Ca_ZNf/psycopg2-binary/pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

[anitabee]

Yes sure, I had two versions that worked for me when I run it from custom image:

Dockerfile:

FROM sentry:9.1-onbuild
RUN apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get install -y -q libxslt1-dev libldap2-dev libsasl2-dev libssl-dev   

COPY req.txt /tmp/
RUN pip install -r /tmp/req.txt

RUN apt-get remove -y -q libxslt1-dev libldap2-dev libsasl2-dev libssl-dev 
RUN rm -rf /var/lib/apt/lists/*
RUN rm -rf /tmp/*
RUN usermod -u 1001 sentry

req.txt version 1:

# TODO
# ERROR: sentry 9.1.2 has requirement Django<1.7,>=1.6.11, but you'll have django 1.11.26 which is incompatible.
Django<1.7,>=1.6.11
psycopg2==2.7.3 --no-binary=psycopg2 

req.txt version 2:

psycopg2-binary>=2.7.0,<2.9.0

Also I switched from getsentry/onbuild to getsentry version, I'm not using this custom image any-more I moved most of this config/installation to chart, so didn't have much use from onbuild...

[CH-anhngo]

Yes sure, I had two versions that worked for me when I run it from custom image:

Dockerfile:

FROM sentry:9.1-onbuild
RUN apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get install -y -q libxslt1-dev libldap2-dev libsasl2-dev libssl-dev   

COPY req.txt /tmp/
RUN pip install -r /tmp/req.txt

RUN apt-get remove -y -q libxslt1-dev libldap2-dev libsasl2-dev libssl-dev 
RUN rm -rf /var/lib/apt/lists/*
RUN rm -rf /tmp/*
RUN usermod -u 1001 sentry

req.txt version 1:

# TODO
# ERROR: sentry 9.1.2 has requirement Django<1.7,>=1.6.11, but you'll have django 1.11.26 which is incompatible.
Django<1.7,>=1.6.11
psycopg2==2.7.3 --no-binary=psycopg2 

req.txt version 2:

psycopg2-binary>=2.7.0,<2.9.0

Also I switched from getsentry/onbuild to getsentry version, I'm not using this custom image any-more I moved most of this config/installation to chart, so didn't have much use from onbuild...

so you mean getsentry 9.1 works for you with SSL require ?

[anitabee]

@CH-anhngo yes I'm running getsentry 9.1 with SSL require, still no luck?

[CH-anhngo]

@CH-anhngo yes I'm running getsentry 9.1 with SSL require, still no luck?

@anitabee
It works for us only with 9.0 as soon as we switch to [9.1](https://github.com/getsentry/docker-sentry/blob/master/9.1/Dockerfile) it throws out the same error. Wondering what 9.1 docker image did you use ? or you build the custom image for 9.1 ?
oh just noticed that we don't try 9.1-onbuild though as mentioned in your Dockerfile above.
thanks,

[BYK]

We're not planning to work on this for now so closing as wontfix.

[Bessonov]

@BYK but if you write for now, then why did you close the issue? From my point of view, the issue isn't resolved and should stay open. BTW, we run in the same issue and because the database allows ssl connections only, we have a hard time to use sentry.

[BYK]

@Bessonov we are dedicating our resources to support the recommended setup here in this repo, based on docker-compose and the issue does not apply to that so we are not going to work on this at all probably.

This repo has a certain direction which does not align with this issue, so closing it as wontfix is a stronger signal to anyone interested, rather than keeping it open forever.