Sentry: Duplicate key value violates unique constraint

[kaplun]

I am running sentry-8.22.0 from the docker-compose configuration.

I have never run a previous version.

From the PostgreSQL logs I see this error per every event being sent to Sentry:

ERROR:  duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
DETAIL:  Key (project_id, environment_id)=(22, 2) already exists.
STATEMENT:  INSERT INTO "sentry_environmentproject" ("project_id", "environment_id") VALUES (22, 2) RETURNING "sentry_environmentproject"."id"

[kaplun]

Note I am using GitLab authentication. Also Note this doesn't cause visible problems. It's just filling up system logs.

[mattrobenolt]

Yeah, this is working as intended. We handle this gracefully in the application, but Postgres does not know that.

If you want to reduce the volume of these errors, configure caching. We cache this value so we know not to try to insert it again.

[kaplun]

Hi @mattrobenolt, thanks for the quick feedback!

What exactly do you mean by "configure caching". I am using almost default docker-compose set up provided by you. There I see:

#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

memcached = env('SENTRY_MEMCACHED_HOST') or (env('MEMCACHED_PORT_11211_TCP_ADDR') and 'memcached')
if memcached:
    memcached_port = (
        env('SENTRY_MEMCACHED_PORT')
        or '11211'
    )
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
            'LOCATION': [memcached + ':' + memcached_port],
            'TIMEOUT': 3600,
        }
    }

# A primary cache is required for things such as processing events
SENTRY_CACHE = 'sentry.cache.redis.RedisCache'

and both memcached and redis seem to be correctly configured. Is there anything else I would need to do?

[JoyceBabu]

I am also having the same issue. The error is appearing several times in a second. So I don't think the caching is working as intended in my case.

postgres_1 | ERROR: duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
postgres_1 | DETAIL: Key (project_id, environment_id)=(2, 1) already exists.
postgres_1 | STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (2, 1, NULL) RETURNING "sentry_environmentproject"."id"

[jeromepin]

@mattrobenolt Could you give more informations about that issue ? Why it's "working as intended" ? And how caching should resolve this ?

[MRigal]

Still a pain with Sentry 9.0.0

[mattrobenolt]

Why is it a pain? We run this at a much larger scale and I’d hardly consider it a pain.

[wittdev]

I hit this thread with the same issue.

@mattrobenolt it's a pain for me because it spams system logs with errors, which, by default, are picked up by intrusion detection + prevention (OSSEC) and emailed / delivered to a sysadmin for followup. Sample error from my /var/log/messages file:

Apr 12 03:54:05 ip-86-77-11-50 journal: 2020-04-12 09:54:05.089 UTC [5416] ERROR:  duplicate key value violates unique constraint "sentry_grouprelease_group_id_46ba6e430d088d04_uniq"
Apr 12 03:54:05 ip-86-77-11-50 journal: 2020-04-12 09:54:05.089 UTC [5416] DETAIL:  Key (group_id, release_id, environment)=(212, 96, production) already exists.
Apr 12 03:54:05 ip-86-77-11-50 journal: 2020-04-12 09:54:05.089 UTC [5416] STATEMENT:  INSERT INTO "sentry_grouprelease" ("project_id", "group_id", "release_id", "environment", "first_seen", "last_seen") VALUES (4, 212, 96, 'production', '2020-04-12 09:54:04+00:00', '2020-04-12 09:54:04+00:00') RETURNING "sentry_grouprelease"."id"

If you're a sysadmin monitoring production closely for any kind of error or anomaly, and having to write a million little custom exceptions to work around applications that carelessly write errors to logs that aren't actually errors, this is definitely a pain. "Fixing" the problem(s) from a sysadmin standpoint (e.g. by messing with postgresql's caching) is also a pain. (We're using the dockerized sentry, so I'm not sure yet how to make permanent caching config changes in the postgresql container.. maybe a workaround would be to ship the next version with caching already tweaked?).

I've been a developer in this position before, too. The hardest thing for me was to convince myself that coding it the "right" way was worth my time. ;) In that case, I beg you to pretty please, reconsider, and handle these "upsert" cases so they handle duplicates without dropping errors in the system log.

Thanks, and keep up the good work!

[mattrobenolt]

We could probably tease this idea now since we run Postgres 9.6 ourselves in production. The primary issue before (not sure if I said this really, I'm not reading the thread again) is that we didn't run a version of Postgres that truly supported UPSERT. This came in 9.5 iirc, and we used to run 9.3.

This meant we'd have to do a bunch of work that we didn't even personally benefit from.

The other issue now remaining is the Django ORM, which is what we're using everywhere. There's no way in the ORM to explicitly do this. So we can augment the ORM with our own custom methods that provide this behavior, which I'd actually like to do now that we require 9.6. Not just for the log spam, but it's also more efficient than creating a new transaction just to isolate the failure.

But this is likely a bunch of work to improve this with arguably still minimal gains even from our end. So this might be something that still sits on the back burner or something I attempt to tackle during our next Hack Week or something.

fwiw Postgres can also be configured to not spew these errors. We could/should probably do that by default on the onpremise we bundle to cut down on the noise generated there. cc @BYK