Onpremise: Not able to upload debug symbols

[qodeboy]

Important Details

How are you running Sentry?

• [x] On-Premise docker [Version 9.0.0]

Thing to note on environment:

• We use docker-compose to manage our containers, config attached below.
• Docker is behind reverse proxy. We use apache for this matter, config attached as well.
• The issue is reproducible even if we connect to docker instance without reverse proxy (on 16100 port).

docker-compose.yml:

version: "3"

services:
  redis:
    image: redis:3.0
    container_name: sentry-redis
    restart: always
    volumes:
      - ./data/redis:/data:rw

  memcached:
    image: memcached:1.4
    container_name: sentry-memcached
    restart: always

  postgres:
    image: postgres:9.6
    container_name: sentry-postgres
    restart: always
    environment:
      POSTGRES_USER: "sentry"
      POSTGRES_PASSWORD: "<redacted>"
    volumes:
      - ./data/postgres:/var/lib/postgresql/data:rw

  web:
    image: sentry:9.0
    container_name: sentry-web
    restart: always
    environment:
      SENTRY_SECRET_KEY: "<redacted>"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: '<redacted>'
      SENTRY_EMAIL_PASSWORD: '<redacted>'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "<redacted>"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/tmp/sentry-files"
    depends_on:
      - redis
      - postgres
    ports:
      - 16100:9000
    volumes:
      - ./data/sentry/files:/var/lib/sentry/files:rw
      - ./data/sentry/config:/etc/sentry:rw

  cron:
    image: sentry:9.0
    container_name: sentry-cron
    restart: always
    environment:
      SENTRY_SECRET_KEY: "<redacted>"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: '<redacted>'
      SENTRY_EMAIL_PASSWORD: '<redacted>'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "<redacted>"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/tmp/sentry-files"
    command: "sentry run cron"
    depends_on:
      - web

  worker:
    image: sentry:9.0
    container_name: sentry-worker
    restart: always
    environment:
      SENTRY_SECRET_KEY: "<redacted>"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: '<redacted>'
      SENTRY_EMAIL_PASSWORD: '<redacted>'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "<redacted>"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/tmp/sentry-files"
    command: "sentry run worker"
    depends_on:
      - web

```apache-sentry.conf:```

<VirtualHost *:80>
  ServerName <redacted>
  ServerSignature Off

  RewriteEngine on
  RewriteCond %{HTTPS} !=on
  RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [NE,R,L]
</VirtualHost>

<VirtualHost *:443>
  SSLEngine on
  SSLProtocol all -SSLv2
  SSLHonorCipherOrder on
  SSLCipherSuite "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
  Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
  SSLCompression Off

  SSLCACertificateFile /etc/letsencrypt/live/<redacted>/fullchain.pem
  SSLCertificateFile /etc/letsencrypt/live/<redacted>/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/<redacted>/privkey.pem

  Include /etc/letsencrypt/options-ssl-apache.conf

  ServerName <redacted>
  ServerSignature Off

  ProxyPreserveHost On

  ProxyPass / http://localhost:16100/
  ProxyPassReverse / http://localhost:16100/
  ProxyPreserveHost On
  RequestHeader set X-Forwarded-Proto "https" env=HTTPS

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  AllowEncodedSlashes NoDecode

  RequestHeader set X_FORWARDED_PROTO 'https'
  RequestHeader set X-Forwarded-Ssl on

  # It is assumed that the log directory is in /var/log/httpd.
  # For Debian distributions you might want to change this to /var/log/apache2.
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog ${APACHE_LOG_DIR}/<redacted>-error.log
  CustomLog ${APACHE_LOG_DIR}/<redacted>-forwarded.log common_forwarded
  CustomLog ${APACHE_LOG_DIR}/<redacted>-access.log combined env=!dontlog
  CustomLog ${APACHE_LOG_DIR}/<redacted>.log combined
</VirtualHost>

## Description Not able to upload debug symbols using sentry-cli/1.37.3 and sentry/9.0.0-onpremise. Created a new project and tried to upload dsyms (downloaded from iTunes Connect) to sentry using ```sentry-cli --url https:/// --auth-token --log-level DEBUG upload-dsym --org sentry --project appDsyms.zip``` The first run resulted in following (notice bolded messages):

 INFO    2019-01-16 11:41:57.045400807 +01:00 Loaded config from /root/.sentryclirc
  DEBUG   2019-01-16 11:41:57.045480156 +01:00 sentry-cli version: 1.37.3, platform: "linux", architecture: "x86_64"
  INFO    2019-01-16 11:41:57.045579013 +01:00 sentry-cli was invoked with the following command line: "sentry-cli" "--url" "https:///" "--auth-token" "" "--log-level" "DEBUG" "upload-dsym" "--org" "sentry" "--project" "" "appDsyms.zip"
  DEBUG   2019-01-16 11:41:57.049436827 +01:00 request GET https:///api/0/organizations/sentry/chunk-upload/
  DEBUG   2019-01-16 11:41:57.049466959 +01:00 using token authentication
  DEBUG   2019-01-16 11:41:57.049480916 +01:00 retry number 0, max retries: 0
  DEBUG   2019-01-16 11:41:57.067201060 +01:00 > GET /api/0/organizations/sentry/chunk-upload/ HTTP/1.1
  DEBUG   2019-01-16 11:41:57.067215167 +01:00 > Host: 
  DEBUG   2019-01-16 11:41:57.067222723 +01:00 > Accept: */*
  DEBUG   2019-01-16 11:41:57.067230001 +01:00 > Connection: TE
  DEBUG   2019-01-16 11:41:57.067236958 +01:00 > TE: gzip
  DEBUG   2019-01-16 11:41:57.067245218 +01:00 > User-Agent: sentry-cli/1.37.3
  DEBUG   2019-01-16 11:41:57.067553112 +01:00 > Authorization: Bearer eb1acc7d***
  DEBUG   2019-01-16 11:41:57.079702450 +01:00 < HTTP/1.1 200 OK
  DEBUG   2019-01-16 11:41:57.079728736 +01:00 < Date: Wed, 16 Jan 2019 10:41:57 GMT
  DEBUG   2019-01-16 11:41:57.079743997 +01:00 < Server: Apache/2.4.37 (Ubuntu)
  DEBUG   2019-01-16 11:41:57.079753656 +01:00 < Content-Length: 220
  DEBUG   2019-01-16 11:41:57.079763728 +01:00 < X-XSS-Protection: 1; mode=block
  DEBUG   2019-01-16 11:41:57.079774430 +01:00 < Content-Language: en
  DEBUG   2019-01-16 11:41:57.079784973 +01:00 < X-Content-Type-Options: nosniff
  DEBUG   2019-01-16 11:41:57.079798772 +01:00 < Vary: Accept-Language,Cookie
  DEBUG   2019-01-16 11:41:57.079809399 +01:00 < Allow: GET, POST, HEAD, OPTIONS
  DEBUG   2019-01-16 11:41:57.079821822 +01:00 < X-Frame-Options: deny
  DEBUG   2019-01-16 11:41:57.079833713 +01:00 < Content-Type: application/json
  DEBUG   2019-01-16 11:41:57.079845297 +01:00 < Strict-Transport-Security: max-age=15768000;includeSubdomains
  DEBUG   2019-01-16 11:41:57.081369870 +01:00 response status: 200
  DEBUG   2019-01-16 11:41:57.081384213 +01:00 body: {"hashAlgorithm": "sha1", "chunkSize": 1048576, "url": "https:///api/0/organizations/sentry/chunk-upload/", "maxRequestSize": 33554432, "concurrency": 4, "chunksPerRequest": 64, "compression": ["gzip"]}
  DEBUG   2019-01-16 11:41:57.081740832 +01:00 searching location appDsyms.zip
  DEBUG   2019-01-16 11:41:57.082294371 +01:00 searching zip archive appDsyms.zip
  DEBUG   2019-01-16 11:41:57.104142327 +01:00 trying to parse dif 59DB9BDF-B3B0-313B-98A9-74C273CA076E.dSYM/Contents/Resources/DWARF/Sentry
  DEBUG   2019-01-16 11:41:57.104159543 +01:00 Ctx: Ctx { container: Little, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104186842 +01:00 Mach-o header: 0xfeedface 12 0x9 DSYM 6 2856 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.104221411 +01:00 0 - LoadCommand { offset: 28, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [89, 219, 155, 223, 179, 176, 49, 59, 152, 169, 116, 194, 115, 202, 7, 110] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104248112 +01:00 1 - LoadCommand { offset: 52, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 800, stroff: 13696, strsize: 29084 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104264969 +01:00 2 - LoadCommand { offset: 76, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 668, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 131072, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 9, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104290403 +01:00 3 - LoadCommand { offset: 744, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 1280, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 131072, vmsize: 32768, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 18, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104308324 +01:00 4 - LoadCommand { offset: 2024, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 56, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 163840, vmsize: 40960, fileoff: 4096, filesize: 38684, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.104334120 +01:00 5 - LoadCommand { offset: 2080, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 804, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 204800, vmsize: 126976, fileoff: 45056, filesize: 123487, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109251524 +01:00 trying to parse dif E371472A-4ADF-33E6-9785-5EC9E801F1BF.dSYM/Contents/Resources/DWARF/Watchdog
  DEBUG   2019-01-16 11:41:57.109277661 +01:00 Ctx: Ctx { container: Big, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109302099 +01:00 Mach-o header: 0xfeedfacf 16777228 0x0 DSYM 6 3536 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.109345892 +01:00 0 - LoadCommand { offset: 32, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [227, 113, 71, 42, 74, 223, 51, 230, 151, 133, 94, 201, 232, 1, 241, 191] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109393421 +01:00 1 - LoadCommand { offset: 56, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 160, stroff: 6656, strsize: 7188 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109420965 +01:00 2 - LoadCommand { offset: 80, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1272, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 32768, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 15, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109455204 +01:00 3 - LoadCommand { offset: 1352, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1192, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 32768, vmsize: 16384, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 14, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109483853 +01:00 4 - LoadCommand { offset: 2544, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 49152, vmsize: 12288, fileoff: 4096, filesize: 9748, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.109520087 +01:00 5 - LoadCommand { offset: 2616, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 952, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 61440, vmsize: 28672, fileoff: 16384, filesize: 24724, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113063013 +01:00 trying to parse dif DCA8B7DA-E7EB-3BDB-AD1E-C7B3C39BE96C.dSYM/Contents/Resources/DWARF/
  DEBUG   2019-01-16 11:41:57.113078925 +01:00 Ctx: Ctx { container: Little, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113091370 +01:00 Mach-o header: 0xfeedface 12 0x9 DSYM 7 2844 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.113109411 +01:00 0 - LoadCommand { offset: 28, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [220, 168, 183, 218, 231, 235, 59, 219, 173, 30, 199, 179, 195, 155, 233, 108] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113139401 +01:00 1 - LoadCommand { offset: 52, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 189, stroff: 6364, strsize: 8417 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113178049 +01:00 2 - LoadCommand { offset: 76, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 56, segname: [95, 95, 80, 65, 71, 69, 90, 69, 82, 79, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 16384, fileoff: 0, filesize: 0, maxprot: 0, initprot: 0, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113197849 +01:00 3 - LoadCommand { offset: 132, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 872, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 16384, vmsize: 32768, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 12, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113225718 +01:00 4 - LoadCommand { offset: 1004, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 1008, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 49152, vmsize: 16384, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 14, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113249457 +01:00 5 - LoadCommand { offset: 2012, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 56, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 65536, vmsize: 12288, fileoff: 4096, filesize: 10685, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.113273250 +01:00 6 - LoadCommand { offset: 2068, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 804, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 77824, vmsize: 16384, fileoff: 16384, filesize: 15056, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210151061 +01:00 trying to parse dif 5AD9457F-16B2-3EDC-B643-DE7A274C9C37.dSYM/Contents/Resources/DWARF/KSCrash
  DEBUG   2019-01-16 11:41:57.210173920 +01:00 Ctx: Ctx { container: Big, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210189962 +01:00 Mach-o header: 0xfeedfacf 16777228 0x0 DSYM 6 3536 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.210232457 +01:00 0 - LoadCommand { offset: 32, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [90, 217, 69, 127, 22, 178, 62, 220, 182, 67, 222, 122, 39, 76, 156, 55] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210252008 +01:00 1 - LoadCommand { offset: 56, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 1844, stroff: 33600, strsize: 65714 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210298230 +01:00 2 - LoadCommand { offset: 80, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 952, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 425984, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210318602 +01:00 3 - LoadCommand { offset: 1032, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1512, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 425984, vmsize: 98304, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 18, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210344900 +01:00 4 - LoadCommand { offset: 2544, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 524288, vmsize: 98304, fileoff: 4096, filesize: 95218, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.210366313 +01:00 5 - LoadCommand { offset: 2616, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 952, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 622592, vmsize: 835584, fileoff: 102400, filesize: 831628, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.308896611 +01:00 trying to parse dif ECF692BB-CD6E-3A71-B66A-B6FC1C04EF4A.dSYM/Contents/Resources/DWARF/KSCrash
  DEBUG   2019-01-16 11:41:57.308944617 +01:00 Ctx: Ctx { container: Little, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.308963137 +01:00 Mach-o header: 0xfeedface 12 0x9 DSYM 6 2856 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.308978206 +01:00 0 - LoadCommand { offset: 28, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [236, 246, 146, 187, 205, 110, 58, 113, 182, 106, 182, 252, 28, 4, 239, 74] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.308995291 +01:00 1 - LoadCommand { offset: 52, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 1886, stroff: 26728, strsize: 67881 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.309009672 +01:00 2 - LoadCommand { offset: 76, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 668, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 376832, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 9, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.309025382 +01:00 3 - LoadCommand { offset: 744, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 1280, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 376832, vmsize: 65536, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 18, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.309040848 +01:00 4 - LoadCommand { offset: 2024, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 56, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 442368, vmsize: 94208, fileoff: 4096, filesize: 90513, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.309055413 +01:00 5 - LoadCommand { offset: 2080, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 804, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 536576, vmsize: 696320, fileoff: 98304, filesize: 694249, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312709020 +01:00 trying to parse dif 6458DBF0-9722-3CB3-971A-2E9763DF9EB0.dSYM/Contents/Resources/DWARF/
  DEBUG   2019-01-16 11:41:57.312726482 +01:00 Ctx: Ctx { container: Big, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312739611 +01:00 Mach-o header: 0xfeedfacf 16777228 0x0 DSYM 7 3448 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.312752980 +01:00 0 - LoadCommand { offset: 32, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [100, 88, 219, 240, 151, 34, 60, 179, 151, 26, 46, 151, 99, 223, 158, 176] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312767607 +01:00 1 - LoadCommand { offset: 56, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 176, stroff: 6912, strsize: 7938 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312781035 +01:00 2 - LoadCommand { offset: 80, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 72, segname: [95, 95, 80, 65, 71, 69, 90, 69, 82, 79, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 4294967296, fileoff: 0, filesize: 0, maxprot: 0, initprot: 0, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312797216 +01:00 3 - LoadCommand { offset: 152, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1192, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4294967296, vmsize: 32768, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 14, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312812093 +01:00 4 - LoadCommand { offset: 1344, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1112, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4295000064, vmsize: 16384, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 13, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312826764 +01:00 5 - LoadCommand { offset: 2456, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 4295016448, vmsize: 12288, fileoff: 4096, filesize: 10754, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.312841166 +01:00 6 - LoadCommand { offset: 2528, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 952, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 4295028736, vmsize: 16384, fileoff: 16384, filesize: 15009, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319329919 +01:00 trying to parse dif 886D9A4F-4850-3F2F-A0B2-6E9571FB73BC.dSYM/Contents/Resources/DWARF/Watchdog
  DEBUG   2019-01-16 11:41:57.319345197 +01:00 Ctx: Ctx { container: Little, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319366540 +01:00 Mach-o header: 0xfeedface 12 0x9 DSYM 6 2924 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.319387682 +01:00 0 - LoadCommand { offset: 28, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [136, 109, 154, 79, 72, 80, 63, 47, 160, 178, 110, 149, 113, 251, 115, 188] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319401929 +01:00 1 - LoadCommand { offset: 52, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 194, stroff: 6424, strsize: 8783 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319429909 +01:00 2 - LoadCommand { offset: 76, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 1008, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 32768, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 14, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319464074 +01:00 3 - LoadCommand { offset: 1084, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 1008, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 32768, vmsize: 16384, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 14, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319497792 +01:00 4 - LoadCommand { offset: 2092, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 56, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 49152, vmsize: 12288, fileoff: 4096, filesize: 11111, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.319521310 +01:00 5 - LoadCommand { offset: 2148, command: Segment32(SegmentCommand32 { cmd: 1, cmdsize: 804, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 61440, vmsize: 36864, fileoff: 16384, filesize: 33533, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338747482 +01:00 trying to parse dif 626FDD78-C006-3BCC-BD3C-2A8F4460E041.dSYM/Contents/Resources/DWARF/Sentry
  DEBUG   2019-01-16 11:41:57.338762587 +01:00 Ctx: Ctx { container: Big, le: Little }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338799917 +01:00 Mach-o header: 0xfeedfacf 16777228 0x0 DSYM 6 3456 0x0 0x0  (from goblin)
  DEBUG   2019-01-16 11:41:57.338813029 +01:00 0 - LoadCommand { offset: 32, command: Uuid(UuidCommand { cmd: 27, cmdsize: 24, uuid: [98, 111, 221, 120, 192, 6, 59, 204, 189, 60, 42, 143, 68, 96, 224, 65] }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338846111 +01:00 1 - LoadCommand { offset: 56, command: Symtab(SymtabCommand { cmd: 2, cmdsize: 24, symoff: 4096, nsyms: 761, stroff: 16272, strsize: 27160 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338859758 +01:00 2 - LoadCommand { offset: 80, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 872, segname: [95, 95, 84, 69, 88, 84, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 0, vmsize: 131072, fileoff: 0, filesize: 0, maxprot: 5, initprot: 5, nsects: 10, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338883773 +01:00 3 - LoadCommand { offset: 952, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 1512, segname: [95, 95, 68, 65, 84, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 131072, vmsize: 49152, fileoff: 0, filesize: 0, maxprot: 3, initprot: 3, nsects: 18, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338898268 +01:00 4 - LoadCommand { offset: 2464, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 72, segname: [95, 95, 76, 73, 78, 75, 69, 68, 73, 84, 0, 0, 0, 0, 0, 0], vmaddr: 180224, vmsize: 40960, fileoff: 4096, filesize: 39336, maxprot: 1, initprot: 1, nsects: 0, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.338930573 +01:00 5 - LoadCommand { offset: 2536, command: Segment64(SegmentCommand64 { cmd: 25, cmdsize: 952, segname: [95, 95, 68, 87, 65, 82, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0], vmaddr: 221184, vmsize: 118784, fileoff: 45056, filesize: 116590, maxprot: 7, initprot: 3, nsects: 11, flags: 0 }) }  (from goblin)
  DEBUG   2019-01-16 11:41:57.339066863 +01:00 finished zip archive appDsyms.zip
  DEBUG   2019-01-16 11:41:57.339080250 +01:00 finished location 
> Found 8 debug information files
> Prepared debug information files for upload
  DEBUG   2019-01-16 11:41:57.349819325 +01:00 request POST https:///api/0/projects/sentry//files/difs/assemble/
  DEBUG   2019-01-16 11:41:57.349839684 +01:00 using token authentication
  DEBUG   2019-01-16 11:41:57.349851086 +01:00 json body: {"b92da8cbef38eb5ac26f2ffa64b96050d5100eaa":{"name":"Watchdog","chunks":["b92da8cbef38eb5ac26f2ffa64b96050d5100eaa"]},"d3fae0adeb21429151cb135fd66c59fd152d8ba4":{"name":"Watchdog","chunks":["d3fae0adeb21429151cb135fd66c59fd152d8ba4"]},"ce04ea9c86a038d2d85255442b3a723b8e56212a":{"name":"KSCrash","chunks":["ce04ea9c86a038d2d85255442b3a723b8e56212a"]},"bc5f3ecf1b5ee32e1c64945210495eaeae96c50d":{"name":"","chunks":["bc5f3ecf1b5ee32e1c64945210495eaeae96c50d"]},"e8efaea3f8a76d46bcdec4bc759e5628d94e904f":{"name":"","chunks":["e8efaea3f8a76d46bcdec4bc759e5628d94e904f"]},"935f2f733571ecc4b2be5d2bc858221ec9e003de":{"name":"Sentry","chunks":["935f2f733571ecc4b2be5d2bc858221ec9e003de"]},"7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95":{"name":"KSCrash","chunks":["7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95"]},"ff97043c3062c516db1e991248d945a277707556":{"name":"Sentry","chunks":["ff97043c3062c516db1e991248d945a277707556"]}}
  DEBUG   2019-01-16 11:41:57.349868444 +01:00 retry number 0, max retries: 5
  DEBUG   2019-01-16 11:41:57.355857545 +01:00 > POST /api/0/projects/sentry//files/difs/assemble/ HTTP/1.1
  DEBUG   2019-01-16 11:41:57.355868794 +01:00 > Host: 
  DEBUG   2019-01-16 11:41:57.355874229 +01:00 > Accept: */*
  DEBUG   2019-01-16 11:41:57.355878320 +01:00 > Connection: TE
  DEBUG   2019-01-16 11:41:57.355882188 +01:00 > TE: gzip
  DEBUG   2019-01-16 11:41:57.355887154 +01:00 > User-Agent: sentry-cli/1.37.3
  DEBUG   2019-01-16 11:41:57.355922810 +01:00 > Authorization: Bearer eb1acc7d***
  DEBUG   2019-01-16 11:41:57.355937185 +01:00 > Content-Type: application/json
  DEBUG   2019-01-16 11:41:57.355941291 +01:00 > Content-Length: 931
  DEBUG   2019-01-16 11:41:57.470222069 +01:00 < HTTP/1.1 200 OK
  DEBUG   2019-01-16 11:41:57.470257460 +01:00 < Date: Wed, 16 Jan 2019 10:41:57 GMT
  DEBUG   2019-01-16 11:41:57.470267473 +01:00 < Server: Apache/2.4.37 (Ubuntu)
  DEBUG   2019-01-16 11:41:57.470273267 +01:00 < Content-Length: 696
  DEBUG   2019-01-16 11:41:57.470278187 +01:00 < X-XSS-Protection: 1; mode=block
  DEBUG   2019-01-16 11:41:57.470283498 +01:00 < Content-Language: en
  DEBUG   2019-01-16 11:41:57.470288059 +01:00 < X-Content-Type-Options: nosniff
  DEBUG   2019-01-16 11:41:57.470293170 +01:00 < Vary: Accept-Language,Cookie
  DEBUG   2019-01-16 11:41:57.470298224 +01:00 < Allow: POST, OPTIONS
  DEBUG   2019-01-16 11:41:57.470303382 +01:00 < X-Frame-Options: deny
  DEBUG   2019-01-16 11:41:57.470308630 +01:00 < Content-Type: application/json
  DEBUG   2019-01-16 11:41:57.470313377 +01:00 < Strict-Transport-Security: max-age=15768000;includeSubdomains
  DEBUG   2019-01-16 11:41:57.472276374 +01:00 response status: 200
  DEBUG   2019-01-16 11:41:57.472308097 +01:00 body: {"7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95": {"state": "created", "missingChunks": []}, "ff97043c3062c516db1e991248d945a277707556": {"state": "created", "missingChunks": []}, "bc5f3ecf1b5ee32e1c64945210495eaeae96c50d": {"state": "created", "missingChunks": []}, "b92da8cbef38eb5ac26f2ffa64b96050d5100eaa": {"state": "created", "missingChunks": []}, "935f2f733571ecc4b2be5d2bc858221ec9e003de": {"state": "created", "missingChunks": []}, "e8efaea3f8a76d46bcdec4bc759e5628d94e904f": {"state": "created", "missingChunks": []}, "d3fae0adeb21429151cb135fd66c59fd152d8ba4": {"state": "created", "missingChunks": []}, "ce04ea9c86a038d2d85255442b3a723b8e56212a": {"state": "created", "missingChunks": []}}
>Nothing to upload, all files are on the server
  DEBUG   2019-01-16 11:41:57.472555623 +01:00 request POST https:///api/0/projects/sentry//reprocessing/
  DEBUG   2019-01-16 11:41:57.472569526 +01:00 using token authentication
  DEBUG   2019-01-16 11:41:57.472575329 +01:00 retry number 0, max retries: 0
  DEBUG   2019-01-16 11:41:57.480856588 +01:00 > POST /api/0/projects/sentry//reprocessing/ HTTP/1.1
  DEBUG   2019-01-16 11:41:57.480885539 +01:00 > Host: 
  DEBUG   2019-01-16 11:41:57.480900816 +01:00 > Accept: */*
  DEBUG   2019-01-16 11:41:57.480910067 +01:00 > Connection: TE
  DEBUG   2019-01-16 11:41:57.480918648 +01:00 > TE: gzip
  DEBUG   2019-01-16 11:41:57.480927250 +01:00 > User-Agent: sentry-cli/1.37.3
  DEBUG   2019-01-16 11:41:57.480953870 +01:00 > Authorization: Bearer eb1acc7d***
  DEBUG   2019-01-16 11:41:57.586634345 +01:00 < HTTP/1.1 200 OK
  DEBUG   2019-01-16 11:41:57.586673643 +01:00 < Date: Wed, 16 Jan 2019 10:41:57 GMT
  DEBUG   2019-01-16 11:41:57.586691046 +01:00 < Server: Apache/2.4.37 (Ubuntu)
  DEBUG   2019-01-16 11:41:57.586701944 +01:00 < Content-Length: 0
  DEBUG   2019-01-16 11:41:57.586710885 +01:00 < X-XSS-Protection: 1; mode=block
  DEBUG   2019-01-16 11:41:57.586725414 +01:00 < Content-Language: en
  DEBUG   2019-01-16 11:41:57.586735736 +01:00 < X-Content-Type-Options: nosniff
  DEBUG   2019-01-16 11:41:57.586746170 +01:00 < Vary: Accept-Language,Cookie
  DEBUG   2019-01-16 11:41:57.586756893 +01:00 < Allow: POST, OPTIONS
  DEBUG   2019-01-16 11:41:57.586767775 +01:00 < X-Frame-Options: deny
  DEBUG   2019-01-16 11:41:57.586778253 +01:00 < Strict-Transport-Security: max-age=15768000;includeSubdomains
  DEBUG   2019-01-16 11:41:57.588896232 +01:00 response status: 200
  DEBUG   2019-01-16 11:41:57.588912939 +01:00 body: 

Error: some symbols did not process correctly/
  DEBUG   2019-01-16 11:41:57.588936471 +01:00 error: running update nagger
  INFO    2019-01-16 11:41:57.589041469 +01:00 Skipping update nagger update check

While it at first says "Nothing to upload, all files are on the server", there are none actually displayed in the UI:

The next run with same files results in following POST request being looped (no "some symbols did not process correctly" errors this time):

  DEBUG   2019-01-16 10:21:55.951882217 +01:00 request POST http://176.9.18.71:16100/api/0/projects/sentry/<redacted>/files/difs/assemble/
  DEBUG   2019-01-16 10:21:55.952030205 +01:00 using token authentication
  DEBUG   2019-01-16 10:21:55.952120100 +01:00 json body: {"ce04ea9c86a038d2d85255442b3a723b8e56212a":{"name":"KSCrash","chunks":["ce04ea9c86a038d2d85255442b3a723b8e56212a"]},"e8efaea3f8a76d46bcdec4bc759e5628d94e904f":{"name":"<redacted>","chunks":["e8efaea3f8a76d46bcdec4bc759e5628d94e904f"]},"b92da8cbef38eb5ac26f2ffa64b96050d5100eaa":{"name":"Watchdog","chunks":["b92da8cbef38eb5ac26f2ffa64b96050d5100eaa"]},"935f2f733571ecc4b2be5d2bc858221ec9e003de":{"name":"Sentry","chunks":["935f2f733571ecc4b2be5d2bc858221ec9e003de"]},"bc5f3ecf1b5ee32e1c64945210495eaeae96c50d":{"name":"<redacted>","chunks":["bc5f3ecf1b5ee32e1c64945210495eaeae96c50d"]},"ff97043c3062c516db1e991248d945a277707556":{"name":"Sentry","chunks":["ff97043c3062c516db1e991248d945a277707556"]},"d3fae0adeb21429151cb135fd66c59fd152d8ba4":{"name":"Watchdog","chunks":["d3fae0adeb21429151cb135fd66c59fd152d8ba4"]},"7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95":{"name":"KSCrash","chunks":["7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95"]}}
  DEBUG   2019-01-16 10:21:55.952193860 +01:00 retry number 0, max retries: 5
  DEBUG   2019-01-16 10:21:55.952577364 +01:00 > POST /api/0/projects/sentry/<redacted>/files/difs/assemble/ HTTP/1.1
  DEBUG   2019-01-16 10:21:55.952671211 +01:00 > Host: 176.9.18.71:16100
  DEBUG   2019-01-16 10:21:55.952751042 +01:00 > Accept: */*
  DEBUG   2019-01-16 10:21:55.952816907 +01:00 > Connection: TE
  DEBUG   2019-01-16 10:21:55.952886323 +01:00 > TE: gzip
  DEBUG   2019-01-16 10:21:55.952953428 +01:00 > User-Agent: sentry-cli/1.37.3
  DEBUG   2019-01-16 10:21:55.953045105 +01:00 > Authorization: Bearer eb1acc7d***
  DEBUG   2019-01-16 10:21:55.953111983 +01:00 > Content-Type: application/json
  DEBUG   2019-01-16 10:21:55.953177453 +01:00 > Content-Length: 931
  DEBUG   2019-01-16 10:21:55.987248503 +01:00 < HTTP/1.1 200 OK
  DEBUG   2019-01-16 10:21:55.987310201 +01:00 < Content-Length: 848
  DEBUG   2019-01-16 10:21:55.987337689 +01:00 < X-XSS-Protection: 1; mode=block
  DEBUG   2019-01-16 10:21:55.987361156 +01:00 < Content-Language: en
  DEBUG   2019-01-16 10:21:55.987381619 +01:00 < X-Content-Type-Options: nosniff
  DEBUG   2019-01-16 10:21:55.987403889 +01:00 < Vary: Accept-Language, Cookie
  DEBUG   2019-01-16 10:21:55.987424168 +01:00 < Allow: POST, OPTIONS
  DEBUG   2019-01-16 10:21:55.987445963 +01:00 < X-Frame-Options: deny
  DEBUG   2019-01-16 10:21:55.987468221 +01:00 < Content-Type: application/json
  DEBUG   2019-01-16 10:21:55.987544387 +01:00 response status: 200
  DEBUG   2019-01-16 10:21:55.987587788 +01:00 body: {"7e6a5a2ca3cfcb404f8a8a5fd8c48d24642f2e95": {"state": "assembling", "missingChunks": [], "detail": null}, "e8efaea3f8a76d46bcdec4bc759e5628d94e904f": {"state": "assembling", "missingChunks": [], "detail": null}, "bc5f3ecf1b5ee32e1c64945210495eaeae96c50d": {"state": "assembling", "missingChunks": [], "detail": null}, "b92da8cbef38eb5ac26f2ffa64b96050d5100eaa": {"state": "assembling", "missingChunks": [], "detail": null}, "935f2f733571ecc4b2be5d2bc858221ec9e003de": {"state": "assembling", "missingChunks": [], "detail": null}, "ff97043c3062c516db1e991248d945a277707556": {"state": "assembling", "missingChunks": [], "detail": null}, "d3fae0adeb21429151cb135fd66c59fd152d8ba4": {"state": "assembling", "missingChunks": [], "detail": null}, "ce04ea9c86a038d2d85255442b3a723b8e56212a": {"state": "assembling", "missingChunks": [], "detail": null}}

I have configured memcache and redis cache stores as well as filestore backend in my instance. Related issue on sentry-cli repository: https://github.com/getsentry/sentry-cli/issues/376. Also my comment on the same matter on the abandoned issue: https://github.com/getsentry/sentry/issues/7595

Steps to Reproduce

See above.

What you expected to happen

Debug symbols successfully uploaded to sentry and visible in the UI.

[ifokeev]

same issue here. did you resolve this?

[qodeboy]

nope :-(

[ifokeev]

and no answer from Sentry, haha

spent a lot of hours to figure out that the problem is with the server and not with my app/configuration =\

[lorenzo-ange]

Same issue here. Someone managed to resolve?

[mikuslaw]

I have the same problem. Sentry 9.0.0 OnPremise, run in Docker. In general it works, can upload minidumps from my remote platforms, but when trying to upload symbols using sentry-cli it goes into a loop and never finishes. It has the same result for both elf and breakpad symbol uploads.
Sentry-cli 1.40.0

Found what was causing issues in my case. Worker and Web containers didn't share a filestore. Maybe there is a better way to do that (not sure yet if memcached should be used for that or is it totally different kind of breed). In general with these changes in your docker-compose.yml it worked for me (I can finaly upload symbols).

version: "3.2"

services:
  redis:
    image: redis:3.0
    container_name: sentry-redis
    restart: always
    volumes:
      - ./data/redis:/data:rw

  memcached:
    image: memcached:1.4
    container_name: sentry-memcached
    restart: always

  postgres:
    image: postgres:9.6
    container_name: sentry-postgres
    restart: always
    environment:
      POSTGRES_USER: "sentry"
      POSTGRES_PASSWORD: "test"
    volumes:
      - ./data/postgres:/var/lib/postgresql/data:rw

  web:
    image: sentry:9.0
    container_name: sentry-web
    restart: always
    environment:
      SENTRY_SECRET_KEY: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: ''
      SENTRY_EMAIL_PASSWORD: 'test'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "test"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/var/lib/sentry/files"
    depends_on:
      - redis
      - postgres
    ports:
      - 16100:9000
    volumes:
      - type: bind
        source: ./data/sentry/files
        target: /var/lib/sentry/files
        bind:
          propagation: shared
      - type: bind
        source: ./data/sentry/config
        target: /etc/sentry
        bind:
          propagation: shared

  cron:
    image: sentry:9.0
    container_name: sentry-cron
    restart: always
    environment:
      SENTRY_SECRET_KEY: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: ''
      SENTRY_EMAIL_PASSWORD: 'test'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "test"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/var/lib/sentry/files"
    command: "sentry run cron"
    depends_on:
      - web

  worker:
    image: sentry:9.0
    container_name: sentry-worker
    restart: always
    environment:
      SENTRY_SECRET_KEY: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
      SENTRY_USE_SSL: "false"
      SENTRY_EMAIL_HOST: "smtp.mailgun.org"
      SENTRY_EMAIL_PORT: 587
      SENTRY_EMAIL_USER: ''
      SENTRY_EMAIL_PASSWORD: 'test'
      SENTRY_EMAIL_USE_TLS: "true"
      SENTRY_SERVER_EMAIL: "<redacted>"
      SENTRY_POSTGRES_HOST: "postgres"
      SENTRY_POSTGRES_PORT: 5432
      SENTRY_DB_USER: "sentry"
      SENTRY_DB_PASSWORD: "test"
      SENTRY_REDIS_HOST: "redis"
      SENTRY_REDIS_PORT: 6379
      SENTRY_MEMCACHED_HOST: "memcached"
      SENTRY_MEMCACHED_PORT: 11211
      SENTRY_FILESTORE_DIR: "/var/lib/sentry/files"
    command: "sentry run worker"
    volumes:
      - type: bind
        source: ./data/sentry/files
        target: /var/lib/sentry/files
        bind:
          propagation: shared
    depends_on:
      - web

[mikuslaw]

By the way. Even with the changes posted in my previous post, the symbol processing worked less then optimal. If you didn't do that yet- build a docker image from master. Really, build it and use it! It's a simple process, and it's a life changer.
Everything works now, upload using sentry-cli and processing of symbols.
Thanks for a great tool Sentry Team!

[guyromb]

I tried using GCS for sharing data between workers and web. still same issues.
Related: https://github.com/helm/charts/pull/11153

[jan-auer]

Hi all, sorry that you're experiencing issues configuring Sentry for Debug File uploads. I'll try to summarize a couple of gotchas (also ones that everyone in this thread got correct, for future reference):

• Both memcached and redis are required to get uploads working
• Sentry worker and web both need access to the same file store backend
• Always use the latest version of sentry-cli as it might also fix certain issues

Also, please upgrade to Sentry v9.1.1 (or build from the latest master), as there are a couple of crucial fixes and improvements to the debug file upload and related UI that will help to debug issues.

[BYK]

Closing this issue as we all the suggestions from @jan-auer in this repo. As @mikuslaw mentioned, master has better support but that requires running more services like Snuba and Symbolicator which we haven't had the time to make easy to run on premise yet.

Feel free to keep the conversation going at the forums: https://forum.sentry.io/c/on-premise