Sentry: IO Error after update

[Ninerian]

I updated sentry to the version 8.5.1 and experiencing IOerrors while sentry tries to access the sourcemap files.

Stacktrace from the worker:

[2016-06-16 06:11:32,821: ERROR/Worker-4] [Errno 2] No such file or directory: u'/tmp/sentry-files/16951/41536/5cb85cccbfef4064a89ee26bfd951b7b'
Traceback (most recent call last):
  File "/var/www/vhosts/diginetmedia.de/sentry/local/lib/python2.7/site-packages/sentry/lang/javascript/processor.py", line 212, in fetch_release_file
    with releasefile.file.getfile() as fp:
  File "/var/www/vhosts/diginetmedia.de/sentry/local/lib/python2.7/site-packages/sentry/models/file.py", line 160, in getfile
    mode=kwargs.get('mode'),
  File "/var/www/vhosts/diginetmedia.de/sentry/lib/python2.7/site-packages/sentry/models/file.py", line 220, in __init__
    self.open()
  File "/var/www/vhosts/diginetmedia.de/sentry/lib/python2.7/site-packages/sentry/models/file.py", line 242, in open
    self.seek(0)
  File "/var/www/vhosts/diginetmedia.de/sentry/lib/python2.7/site-packages/sentry/models/file.py", line 260, in seek
    self._nextidx()
  File "/var/www/vhosts/diginetmedia.de/sentry/lib/python2.7/site-packages/sentry/models/file.py", line 231, in _nextidx
    self._curfile = self._curidx.blob.getfile()
  File "/var/www/vhosts/diginetmedia.de/sentry/local/lib/python2.7/site-packages/sentry/models/file.py", line 131, in getfile
    return storage.open(self.path)
  File "/var/www/vhosts/diginetmedia.de/sentry/local/lib/python2.7/site-packages/django/core/files/storage.py", line 33, in open
    return self._open(name, mode)
  File "/var/www/vhosts/diginetmedia.de/sentry/local/lib/python2.7/site-packages/django/core/files/storage.py", line 159, in _open
    return File(open(self.path(name), mode))
IOError: [Errno 2] No such file or directory: u'/tmp/sentry-files/16951/41536/5cb85cccbfef4064a89ee26bfd951b7b'

Here my supervisord.conf entries:

[program:sentry-web]
directory=/var/www/vhosts/diginetmedia.de/sentry/
environment=SENTRY_CONF="/home/sentry/.sentry"
command=/var/www/vhosts/diginetmedia.de/sentry/bin/sentry run web
user=sentry
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stopsignal=INT
redirect_stderr=false
stdout_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/web.out.log
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stderr_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/web.err.log
stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)


[program:sentry-worker]
directory=/var/www/vhosts/diginetmedia.de/sentry/
environment=SENTRY_CONF="/home/sentry/.sentry"
command=su -l sentry -c "/var/www/vhosts/diginetmedia.de/sentry/bin/sentry run worker -c 4 -l WARNING -n worker-%(process_num)02d"
process_name=%(program_name)s_%(process_num)02d
numprocs=16
numprocs_start=0
startsecs=1
startretries=3
stopsignal=TERM
stopwaitsecs=10
stopasgroup=false
killasgroup=true
user=root
autostart=true
autorestart=true
redirect_stderr=false
stdout_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/worker1.out.log
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stderr_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/worker1.err.log
stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)

[program:sentry-cron]
directory=/var/www/vhosts/diginetmedia.de/sentry/
environment=SENTRY_CONF="/home/sentry/.sentry"
command=/var/www/vhosts/diginetmedia.de/sentry/bin/sentry run cron -l WARNING
user=sentry
autostart=true
autorestart=true
redirect_stderr=false
stdout_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/cron.out.log
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stderr_logfile=/var/www/vhosts/system/sentry.diginetmedia.de/logs/cron.err.log
stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)

[mattrobenolt]

This error is just as it says, it can't find the files. Did you delete or move them?

[Ninerian]

No, I didn't delete any files. It affects both old and new releases with uploaded sourcefiles.

[dcramer]

@Ninerian my guess is something cleans up your /tmp directory. You need to configure this to be a persistent device.

[Ninerian]

It seems, the standard supervisord.conf cleans the tmp directory on each start. Is there an option to recreate the files?

[dcramer]

@Ninerian how would Sentry recreate them? You'd need to upload them again.

[Ninerian]

Can I delete all the existing file entries without deleting each release?

[dcramer]

You could remove the ReleaseFile and File entries from the db which would wipe everything that's not the core release metadata

[Ninerian]

I will try. Thank you.

[Ninerian]

It worked. Thanks for the help.

[Ninerian]

@dcramer Unfortunately the error reappears. I also looked into the tmp directory, while uploading the files. There is only one file created.
In the database in the table sentry_fileblob are all files with paths listed.

Does Sentry creates the files on upload? Is there any rate limit, how fast the files could be processed? I use a script to upload them. Do I have the possibility to log the process?

[dcramer]

You should not be using /tmp for uploads either way

[Ninerian]

I didn't changed the default option in the sentry.conf.py file. Now I use a tmp directory which doesn't get cleaned. But I don't think, that this is the source for the problems.
Before 8.5 these errors didn't occur.

[dcramer]

Either way these errors are not Sentry related (and feel free to correct me if you do prove that wrong). An IO error suggests whatever process is reading it, can't. Which means that either the permissions are valid, the mount is not functional, or the file simple doesn't exist.

[Ninerian]

When I set the wrong permissions for the directory it was stated correctly in the logs. Now there is no failure, while uploading the source files via your API.
For my understanding, the uploaded file is saved in the given directory renamed to a random value. This value is saved to the database, so Sentry could find it.
Currently after uploading the source files, Sentry generates none or only one file.
So yes, the error occurs, cause the file isn't present. But I want to know, why it's not generated.

Am 16. Juni 2016, 20:06, um 20:06, David Cramer [email protected] schrieb:

Either way these errors are not Sentry related (and feel free to
correct me if you do prove that wrong). An IO error suggests whatever
process is reading it, can't. Which means that either the permissions
are valid, the mount is not functional, or the file simple doesn't
exist.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/getsentry/sentry/issues/3498#issuecomment-226565899

[bndr]

I think the default value of /tmp/sentry-files should be changed to something more permanent, like /var/sentry-files

[mattrobenolt]

@istommao This would be entirely unrelated. Can you open a new ticket for that?

[istommao]

Ok

[grunmin]

Because of directory /tmp may clean files that were not changed in past 30 days, I changed value of filestore.options.location at that file config.yaml from /tmp/sentry-files to /var/sentry-files and restart sentry-(worker, core, cron) and upload sourcemap file again.

filestore.backend: 'filesystem'
filestore.options:
  location: '/var/sentry-files'

and error occurred again

IOError: [Errno 2] No such file or directory: u'/var/sentry-files/17430/21586/ad91545c77f8498389183cd2f6a7cb61'

It seems that web service recognized the change.

I look into directory /var/sentry-files, it's empty, and not file existed in /tmp/sentry-files either(old files had removed automatically). I removed directory /var/sentry-files and uploaded sourcemap again hoping sentry would create it automatically, but it didn't.

So, where is sourcemap files store after I change filestore.options.location? How can I make it work? Thanks

sentry version is 8.18

[kkdev163]

@grunmin if /var/sentry-files is empty, you may to confirm that is sentry has permission to write to this dir. you can try to run ' chmod sentry:sentry /var/sentry-files ', then upload agian.