Sentry-cli: Add batch upload option for 'release' command to send multiple source files at once

Created on 1 Apr 2020 · 17Comments · Source: getsentry/sentry-cli

This was discussed offline with @jan-auer.

Currently we have to execute this for thousand of files:

sentry-cli releases files "914" upload filename.lua

We would like to be able to pass a folder name, or a zip file, or a 'command file', or a list of files from stdin, ..., or whatever you guys think is best to this command.

In our case, if the input is a directory name, we would want that directory name to be stripped.

feature

Source

bsergean

👍1

All 17 comments

hi, any update on #715, when we can expect it to be released?

dvshsahu on 25 Jun 2020

You have to ask @jan-auer :>

kamilogorek on 26 Jun 2020

👍1

hi @jan-auer,

regarding #715, when we can expect it to be released?

dvshsahu on 26 Jun 2020

@dvshsahu will do a release tomorrow, stay tuned.

kamilogorek on 30 Jun 2020

👍1

Thnx a ton

dvshsahu on 30 Jun 2020

@dvshsahu 1.55.0 has been released. Let me know if it works when you have a chance to test it out. Thanks!

kamilogorek on 1 Jul 2020

hi @kamilogorek, thanks for the feature but I am not able to use it, can you please provide some documentation on how to use it, it would be of great help?

dvshsahu on 22 Jul 2020

@dvshsahu there are no changes to the API, it works in the exact same way as regular single-file upload, with the exception that you can now point it to the directory instead of files, so instead of:

sentry-cli releases files some-release upload src/file1.rs
sentry-cli releases files some-release upload src/file2.rs
sentry-cli releases files some-release upload src/file3.rs
sentry-cli releases files some-release upload src/file4.rs
sentry-cli releases files some-release upload src/file5.rs

you can just call this instead:

sentry-cli releases files some-release upload src

kamilogorek on 22 Jul 2020

👍1

I have a flat structure (files are not inside folder, but at the top level), and I'm using . (dot) for the folder which seems to work nicely so far.

On Jul 22, 2020, at 7:56 AM, Kamil Ogórek notifications@github.com wrote:

@dvshsahu https://github.com/dvshsahu there are no changes to the API, it works in the exact same way as regular single-file upload, with the exception that you can now point it to the directory instead of files, so instead of:

sentry-cli releases files some-release upload src/file1.rs
sentry-cli releases files some-release upload src/file2.rs
sentry-cli releases files some-release upload src/file3.rs
sentry-cli releases files some-release upload src/file4.rs
sentry-cli releases files some-release upload src/file5.rs
you can just call this instead:

sentry-cli releases files some-release upload src
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/getsentry/sentry-cli/issues/704#issuecomment-662502254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2O6UM4IMTZBVUV5OF7XX3R4342VANCNFSM4LYDUYLA.

bsergean on 22 Jul 2020

@kamilogorek & @bsergean thanks, it works smoothly now

dvshsahu on 23 Jul 2020

🎉1

Hi there ... I actually have some problems using the batch upload way.

Uploading files one by one works though. Not sure what's going on yet but just a heads up.

bsergean on 15 Sep 2020

sentry-cli --version
sentry-cli 1.56.1

If I use the batch upload (using . as the folder) I don't see my code contexts for my crashes.
If I upload files individually (which takes about one hour as we have 4000 of them) my code context do work.

bsergean on 15 Sep 2020

@bsergean could you provide some debug logs for both cases? (SENTRY_LOG_LEVEL=debug or --log-level debug)
These upload methods should be fully interchangeable 🤔

kamilogorek on 15 Sep 2020

Batch upload

09/15/2020 09:08:46 AM - > Found 4000 release files
09/15/2020 09:08:46 AM -   DEBUG   2020-09-15 09:08:46.768346 -07:00 request GET https://sentry.io/api/0/organizations/mz-inc/chunk-upload/
09/15/2020 09:08:46 AM -   DEBUG   2020-09-15 09:08:46.768377 -07:00 using token authentication
09/15/2020 09:08:46 AM -   DEBUG   2020-09-15 09:08:46.768390 -07:00 retry number 0, max retries: 0
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116387 -07:00 > GET /api/0/organizations/mz-inc/chunk-upload/ HTTP/1.1
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116419 -07:00 > Host: sentry.io
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116429 -07:00 > Accept: */*
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116437 -07:00 > Connection: TE
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116445 -07:00 > TE: gzip
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.116453 -07:00 > User-Agent: sentry-cli/1.56.1
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.117112 -07:00 > Authorization: Bearer XXX-EDITED-XXX***
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228409 -07:00 < HTTP/1.1 200 OK
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228468 -07:00 < Server: nginx
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228492 -07:00 < Date: Tue, 15 Sep 2020 16:08:47 GMT
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228510 -07:00 < Content-Type: application/json
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228581 -07:00 < Content-Length: 281
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228625 -07:00 < Connection: keep-alive
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228670 -07:00 < Access-Control-Allow-Headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228684 -07:00 < X-Content-Type-Options: nosniff
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228695 -07:00 < Content-Language: en
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228766 -07:00 < Access-Control-Expose-Headers: X-Sentry-Error, Retry-After
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228789 -07:00 < Vary: Accept-Language, Cookie
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228806 -07:00 < X-XSS-Protection: 1; mode=block
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228823 -07:00 < Allow: GET, POST, HEAD, OPTIONS
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228841 -07:00 < Access-Control-Allow-Origin: *
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228859 -07:00 < Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228877 -07:00 < X-Frame-Options: deny
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228905 -07:00 < X-Served-By: web-d6992418
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228926 -07:00 < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.228988 -07:00 response status: 200
09/15/2020 09:08:47 AM -   DEBUG   2020-09-15 09:08:47.229006 -07:00 body: {"maxFileSize":2147483648,"hashAlgorithm":"sha1","concurrency":8,"chunkSize":8388608,"maxRequestSize":33554432,"url":"https://sentry.io/api/0/organizations/mz-inc/chunk-upload/","chunksPerRequest":64,"accept":["debug_files","release_files","pdbs","sources"],"compression":["gzip"]}
09/15/2020 09:08:48 AM - > Bundled 4000 files for upload
09/15/2020 09:08:49 AM - > Uploaded release files to Sentry
09/15/2020 09:08:50 AM - > File upload complete
09/15/2020 09:08:50 AM -   DEBUG   2020-09-15 09:08:50.435345 -07:00 skipping update nagger because session is not attended

Individual file upload

Those are failing with a 409 error because I uploaded them twice. Note that I used --finalize to finalize the release.

09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.248188 -07:00 request POST https://sentry.io/api/0/organizations/mz-inc/releases/<release_number_EDITED>/files/
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.313314 -07:00 > Accept: */*
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.244370 -07:00 request POST https://sentry.io/api/0/organizations/mz-inc/releases/<release_number_EDITED>/files/
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.248215 -07:00 using token authentication
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.313321 -07:00 > Connection: TE
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.244390 -07:00 using token authentication
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.248229 -07:00 sending form data
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.313328 -07:00 > TE: gzip
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.244401 -07:00 sending form data
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.248244 -07:00 retry number 0, max retries: 5
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.313334 -07:00 > User-Agent: sentry-cli/1.56.1
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.244455 -07:00 retry number 0, max retries: 5
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.314043 -07:00 > Authorization: Bearer 0737b472***
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.314069 -07:00 > Content-Length: 4168
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.314080 -07:00 > Content-Type: multipart/form-data; boundary=------------------------5e3578b2576be39a
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358207 -07:00 < HTTP/1.1 409 Conflict
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358224 -07:00 < Server: nginx
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358230 -07:00 < Date: Tue, 15 Sep 2020 16:06:21 GMT
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358237 -07:00 < Content-Type: application/json
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358250 -07:00 < Content-Length: 75
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358256 -07:00 < Connection: keep-alive
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358267 -07:00 < Access-Control-Allow-Headers: X-Sentry-Auth, X-Requested-With, Origin, Accept, Content-Type, Authentication, Authorization, Content-Encoding
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358273 -07:00 < X-Content-Type-Options: nosniff
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358279 -07:00 < Content-Language: en
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358284 -07:00 < Access-Control-Expose-Headers: X-Sentry-Error, Retry-After
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358289 -07:00 < Vary: Accept-Language, Cookie
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358294 -07:00 < X-XSS-Protection: 1; mode=block
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358299 -07:00 < Allow: GET, POST, HEAD, OPTIONS
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358304 -07:00 < Access-Control-Allow-Origin: *
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358309 -07:00 < Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358316 -07:00 < X-Frame-Options: deny
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358324 -07:00 < X-Served-By: web-17b74f97
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358332 -07:00 < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358363 -07:00 response status: 409
09/15/2020 09:06:21 AM - error: File already present!
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358791 -07:00 client close; no transport to shut down  (from sentry)
09/15/2020 09:06:21 AM -   DEBUG   2020-09-15 09:06:21.358806 -07:00 skipping update nagger because session is not attended

bsergean on 15 Sep 2020

https://github.com/getsentry/sentry-cli/pull/715/files#diff-cb855f2407956db7a6542289a0415ed6R803

I wonder if the problem is because I have no url prefix (like no ./ and so the files are not matched). Have you tried with . as the root folder (like picking all the files in the current directory ?).

bsergean on 15 Sep 2020

@bsergean can you confirm that the problem itself is that files are uploaded but not picked up? Or the other way around?
You should be able to distinguish it by verifying that files were uploaded in one of this section in the settings (depending on what you are uploading):

kamilogorek on 16 Sep 2020

Summary: the upload appear to work fine, no error from the cli.

But when I look at crashes, the backtraces are missing code context.

On Sep 16, 2020, at 3:54 AM, Kamil Ogórek notifications@github.com wrote:

@bsergean https://github.com/bsergean can you confirm that the problem itself is that files are uploaded but not picked up? Or the other way around?
You should be able to distinguish it by verifying that files were uploaded in one of this section in the settings (depending on what you are uploading):

https://user-images.githubusercontent.com/1523305/93328249-c6b86480-f81b-11ea-8d3b-29096917a077.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/getsentry/sentry-cli/issues/704#issuecomment-693329588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2O6UK2EC4GBBGRX5KLFIDSGCKPPANCNFSM4LYDUYLA.