Sentry-cli: Slow sourcemap upload
I'm having issues with the upload time to sentry.io.
Our sourcemaps are ~30mb, and it takes over 5 minutes to upload them. As we do this as part of CI it slows our build times down.
Unlike https://github.com/getsentry/sentry-cli/issues/17 the upload is consistently slow.
The upload is happening from our corporate network (very fast internet), so it seems unlikely it is related to our connection. (e.g. uploading the same sourcemaps to other cloud providers takes just seconds)
Our build script calls this (after first verifying if a release exists, and creating one if not):
sentry-cli releases -p OUR_PROJECT files OUR_REVISION upload-sourcemaps ./path/to/built/code
There is a ~/.sentryclirc which specifies some defaults (our org, auth token, log level etc)
Can anyone suggest ways to speed this up / further investigate why it's slow?
I've run the upload with log level trace but didn't see anything illuminating in the log.
EDIT: one thing I just noticed is for some files it first requests a DELETE for that file before doing the POST. I'm not sure why it would do this as I'm creating a new release each time before uploading, so there should be no existing files associated with the release.
elwynelwyn
All 20 comments
I can confirm, this is because we do not properly gzip before uploading symbols.
jan-auer
on 23 May 2018
Is there any update with this?
TareqElMasri
on 24 Jun 2018
Any updates? It takes 8 minutes in our case.
ex3ndr
on 2 Jul 2018
Also seem to run into these problems, the time seems to vary, but it does take longer than I would expect. Project is also fairly large.
SMP ⏱
General output time took 3 mins, 55.11 secs
SMP ⏱ Plugins
SentryCliPlugin took 2 mins, 53.42 secs
ebuchmann
on 25 Sep 2018
Same problem here, it triples the time of the build when using the sentry webpack plugin when building.
diegofaria
on 28 Sep 2018
Yes, sourcemap uploading or discovering is extremely slow..
sheerun
on 30 Sep 2018
As noticed in now-closed #405 probably the main reason of slow upload isn't lack of compression, but rather uploading of tenths or hundreds of files separately instead archiving them and uploading as one. I hope this issue will be closed only when both improvements are applied or #405 is reopened.
sheerun
on 3 Oct 2018
@jan-auer any updates? This is causing intermittent flakiness in our CI/CD process. Are there any suggested workarounds? Anything I can do to help debug this more?
oderby
on 11 Oct 2018
Hello,
Any update on this ? Source upload is now 20% of our build time :(
eltonio450
on 27 Nov 2018
Any update here? It's taking a major chunk of deployment time.
balramkhichar
on 18 Dec 2018
We are also loosing a lot of time because of that. Is there any update? As stated in #405 there is a fix planned and on the roadmap. Would you mind sharing a timeline?
mruckli
on 8 Jan 2019
Uploading with webpack-sentry-plugin is much faster because it makes concurrent requests to sentry api. My big project was uploaded in seconds compared to over 30 minutes (!!!) i get with @sentry/[email protected]
rafalmaciejewski
on 23 May 2019
We are actively working on this at the moment. I will try to share more concrete timelines soon.
mitsuhiko
on 23 May 2019
hi @mitsuhiko , any updates? it's taking forever to upload the sourcemaps....
ghost
on 28 Jun 2019
@wppd this should no longer happen with recent versions of sentry-cli against up to date sentry servers. Are you getting slow uploads on recent versions against sentry.io?
mitsuhiko
on 28 Jun 2019
The most recent updates added parallel upload support as well as compression which reduced the upload times from minutes to seconds for our test cases.
mitsuhiko
on 28 Jun 2019
This has been resolved as of sentry-cli v1.44.0. It requires sentry.io, or for on-premise a recent installation of master. The Sentry 10 release will also include the server-side changes.
jan-auer
on 28 Jun 2019
@jan-auer Our team really need this. It takes looooong time to deploy.. I tried sentry 10 today and I failed mostly because of snuba... I am using self-hosted sentry 9.1.2 (non docker). Will you make a 9.x release to include the faster sourcemap feature?? thanks!
ghost
on 8 Sep 2019
@BYK might be able to chime in. AFAIK, there is no more 9.x feature release planned but let's wait for a definitive answer.
jan-auer
on 8 Sep 2019
Sorry, there won't be a 9.2 release and this seems like a large change for an unlikely 9.1.3. That said v10 shouldn't be too far away 😉
BYK
on 8 Sep 2019
Related issues
fungilation
·
4Comments
jef
·
5Comments
ACoolmanBigHealth
·
4Comments
BioPhoton
·
3Comments
Tymolc
·
3Comments
Most helpful comment
@jan-auer any updates? This is causing intermittent flakiness in our CI/CD process. Are there any suggested workarounds? Anything I can do to help debug this more?