Che: VS Code marketplace rate limit error

Created on 6 Mar 2019  路  24Comments  路  Source: eclipse/che

Description

VS Code marketplace impose rate limiting on access to the extensions binaries.
PR https://github.com/eclipse/che-plugin-broker/pull/35 makes broker do retries and send pretty error to Che but still doesn't prevent blocking of workspace start because of rate limiting.
This often happens when user sets many VS Code extensions in his workspace configuration or several users share public IP which gets blocked when access rate is too high.
It seems VS Code itself bypasses this problem, but I haven't found how. We need to investigate how exactly it is done.
Another possible improvement is not to download extensions when workspace plugins list doesn't change, so provisioning could be cached.

Related issues:
https://github.com/eclipse/che/issues/12831
https://github.com/eclipse/che/issues/12762

Reproduction Steps

OS and version:

Diagnostics:

kinbug lifecyclstale
Source
picture garagatyi

Most helpful comment

The latest vsix have been "released" on github:

picture fbricon on 18 Apr 2019
👍2

All 24 comments

@l0rd it is problem we discussed today.
@ibuziuk @skabashnyuk @benoitf @evidolob FYI

garagatyi picture garagatyi on 6 Mar 2019

Full exception as reported by @benoitf:

{
"$id":"1",
"innerException":null,
"message":"Anonymous usage from IP address ***** has exceeded our rate limits and is being blocked. Anonymous access from this IP address will be unblocked within the next 1 minute. You can also consider logging in to get unblocked. Logged in users have higher limits. Learn more: https://go.microsoft.com/fwlink/?LinkId=823950.",
"typeName":"Microsoft.TeamFoundation.Framework.Server.RequestBlockedException,Microsoft.TeamFoundation.Framework.Server",
"typeKey":"RequestBlockedException",
"errorCode":0,
"eventId":3000
}

cc @slemeur

l0rd picture l0rd on 11 Mar 2019

The possible solution for this problem can range from a lazy caching mechanism (once we request one vs code extensions we don't request it anymore) to a vs code extension mirror (we pre-fetch all the extensions).

In the short term, to mitigate the problem, we could look at these solutions:

  • embed the vs code extensions that are currently in the che-plugin-registry in their sidecar image
  • embed the extensions in the che-plugin-registry docker image and change the meta.yml to point to it.
  • look at some already existing alternative URLs to download the extension (i.e. the artifact produced by the CI/CD of the extensions)
l0rd picture l0rd on 18 Mar 2019

Additional details for the assignee:
We have 10 VS Code plugins in the registry that are useful (I do not treat sonarlint extension as such).
4 of them are published by Red Hat.
We should check if any of plugins (especially non-redhat owned ones) are publishing github releases or host archive binaries anywhere else.

garagatyi picture garagatyi on 29 Mar 2019
👍2

I also hit this issue having just 1 or 2 plugins (I reproduced several times with a mix of Camel, Java and XML) in my stack. this is quite annoying and it was not easy to find the root cause.

What about an even shorter-term solution which is reporting a better error message to user, inviting her/him to retry later pointing that there is a rate limit reached?

apupier picture apupier on 15 Apr 2019

The latest vsix have been "released" on github:

fbricon picture fbricon on 18 Apr 2019
👍2

@skabashnyuk @garagatyi @l0rd guys, I'm not very fluent with plugin versioning update process. Could you please clarify the policy? e.g. both xml and yaml plugins used in our registry are obsolete. Should we create dedicated folders with versioning? So, far I created https://github.com/eclipse/che-plugin-registry/pull/113 just for url switch

ibuziuk picture ibuziuk on 18 Apr 2019

@ibuziuk problematic part is that we do not have proper ways for a user to use newer version of plugins in a workspace. So, we usually update just URL to the binary of the plugin.
But since VS Code plugins are properly versioned this seems a bad practice.
We should definitely invest in https://github.com/eclipse/che/issues/12937

At this point of time I'm +1 to create new version for yaml and xml plugins.

garagatyi picture garagatyi on 19 Apr 2019

At this point of time I'm +1 to create new version for yaml and xml plugins.

@garagatyi @l0rd so are you ok if UD would display 'n' versions of plugin atm once we add latest versions to the registry?

ibuziuk picture ibuziuk on 19 Apr 2019

@ibuziuk yes that's ok. That's a UD problem and we are removing the list of plugins and editors from the create workspace panel in UD #12896 and moving it inside che-theia #12905.

l0rd picture l0rd on 19 Apr 2019
👍1

@ibuziuk since you have done the first PR can you add the newest version of JDT LS as well? Hence adding new meta.yaml.

l0rd picture l0rd on 19 Apr 2019

@l0rd sure thing will do
@fbricon @gorkem could you please give a hand with https://github.com/redhat-developer/vscode-java/issues/892

ibuziuk picture ibuziuk on 19 Apr 2019

Some updates before updating plugin registry on prod:

ibuziuk picture ibuziuk on 23 Apr 2019

vscode-yaml 0.3.0 and vscode-redhat.java recent builds have now been attached to github releases

fbricon picture fbricon on 23 Apr 2019
👍1

Is usage of Github release artefact a temporary hack?
We are loosing the metrics having them.
it is forcing to add an extra step to configure jobs to push the artefact too.

apupier picture apupier on 24 Apr 2019

@apupier yes, this might be temporary if we find a better way to avoid rate limit issues, so far there was no straight forward solution found

ibuziuk picture ibuziuk on 24 Apr 2019
👍1

@ibuziuk @apupier this is a temporary solution to mitigate a blocking problem. After Che 7 GA is released we should start working on a lazy caching mechanism or a VS Code Marketplace mirroring as described https://github.com/eclipse/che/issues/12840#issuecomment-473870618.

l0rd picture l0rd on 24 Apr 2019
👍1

@l0rd atm there are only 2 plugins that still use marketplace (node-debug / node-debug2), so I sent PR for updating prod (also includes naming convention changes) - https://github.com/openshiftio/saas-openshiftio/pull/1255

Once merged I believe we can stop working on this issue at least till GA since the error should not appear with the updated registry. It might make sense to add some section in README to discourage referencing marketplace directly for the time being WDYT?

ibuziuk picture ibuziuk on 25 Apr 2019

@ibuziuk thanks makes sense

l0rd picture l0rd on 25 Apr 2019

Sometimes hit this problem during installation of java debugger - https://github.com/microsoft/vscode-java-debug/issues/583

ibuziuk picture ibuziuk on 7 May 2019

@ibuziuk
would these links work ? AFAIK there is no limit rate on the assets
https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/java/0.44.0/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage

benoitf picture benoitf on 28 May 2019

@benoitf I haven't tried but I believe any link with valid vsix would work

ibuziuk picture ibuziuk on 4 Jun 2019

I mean, these links are generics, it's easy to have links for any vs code extension and it's like the final redirect of marketplace download so there is no limit rate

benoitf picture benoitf on 5 Jun 2019

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

che-bot picture che-bot on 5 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

SkorikSergey picture SkorikSergey  路  3Comments
sleshchenko picture sleshchenko  路  3Comments
dmytro-ndp picture dmytro-ndp  路  3Comments
skabashnyuk picture skabashnyuk  路  3Comments
luckymore0520 picture luckymore0520  路  3Comments