Appcenter: 500 error when uploading artifacts

[tasomaniac]

Describe issue
When we are uploading to AppCenter, we are receiving 500 errors. Currently, this is blocking our operations. We would appreciate it if you look into this urgently. The status on appcenter says "all systems operational"

{"code":"internal_server_error","message":"internal error (Support ID: af446ffe-15c9-490b-8047-417de105ae3a)"}

[joerg04]

The status also mentions:

Also, if you are using custom scripts with our swagger API, please adjust your script based off the implementation of App Center CLI.

(https://status.appcenter.ms/incidents/4nbhlm0lxsvt)

Does that mean api.appcenter.ms is not working any longer?

[joerg04]

Just to make sure, I'm not using the CLI, but plain curl requests:
curl -X POST -H 'accept: application/json' -H 'Content-Type: application/json' -H 'X-API-Token: ***' https://api.appcenter.ms/v0.1/apps/foo/bar/release_uploads {"code":"internal_server_error","message":"internal error (Support ID: 80e773ab-9d71-4684-9b57-746253f08584)"}

[joerg04]

Support mentioned a new API path: https://openapi.appcenter.ms/#/distribute/releases_createReleaseUpload
with small differences in the path:

• new /v0.1/apps/{owner_name}/{app_name}/uploads/releases
• old /v0.1/apps/{owner_name}/{app_name}/release_uploads

[tasomaniac]

Hmmm, I don't understand why the old API returns 500 tho? If it is not supported anymore, shouldn't it be 404?

[BastianKusserow]

I'd also be interested if the old API isn't supported anymore and if we should fix this on our side or if this will be fixed on appcenter side?

[TobiasJoergensen]

I have upgraded Appcenter-cli as mentioned in the official post here: App Center CLI Upgrade Required to Continue Using Distribution

Now it seems that the "checking the uploaded file"-step takes quite some time, and I wonder if it's because of the maintenance. Anyone else experiencing this?

EDIT:
Last two distributions took 35 minutes

[Larusso]

We also use custom url calls and I'm trying to move to the correct APIs but uploading the binary is now very complicated when following the sources here: https://github.com/microsoft/appcenter-cli/blob/9670fae935d2d2e90bf9e02c7187854cfcb3e6f6/appcenter-file-upload-client-node/src/ac-fus-uploader.ts#L654

I simply want to upload my binary to appcenter. Where is the api for https://file.appcenter.ms defined. I simply want to upload the binary in one go. I don't want to rollout node to all ci machines + developers just to publish a file. With hockeyapp that was one simple post command ...

[esemle]

We are using a bash script with curl. No issues with the file upload.

Looks like they added an inline step to scan for malware. I think the 35 minute wait was for the scan to complete.

Below are the new steps I've figured out so far:

1. Create new release upload
2. POST: https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases
3. Upload the release
4. POST: upload_domain+token
5. Commit release
6. PATCH https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases/{upload_id}
7. Start polling the follow web service until the upload_status is "readyToBePublished". This step can fail with "malwareDetected". Hence why I think the wait is long.
8. GET https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases/{upload_id}"
9. Get the release_distinct_id from the response and distribute the release
10. PATCH https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/releases/{release_id}

I assume the last step is to distribute the release using the value of the release_distinct_id, but I haven't gotten that far yet. The status has been stuck in "uploadFinished" for the last 20 minutes.

The company I work for uses Bitbucket Pipelines. We'll have to move to a different distribution method, because the wait to distribute will eat into our monthly minute allotment. Kinda sucks..

[Larusso]

@esemle I tried POST: upload_domain+token but I'm on groovy and use apache http and this stupid library doesn't like when I set uri + params. I'll test some more.

[jhass]

@esemle Would you mind sharing the exact curl call you had success with? :)

[lassana]

I wonder why the breaking change was made in the API at all. New API has the same 0.1 version 😕

[TobiasJoergensen]

Looks like they added an inline step to scan for malware. I think the 35 minute wait was for the scan to complete.

Do you know if this has been documented anywhere?

[esemle]

@Larusso and @jhass here is an example for Android. I haven't tried iOS yet.
curl -X POST ${UPLOAD_URL} -F "apk=@${APK_FILE}"

@TobiasJoergensen
I found a reference to malware detection in the API documentation for get release upload status. "malwareDetected" is one of the statuses.

https://openapi.appcenter.ms/#/distribute/releases_getReleaseUploadStatus

upload_status* | The current upload status. [ uploadStarted, uploadFinished, readyToBePublished, malwareDetected, error ]

[Larusso]

@esemle thanks. I'm testing ipa file upload at the moment and upload_domain+token returns 404 for me.

[stefandb1995]

@Larusso and @jhass here is an example for Android. I haven't tried iOS yet.
curl -X POST ${UPLOAD_URL} -F "apk=@${APK_FILE}"

@TobiasJoergensen
I found a reference to malware detection in the API documentation for get release upload status. "malwareDetected" is one of the statuses.

https://openapi.appcenter.ms/#/distribute/releases_getReleaseUploadStatus

upload_status* | The current upload status. [ uploadStarted, uploadFinished, readyToBePublished, malwareDetected, error ]

Just to confirm. Is the this upload_domain that returns from "https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases". And then is it the upload domain that is return with the token added to it?

[jhass]

Yeah uploadDomain + token just 404's for me too, @esemle would you mind sharing how exactly you build up $UPLOAD_URL?

[esemle]

@jhass Sorry. I was wrong about the file upload. It is failing with a 404.

Looked at the source code for the cli. It's way more complicated than I expected. A metadata request is required before uploading the file. The actual file upload is chunked. No thanks...

https://github.com/microsoft/appcenter-cli/blob/9670fae935d2d2e90bf9e02c7187854cfcb3e6f6/appcenter-file-upload-client-node/src/ac-fus-uploader.ts#L502

We are looking into other distribution methods. Good luck!

[jhass]

Yeah, I was looking at this and hoped you somehow figured out a simpler way :D

[PeterBenko]

For what it's worth, we're getting 502s even when using appcenter-cli for the distribute release task, after being stuck on "Checking the uploaded file..." for a longer period of time.
Manual upload via website fails as well.

[Larusso]

@esemle Ok I was down this rabbithole 2h ago and thought no way I'm I porting this. It would be nice if somebody from the team could explain how to use the API without the cli tool. In this state it's nearly impossible to upload an artifact without the cli. In the past the doc pages at least contained some curl commands explaining the process how to publish. Since hockey is gone this seems to be gone as well.

[Larusso]

Ok just found the fastlane plugin from MS. here the code is a little bit less convoluted:
https://github.com/microsoft/fastlane-plugin-appcenter/blob/2b8aef1aa9a75faf09f14428d50d65d9d35b34e7/lib/fastlane/plugin/appcenter/actions/appcenter_upload_action.rb#L129

It was patched recently

[manivacube]

@esemle when i tried(for ios) 2nd step which is curl -F "ipa=@./$build_app" ${uploadUrl}, i am getting below error.
502 - Web server received an invalid response while acting as a gateway or proxy server.

Could you please help me out to proceed further..

[stefandb1995]

@manivacube I had some success with using curl -X POST ${UPLOAD_URL} -F "apk=@${APK_FILE}". Where the upload_url is the one retrieved from the https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases. I did this without the token though. But when running
GET https://api.appcenter.ms/v0.1/apps/{owner_name}/{app_name}/uploads/releases/{upload_id}"
It indicates that there was "an error extracting your app" with no real reason as to the issue. I decided to give the CLI a try and looks to be working fine now.
Hope the new curl code helps!

[manivacube]

@stefandb1995 Tried the same for ios with upload_url as upload_domain from https://api.appcenter.ms/v0.1/apps/%7Bowner_name%7D/%7Bapp_name%7D/uploads/releases api. Still facing the same issues. I think you might have tried to upload android build.. Could you please take a look for ios build upload..

[stefandb1995]

@manivacube I tried running it on our IPA and it also worked (didn't get any errors). Not sure if you need to specific something in the command to get the output the same as you have it? I'm pretty new to using curl.

[Charis1331]

@stefandb1995 I am not sure how you achieved that, since, as @Larusso posted earlier for the fastlane plugin, the process has quite some steps including polling. Also, just to make sure, the upload_url you used is the concatenation of the upload_domain + token from the response of this endpoint, correct?

P.S. I tried uploading an APK file and got the same error as @manivacube, here.

[Larusso]

With a nice nightshift I managed to get it to work in groovy. I copied the steps from the fastlane code as the ruby code is clearer to read. I need to cleanup the code and be hopefully up and running again today.

[stefandb1995]

@Charis1331 No the upload_url is only upload_domain. No token with it. If I applied the token I also get an error. Although as stated I get errors down the line so it's possible that mine is uploading without some needed data and when it's finished uploading and the upload request is closed I then get different errors because of validation. Although this error is very vague so I can't be sure.

[Larusso]

@stefandb1995 The whole flow is very stateful. I initially uploaded not all chunks and the finish call errors. You really need to execute everything in order.

[narciszait]

Microsoft reinstated the deprecated APIs: https://status.appcenter.ms/incidents/7930rkfgtg1y until February 1st 2021.

[tasomaniac]

Somebody needs to teach them the difference between deprecation and removal of APIs.

[King-of-Spades]

Hi all, I'm currently working on trying to get these instructions restored with the current APIs, rather than the deprecated ones. If it's helpful for you folks trying to workaround, here's the historical instructions for the old methods:

https://github.com/MicrosoftDocs/appcenter-docs/blob/7a2bac806ffa51ec4ac76ade58f9b59a36181ab9/docs/distribution/uploading.md#distributing-using-the-apis

Based on my experiments, it seems like step 3 is the main part that needs to be changed, referencing the deprecation APIs. All the rest of the references, unless I had some oversight, are pointing to APIs that aren't deprecated so seems like they should still work as originally described.

I'm still researching and experimenting with this to try to get something that works reliably with the new APIs.

[Larusso]

@King-of-Spades Step 2 and 3 is completely different.
In step two you can't no longer just do a post against the provided URL.
You have to call at least 3 URLs (depending how big your artifact is)

1. call a metadata endpoint with the file name and file size. This returns a chunksize value
2. call the upload url with the binary in chunks. (for me mostly 4MB each. So a typical apk needs 10 - 20 calls)
3. call a finish url to declare that you uploaded all chunks. The return value here tells you if all chunks have been uploaded completely.

Then you go to step 3 which is different then before because the old commit call returned the releaseId.
You have to do the following to get the release id.

1. change the state of the release uplaod resource to finishedUploading
2. polll the upload resource endpoint and wait untile the state changed again to readyToBePublished
3. when this state is returned the same payload also contains the release id

Now you can continue with step 4 like before (distribute your release to groups, set release notes etc).

I ported everything to groovy here. At the moment it's still in a rough state but readable enough.

The calling flow is still in a different class here:
https://github.com/wooga/atlas-appcenter/blob/c71a4a8af5c10de3c3a8cc5115ecaa0e1170be6d/src/main/groovy/wooga/gradle/appcenter/tasks/AppCenterUploadTask.groovy#L201-L223

[grndvl1]

God what a clusterF*ck this is. Depracation does not mean EOL. It means it still works but will be removed in the future. Which also means you let people know in the documentation before you kill it! What a bunch of idiots, I was on the Bluetooth deprecation committee and we took years to remove support for anything. Why is it people do this? The endpoint was fine, why rename it and then just basically do the same thing? Probably they added the malware step, but now this looks like this could mess up with timing issues since it takes way too long. I'm gonna have to recommend a different service after this experience. Too bad these idiots bought out Hockeyapp, they ruined a perfectly good company and product.

[King-of-Spades]

@grndvl1 Sorry for the frustration, and thanks for the feedback. I agree this should've been handled better and sooner by our team.

I have some quick findings to share with any folks who are still struggling with this:

1. The Depreciated Workflow can be used (until Feb. 1st 2021) since the APIs were reactivated. The steps described here are a series of CURL commands. They're a little tricky and the upload is slow, but I've confirmed it works when configured properly.

2. Conceptually @Larusso's solution looks great to me, but I haven't yet confirmed that approach. The key issue is the new process our CLI chunks the upload which is why the multiple uploads are needed in the new API.

3. One of our users created a gist with an open-source bash script. I also haven't tested it yet, but it looks like the cleanest solution I've seen yet: https://gist.github.com/ouchadam/c74fa26c639a50d68bc35ee5749f868c

I'm working on updating this in our docs to try and have a minimized, clean and clear process; however I might not be able to finish by the holidays.

[grndvl1]

@King-of-Spades sad that we have about 20 lines of code is now turned into almost 90 lines of code... and before that with Hockeyapp it was about 5 lines. This does not make me have confidence in this product, and that a user had to write a script instead of someone on your team. If it wasn't that my client uses Appcenter I would quickly abandon this platform.

[Larusso]

@grndvl1 We used HockeyApp to the last minute because of that. When appcenter was introduced I looked over the API and thought: "OK that is all still in Beta they will add a higher level API later to make this easier." Bottom line: The API still refers to itself as 0.2, I see a lot of deprecated methods in the swagger page and removals without a change in the version and the deploy flow become even harder.

[grndvl1]

@Larusso We used Hockeyapp to the end too but we did not hear about when the date of EOL was so we thought oh its deprecated, meaning it will still work. Then Feb came and no uploads. We fixed, then they broke something in the summer time. We fixed and we adjusted the the API calls, not realizing the API calls we used were deprecated, No document to say so, they just do it in Nov, and messed up our endpoint, we fixed. Now this again and I have to find solution by the 1st. Appcenter owes my client a lot of money for wasting my time. I wish I understood fastlane for Android, just can't wrap my head around it, then if I did get it to work then its on Appcenter if they break it.

[King-of-Spades]

@Larusso @grndvl1 I 100% hear your frustration with this. Full disclosure: though I've been working with the Distribution team on this project, my expertise is more specifically in support...when things break or need better docs.

I'm continuing to work on docs, as well as providing feedback & post-mortem to the product group. I'd hoped to write + QA a cleaner process by today, but there's still too many moving pieces. Poorly QA'd docs can add to the confusion. I highlighted user solutions to potentially unblock folks immediately, but that's not a substitute for better docs/better systems.

P.S. "I wish I understood fastlane for Android, just can't wrap my head around it..."
Story of my professional life right there: software is better for everyone when it's as accessible as feasible. :)

[grndvl1]

@King-of-Spades well it is much appreciated. I'll be looking into this come the new year, already have a ticket in place for our client

[King-of-Spades]

Quick update now that I'm back from the holidays:

With only minor tweaks, I was able to get the user-shared script working on my side. The tweaks I made are here. Some may be specific to macOS where I'm working from: https://gist.github.com/ouchadam/c74fa26c639a50d68bc35ee5749f868c#gistcomment-3583629

I'm working on applying the script, with some more tweaks, to some of our samples: https://github.com/microsoft/appcenter-Xamarin.Forms-Gallery-Demo. That way the samples can be used both to see the process in operation, and provide support QA.

[King-of-Spades]

New process: https://docs.microsoft.com/en-us/appcenter/distribution/uploading#distributing-using-the-api
Sample implementation: https://github.com/microsoft/appcenter-Xamarin.UITest-Demo/blob/main/ac-distribute.sh

The sample makes heavy use of custom environment variables: https://docs.microsoft.com/en-us/appcenter/build/custom/variables/#variables-declared-in-build-configuration.

I used custom env. variables so that the script is more broadly compatible across scenarios, as well as to illustrate the different things to configure. One thing that still doesn't _quite_ work using a custom env variable is setting the distribution group, though that's a non-blocking issue because App Center still allows the upload even with an invalid or missing distribution group.

I've tested extensively in order to make sure I know how it works and to try to make it as organized as possible...but there's probably parts that are confusing / could be made clearer, so I'm watching for early feedback to make further improvements.