Onedrive-api-docs: Problems uploading into uploadUrl: 401 Unauthorized

Created on 25 Apr 2017  路  45Comments  路  Source: OneDrive/onedrive-api-docs

This started happening sometime between April 18-22. We can obtain the upload URL for a new file (so we create the upload session), but when uploading the first chunk, it fails with a 401 Unauthorized error. This only happens when using an ADMINISTRATOR access token. When using a single-user access token, this does not happen. So this applies to OneDrive for Business (with administrator token) and not single user.

Request (to get the upload URL):

> POST /personal/{email}/_api/v2.0/drive/root:/robots2.txt:/createUploadSession HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.29-1+deb.sury.org~trusty+1
Content-Type: application/json
Host: {tenant}-my.sharepoint.com
Content-Length: 56

Response:

< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< Expires: Mon, 10 Apr 2017 21:30:46 GMT
< Last-Modified: Tue, 25 Apr 2017 21:30:46 GMT
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-SharePointHealthScore: 0
< X-SP-SERVERSTATE: ReadOnly=0
< ODATA-VERSION: 4.0
< SPClientServiceRequestDuration: 1100
< SPRequestDuration: 1235
< X-AspNet-Version: 4.0.30319
< SPRequestGuid: 2f93eb9d-e033-4000-e1b2-d0dc23d8504c
< request-id: 2f93eb9d-e033-4000-e1b2-d0dc23d8504c
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6413
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: 0CB89A006F6D4DDDB7067CD67FBDAD30 Ref B: CH1EDGE0521 Ref C: Tue Apr 25 14:30:47 2017 PST
< Date: Tue, 25 Apr 2017 21:30:47 GMT
<

We got the uploadUrl, we now try to upload:

> PUT /personal/{email}/_api/v2.0/drive/items/01VKXHYLN6Y2GOVW7725BZO354PWSELRRZ/uploadSession?guid='guid'&path='~tmpE3_robots2.txt'&overwrite=True&rename=False&access_token={access_token}
HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.29-1+deb.sury.org~trusty+1
Host: {tenant}-my.sharepoint.com
Content-Range: bytes 0-22379/22380
Content-Length: 22380

Response:

< HTTP/1.1 401 Unauthorized
< Content-Length: 0
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< WWW-Authenticate: Bearer realm="96f92206-b32e-4ed0-8020-9e0984ce71c4",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize"
< SPRequestGuid: 2f93eb9d-007d-4000-e6b6-6c8efd18b843
< request-id: 2f93eb9d-007d-4000-e6b6-6c8efd18b843
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< x-ms-suspended-features: features=""
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6413
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: 4A6D4FE3B7494A589915F680E32C91DD Ref B: CH1EDGE0521 Ref C: Tue Apr 25 14:30:48 2017 PST
< Date: Tue, 25 Apr 2017 21:30:47 GMT
* HTTP error before end of send, stop sending
<

This is happening consistently with different admin tokens and for different tenants and admin accounts. Has anything changed that might only be affecting ODB with admin tokens?

resolved
Source
picture dluces
👍2

Most helpful comment

The patch should be, for all intents and purposes, fully deployed. Can anyone that was having this issue ( @dluces @amagliul @ad2012 ) please try again and confirm we've resolved the issue?

picture ificator on 14 Aug 2017
🎉2

All 45 comments

I'm observing the same behavior using app-tokens for ODB. Started happening in the last week or so.

desmondyeung picture desmondyeung on 26 Apr 2017
👍1

Great report. If it's helpful, here another example request/response:

Create Upload Request:

POST https://devman1-my.sharepoint.com/personal/admin_devman1_onmicrosoft_com/_api/v2.0/drives/b!TRL-vrd7NUuJC6RejF5gcNs0MNAAvJ5BrstNszo2dl0D5LTbVMGXQrq0G6FOHjDx/items/01CSENZRQCRTBA5PFUGBFKOSAEEXKPZNZS:/forms.less:/createUploadSession
Authorization: Bearer <access token>


Create Upload Response:

200 OK

{"@odata.context":"https://devman1-my.sharepoint.com/personal/admin_devman1_onmicrosoft_com/_api/v2.0/$metadata#oneDrive.uploadSession","expirationDateTime":"2017-04-25T20:45:35.5730268Z","nextExpectedRanges":["0-"],"uploadUrl":"https://devman1-my.sharepoint.com/personal/admin_devman1_onmicrosoft_com/_api/v2.0/drives/b!TRL-vrd7NUuJC6RejF5gcNs0MNAAvJ5BrstNszo2dl0D5LTbVMGXQrq0G6FOHjDx/items/01CSENZRQCRTBA5PFUGBFKOSAEEXKPZNZS/uploadSession?guid='abd3b7d9-0ff5-4274-b80b-219b1fc2e289'&path='~tmp49_forms.less'&overwrite=True&rename=False&access_token=<upload access token>&prooftoken=<proof token>"}

PUT Request:

PUT https://devman1-my.sharepoint.com/personal/admin_devman1_onmicrosoft_com/_api/v2.0/drives/b!TRL-vrd7NUuJC6RejF5gcNs0MNAAvJ5BrstNszo2dl0D5LTbVMGXQrq0G6FOHjDx/items/01CSENZRQCRTBA5PFUGBFKOSAEEXKPZNZS/uploadSession?guid='abd3b7d9-0ff5-4274-b80b-219b1fc2e289'&path='~tmp49_forms.less'&overwrite=True&rename=False&access_token=<upload access token>&prooftoken=<proof token>
Content-Range: bytes 0-15835/15836
Authorization: Bearer <access token> 

<15836 byte body>

PUT Response:

401 Unauthorized
Server: Microsoft-IIS/8.5, P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
WWW-Authenticate: Bearer realm="bae0dc89-7ffc-430a-ba94-f32f319d0fd7",client_id=00000003-0000-0ff1-ce00-000000000000,trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize"
SPRequestGuid: bd8feb9d-60aa-3000-edf3-db51a8ffeb31, request-id: bd8feb9d-60aa-3000-edf3-db51a8ffeb31
Strict-Transport-Security: max-age=31536000
X-FRAME-OPTIONS: SAMEORIGIN, x-ms-suspended-features: features=""
X-Powered-By: ASP.NET, MicrosoftSharePointTeamServices: 16.0.0.6413
X-Content-Type-Options: nosniff, X-MS-InvokeApp: 1; RequireReadOnly
X-MSEdge-Ref: Ref A: A1ACF91841B44824AC947F3C8421F05A Ref B: BLUEDGE0712 Ref C: Tue Apr 25 13:30:36 2017 PST
Date: Tue, 25 Apr 2017 20:30:36 GMT

<empty body>
ANorwell picture ANorwell on 26 Apr 2017
👍1

Update:

A few months ago I remember a similar issue happening. MS acknowledged it and fixed it. However, they provided a workaround: to remove the Authorization header when uploading into the upload URL. This workaround does NOT work this time, as doing that just returns 404 Not Found instead.

dluces picture dluces on 26 Apr 2017
👍1

Update:

We've started seeing this with normal ODB single user tokens. It happens for some tenants and works fine for others.

Create upload session request:

> POST /_api/v2.0/drive/items/01RHVPNSSNKCSKZYL5U5FJFD26KH5T6RSN:/Quite%20Long%20Filename.pdf:/createUploadSession HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.29-1+deb.sury.org~trusty+1
Content-Type: application/json
Host: {tenant}-my.sharepoint.com
Content-Length: 56

Response:

< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Transfer-Encoding: chunked
< Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
< Expires: Tue, 11 Apr 2017 21:06:57 GMT
< Last-Modified: Wed, 26 Apr 2017 21:06:57 GMT
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< X-SharePointHealthScore: 0
< X-SP-SERVERSTATE: ReadOnly=0
< ODATA-VERSION: 4.0
< SPClientServiceRequestDuration: 867
< SPRequestDuration: 981
< X-AspNet-Version: 4.0.30319
< SPRequestGuid: 38e4eb9d-f013-4000-b335-dc5283ac2bbb
< request-id: 38e4eb9d-f013-4000-b335-dc5283ac2bbb
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6413
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< Date: Wed, 26 Apr 2017 21:06:58 GMT

Upload:

> PUT /personal/{user_site}/_api/v2.0/drive/items/01RHVPNSSNKCSKZYL5U5FJFD26KH5T6RSN/uploadSession?guid='{guid}'&path='~tmp22_Quite%20Long%Filename.pdf'&overwrite=True&rename=False&access_token={access_token}
HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.29-1+deb.sury.org~trusty+1
Host: {tenant}-my.sharepoint.com
Content-Range: bytes 0-1/2
Content-Length: 2

Response:

< HTTP/1.1 401 Unauthorized
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< SPRequestGuid: 38e4eb9d-6053-4000-b335-d02ced5de48a
< request-id: 38e4eb9d-6053-4000-b335-d02ced5de48a
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< x-ms-suspended-features: features=""
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6413
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< WWW-Authenticate: Bearer realm="2f25c27f-0a84-4960-886a-685cbabea3ab",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize"
< Date: Wed, 26 Apr 2017 21:06:58 GMT
< Connection: close
< Content-Length: 0
dluces picture dluces on 26 Apr 2017

For everyone that's currently having issues can you please let us know the total length of the URL (including query parameters etc). If you have working scenarios please include those as well (but please call out which ones work and which ones don't.

ificator picture ificator on 27 Apr 2017

@ificator
Tenant 1:
Not working URL length (including protocol): 2296 (single user token)

Tenant 2 (to upload the SAME file):
Not working URL length (including protocol): 2268 (admin token)
Working URL length (including protocol): 2051 (single user token)

dluces picture dluces on 27 Apr 2017

Thanks @dluces, do you mind cracking the access_token for Tenant 1 single user and Tenant 2 single user and pasting them as a comment? Feel free to remove the audience / user id if you like

ificator picture ificator on 27 Apr 2017

@ificator For sure.

Tenant 1:

{
  "aud": "https://{another_tenant}-my.sharepoint.com/",
  "iss": "https://sts.windows.net/2f25c27f-0a84-4960-886a-685cbabea3ab/",
  "iat": 1493247445,
  "nbf": 1493247445,
  "exp": 1493251345,
  "acr": "1",
  "aio": "ASQA2/8DAAAAtAZ0Z/uexNZSwNFt+j+Uvvn/XWKmBmVMRCwX030GdPQ=",
  "amr": [
    "pwd"
  ],
  "app_displayname": "Mover OneDrive Single User",
  "appid": "89b4a753-031a-4c47-95aa-168ce8418c8a",
  "appidacr": "1",
  "deviceid": "869b6526-d823-403a-b069-2c4f8afaf4dd",
  "family_name": "{family_name}",
  "given_name": "{given_name}",
  "ipaddr": "{ip_address}",
  "name": "{full_name}",
  "oid": "21c9cc72-11d6-46d7-93e1-8b6fc992a37d",
  "onprem_sid": "S-1-5-21-379033014-1428274042-417481102-9913",
  "platf": "3",
  "puid": "10033FFF8066E87D",
  "scp": "MyFiles.Write",
  "signin_state": [
    "kmsi"
  ],
  "sub": "GvDtOMfOr_9_R8FHrdWFF892rwu3UmkEBd9EFyr0Tyg",
  "tid": "2f25c27f-0a84-4960-886a-685cbabea3ab",
  "unique_name": "{an_email}",
  "upn": "{an_email}",
  "ver": "1.0"
}

Tenant 2:

{
  "aud": "https://{tenant}-my.sharepoint.com/",
  "iss": "https://sts.windows.net/96f92206-b32e-4ed0-8020-9e0984ce71c4/",
  "iat": 1493247298,
  "nbf": 1493247298,
  "exp": 1493251198,
  "acr": "1",
  "aio": "Y2ZgYHiots7Y4U59hvGfzc7y0w2U1SKUkoRSIzze/ThTKhXr0gMA",
  "amr": [
    "pwd"
  ],
  "app_displayname": "Mover OneDrive Single User",
  "appid": "89b4a753-031a-4c47-95aa-168ce8418c8a",
  "appidacr": "1",
  "family_name": "{family_name}",
  "given_name": "{given_name}",
  "ipaddr": "{ip_address}",
  "name": "{full_name}",
  "oid": "0e24c068-11c3-4a03-9e2c-dcd8c4286865",
  "platf": "5",
  "puid": "10033FFF98B34687",
  "scp": "MyFiles.Write",
  "sub": "vy_zcGOtvqVVEE_wGRe8heyhzLNp0fJnzaDf06xIQc4",
  "tid": "96f92206-b32e-4ed0-8020-9e0984ce71c4",
  "unique_name": "{my_email}",
  "upn": "{my_email}",
  "ver": "1.0"
}
dluces picture dluces on 27 Apr 2017

Hi, I've the same problem on ODB, this is my url (URL length : 2233)

(https://[TENANT]-my.sharepoint.com/personal/tecnoconference_tecnoconference1_onmicrosoft_com/_api/v2.0/drive/items/01YS2LNEH4C4TWLS23OVA2T7KMPRJJ76V7/uploadSession?guid=[MY_GUID]&path='~tmp73_daricordare.pptx'&overwrite=True&rename=False&access_token=[my_access_token])

frenchdonah picture frenchdonah on 27 Apr 2017

Thanks @dluces it does seem like the token has some extra stuff in it, but not a great deal.

Here's something we can try - if it works it's a work around and if it doesn't it'll disapprove one of our theories :).

The upload session URL will look something like this:
https://{tenant}-my.sharepoint.com/personal/{targetspidentifier}/_api/v2.0/drive/items/{parentitemid}/uploadSession?guid='{sessionid}'&path='{filename}'&overwrite=True&rename=False&access_token={accesstoken}&prooftoken={prooftoken}

If you remove the access_token and prooftoken query parameters, and add their values as HTTP headers so you request looks more like this let's see if that works:

PUT /personal/{targetspidentifier}/_api/v2.0/drive/items/{parentitemid}/uploadSession?guid='{sessionid}'&path='{filename}'&overwrite=True&rename=False
Authorization: bearer {accesstoken}
X-PROOF_TOKEN: {prooftoken}

ificator picture ificator on 27 Apr 2017

Hi @ificator, I've just tried that without any luck:

Tenant 1, single user token:
Request (URL length: 368):

> PUT /personal/{user1}/_api/v2.0/drive/items/01RHVPNSSNKCSKZYL5U5FJFD26KH5T6RSN/uploadSession?guid='69803d79-641b-4044-a485-9c121042d34a'&path='~tmpDE_{Quite%20Long%20Filename}.pdf'&overwrite=True&rename=False HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.30-7+deb.sury.org~trusty+1
Host: {tenant1}-my.sharepoint.com
Content-Range: bytes 0-1/2
X-PROOF_TOKEN: {prooftoken}
Content-Length: 2

* upload completely sent off: 2 out of 2 bytes

Response:

< HTTP/1.1 401 Unauthorized
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< SPRequestGuid: ec25ec9d-00ba-4000-1cb2-56913b48f3ab
< request-id: ec25ec9d-00ba-4000-1cb2-56913b48f3ab
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< SPRequestDuration: 10
< SPIisLatency: 4
< x-ms-suspended-features: features=""
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6420
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< WWW-Authenticate: Bearer realm="2f25c27f-0a84-4960-886a-685cbabea3ab",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize"
< Date: Thu, 27 Apr 2017 16:15:15 GMT
< Content-Length: 102
<

Tenant 2, admin token:

Request (URL length: 220):

> PUT /personal/{user}/_api/v2.0/drive/items/01VKXHYLN6Y2GOVW7725BZO354PWSELRRZ/uploadSession?guid='c2862c23-a324-4216-aad7-8e292944d211'&path='robots2.txt'&overwrite=True&rename=False
HTTP/1.1
Accept: application/json
Authorization: Bearer {access_token}
User-Agent: GuzzleHttp/6.1.1 curl/7.35.0 PHP/5.6.30-7+deb.sury.org~trusty+1
Host: {tenant}-my.sharepoint.com
Content-Range: bytes 0-1/2
X-PROOF_TOKEN: {prooftoken}
Content-Length: 2

* upload completely sent off: 2 out of 2 bytes

Response:

< HTTP/1.1 401 Unauthorized
< Content-Length: 102
* Server Microsoft-IIS/8.5 is not blacklisted
< Server: Microsoft-IIS/8.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< WWW-Authenticate: Bearer realm="96f92206-b32e-4ed0-8020-9e0984ce71c4",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize"
< SPRequestGuid: ad25ec9d-e0e6-4000-f2a0-ebe83c996e77
< request-id: ad25ec9d-e0e6-4000-f2a0-ebe83c996e77
< Strict-Transport-Security: max-age=31536000
< X-FRAME-OPTIONS: SAMEORIGIN
< SPRequestDuration: 167
< SPIisLatency: 4
< x-ms-suspended-features: features=""
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 16.0.0.6420
< X-Content-Type-Options: nosniff
< X-MS-InvokeApp: 1; RequireReadOnly
< X-MSEdge-Ref: Ref A: C081288E89C44596A5A735D444B43884 Ref B: CO1EDGE0107 Ref C: Thu Apr 27 09:10:58 2017 PST
< Date: Thu, 27 Apr 2017 16:10:57 GMT
<
dluces picture dluces on 27 Apr 2017

Hi @dluces can you confirm the value of access_token you put in the Authorization header for the above PUT requests was the one from the query parameter of the upload URL we returned and NOT the access token you used to create the upload session?

ificator picture ificator on 27 Apr 2017

@ificator Unfortunately, I did not use the access token from the upload URL. When using the access token from the upload URL query string, both cases DO work. So I guess that is a workaround. I'll test some more.

dluces picture dluces on 27 Apr 2017

Thanks for confirming - that does seem to suggest that URL length is factoring in somehow.

ificator picture ificator on 27 Apr 2017

@ificator I don't know if this is related to the same issue, but if you noticed the URL parameters, we specified that it should overwrite existing files if there were any with the same name. However, now that the URL works, if I upload the same file twice, I get:

409 Conflict

{"error":{"code":"nameAlreadyExists","message":"The specified item name already exists."}}

Might this be related or should I open a new ticket?

dluces picture dluces on 27 Apr 2017

The workaround does not appear to work with app-tokens. When using the access_token from the session url as a header I get a different 401 with the following body:
```
{"error":"invalid_request","error_description":"This access token is not valid on this endpoint."}
````

desmondyeung picture desmondyeung on 27 Apr 2017

@desmondyeung did you also include the X-PROOF_TOKEN header?

@dluces shouldn't be related to this thread so go ahead and create a new issue and we'll investigate

ificator picture ificator on 27 Apr 2017
👍1

@ificator Yes I did include the X-PROOF_TOKEN header.

This is what was in the access_token from the session url if it helps

{
   "aud":"00000003-0000-0ff1-ce00-000000000000/[TENANT]-my.sharepoint.com@e7e96b68-1f05-4fef-a2ff-cba361f2fad7",
   "iss":"00000003-0000-0ff1-ce00-000000000000",
   "nbf":1493310896,
   "exp":1493397296,
   "endpointurl":"KA+cjmjDheW+qu1L8qEY+ROnIX0s1L9qIZelkm1WUO8=",
   "endpointurlLength":"242",
   "isloopback":"True",
   "nameid":"8b41ecbb-feaa-424a-bf3b-ce0f538a615a@e7e96b68-1f05-4fef-a2ff-cba361f2fad7",
   "ver":"apponlyprooftoken",
   "roles":"[\"User.ReadWrite.All\",\"TermStore.ReadWrite.All\",\"Sites.Manage.All\",\"Sites.ReadWrite.All\",\"Sites.FullControl.All\"]",
   "usePersistentCookie":null
}
desmondyeung picture desmondyeung on 27 Apr 2017

@desmondyeung hmm, did you touch any other part of the URL or just removed the access_token and prooftoken query parameters?

ificator picture ificator on 27 Apr 2017

@ificator only removed the access_token and prooftoken query parameters

desmondyeung picture desmondyeung on 27 Apr 2017

Thanks for confirming @desmondyeung. If possible could you paste in the 401 response with all of the headers? I'll see if it gives an indication of what's going on.

ificator picture ificator on 27 Apr 2017

@ificator

Status:

401 Unauthorized

Headers: 

content-length: 98,
server: Microsoft-IIS/8.5,
p3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI",
www-authenticate: Bearer realm="e7e96b68-1f05-4fef-a2ff-cba361f2fad7",client_id="00000003-0000-0ff1-ce00-000000000000",trusted_issuers="00000001-0000-0000-c000-000000000000@*,https://sts.windows.net/*/,00000003-0000-0ff1-ce00-000000000000@90140122-8516-11e1-8eff-49304924019b",authorization_uri="https://login.windows.net/common/oauth2/authorize",
x-ms-diagnostics: 3000001;reason="This access token is not valid on this endpoint.";category="invalid_request",
sprequestguid: 5c2bec9d-c057-3000-f4c1-e089a07a18a4,
request-id: 5c2bec9d-c057-3000-f4c1-e089a07a18a4,
strict-transport-security: max-age=31536000,
x-frame-options: SAMEORIGIN,
sprequestduration: 234,
spiislatency: 4,
x-powered-by: ASP.NET,
microsoftsharepointteamservices: 16.0.0.6413,
x-content-type-options: nosniff,
x-ms-invokeapp: 1; RequireReadOnly,
x-msedge-ref: Ref A: BD0200256AE64B0D99D9A300B30F0B5F Ref B: BLUEDGE0711 Ref C: Thu Apr 27 10:50:15 2017 PST,
date: Thu, 27 Apr 2017 17:50:14 GMT,
connection: close

Body:

{"error":"invalid_request","error_description":"This access token is not valid on this endpoint."}
desmondyeung picture desmondyeung on 27 Apr 2017

@desmondyeung the system is detecting a change to the URL and so is rejecting it. Normally this kind of thing is a pain to root cause, but in your case I got lucky.

The generated URL appears to have been:
https://tenant-my.sharepoint.com/personal/user/_api/v2.0/drive/items/01WEKZHRF6Y2GOVW7725BZO354PWSELRRZ/uploadSession?guid='1acb75f9-f9d9-47f8-a295-f0d26a942867'&path='~tempfilename'&overwrite=True&rename=False

While the URL hit to upload was:
https://tenant-my.sharepoint.com/personal/user/_api/v2.0/drive/items/01WEKZHRF6Y2GOVW7725BZO354PWSELRRZ/uploadSession?guid='1acb75f9-f9d9-47f8-a295-f0d26a942867'&overwrite=True&path='~tempfilename'&rename=False

Note the overwrite parameter has been switched around... almost like the parameters have been sorted alphabetically? Is it possible that was done on your side?

ificator picture ificator on 28 Apr 2017

@ificator found out the part of the issue was on my end. I'm able to successfully upload with an app-token with the Authorization and X-PROOF_TOKEN headers.

The http client library I was using was unexpectedly percent-encoding some of the characters in the session url which would cause the endpoint mismatch.

I noticed what was happening once I successfully uploaded using curl.

Thanks for all the help/responsiveness.

desmondyeung picture desmondyeung on 28 Apr 2017

I just wanted to update you guys on a couple of things.

  1. There was a recent change in upload urls for app-only scenarios that increased their lengths so we're rolling that back over the weekend.
  2. We're doing work to decrease the size of the urls for app+user scenarios.

Be aware that the work done for item 2 above will change the structure of the URLs, and so if you're utilizing the work around we discussed earlier you'll want to make sure it's resilient (e.g. only do it when you see both the access_token and prooftoken query string parameters)

ificator picture ificator on 29 Apr 2017
👍1

@ificator
Does the workaround still works? I got 404 again

verybada picture verybada on 23 May 2017

@ificator
Server will return 404 not found once filename greater than 62. Is it the same issue?

verybada picture verybada on 24 May 2017

It could be - anything that increases the overall size of the query string has the potential to push it over the limit. Large file names could certainly do this.

We're still working through the set of dependencies that we need to push through to get this working across all scenarios. In the meantime, the work around I outlined in my previous comment is really the only option :(.

ificator picture ificator on 24 May 2017

I had same problem (404 not found error).
I read the answers above and created the sample code.

public static string GetQueryStringByKey(string url, string key)
{
    var uri = new Uri(url);
    var newQueryString = HttpUtility.ParseQueryString(uri.Query);
    return newQueryString.Get(key);
}

public static string RemoveQueryStringByKey(string url, string key, string value)
{
    var uri = new Uri(url);

    string target = "&" + key + "=" + value;
    string query = uri.Query;
    int position = query.IndexOf(target);
    if(position > 0)
    {
        query = query.Remove(position, target.Length);
    }

    string pagePathWithoutQueryString = uri.GetLeftPart(UriPartial.Path);

    if(query.StartsWith("?"))
        return String.Format("{0}{1}", pagePathWithoutQueryString, query);
    else
        return String.Format("{0}?{1}", pagePathWithoutQueryString, query);
}

public bool UploadChunk(string session_url, byte[] data, long position, int length)
{
    string url = session_uri;
    string prof_token = GetQueryStringByKey(url, "prooftoken");
    url = RemoveQueryStringByKey(url, "prooftoken", prof_token);

    var Request = (HttpWebRequest)WebRequest.Create(new Uri(url));
    Request.Method = "PUT";
    Request.Headers.Add("X-PROOF_TOKEN: " + prof_token);
    ...
}

public void UploadCancel(string session_url)
{
    string url =session_uri;
    string prof_token = GetQueryStringByKey(url, "prooftoken");
    url = RemoveQueryStringByKey(url, "prooftoken", prof_token);

    var Request = (HttpWebRequest)WebRequest.Create(new Uri(url));
    Request.Method = "DELETE";
    Request.Headers.Add("X-PROOF_TOKEN: " + prof_token);
    ...
}
lanopk picture lanopk on 17 Jul 2017

Hello @ificator

I might be seeing this issue happening again since Friday at about 4pm MT. The upload URL does not contain access_token or prooftoken anymore. Instead, it now has a tempauth parameter that seems to be a valid JWT. However, the URL is about 1178 characters long with this parameter, so I'm guessing it's pushing the URL over the limit.

I've tried removing this parameter and putting it in the Authorization header without luck. I've tried removing this parameter and using the regular access token (the one used to create the upload session) and it did not work either. I've tried just removing the parameter and of course that didn't work.

Also, this doesn't break our tests for regular filenames but only for filenames with UTF-8 characters in them, which become way longer when URL-encoding them. This can potentially push the limit for the URL.

This is how the payload of the JWT in the tempauth parameter looks:

{
  "aud": "00000003-0000-0ff1-ce00-000000000000/{tenant}-my.sharepoint.com@{some-guid}",
  "iss": "00000003-0000-0ff1-ce00-000000000000",
  "nbf": "1502209530",
  "exp": "1502295930",
  "endpointurl": "zxoxXci0hS33Z8PG2d2cS8y93I9jC1pLY9m1T98HHKY=",
  "endpointurlLength": "237",
  "isloopback": "True",
  "cid": "NzI0ZDBkOWUtMDBmOS00MDAwLWUwNzctMTI3YWM0NWFhOGUy",
  "ver": "hashedprooftoken",
  "siteid": "MmQ1NWNlN2UtZTE2My00ZWRhLWIyYWEtMjhmNTIzZTdkMWE1",
  "nameid": "0#.f|membership|test@{tenant}",
  "nii": "microsoft.sharepoint",
  "isuser": "true",
  "cachekey": "0h.f|membership|[email protected]",
  "tt": "0",
  "usePersistentCookie": null
}

Any ideas? Workaround?

dluces picture dluces on 8 Aug 2017
👍1

@ificator Interesting details:

  • When creating the upload session with an admin token for a specific user, the upload URL works fine even for files with UTF-8 characters in their name
  • The issue does not happen with OneDrive Personal or OneDrive for Business with an admin token, just for OneDrive for Business with a single user token
  • The issue does not seem to happen for OneDrive for Business (single user) for long filenames but it happens for filenames with non-ASCII characters
dluces picture dluces on 8 Aug 2017

We are seeing the same issue that @dluces is describing above where filenames with non-ASCII characters will generate a 401 exception when attempting an upload.

For example, trying to upload a file with the name a虉.jpg in the root of a user's drive will produce a 401 response with error message: "This access token is not valid on this endpoint."

amagliul picture amagliul on 9 Aug 2017
👍1

Thanks for the info @dluces and @amagliul - the new tempauth parameter is our first step in trying to shrink the auth tokens down to a more manageable size. I'll play around with Unicode scenarios today and see if I can reproduce the issues you're describing.

ificator picture ificator on 9 Aug 2017

Hmm, I tried with a虉.jpg and it "worked on my machine" :). Do either of you have a SPRequestGuid and Date response header values you can provide to help debug?

ificator picture ificator on 9 Aug 2017

@ificator For sure, here's some response headers:

Content-Length: 98
P3P: CP=\"ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI\"
WWW-Authenticate: {...}
x-ms-diagnostics: 3000001;reason=\"This access token is not valid on this endpoint.\";category=\"invalid_request\"
SPRequestGuid: 47ab0d9e-a0a7-4000-17b9-7ec038c22d19
request-id: 47ab0d9e-a0a7-4000-17b9-7ec038c22d19
Strict-Transport-Security: max-age=31536000
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 28
SPIisLatency: 2
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 16.0.0.6802
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
X-MSEdge-Ref: Ref A: DE364F3A41EF4A0A80AD2D0CF8E591E6 Ref B: CO1EDGE0517 Ref C: 2017-08-09T19:45:18Z
Date: Wed, 09 Aug 2017 19:45:17 GMT

Another:

Content-Length: 98
P3P: CP=\"ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI\"
WWW-Authenticate: {...}
x-ms-diagnostics: 3000001;reason=\"This access token is not valid on this endpoint.\";category=\"invalid_request\"
SPRequestGuid: 47ab0d9e-e09a-4000-c326-681d352aebef
request-id: 47ab0d9e-e09a-4000-c326-681d352aebef
Strict-Transport-Security: max-age=31536000
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 20
SPIisLatency: 1
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 16.0.0.6802
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
X-MSEdge-Ref: Ref A: 6CF8EA9549D845BABBB585F43A094E69 Ref B: CO1EDGE0517 Ref C: 2017-08-09T19:45:18Z
Date: Wed, 09 Aug 2017 19:45:17 GMT
dluces picture dluces on 9 Aug 2017

Hi @dluces, are you willing to share an example upload session URL that I can take a look at? You can make it invalid by removing the last segment of the tempauth JWT so the hash check fails, but I'll need the rest intact. If you have concerns about exposing tenant / client data let me know and we can figure out another way of getting me the information.

ificator picture ificator on 10 Aug 2017

@ificator this upload URL is for a test tenant so it shouldn't matter. I've only removed the signature:

https://moverio-my.sharepoint.com/personal/test_mover_io/_api/v2.0/drive/items/01TXLQNTIK2MXG7SDLQ5D3ZV473RI5YWHT/uploadSession?guid=%27a8d4037c-f304-4781-b7d5-30a64d62a89a%27&path=%27%7etmp84_dzi%u0119kuj%u0119+wszystkim+z+po%u0142udnia+-+%u9633%u5149.txt.un%7e%27&overwrite=True&rename=False&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvbW92ZXJpby1teS5zaGFyZXBvaW50LmNvbUA5NmY5MjIwNi1iMzJlLTRlZDAtODAyMC05ZTA5ODRjZTcxYzQiLCJpc3MiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAiLCJuYmYiOiIxNTAyMzY5MjIwIiwiZXhwIjoiMTUwMjQ1NTYyMCIsImVuZHBvaW50dXJsIjoiRWljT3huakZjdzVTcHExaHBRb2NDUjExQUR4Qk1yVmhURWlhaEdOeUxoUT0iLCJlbmRwb2ludHVybExlbmd0aCI6IjI1NiIsImlzbG9vcGJhY2siOiJUcnVlIiwiY2lkIjoiWW1SbE5UQmtPV1V0TmpCak15MDBNREF3TFRFM1lqa3ROMk5pWVRVeU0yTTFNV1E1IiwidmVyIjoiaGFzaGVkcHJvb2Z0b2tlbiIsInNpdGVpZCI6Ik1tUTFOV05sTjJVdFpURTJNeTAwWldSaExXSXlZV0V0TWpobU5USXpaVGRrTVdFMSIsIm5hbWVpZCI6IjAjLmZ8bWVtYmVyc2hpcHx0ZXN0QG1vdmVyLmlvIiwibmlpIjoibWljcm9zb2Z0LnNoYXJlcG9pbnQiLCJpc3VzZXIiOiJ0cnVlIiwiY2FjaGVrZXkiOiIwaC5mfG1lbWJlcnNoaXB8MTAwMzNmZmY5YWIyMWUyMkBsaXZlLmNvbSIsInR0IjoiMCIsInVzZVBlcnNpc3RlbnRDb29raWUiOm51bGx9.

dluces picture dluces on 10 Aug 2017

This was very useful, thanks @dluces!

Here's what I've found:

  1. Our code that adds the new query string parameter is different for the new token than it was for the old, and it's doing it in a slightly dangerous way. It shouldn't actually cause any functional issue, but it is resulting us returning a URL that looks a little different then expected (e.g. there's %uXXXX encoding for some characters instead of UTF8 percent encoding).
  2. The URL that was provided to us did NOT match the one you provided above. Specifically the path component was modified to be %27%7etmp84_dzi%25u0119kuj%25u0119+wszystkim+z+po%25u0142udnia+-+%25u9633%25u5149.txt.un%7e%27 - note the encoded %25 in front of, for example, u0142 instead of just a raw %

Can you take a look at your code and see if you can find something that explains how this limited double encoding would occur? I'll make a change on our side to improve the code that appends the query string.

ificator picture ificator on 10 Aug 2017

I'm running in exactly the same problem here.

$FileName = "Bonit盲ts眉bersicht.xlsx"
$FilePath = "$env:tempdel\$FileName" # The file path you prepared to upload
$OneDriveFilePath = "Test/$FileName" # The target path in your onedrive
[...]

    $EncOneDriveTargetPath=$OneDriveFilePath 
    #$EncOneDriveTargetPath = [System.Web.HttpUtility]::UrlEncode($OneDriveTargetPath) #doesn't help
    #$EncOneDriveTargetPath = [uri]::EscapeUriString($OneDriveTargetPath) #doesn't help

    Write-Host "   Starting upload process..."
    $uploadSession = Invoke-RestMethod -Headers $Header -Method Post -Uri "$ApiRootUrl/drive/root:/${EncOneDriveTargetPath}:/createUploadSession"

[...]
        # call upload api
        $uploadUri = $uploadSession.uploadUrl #[System.Uri]::EscapeUriString($uploadSession.uploadUrl) 
        #-# Write-Host "UploadUri=", $uploadUri -ForegroundColor DarkGray
        $uploadResult = Invoke-RestMethod -Headers $Header -Method Put -Uri $uploadUri -Body $filePartlyBytes

Sample of $uploadUri:

https://tenant-my.sharepoint.com/personal/useruri/_api/v2.0/drive/items/01I4ZPDBMWKPH7KUEWBRDILTOZJNYMC2FN/uploadSess
ion?guid=%279398917d-dca4-4978-b829-1fccd49f0b50%27&path=%27%7etmp7A_Bonit%u00e4ts%u00fcbersicht.xlsx%27&overwrite=True&rename=False&tempauth=eyJ0eX
AiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvYXJ2aWNvci1teS5zaGFyZXBvaW50LmNvbUBjNThkZmYzNC1lOTlkLTQxNz
QtYThhMS05ZTk2MTMwZGM3N2QiLCJpc3MiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAiLCJuYmYiOiIxNTAyMzgyNzY3IiwiZXhwIjoiMTUwMjQ2OTE2NyIsImVuZHBvaW
50dXJsIjoiUHQ2TktlRWJON3pWOGdSYXc1VGtGODVjS21paW5Vc0ovZ2xab2VQRGxvWT0iLCJlbmRwb2ludHVybExlbmd0aCI6IjI0MCIsImlzbG9vcGJhY2siOiJUcnVlIiwiY2lkIjoiT1dRME
9UbGpaakV0Tm1ObVl5MDBOamczTFdFM04yVXROalJqTkRJelpqVTROVEF3IiwidmVyIjoiaGFzaGVkcHJvb2Z0b2tlbiIsInNpdGVpZCI6IlpEVmpPVFE1TXpjdFpUSTNPUzAwTlRSaExXRmhaV0
l0WmpNM04yRXpZMkZqTVRWaSIsIm5hbWVpZCI6IjAjLmZ8bWVtYmVyc2hpcHxhcm5kdEBhcnZpY29yLm5ldCIsIm5paSI6Im1pY3Jvc29mdC5zaGFyZXBvaW50IiwiaXN1c2VyIjoidHJ1ZSIsIm
NhY2hla2V5IjoiMGguZnxtZW1iZXJzaGlwfDEwMDM3ZmZlOGJmMTEzZThAbGl2ZS5jb20iLCJ0dCI6IjAiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.QU1SVmNRbGc5OEhTczEwOFVubFJ
HNEc4ZXBPVkxQZ21aSWpiQ3Ivaitidz0

In O4B online a file with size=$null will be visible afterwards, named "~tmp7A_Bonit盲ts眉bersicht"

ad2012 picture ad2012 on 10 Aug 2017

It looks like .NETs Uri class will auto encode %uXXXX to %25uXXXX so this may be pretty hard to work around on the clients side (at least for .NET). I'm going to push a code change now and see if we can fast track deployment.

ificator picture ificator on 10 Aug 2017
👍1

@ificator the same problem when using Guzzle 6. There's no way to bypass the query auto encoding. Not even with reflection as it will re-do it right before sending it out. I was considering looking into a middleware right before the request is executed but it seems troublesome. If you could push a change for this, it'd be awesome :)

dluces picture dluces on 10 Aug 2017

The patch should be, for all intents and purposes, fully deployed. Can anyone that was having this issue ( @dluces @amagliul @ad2012 ) please try again and confirm we've resolved the issue?

ificator picture ificator on 14 Aug 2017
🎉2

Tried it successfully 馃憤
Thank you very much! Great work, fast fix 馃

This is the received uploadUri:

https://tenant-my.sharepoint.com/personal/useruri/_api/v2.0/drive/items/01I4ZPDBMWKPH7KUEWBRDILTOZJNYMC2FN/uploadSession?guid='cf753484-891d-4f31-8d
ac-61ec7cd85773'&path='Bonit%C3%A4ts%C3%BCbersicht.xlsx'&overwrite=True&rename=False&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2Uw
MC0wMDAwMDAwMDAwMDAvYXJ2aWNvci1teS5zaGFyZXBvaW50LmNvbUBjNThkZmYzNC1lOTlkLTQxNzQtYThhMS05ZTk2MTMwZGM3N2QiLCJpc3MiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAiLCJuYm
YiOiIxNTAyNzI0ODE5IiwiZXhwIjoiMTUwMjgxMTIxOSIsImVuZHBvaW50dXJsIjoic2FVaS83RWh2U3YrcnhNY3VMZWFUdmhrWnd3bktnSUlDUzd2L0VJa2dGWT0iLCJlbmRwb2ludHVybExlbmd0aCI6IjIzMyIsImlzbG9v
cGJhY2siOiJUcnVlIiwiY2lkIjoiWldFellUazFObUV0TXpJMU9DMDBOV1kyTFRsall6Y3RPVGhpWkdSbFlUUTBPVGM0IiwidmVyIjoiaGFzaGVkcHJvb2Z0b2tlbiIsInNpdGVpZCI6IlpEVmpPVFE1TXpjdFpUSTNPUzAwTl
RSaExXRmhaV0l0WmpNM04yRXpZMkZqTVRWaSIsIm5hbWVpZCI6IjAjLmZ8bWVtYmVyc2hpcHxhcm5kdEBhcnZpY29yLm5ldCIsIm5paSI6Im1pY3Jvc29mdC5zaGFyZXBvaW50IiwiaXN1c2VyIjoidHJ1ZSIsImNhY2hla2V5
IjoiMGguZnxtZW1iZXJzaGlwfDEwMDM3ZmZlOGJmMTEzZThAbGl2ZS5jb20iLCJ0dCI6IjAiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.M0hYc2ZGZEovbG1vOEluQ3ZBYlRBaUhkS01KSkR3d0ZRUGMzMWVsdTlEdz0
ad2012 picture ad2012 on 14 Aug 2017
👍1

@ificator Same here, it seems to be working now. We have about 10 tests for filenames with UTF-8 characters that were failing before and are passing now. Thank you!

dluces picture dluces on 14 Aug 2017
👍1

I can also confirm that this has resolved the issue for us. Great work @ificator! I really appreciate the swift resolution.

amagliul picture amagliul on 14 Aug 2017
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

upxact picture upxact  路  9Comments
StewartRyanSony picture StewartRyanSony  路  9Comments
ANorwell picture ANorwell  路  7Comments
rkulow picture rkulow  路  7Comments
geoffrey-mantel picture geoffrey-mantel  路  6Comments