Harbor: Harbor 2.0.2 CSRF Token Invalid

[boxboatmatt]

Updated Harbor from 1.10.4 to 2.0.2 - using the harbor helm chart. I have Okta OIDC as my login provider. Anything that is a POST in the UI results in a CSRF token invalid message. I have SSL termination on an AWS ALB. I see this in both Safari and Chrome. No issues prior to updating to 2.0.2

[reasonerjt]

Hi,
In 2.x the CSRF mechanism has been reworked. This may due to some settings in the ALB makes it drop some headers/cookies.

Does the CSRF error only happens to regular user that was authenticated via OIDC?

Does admin user also see this problem?

Could you please also capture the failed request via your browser? and the log of harbor-core?

[boxboatmatt]

Hi,
In 2.x the CSRF mechanism has been reworked. This may due to some settings in the ALB makes it drop some headers/cookies.

Does the CSRF error only happens to regular user that was authenticated via OIDC?

Does admin user also see this problem?

I can't login as the admin local user anymore on the web ui - I get a 403 (I can docker login still)

{
    "errors": [
        {
            "code": "FORBIDDEN",
            "message": "CSRF token invalid"
        }
    ]
}

Could you please also capture the failed request via your browser? and the log of harbor-core?

I don't see the POST in the harbor-core logs.
Here is the request captured in the browser (I removed the jwt and changed the domain name to example.com)

URL: https://harbor.example.com/api/v2.0/projects
Status: 403
Source: Network
Address: <ip>:443
Initiator: 
harbor.example.com:1:780


Request
:method: POST
:scheme: https
:authority: harbor.example.com
:path: /api/v2.0/projects
Content-Type: application/json
Pragma: no-cache
Accept: application/json
Accept-Language: en-us
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
Host: harbor.example.com
Origin: https://harbor.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Safari/605.1.15
Referer: https://harbor.example.com/harbor/projects
Content-Length: 70
Connection: keep-alive
Cookie: __csrf=tLJZRYSJ8TswdS5AZoZCJcl8Q0XWeq0qffdDZQqooZ2eb8kgu6UPyTxSC5s0aEKcWgOLr4uP2KYRHMR7fxyiaA==; _gorilla_csrf=MTU5NzA4MTQwM3xJa3QwTWxGYVZEaHpMM1pKVFVwNVdHSlZkVFJCZFZwT0wzbFBjR1E1V0ZkTllrOTFTRWh1VnpCQkwxVTlJZ289fNwwqBCOvViZRZcSf2HO_DkUftrgndAtuU-UqaQ1Uurz; sid=7cd520c95e6f5baac941d4d709600465; okta-jwt=<okta-jwt-redacted>; intercom-session-cfy7ppqw=; ajs_anonymous_id=%22df5b2259-4641-454a-9c0d-6d4605d2bc41%22

Response
:status: 403
Set-Cookie: __csrf=fqGWVJ/aVU3OMB8Sj7+ZDTtksuu5xIEI5AomDg9IYflUfAYxoParv8IXOsndUZm0qBt6AeQx9ISI4aEQevxiDA==; Path=/; Secure; SameSite=Strict
Content-Type: application/json; charset=utf-8
Content-Length: 65
Date: Mon, 10 Aug 2020 17:48:36 GMT
x-request-id: 0378d79b00dab72239b1a931604b1d73
Server: nginx/1.14.0

Request Data
MIME Type: application/json
Request Data: {"project_name":"foo","metadata":{"public":"true"},"storage_limit":-1}

[reasonerjt]

@boxboatmatt
I don't think it is relevant to OIDC.

For the request to be successful the request has to carry the header

X-Harbor-CSRF-Token: eN+pJhySPcY/PuWz2TrgYvkrH4lqLb+/od9/yUC378mVFSr3iRoK2m/cCAdI85zspbOBd0XIGr2MWKiF0gn4cg==

And UI code will access the cookie __csrf to read the value and attach this header. Note that there is SameSite attribute in the cookie.
b/c you are using an extra LB, it's possible the SamteSite=strict makes the UI code unable to access the cookie and attach the header.

How you configured the ALB to proxy the request to nginx? I have not used ALB, but is it possible the way it proxy the request makes it look like cross-site request?
If you access the nginx directly, does it work?

[boxboatmatt]

@reasonerjt The ALB is not configured to drop any header fields. And the header is not being set by the UI the majority of the time... However troubleshooting today with the developer console open to watch network traffic, I have seen the ui get in a state where it is setting the X-Harbor-CSRF-Token in the request and then it does succeed but I have no idea what is causing the UI to bounce in and out of setting that request header. When it happens the UI will work until you refresh the page or navigate to another section of the UI.

[dolv]

The same issue here. We upgraded from 1.10.1 to 2.0.2 and harbor does not work with EnableXSRF = true in app.conf. We need to disable this feature or wait until this issue is fixed. We have Keycloak as OIDC provider.

[dolv]

Admin user is not able to even log-in with EnableXSRF = true

[dolv]

More findings: if we send the X-Harbor-CSRF-Token to harbor host using curl - it appears in logs on the ingress-controller. But when using UI the X-Harbor-CSRF-Token is absent...

[reasonerjt]

@boxboatmatt

UI sets whatever in cookie __csrf to header X-Harbor-CSRF-Token
please see my comment:
https://github.com/goharbor/harbor/issues/12676#issuecomment-671904558
Is it possible it's due to the "SameSite" attribute?

@dolv
The backend only checks the CSRF token for request coming from UI.
Please refer to comments above and capture requests as was did in:
https://github.com/goharbor/harbor/issues/12676#issuecomment-671501995

[boxboatmatt]

@reasonerjt harbor is installed with EXT_ENDPOINT matching the DNS hostname.. The ingress is configured to use-forwarded-headers.

Also this only happens when OIDC is enabled. On the same cluster, I just installed an identical 2nd copy of harbor at harbor2.domain and with no OIDC provider configured I can login and perform any POST/PUT... In the requests you can see X-Harbor-CSRF-Token being set.

[zdhamasha]

Running into same issue, any updates ??

[SlashTeen]

encounting the same issue, any updates?

[SlashTeen]

Hi,
In 2.x the CSRF mechanism has been reworked. This may due to some settings in the ALB makes it drop some headers/cookies.

Does the CSRF error only happens to regular user that was authenticated via OIDC?

Does admin user also see this problem?

Could you please also capture the failed request via your browser? and the log of harbor-core?

encounting the same issue, any updates?

[savealive]

I've got the same issue. Harbor 2.0.2 on Kubernetes with nginx-ingress (aws classic LB, tcp backend proto, ssl termination on LB).
Getting CSRF token invalid when attempting to login with OIDC for the first time and set username.
Local admin user login is successful.

With enabled EnableXSRF = true in app.conf admin user is also unable to login with message
2020-09-06T02:11:36Z [DEBUG] [/server/error/error.go:58]: {"errors":[{"code":"FORBIDDEN","message":"CSRF token invalid"}]}

[ianfergu]

Same issue... any updates?

[enderv]

Was running into this as well using harbor helm charts. We finally got it working when we trimmed our xsrf key in the values. It used to be 40 characters now there is a limit of 32. If the key is over the limit core will generate its own and cause this.

[boxboatmatt]

The xsrf key is already set to 32 characters as part of the helm chart update and shortening it has no affect on my issue

[boxboatmatt]

However - I did just update to harbor 2.1.0 and this problem now seems to be resolved in this release

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.