Pulsar: OAuth2 url .well-known not parsed correctly

[harissecic]

When trying to make Pulsar work with Keycloak (supporting multiple realms) I get this:

# bin/pulsar-admin --admin-url https://localhost:8080 --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 --auth-params '{"privateKey":"file:///pulsar_test.json","issuerUrl": "https://*****/auth/realms/***/", "audience":"localhost"}' tenants list
#19:29:25.154 [main] ERROR org.apache.pulsar.client.impl.auth.oauth2.FlowBase - Unable to retrieve OAuth 2.0 server metadata
java.io.IOException: Cannot obtain authorization metadata from https://***/.well-known/openid-configuration
        at org.apache.pulsar.client.impl.auth.oauth2.protocol.DefaultMetadataResolver.resolve(DefaultMetadataResolver.java:78) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.client.impl.auth.oauth2.FlowBase.initialize(FlowBase.java:50) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.client.impl.auth.oauth2.ClientCredentialsFlow.initialize(ClientCredentialsFlow.java:66) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2.start(AuthenticationOAuth2.java:96) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.client.admin.PulsarAdmin.<init>(PulsarAdmin.java:159) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:45) ~[org.apache.pulsar-pulsar-client-admin-original-2.6.1.jar:2.6.1]
        at org.apache.pulsar.admin.cli.PulsarAdminTool.lambda$main$2(PulsarAdminTool.java:273) ~[org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
        at org.apache.pulsar.admin.cli.PulsarAdminTool.setupCommands(PulsarAdminTool.java:146) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
        at org.apache.pulsar.admin.cli.PulsarAdminTool.run(PulsarAdminTool.java:226) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
        at org.apache.pulsar.admin.cli.PulsarAdminTool.main(PulsarAdminTool.java:282) [org.apache.pulsar-pulsar-client-tools-2.6.1.jar:2.6.1]
Caused by: java.io.FileNotFoundException: https://zdb-users.azurewebsites.net/.well-known/openid-configuration
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896) ~[?:1.8.0_252]
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) ~[?:1.8.0_252]
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) ~[?:1.8.0_252]
        at org.apache.pulsar.client.impl.auth.oauth2.protocol.DefaultMetadataResolver.resolve(DefaultMetadataResolver.java:72) ~[org.apache.pulsar-pulsar-client-original-2.6.1.jar:2.6.1]
        ... 9 more
class org.apache.pulsar.client.api.PulsarClientException$AuthenticationException: Unable to retrieve OAuth 2.0 server metadata

So it appears that extra path after the base URL is ignored and stripped. It would be really useful to have it respect full path and work with Keycloak. As visible, /auth/realms/{RELAM_NAME} is ignored and it's trying to access base URL by appending /.well-known/openid-configuration instead of appending it at the end of provided issuer

[cuzyoucant]

Hey.

We ran into the same problem as you described above. The better solution would be to have a config somewhere where we can set what gets appended to the issuerURL.

I implemented a small ghetto fix for myself and build pulsar from source until this gets officially implemented.
If this is something you want to do, here is what you need to change:

To "fix" your problem you need to change the line 100 in the DefaultMetadataResolver.java
from this: return new URL(issuerUrl, "/.well-known/openid-configuration");
to this: return new URL(issuerUrl.toString() + "/.well-known/openid-configuration");

Now the URL gets appended correctly and it works as expected.

To also get the client(producer/consumer) to work you need to change the following since they also do not work with keycloak URLs:

1. In the file /pulsar-client-cpp/lib/auth/AuthOauth2.cc in line 187 comment out the line.
   issuerUrl_.append("/oauth/token");

1. Now you need to specify the full token endpoint from keycloak as the issuer_url in your client.
   example: https://example.com/auth/realms/myrealm/protocol/openid-connect/token

The reason you need to do that is that after the first token expires, it tries to get a new one with the settings from the /.well-known configuration and it appends /oauth/token again to the tokenURL which obviously doesnt work.

I know its kind of a bad fix for now but i am to inexperienced to add a complete feature myself. We just want it to work for now and we will not be using it for production anytime soon. Just want to test features and infrastructure stuff and for that it works fine.

Hope it helps!

[harissecic]

@cuzyoucant Thank you for detailed description. I'll consider it. However I'm currently using JWT until this gets fixed and I'm trying to make integration with a certain framework which is better to put off than to test with custom code. First case for my client might benefit from building Pulsar from code. Thanks again.

[sijie]

@jiazhai Can you take a look?

[zymap]

Hi, @harissecic By this error java.io.IOException: Cannot obtain authorization metadata from https://***/.well-known/openid-configuration, it seems the client can not access the configuration URL. Have you tried to using curl to get it directly?

[harissecic]

Hi, @harissecic By this error java.io.IOException: Cannot obtain authorization metadata from https://***/.well-known/openid-configuration, it seems the client can not access the configuration URL. Have you tried to using curl to get it directly?

I'm afraid you didn't read it entirely.

1. https://*/.well-known/openid-configuration is incorrect and that's on Pulsar side 100%
2. https:///auth/realms/* is passed in and is quite clear and pointed out that /auth/realms/realmId* is being stripped inside Pulsar code and /.well-known/openid-configuration is being appended at the base URL when my realms are stripped as I stated in the issue as first sentence after the log.
   > So it appears that extra path after the base URL is ignored and stripped
3. Endpoint I provide is 100% correct and working. My APIs are using that same URL.

Please note that stars are used to hide the actual BASE URL and REALM part is left to be pointed out.

[cuzyoucant]

@zymap I can agree what @harissecic says. The problem is the line 100 in the DefaultMetadataResolver.java
return new URL(issuerUrl, "/.well-known/openid-configuration");

Because the URL constructor does not work as someone would expect. It does not concatenate the two inputs. You can read that up in the java documentation. It strips everything from the issuerURL after the www.xyz.com and adds the "/.well-known/openid-configuration".

So no matter what you pass in as issuerURLwww.xyz.com orwww.xyz.com/auth/realms/myrealm/ (which we want to pass in, or rather what you need to pass in if you use keycloak) it always ends up as www.xyz.com/.well-known/openid-configuration.

Thats why i just changed it the line to return new URL(issuerUrl.toString() + "/.well-known/openid-configuration"); so it actually gets appended.

Hope that helps!

[zymap]

@harissecic @cuzyoucant Sorry for my mistake and thanks for your patience.
Would you like to create a patch for the Java implementation?

For the CPP client, it will be better if we have the same implementation as Java

[EronWright]

Regarding the openid configuration document location, @cuzyoucant is correct that the base path of the issuerUrl should be preserved, to better interoperate with Keycloak and others. The following expression would work robustly:
URI.create(issuerUrl.toExternalForm() + "/.well-known/openid-configuration").normalize().toURL();

Regarding the CPP client issue, I believe the token endpoint should be obtained from the openid configuration document, specifically the token_endpoint value, as documented here:
https://tools.ietf.org/html/rfc8414#section-2

The overarching idea is, from the configured issuerUrl we get the openid configuration document, and from that we get the token endpoint.