Quarkus: Problem with environment variables when using multiple datasources

[felipewind]

Describe the bug
I belive there's a problem with environment variables when using multiple datasources.
Looks like Quarkus isn't recognizing the environment variables that have datasource namespaces, like quarkus.datasource."users".jdbc.url .
Or perhaps I'm not passing the variables correctly.

Expected behavior
Quarkus should read the environment variables with the datasource namespace.

Actual behavior
Quarkus isn't reading reading the environment variables with the datasource namespace.

To Reproduce

1. Build the project with mvn clean package
2. Export the environment variables
3. Execute the runnable jar.
4. Quarkus says that the variables weren't defined.

Configuration

quarkus.datasource."users".db-kind=postgresql
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.hibernate-orm."users".datasource=users
quarkus.hibernate-orm."users".packages=com.helesto.models.users
quarkus.hibernate-orm."users".database.generation = none
quarkus.hibernate-orm."users".sql-load-script=no-file
quarkus.hibernate-orm."users".log.sql=false

quarkus.datasource."orders".db-kind=mariadb
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}
quarkus.hibernate-orm."orders".datasource=orders
quarkus.hibernate-orm."orders".packages=com.helesto.models.orders
quarkus.hibernate-orm."orders".database.generation=none
quarkus.hibernate-orm."orders".sql-load-script=no-file
quarkus.hibernate-orm."orders".log.sql=false

Environment:

• uname -a:
  Linux machine 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• java -version:
  openjdk version "11.0.8" 2020-07-14
  OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
  OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

• Quarkus version or git rev:
  1.9.0.Final

• mvnw --version:
  Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
  Maven home: /home/helesto/.m2/wrapper/dists/apache-maven-3.6.3-bin/1iopthnavndlasol9gbrbg6bf2/apache-maven-3.6.3
  Java version: 11.0.8, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64
  Default locale: en_US, platform encoding: UTF-8
  OS name: "linux", version: "5.4.0-52-generic", arch: "amd64", family: "unix"

Test with my project - happy end

It's possible to reproduce this problem with a project I created to test the use of multiple databases:

• https://github.com/felipewind/poc-quarkus-multiple-datasources

With this project, I did the following test:

1. Build it with ./mvnw clean package
2. Start the docker containers of the databases (PostgreSQL and MariaDB):

docker run -d --name postgres-db -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=usersdb postgres
docker run -d --name mariadb-db -p 3306:3306 -e MYSQL_USER=maria -e MYSQL_ROOT_PASSWORD=maria -e MYSQL_PASSWORD=maria -e MYSQL_DATABASE=ordersdb mariadb

1. Run the project in dev mode just to create the tables and insert some rows:

./mvnw compile quarkus:dev

1. Enter inside the /target folder and run the runnable jar passing all parameters:

java -jar -Dquarkus.datasource.users.jdbc.url=jdbc:postgresql://localhost:5432/usersdb -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar

1. Enter inside the http://localhost:8080/swagger-ui , execute the tests and confirm everything is ok.

Now the problem.

1. I try to export just one variable with, like this one:

export QUARKUS_DATASOURCE_USERS_JDBC_URL=jdbc:postgresql://localhost:5432/usersdb 

1. Then I try to run the same command without this property ( quarkus.datasource."users".jdbc.url ), just to test:

java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar

1. Quarkus ignore my environment variable and I receive this error:

WARN  [io.agr.pool] (main) Datasource 'users': Driver does not support the provided URL: prod-url

• prod-url is my default value:

quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}

Now my workaround

As Quarkus isn't reading the property variables with the datasource namespace, I've created the variables below:

quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}

For example:

1. I export the POSTGRESQL_URL:

export POSTGRESQL_URL=jdbc:postgresql://localhost:5432/usersdb 

1. Now I run the application without passing this variable ( quarkus.datasource."users".jdbc.url ) and everything works fine:

java -jar -Dquarkus.datasource.users.username=postgres -Dquarkus.datasource.users.password=postgres -Dquarkus.datasource.orders.jdbc.url=jdbc:mariadb://localhost:3306/ordersdb -Dquarkus.datasource.orders.username=maria -Dquarkus.datasource.orders.password=maria poc-quarkus-multiple-datasources-1.0.0-SNAPSHOT-runner.jar

I'm using this solution to run my real application inside Kubernetes. That is, instead of passing QUARKUS_DATASOURCE_USERS_JDBC_URL and the other variables, I'm passing this variables I've created, like POSTGRESQL_URL.

[felipewind]

@viniciusfcf solved my problem.

I shouldn't have put quotation marks in the datasource name inside my application.properties

I've changed from:

quarkus.datasource."users".db-kind=postgresql
quarkus.datasource."users".jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource."users".username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource."users".password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.hibernate-orm."users".datasource=users
quarkus.hibernate-orm."users".packages=com.helesto.models.users
quarkus.hibernate-orm."users".database.generation = none
quarkus.hibernate-orm."users".sql-load-script=no-file
quarkus.hibernate-orm."users".log.sql=false

quarkus.datasource."orders".db-kind=mariadb
quarkus.datasource."orders".jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource."orders".username=${MARIADB_USERNAME:prod-username}
quarkus.datasource."orders".password=${MARIADB_PASSWORD:prod-password}
quarkus.hibernate-orm."orders".datasource=orders
quarkus.hibernate-orm."orders".packages=com.helesto.models.orders
quarkus.hibernate-orm."orders".database.generation=none
quarkus.hibernate-orm."orders".sql-load-script=no-file
quarkus.hibernate-orm."orders".log.sql=false

To this:

quarkus.datasource.users.db-kind=postgresql
quarkus.datasource.users.jdbc.url=${POSTGRESQL_URL:prod-url}
quarkus.datasource.users.username=${POSTGRESQL_USERNAME:prod-username}
quarkus.datasource.users.password=${POSTGRESQL_PASSWORD:prod-password}
quarkus.hibernate-orm.users.datasource=users
quarkus.hibernate-orm.users.packages=com.helesto.models.users
quarkus.hibernate-orm.users.database.generation = none
quarkus.hibernate-orm.users.sql-load-script=no-file
quarkus.hibernate-orm.users.log.sql=false

quarkus.datasource.orders.db-kind=mariadb
quarkus.datasource.orders.jdbc.url=${MARIADB_URL:prod-url}
quarkus.datasource.orders.username=${MARIADB_USERNAME:prod-username}
quarkus.datasource.orders.password=${MARIADB_PASSWORD:prod-password}
quarkus.hibernate-orm.orders.datasource=orders
quarkus.hibernate-orm.orders.packages=com.helesto.models.orders
quarkus.hibernate-orm.orders.database.generation=none
quarkus.hibernate-orm.orders.sql-load-script=no-file
quarkus.hibernate-orm.orders.log.sql=false

And everything worked perfectly.

Thanks @viniciusfcf

[gsmet]

If it's not working with quotes, we have an issue, it should work.

[gsmet]

@radcortez does it ring a bell?

I wouldn't expect quoted or unquoted configuration properties to work differently. As far as I can see, the OP was using quoted properties in the config file and tried to override them with an unquoted version.

[radcortez]

Hum... can't remember anything related, but I'm going to investigate.

[radcortez]

Regarding the quotes, we treat the config property name as is, without any further modification.

What I found a little bit confusing in the repro steps is @felipewind using an unquoted system property configuration to override a quoted one. @felipewind Can you confirm that you are able to override a quoted name with an unquoted one with system properties? Or the other way around?

[felipewind]

Regarding the quotes, we treat the config property name as is, without any further modification.

What I found a little bit confusing in the repro steps is @felipewind using an unquoted system property configuration to override a quoted one. @felipewind Can you confirm that you are able to override a quoted name with an unquoted one with system properties? Or the other way around?

@radcortez I was able to override a quoted property passing an unquoted -D argument to the java -jar execution. But when exporting the same unquoted property (with export var = ... ) and then executing the jar, it didn't work.

Just to update, I've changed my test repository (felipewind/poc-quarkus-multiple-datasources) removing the quotes from the properties.

If I wasn't clear enough, please tell me

Thanks.

[radcortez]

@radcortez I was able to override a quoted property passing an unquoted -D argument to the java -jar execution. But when exporting the same unquoted property (with export var = ... ) and then executing the jar, it didn't work.

Weird. As far as I know, we don't treat quoted property names differently, in the sense that the quote is part of the name, meaning that you require the quote in the system property name to be able to override it. I've tried in one of my projects and I was only able to override it when using the quoted version.

The issue with environment variables is obvious, since you are not able to use quotes on their name it renders any override useful for quote property names. Also an environment variable name is just converted to its property name counter part, meaning that something like FOO_BAR gets converted to foo.bar to perform the lookup.

@dmlloyd do you think we should convert quoted property names to their unquoted counterparts transparently? Right now, they are just treated as different properties.

[dmlloyd]

In Quarkus we do canonicalize (to a degree) the quoting of property names, so foo\.bar should be equivalent to "foo.bar", and foo should be equivalent to "foo". If we're seeing behavior that contradicts this, then I'd consider it a bug in the properties mapping.

On the topic of environment variable names: it's quite tricky because the mapping only goes one way. Your description is therefore slightly inaccurate: we do not convert FOO_BAR to foo.bar but we do convert foo.bar to FOO_BAR when we probe the environment to see what properties are present. The conversion cannot be two-way because converting to environment vars is a lossy operation. For example we would consider foo.bar_baz and foo_bar.baz to be distinct configuration property names, but both of these would be mapped to an environment variable named FOO_BAR_BAZ.

[radcortez]

In Quarkus we do canonicalize (to a degree) the quoting of property names, so foo\.bar _should_ be equivalent to "foo.bar", and foo _should_ be equivalent to "foo". If we're seeing behavior that contradicts this, then I'd consider it a bug in the properties mapping.

The properties mapping is fine I believe. The issue is the lookup. While Quarkus will consider these to be equivalent, ConfigSources may not, since they usually do a simple lookup with the defined key. In conclusion, ConfigSource A may set foo.bar and ConfigSource B may set "foo.bar" and they don't override each other, since they are treated as different keys.

We could apply the same canonicalize logic to the sources we control, but it may be tricky for user defined sources.

[dmlloyd]

Right, under no conditions would foo.bar be the same as "foo.bar". We'd canonicalize "foo.bar" to foo\.bar. So looking up a property name with quotes would never work, I believe, unless we added a special transformation step for that purpose.

[radcortez]

Yes, so in that particular case, we cannot override a quoted property name with an environment variable because there is no way to represent quotes in the environment variable name to the match to succeed.

So the question is, should we add an extra rule to Env Config Source to loose any quotes in the key name, or should we do this in a more general way to all the sources.

[dmlloyd]

That's not strictly true; if I recall correctly, interior . are mapped as double underscores (one for the \ and one for the .), so if you wanted to map foo."bar.baz".zap to an env var, it'd be FOO_BAR__BAZ_ZAP. I'm fairly sure that this is tested.

[radcortez]

Ah exactly!

@felipewind you should be able to override if you QUARKUS_DATASOURCE__BOOKS__JDBC_URL. Let me know if that works.

[dmlloyd]

I'm not sure this is correct; unless I'm completely misremembering our canonicalization rules, you'd only need to double the _ when there is an interior . that would have been within the quotes. This is because we would canonicalize "foo.bar" to foo\.bar. So if the data source is called "books", you'd still have QUARKUS_DATASOURCE_BOOKS_JDBC_URL.

[felipewind]

Ah exactly!

@felipewind you should be able to override if you QUARKUS_DATASOURCE__BOOKS__JDBC_URL. Let me know if that works.

@radcortez I did the test and worked perfectly.

I could override my property quarkus.datasource."users".jdbc.url with export QUARKUS_DATASOURCE__USERS__JDBC_URL=

Thanks a lot for the help.

@gsmet and @dmlloyd , for my user perspective everything is ok. Please feel free to close the issue if you prefer.

[radcortez]

Great! Thanks!