Testcontainers-java: Support custom Docker images in the JDBC URL

[swissarmykirpan]

Make it possible to specify a full docker image name in the JDBC URL and/or custom docker registries, and these might be necessary to allow teams to use their own privately managed images.

[rnorth]

I was looking into this yesterday - would be keen to get your thoughts and the ideas from folks in your team.

Right now, I _think_ if you have a custom container class with image name of the form hostname:port/image:tag then docker will be able to pull from a private registry (assuming the user has already authenticated). This seems like a nice zero-effort solution to me, though I need to test it actually works.

An alternative would be a way to provide private registry hostname/credentials in some way, but this either needs to be encoded in the JDBC URL (getting busy) or as environment variables (less discoverable; requires more documentation).

What do you think?

[rnorth]

@outofcoffee, @haines, any thoughts?

[haines]

I think adding more JDBC URL query string parameters is ok for private registry hostname/credentials.

[swissarmykirpan]

I guess it depends on where the private registry is and how they implement authentication. I know with docker, you can authenticate via the command line:

docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS

Then you can docker push or pull

[rnorth]

I think this is sufficiently complex that we should leave it for a little
while and prioritise other things first.

On Fri, Jun 26, 2015 at 9:17 AM, gusohal [email protected] wrote:

I guess it depends on where the private registry is and how they implement
authentication. I know with docker, you can authenticate via the command
line:

docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS

Then you can docker push or pull

—
Reply to this email directly or view it on GitHub
https://github.com/rnorth/test-containers/issues/4#issuecomment-115438147
.

[outofcoffee]

Would like to see if @rnorth's suggestion works - if so, this seems like a sensible (and standard AFA Docker syntax is concerned) solution.

[rnorth]

We need to try this out - sometime soon!

[rnorth]

This may need further work before it is ready, see #70:

Thank you very much for the fix. Unfortunately it does seem to break support for local registries.
If I understand #4 correctly, the current proposed solution for using custom local image registries is to specify the image name like this: registryHost:port/imageName.
If I specify something like localhost:5000/my-image as image name, the build repeatedly pulls the image (similar behaviour as posted by buckett).
If I use localhost:5000/my-image:latest it says "No image tag was specified in docker image name etc.".

[rnorth]

This is now implemented - see #99

[bsideup]

I re-opened the issue (4th issue of the project!) since it was talking about using custom images in the JDBC URL and it is still not implemented :D

[bsideup]

Context

Example URL: jdbc:tc:mysql:5.6.23:///databasename
Here mysql is the driver's name, and :5.6.23 is a tag of mysql official image.

Problem 1

We cannot easily change it to jdbc:tc:custom-mysql:5.6.23:///databasename because then it will fail to resolve custom-mysql driver.

Problem 2

Custom registries (e.g. registry.my.corp:5000/foo/custom-mysql) are hard to encode in the URL.

Idea

de-couple the driver & image and introduce aliases:

// Must be called before using the URL
JdbcContainer.registerAlias("corporate-mysql", "mysql", "registry.my.corp:5000/foo/custom-mysql");

// And then start it with:
String jdbcUrl = "jdbc:tc:corporate-mysql:5.6.23:///databasename";

Where:

public static void registerAlias(String aliasName, String driverId, String imageName);

We could also add it to testcontainers.properties for those who don't want to use static methods

cc @rnorth @kiview

[SaschaJohn]

I had a discussion with @bsideup in Slack about this because I just recognized that testcontainers.properties are not considered for mysql module, while I was trying to overwrite mysql.container.image = mysql:8.0.19

IMHO aliases are a possibility, but I'm not sure if it's necessary.
Currently only few drivers are supported overwriting the images in testcontainers.properties is the current behavior in testcontainers.
Therefore it would also be possible to keep the coupling between driver and image and only need a custom image for a specific driver if necessary.

[rnorth]

I think I like @bsideup's suggestion. I _suspect_ that configuring via testcontainers.properties is definitely needed as a lowest common denominator, because it's sometimes hard for people to invoke a static method either early enough or even in the right JVM process.

But having the static method to register aliases offers some power too - it means that it's possible to look up an image using custom code, or even build one on the fly with ImageFromDockerfile.

[SaschaJohn]

@rnorth , @bsideup
Feature is ready so far, was faster than expected :)
see: https://github.com/SaschaJohn/testcontainers-java/commit/bb1652708017c806897f70bb16b15d0cc1dd5737
Have to do a little cleanup, provide some tests and documentation as well

Code approach:
ContainerDatabaseDriver.registerAlias("mysqlserver", "sqlserver", "mcmoe/mssqldocker", "latest");
ContainerDatabaseDriver.registerAlias("myownmysql", "mysql", "mysql");

testcontainers.properties:
jdbc.alias.mysqlserver.type=sqlserver
jdbc.alias.mysqlserver.image=mcmoe/mssqldocker
jdbc.alias.mysqlserver.tag=latest

[binkley]

@SaschaJohn Very glad to see your commit. I just stumbled on this issue yesterday when I tried using a custom postgres image which includes the tableversion extension.

Is there a way for me to try "nightly" builds? I'd like to try this out, and provide feedback.

[bsideup]

@binkley the PR is not merged yet, but you can use http://jitpack.io to obtain a version of that code to test it without having to compile/install it yourself, just use PR2294-SNAPSHOT as version.

More info here:
https://www.testcontainers.org/jitpack_dependencies/

[SaschaJohn]

@binkley looking forward for feedback :)

[binkley]

@SaschaJohn Sorry for the slow reply. I've been struggling getting the dependencies working.

1) The testcontainers documentation for jitpack is incorrect (contrast with the instructions on the jitpack site)
2) I am unable to update _all_ testcontainer dependencies to the same PR version; I got funky errors when only adding the testcontainers-java dep, and leaving junit-jupiter and postgresql deps at version 12.5

Since I still see "java.lang.IllegalArgumentException: JDBC URL matches jdbc:tc: prefix but the database or tag name could not be identified", it looks like I'm not using your PR branch version of testcontainers-java, but it's plain in pom.xml that it's the first testcontainers dependency listed.

I'm still poking at it.

[jyrno]

This is how we solved it for postgres:
Added docker.registry.url=my.custom.registry to src/test/resources/testcontainers.properties
Extended testcontainers postgres container with external registry support that handles our custom postgresqlext db types:

package my.package.postgresext;

import org.apache.commons.lang3.StringUtils;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.utility.TestcontainersConfiguration;

public class ExternalRegistryPostgreSQLContainer<SELF extends ExternalRegistryPostgreSQLContainer<SELF>> extends PostgreSQLContainer<SELF> {

    public ExternalRegistryPostgreSQLContainer(final String dockerImageName) {
        super(getRegistryUrl() + dockerImageName);
    }

    private static String getRegistryUrl() {
        String registryUrl = (String) TestcontainersConfiguration.getInstance().getProperties().get("docker.registry.url");
        if (StringUtils.isEmpty(registryUrl)) {
            return StringUtils.EMPTY;
        }
        return StringUtils.appendIfMissing(registryUrl, "/");
    }
}

package my.package.postgresext;

import org.testcontainers.containers.JdbcDatabaseContainer;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.containers.PostgreSQLContainerProvider;

public class ExternalRegistryPostgreSQLContainerProvider extends PostgreSQLContainerProvider {

    private static final String NAME = "postgresqlext";

    @Override
    public boolean supports(String databaseType) {
        return NAME.equals(databaseType);
    }

    @Override
    public JdbcDatabaseContainer newInstance(String tag) {
        return new ExternalRegistryPostgreSQLContainer(PostgreSQLContainer.IMAGE + ":" + tag);
    }

}

Added that container class to src\test\resources\META-INF\services\org.testcontainers.containers.JdbcDatabaseContainerProvider
my.package.ExternalRegistryPostgreSQLContainerProvider

And then just declare your DB url with postgresqlext: jdbc:tc:postgresqlext://localhost/testdb

[bsideup]

@jyrno FYI something like that soon will be natively supported by Testcontainers. See #2599 for an example of how it is proposed to be done for R2DBC