Testcontainers-java: no way to map container port to different host port

[Bill]

docker -p 8003:5432 will expose container port 5432 on host port 8003.

This is extremely valuable when running parallel tests. It lets us avoid port conflicts.

I see no straightforward way to accomplish this with testcontainers.

Container.withExposedPorts(Integer... ports) takes a port array and maps each container port in the array, to the corresponding port on the host.

I expected to find a method with this signature something like this:

Container.withExposedPorts(Map<Integer,Integer> mappings)

Or maybe even:

Container.withExposedPortMappings(Integer... mappings)

…that would take the flattened mapping.

[bsideup]

It lets us avoid port conflicts.

TestContainers doesn't allow you (publically) to use a non-random port. When you call withExposedPorts(5432) it will use -p 5432, it means that the port will be random by default.

Then you just use getMappedPort(5432)

https://www.testcontainers.org/usage/generic_containers.html#accessing-a-container-from-tests

[Bill]

Thank you!!

[NickEm]

BTW, guys there is a possibility to specify host port as well. F.e:

int hostPort = 6380;
int containerExposedPort = 6379;
Consumer<CreateContainerCmd> cmd = e -> e.withPortBindings(new PortBinding(Ports.Binding.bindPort(hostPort), new ExposedPort(containerExposedPort)));

GenericContainer redisContainer = new GenericContainer("redis:4.0.10")
                    .withExposedPorts(containerExposedPort)
                    .withCreateContainerCmdModifier(cmd);

[ocker-docker-gummi-klokker]

import org.testcontainers.containers.GenericContainer
import org.testcontainers.containers.FixedHostPortGenericContainer

Use GenericContainer container = new GenericContainer("name") when working with random ports.

Use GenericContainer container = new FixedHostPortGenericContainer("name") when working
with fixed ports, and then do container.withFixedExposedPort(hostPort, containerPort)

[kiview]

While this works, we strongly advise against using fixed ports, since this will automatically lead to integrated tests (which are an anti pattern).

The definition of integrated tests based on this Spotify blog post:

A test that will pass or fail based on the correctness of another system.

[ocker-docker-gummi-klokker]

I understand and totally agree.
All our automated tests use random ports, but we have a single local acceptance test (disabled in our CI environment) which for good reasons use fixed ports.
In special situations, it is convenient to be able to use fixed ports.

[D0rmouse]

How do I bind the random host port to my @SpringBootTest Spring Data property? For instance: spring.data.mongodb.port=xxxxx.

Would be nice if I could do something like spring.data.mongodb.port=${testcontainers.host.port}.

Now I need to use the FixedHostPortGenericContainer, because I havent figured a way around this yet.

[bsideup]

@D0rmouse please look at our Spring Boot Example

[cal97g]

While this works, we strongly advise against using fixed ports, since this will automatically lead to integrated tests (which are an anti pattern).

The definition of integrated tests based on this Spotify blog post:

A test that will pass or fail based on the correctness of another system.

Tenuous

[andrejpetras]

While this works, we strongly advise against using fixed ports, since this will automatically lead to integrated tests (which are an anti pattern).

The definition of integrated tests based on this Spotify blog post:

A test that will pass or fail based on the correctness of another system.

Could you please make the method withFixedExposedPort public in the GenericContainer?
I believe the developers should have the option to decide what they need.

Currently we need to do workaround for example for the PostgreSQLContainer and we are building the library around the testcontainers.

[thergbway]

As far as I see there are some ways to create fixed port bindings for single containers but there is no any way to make fixed port binding for DockerComposeContainer. In our case we want a fixed exposed port for debug the java application we are testing. The target of out tests is one single service of started docker-compose environment.
How to make fixed exposed ports for DockerComposeContainer?

[bsideup]

We do not recommend using fixed ports, even with single containers.

For debugging, consider using the approach described in the following blogpost:
https://bsideup.github.io/posts/debugging_containers/

[bsideup]

@andrejpetras

even if we expose withFixedExposedPort it won't work in some environments (like running a build inside a container, or even Travis). It is protected for a good reason, and there are other ways of doing this (e.g. by proxying, see my previous comment on this issue)

[chinthaka-dinadasa]

BTW, guys there is a possibility to specify host port as well. F.e:

int hostPort = 6380;
int containerExposedPort = 6379;
Consumer<CreateContainerCmd> cmd = e -> e.withPortBindings(new PortBinding(Ports.Binding.bindPort(hostPort), new ExposedPort(containerExposedPort)));

GenericContainer redisContainer = new GenericContainer("redis:4.0.10")
                    .withExposedPorts(containerExposedPort)
                    .withCreateContainerCmdModifier(cmd);

Unfortunately withPortBindings Deprecated in newest version

[BloodChiefStromberg]

@D0rmouse please look at our Spring Boot Example

Maybe we could add this to the docs? https://github.com/testcontainers/testcontainers-java/pull/2484

[vietj]

It can be very useful to define a fixed port when you want to use a packet sniffer for debugging locally.

Unfortunately the method for defining a fixed mapping is protected. I could create a subclass to achieve fixed port mapping and be able to define a port I could sniff with Wireshark.

Here is what I did using an inner class:

  public static class FixedPostgreSQLContainer extends PostgreSQLContainer {
    public FixedPostgreSQLContainer(String dockerImageName) {
      super(dockerImageName);
    }
    public FixedPostgreSQLContainer configurePort() {
      super.addFixedExposedPort(5432, 5432);
      return this;
    }
  }

I'm posting this because it might be useful to others.

[bsideup]

@vietj please see https://bsideup.github.io/posts/testcontainers_fixed_ports/ and then https://bsideup.github.io/posts/debugging_containers/

The method is protected for reason (in fact, I really want to see it deprecated and removed soon /cc @rnorth )

[vietj]

I am merely saying it should be always available but not the default.

There should be the option to use it when it is useful. In the project I'm doing we are not using it by default but when we need to use Wireshark to understand what happens at the protocol level then we can simply replace the default container by the code I exhibited and be able to have access easily to the information we need.

So removing this "backdoor" would make life of developers like me harder.

[bsideup]

@vietj one can always modify the CreateContainerCmd and apply any (potentially dangerous) modification (s)he wants.

Just we don't want to have this API because it is known to be dangerous, especially for those who don't understand the problem and want to use it for something other than debugging (e.g. they hardcode localhost:5432 in their test app's config)

[bsideup]

@@vietj also, have you checked the second link? Is there any issue with using it with Wireshark or anything?

[vietj]

@bsideup I can't use a proxy in my tests because it is important to have the entire IP traffic between the hosts. Beyond observing what happens on the wire, I often need to check the all IP packets such as ACK or FINACK or RST between the two parties.

[Samehadar]

I had to use @vietj dirty hack today.
I need this coz I use postgresql db in container as in-memory db for my app.
And for querying/debugging data in my db I use IntelliJ Idea or DBeaver.
Without this API I have to look for the containers port each time I start my app.
Not that it is really important case, but I still don't wanna change my Data Source in IntelliJ Idea every time I start my application.

[jdeppe-pivotal]

I'd like you to consider NOT deprecating and removing the port binding API. Although I can understand you urging folks not to use it, I don't think you can always foresee all use cases and being overly strict doesn't end up helping :).

In my particular case I'm trying to build up a Redis cluster which should be externally accessible from the host system. Redis clustering has a command (CLUSTER SLOTS) which reports the IP and ports that members are accessible on. Typically this reports the internal docker IP and the same port for every system making up the cluster. Now, it's a chicken-n-egg problem if I want to use randomly assigned external ports which also need to be known when the cluster is started. Since there's no way for CLUSTER SLOTS to magically report externally exposed ports.

[Aside] To avoid the hardcoded port problem; in my case I'm going to use externally determined 'free' ports and use those when configuring the cluster.

In the case of Redis clustering there is a solution for Redis 6.0 (https://github.com/bitnami/bitnami-docker-redis-cluster/issues/7), however my point hopefully serves more as an example of a situation that would benefit from having the port binding API remain in place (undeprecated).

[bsideup]

@jdeppe-pivotal this sounds very similar to Kafka there we solved the problem by deferring starting the process inside a container. See https://github.com/testcontainers/testcontainers-java/blob/eaf9b9fe2aab4e60a4b0079e7c15afb38bb44beb/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L106

Which only proves that there are valid options to the problem that do not require fixed ports.

[maxandersen]

I use testcontainer api to do quick'n'dirty experiments where fixed ports are good.

Being able to use testcontainers api to setup a set of containers on random ports is awesome; but equally awesome is it to set it up explicitly to fit into environment/usecase where static port is actually useful. i.e. for easy access to a database from external tools as already mentioned, but also for cases where the setup is hardwired in at a stage I can't (yet) change.

Big +1 for not removing the api.

[bsideup]

Note that it will always be possible to fix the ports because Docker supports it, and we allow modifying the underlying CreateContainerCmd. Just we won't offer a _convenient_ API for that, as we find it dangerous.

Last but not least, one can use a proxy to guarantee that the host will be localhost (one of the biggest problem of fixed ports is not the ports themselves, but the fact that users hardcode the host, and, unlike ports, host isn't always static):
https://bsideup.github.io/posts/debugging_containers/

[desiderati]

BTW, guys there is a possibility to specify host port as well. F.e:

int hostPort = 6380;
int containerExposedPort = 6379;
Consumer<CreateContainerCmd> cmd = e -> e.withPortBindings(new PortBinding(Ports.Binding.bindPort(hostPort), new ExposedPort(containerExposedPort)));

GenericContainer redisContainer = new GenericContainer("redis:4.0.10")
                    .withExposedPorts(containerExposedPort)
                    .withCreateContainerCmdModifier(cmd);

Becuase this method (withPortBindings) is deprecated, you can do it now like this:

static final MySQLContainer<?> mysql =
    new MySQLContainer<>("mysql:5.6")
        .withExposedPorts(34343)
        .withCreateContainerCmdModifier(cmd -> cmd.withHostConfig(
            new HostConfig().withPortBindings(new PortBinding(Ports.Binding.bindPort(34343), new ExposedPort(3306)))
        ));

[mklueh]

BTW you need to cast to Consumer if you are using the lambda way of implementation

@Container
@SuppressWarnings({"rawtypes", "unchecked"}) //otherwise everything yellow in the IDE
private final GenericContainer sftp = new GenericContainer("atmoz/sftp")
        .withExposedPorts(PORT)
        .withCreateContainerCmdModifier((Consumer<CreateContainerCmd>) cmd -> cmd.withHostConfig(
                new HostConfig().withPortBindings(new PortBinding(Ports.Binding.bindPort(PORT), new ExposedPort(22222)))
        ));

[bsideup]

@mklueh Or like this:

@Container
private final GenericContainer<?> sftp = new GenericContainer<>("atmoz/sftp")
        .withExposedPorts(PORT)
        .withCreateContainerCmdModifier(cmd -> cmd.withHostConfig(
                new HostConfig().withPortBindings(new PortBinding(Ports.Binding.bindPort(PORT), new ExposedPort(22222)))
        ));