Terraform-provider-azurerm: azurerm_container_group: Error when trying to create container group instances without public IP address

[joakimhellum]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

• provider.azurerm v1.11.0

Affected Resource(s)

• azurerm_container_group

Terraform Configuration Files

resource "azurerm_resource_group" "vsts_agent" {
  name     = "vsts-agent-rg"
  location = "west europe"
}

resource "azurerm_container_group" "vsts_agent" {
  name                = "vsts-agent-aci"
  location            = "${azurerm_resource_group.vsts_agent.location}"
  resource_group_name = "${azurerm_resource_group.vsts_agent.name}"
  os_type             = "linux"

  container {
    name   = "vsts-agent"
    image  = "microsoft/vsts-agent"
    cpu    = "1"
    memory = "2"
  }

}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

To successfully create Azure Container Instances (ACI) where IP address is not exposed to public Internet.

Actual Behavior

Error: autorest/azure: Service returned an error. Status=400 Code="MissingIpAddressPorts" Message="The ports in the 'ipAddress' of container group 'vsts-agent-aci' cannot be empty."

Error: Error applying plan:

1 error(s) occurred:

* azurerm_container_group.vsts_agent: 1 error(s) occurred:

* azurerm_container_group.vsts_agent: containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="MissingIpAddressPorts" Message="The ports in the 'ipAddress' of container group 'vsts-agent-aci' cannot be empty."

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

The documentation for azurerm_container_group resource describes optional ip_address_type and container port arguments without default values:

- ip_address_type - (Optional) Specifies the ip address type of the container. Public is the only acceptable value at this time. Changing this forces a new resource to be created.

The container block supports:
- port - (Optional) A public port for the container. Changing this forces a new resource to be created.

It really sounds like ip_address_type and container port is optional and not set by default.

But the schema defined in resource_arm_container_group.go conflicts with the documentation since it sets Public as default value and then makes Public the only acceptable value. There are also some considerations in the resourceArmContainerGroupCreate() if we are suppose to create container group instance without public IP address.

[metacpp]

@joakimhellum-in Thanks for reporting this issue.

Currently, we only support Public as ip_address_type in ACI. We're working on #2314 to support Private (Virtual Network).

Ports field is designed to deprecate Port, in order to have both backward and forward compatibility, we need to make both of them as optional ones.

Considering that service API requires the port, if we still keep optional for ports, w'd better have some default value.

[joakimhellum]

@metacpp Thanks for your reply, very happy about support for private (VNet).

However it's also possible to create container instances without public ip address or private VNet at the moment, for example:

1. Using Azure CLI:

az container create \
  --resource-group log-exporter-rg \
  --name log-exporter-aci \
  --image myorg/log-exporter

1. Using the Portal:
   
   
   Show ARM Template

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "type": "Microsoft.ContainerInstance/containerGroups",
            "name": "log-exporter-aci",
            "apiVersion": "2018-06-01",
            "location": "westeurope",
            "properties": {
                "containers": [
                    {
                        "name": "log-exporter",
                        "properties": {
                            "image": "myorg/log-exporter",
                            "resources": {
                                "requests": {
                                    "memoryInGB": 1.5,
                                    "cpu": 1
                                }
                            }
                        }
                    }
                ],
                "restartPolicy": "Always",
                "osType": "Linux"
            }
        }
    ]
}

It would be nice to have this support in the Terraform provider as well.

[aregier]

However it's also possible to create container instances without public ip address or private VNet at the moment

I'm also seeing that it is easy to use the az CLI to create container instances with private IP addresses, but no VNet. Terraform does not allow this, however. (at least as of 0.12.9)

# Yields "Private IP address is only supported when network profile is defined"
resource "azurerm_container_group" "my-aci" {
  name                = "my-aci"
  location            = "${azurerm_resource_group.main.location}"
  resource_group_name = "${azurerm_resource_group.main.name}"
  ip_address_type     = "private"
  os_type             = "Linux"

  container {
    name   = "gradguard-integration-service"
    image  = "myacr.azurecr.io/my-image:v1"
    cpu    = "0.5"
    memory = "1.0"
  }
}

# Works
az container create \
  --name my-aci \
  --image myacr.azurecr.io/my-image:v1 \
  --registry-login-server myacr.azurecr.io \
  --registry-username myacr \
  --registry-password changeit \
  --ip-address private \
  --resource-group my-resource-group

[kensykora]

I'm having a similar issue, where my scenario is i want to deploy an ACI into a virtual network that doesn't expose any ports, but needs a private IP address to work inside the virtual network. It makes outbound calls, but does not need any direct inbound access.

Use case is an Azure Relay hybrid listener running in a virtual network. I'm trying to get a container running to be the listening end of this.

resource "azurerm_container_group" "hybrid-listener" {
  name                = "aci-listener"
  location            = local.location
  resource_group_name = azurerm_resource_group.t_relay.name
  ip_address_type     = "private"
  network_profile_id  = azurerm_network_profile.acr_profile.id
  os_type             = "Linux"

  image_registry_credential {
    server   = data.azurerm_container_registry.contoso.login_server
    username = data.azurerm_container_registry.contoso.admin_username
    password = data.azurerm_container_registry.contoso.admin_password
  }
                                                  1                                             2
  container {
    name   = "azbridge"
    image  = "${data.azurerm_container_registry.contoso.login_server}/azbridge:latest"
    cpu    = "0.5"
    memory = "1.5"
  }
}

yields

Error: Error creating/updating container group "aci-listener" (Resource Group "t-relay"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: 
Code="MissingIpAddressPorts" Message="The ports in the 'ipAddress' of container group 'aci-listener' cannot be empty."

the workaround seems to be to throw a bogus port in there, which probably isn't a big deal because (a) there is no public IP, and (b) health checks aren't yet supported in ACI on vnets. I'm testing int now.

resource "azurerm_container_group" "hybrid-listener" {
  name                = "aci-listener"
  location            = local.location
  resource_group_name = azurerm_resource_group.t_relay.name
  ip_address_type     = "private"
  network_profile_id  = azurerm_network_profile.acr_profile.id
  os_type             = "Linux"

  image_registry_credential {
    server   = data.azurerm_container_registry.contoso.login_server
    username = data.azurerm_container_registry.contoso.admin_username
    password = data.azurerm_container_registry.contoso.admin_password
  }
                                                  1                                             2
  container {
    name   = "azbridge"
    image  = "${data.azurerm_container_registry.contoso.login_server}/azbridge:latest"
    cpu    = "0.5"
    memory = "1.5"
    ports {
      port     = 9998
      protocol = "UDP"
    }
  }
}

[kensykora]

It seems like this is an issue with the underlying Azure ACI model on the management API itself. It seems to require at least one port is exposed. When i tried to rewrite this as an ARM template, I got a similar error:

Error: Error waiting for deployment: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details." Details=[{"code":"BadRequest","message":"{\r\n  \"error\": {\r\n    \"code\": \"MissingIpAddressPorts\",\r\n    \"message\": \"The ports in the 'ipAddress' of container group 'azwe01relay01-t-aci-listener' cannot be empty.\"\r\n  }\r\n}"}]

Got the same error when trying to remove the ports array entirely. Here's the template:

resource "azurerm_template_deployment" "db01-t-hybrid-listener" {
  name                = "aci-template-hybrid-listener-01"
  deployment_mode     = "Incremental"
  resource_group_name = azurerm_resource_group.hybridcx_relay.name
  template_body       = <<DEPLOY
{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
  },
  "variables": {
  },
  "resources": [
    {
        "name": "hybridcxrelay01-t-aci-listener",
        "type": "Microsoft.ContainerInstance/containerGroups",
        "apiVersion": "2018-10-01",
        "location": "${local.location}",
        "properties": {
          "containers": [
            {
              "name": "azbridge",
              "properties": {
                "image": "${data.azurerm_container_registry.contoso.login_server}/azbridge:latest",
                "command": [
                  "/bin/bash", "-c",
                  "azbridge -R ${azurerm_relay_hybrid_connection.hybridcxrelay01_t_db01_t.name}:${local.test_server_ip}:${local.sql_port} -e ${split("=", split(";", azurerm_relay_namespace.hybridcxrg01relay01_t.primary_connection_string)[0])[1]} -K ${split("=", split(";", azurerm_relay_namespace.hybridcxrg01relay01_t.primary_connection_string)[1])[1]} -k $SHARED_ACCESS_KEY"
                ],
                "ports": [],
                "environmentVariables": [
                  {
                    "name": "SHARED_ACCESS_KEY",
                    "secureValue": "${regex("SharedAccessKey=(?P<key>.*)", split(";", azurerm_relay_namespace.hybridcxrg01relay01_t.primary_connection_string)[2]).key}"
                  }
                ],
                "resources": {
                  "requests": {
                    "memoryInGB": "1.5",
                    "cpu": "0.5"
                  }
                }
              }
            }
          ],
          "imageRegistryCredentials": [
            {
              "server": "${data.azurerm_container_registry.contoso.login_server}",
              "username": "${data.azurerm_container_registry.contoso.admin_username}",
              "password": "${data.azurerm_container_registry.contoso.admin_password}"
            }
          ],
          "ipAddress": {
            "type": "Private"
          },
          "osType": "Linux",
          "networkProfile": {
            "id": "${azurerm_network_profile.hybridcxacr01_profile.id}"
          },
          "dnsConfig": {
            "nameServers": [
              "10.224.3.5", "10.224.3.6"
            ],
            "searchDomains": "contoso.internal"
          }
        }
      }
  ],
  "outputs": {
  }
}
DEPLOY
}

[neil-yechenwei]

Thanks for opening this issue. After investigated, I assume default value of ip_address_type needs to be removed to allow not setting ip address according by the behavior of azure portal. So I made a fix for this issue.