Terraform: [azurerm] Shared Resource between workspace

[dantheman999301]

I've been wondering what the "best practice" is for having a multi region setup that also has some resources that belong to no region at all (e.g. CosmosDB, Traffic Manager).

I've seen a few ways after some Googling but most of them seem pretty hacky:

• Have a lot of the infrastructure as Modules, then have all the georedundant bits at the top level. List each region you want to deploy to with modules.
• Have the georedundant parts as their own workspace entirely.
• Manually import resources into the workspace.
• Some other weird deployment methods involving remote state.

I'm just wondering if there was a best practice for this that I'm missing. I can't find much in the way of suggested methods of doing this.

This is the set up I currently have infrastructure wise if it helps:

• Environment (test, prod)
  
  
  
  • Regions
    
    
    
    
    
    • Function Apps, web apps, service bus etc.
    
    
    
  
  
  • Cosmos
  
  
  • Traffic Manager
  
  

My original idea would be to have each Environment and Region as their own state (e.g. test.uksouth). Whilst not perfect it would be fairly easy to manager but obviously it falls down with anything that would exist across regions.

Any help would be greatly appreciated.

[apparentlymart]

Hi @dantheman999301,

I don't personally have Azure expertise to know specific details about the services you're talking about here, but I can make some general comments on how Terraform is often used in situations like these.

This usually begins with a key decision point: do you want your multi-region infrastructure all in one configuration to be updated together, or do you want it split into separate configurations so that they can be updated independently? While having everything in one configuration can be convenient, many users have chosen to keep these things separate so that they can safely make localized changes to regional infrastructure if e.g. an outage makes one region unavailable.

Both cases call for putting infrastructure that will be essentially duplicated across multiple regions in a Terraform module. The question then is whether that module is called multiple times in a single configuration or whether it's called from a number of separate configurations that each only configure one region. There isn't a strong best-practice here since it's an architectural tradeoff depending on the structure and needs of your system.

For the sake of this discussion, I generally think of "regionless" infrastructure as just another funny sort of region, since "region" isn't a Terraform concept and so the techniques are the same either way. However, I assume from what you've said here that in your case a special aspect of these regionless items is that the regional infrastructure needs to be configured to talk to them, and so will need access to attributes of the regionless objects.

Regardless of how it is done, when describing a non-trivial system with Terraform it is best to decompose it _somehow_. The common axes of decomposition are regions (as you are discussing here), rate of change (keep "cattle" objects that change regularly separated from valuable long-lived "pet" objects), development vs. production environments, and application-level architectural boundaries. The units of decomposition in Terraform are modules and configurations, with separate configurations giving greater isolation but at the expense of more complexity in connecting these independent units.

It sounds like you're already leaning towards one _state_ per region, which leads me to suggest a separate configuration per region and a separate configuration for the regionless items. The "regionless" configuration would consist only of the objects that do not belong to a region, while the "region" configurations would consist mainly of calls to shared modules.

It's hard to describe this concisely here in a GitHub comment, but I'll try by giving an example logical structure. (You could think of this as one big git repository if you like, or a separate repository for each of these logical components, depending on your organization's preference.)

• regions
  
  
  
  • global
  
  
  • uk-south
  
  
  • france-south
  
  
• modules
  
  
  
  • global-dependencies
  
  
  • standard-regional-infra
  
  

Each of the "region" configurations in this list I would make a separate configuration, by which I mean a separate place to configure a backend and run terraform apply. They therefore have lifecycles independent of one another and can be updated separately.

The global one, as I mentioned, would consist mainly of specific resources to create the needed global infrastructure, and would also have outputs for the ids/names of objects that the downstream regional configurations will depend on.

The regional ones are likely to consist mostly of the following:

provider "azurerm" {
  # Each regional configuration would have a different provider configuration
  region = "South UK"
}

module "global" {
  source = "../../modules/global-dependencies"

  resource_group_name = "example"
}

module "regional" {
  source = "../../modules/standard-regional-infra"

  traffic_manager_endpoint_id = "${module.global.traffic_manager_endpoint_id}"
  cosmosdb_account_id         = "${module.global.cosmosdb_account_id}"
}

# Can include here any other region-specific exceptional objects, if appropriate.

As you can see, the root module of the regional configurations exists to bind together the global dependencies and the local resources. The responsibility of the global-dependencies module is to find the traffic_manager_endpoint_id and cosmosdb_account_id that the global configuration created earlier. Remote state is a straightforward way to do that if the configurations are able to access each other's states, but another option is to use individual data sources to locate the objects directly via the Azure API, like this:

variable "resource_group_name" {
}

data "azurerm_cosmosdb_account" "global" {
  name                = "well-known-cosmosdb-account-name"
  resource_group_name = "${var.resource_group_name}"
}

# I couldn't see a data source for a traffic manager endpoint,
# so this might require a feature request to the AzureRM provider
# team to do this technique. Remote state works independently of
# the provider API, so doesn't have this problem.

output "cosmosdb_account_id" {
  value = "${data.azurerm_cosmosdb_account.global.id}"
}

The specific details aside, the general idea here is to use modules to create well-defined interfaces between system components so that they can be combined together in whatever way makes sense. Because the standard-regional-infra module doesn't "know" where those two ids are coming from, it would work just as well to call it directly from the global configuration and pass through the values directly, without any modification to that module.

This question is timely because a key focus of the forthcoming 0.12 release is to make this sort of module architecture easier to produce by removing some of the limitations that exist in Terraform 0.11 prior with what types of values can pass between modules, how variables can be used to produce dynamic resource configurations, etc. Therefore we're planning to write more extensive documentation about this on the main website once that work is complete and we can show a more complete example using the new features. I'm going to use this issue to represent that documentation task, which will be a more complete and better-organized description of some of the tradeoffs and examples I gave above.

[mildwonkey]

There have been _many_ updates to the module development documentation, so I am going to close this as an out-of-date item. please feel free to open a new issue or feature request if you see problems with the current documentation.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.