Terraform: Individual HTTP proxy settings per provider

[bitbrain]

Hello,

right now it is possible to use an HTTP proxy by setting the http_proxy and https_proxy environment variable on the box. However, I only require this for the AWS provider. For certain providers, I do not want to use a proxy (e.g. Grafana provider).

It would be beneficial to support some kind of proxy settings per-provider instead of setting it globally on the box.

[pawelsocha]

@apparentlymart it's looks like trivial task.

diff --git a/aws/config.go b/aws/config.go
index d357fb40..890f4b5c 100644
--- a/aws/config.go
+++ b/aws/config.go
@@ -6,6 +6,7 @@ import (
        "fmt"
        "log"
        "net/http"
+       "net/url"
        "os"
        "strings"
        "time"
@@ -98,6 +99,7 @@ type Config struct {
        Profile       string
        Token         string
        Region        string
+       ProxyURL      string
        MaxRetries    int

        AssumeRoleARN         string
@@ -300,10 +302,23 @@ func (c *Config) Client() (interface{}, error) {
                opt.Config.Logger = awsLogger{}
        }

-       if c.Insecure {
+       if c.Insecure || c.ProxyURL != "" {
+
                transport := opt.Config.HTTPClient.Transport.(*http.Transport)
-               transport.TLSClientConfig = &tls.Config{
-                       InsecureSkipVerify: true,
+
+               if c.Insecure {
+                       transport.TLSClientConfig = &tls.Config{
+                               InsecureSkipVerify: true,
+                       }
+               }
+
+               if c.ProxyURL != "" {
+                       proxy, err := url.Parse(c.ProxyURL)
+                       if err != nil {
+                               return nil, fmt.Errorf("Error parsing proxy url: %s", err)
+                       }
+
+                       transport.Proxy = http.ProxyURL(proxy)
                }
        }

[bitbrain]

@pawelsocha would you mind creating a Pull Request? Or do I have the pleasure to do it? :-)

[pawelsocha]

@bitbrain I will create PR for AWS provider.

[bitbrain]

@pawelsocha @apparentlymart Is it somehow possible to integrate this into the Terraform Core? What if someone needs to have proxy settings individually per provider? A solution would be to equally implement the same proxy "feature" into each provider but maybe we can avoid duplication.

[pawelsocha]

For me, it looks like attribute per provider. But ... I am a newbie in devel of terraform ;P

[pawelsocha]

@bitbrain if I build a binary can you test it in your enviroment?

[apparentlymart]

Hi all! Sorry for the slow response here.

Unfortunately the HTTP configuration is generally managed separately for each SDK, and is not something Terraform Core can control. The reason the HTTP_PROXY environment variable works is simply that the HTTP client in each SDK is looking for it separately, making it appear like a single global feature.

With that said, the way we'd need to configure a proxy would vary a lot by provider and may, in some cases, not be configurable at all.

In principle I suppose we could try to override the HTTP_PROXY environment variable for the child process of the plugin, but that's not an exact science since multiple provider blocks can share the same child process in some cases.

I think for now I'd be cautious about trying to design and add a possibly-complex core feature for this until we understand better what the needs are. If certain provider dev teams are willing to accept changes like the one @pawelsocha proposed in the mean time (assuming their underlying SDKs _have_ such a feature) then that could be a good way to gather some experience with this feature and see how it might generalize for different underlying SDKs.

[bitbrain]

@pawelsocha Sure, go ahead!

@apparentlymart Okay. :-)

[paultyng]

FWIW Go respects the NO_PROXY environment variable as well.

[pawelsocha]

@paultyng Thanks for your Review.
And the question to @bitbrain what you think about NO_PROXY - it solves your problem?

[bitbrain]

@pawelsocha only in a hacky way. In the same terraform apply I am using a Grafana and an AWS provider. The AWS requires a proxy, Grafana not. Setting the NO_PROXY environment would by default apply to all providers.

A hacky solution _could_ be to hack into the lifecycle like this:

resource "null_resource" "grafana_setting" {
  triggers {
    id = "${uuid()}"
  }
  provisioner "local-exec" {
    command = "export NO_PROXY=\"${var.grafana_host}\""
  }  
}

resource "grafana_dashboard" "metrics" {
  config_json = "${file("grafana-dashboard.json")}"
  depends_on = ["null_resource.grafana_setting"]
}

resource "null_resource" "other_setting" {
  triggers {
    id = "${uuid()}"
  }
  provisioner "local-exec" {
    command = "export HTTP_PROXY=\"${var.proxy_url}\""
  }
  depends_on = ["grafana_dashboard.metrics"]
}

/* other AWS resource stuff */

However, this might not even work. Any alternative suggestion?

[pawelsocha]

@bitbrain Check this binaries :-)
https://drive.google.com/drive/folders/1GhWBv9q3QkloKCNab40AWlmrZ0oaPGW1

you can try proxy_url in aws provider definition

[paultyng]

NO_PROXY lets you specify a list of the destinations to not use proxy (so you would list grafana in that list). So while the setting applies to all providers, it does not apply to all destinations.

See https://godoc.org/net/http#ProxyFromEnvironment

So you could do something like:

export HTTPS_PROXY="https://my-proxy"
export NO_PROXY="grafana.local,10.10.10.1,etc.com"

[pawelsocha]

@paultyng thanks for the explanation :-)
@bitbrain can U check this?