Packer: Add ability to specify GCP Compute image storage location

[yuri-sergiichuk]

Starting December 10, 2019, GCP will bill GCE users for network egress while creating GCE instances from images. The billing is based on the default GCP network egress rules.

The problem here is that by default if the --storage-location option is not specified for the gcloud beta compute image create command, the image is stored in the nearest to the image disk multi-region location. And with the new policy updates, it may be a problem if instances are created in non-multi-region locations.

A workaround that can be used right now:

1. create an image with packer;
2. create a new image from the one packer created using GCP Console UI and store it in a proper location (or using gcloud beta compute images create --source-image=<packer-created-image> --storage-location=<desired-location>).

[SwampDragons]

I'm having trouble understanding what we need to do just via your description. Can you provide an example of a template that will have this problem and an option that would solve it?

[yuri-sergiichuk]

Hi @SwampDragons. What I need is to be able to specify --storage-location in my packer configuration. There is currently no way to specify this parameter in the template.

Here is a template example:

{
  "variables": {
    "gcloud_project_id": "test-project-id"
  },
  "builders": [
    {
      "type": "googlecompute",
      "project_id": "{{user `gcloud_project_id`}}",
      "source_image": "centos-7-v20190905",
      "zone": "us-central1-a",
      "disk_size": "30",
      "image_name": "runner-v{{isotime \"2006-01-02\"}}",
      "image_family": "runner",
      "ssh_username": "packer-builder",
      "machine_type": "n1-standard-1",
      "preemptible": true,
      "storage-location": "us-central1-a"   <---- I need this option
    }
  ]
}

[rickard-von-essen]

We need to allow sending the sourceLocations. But it's only available in the beta API.

[rickard-von-essen]

I couldn't find any official announcement about this, but I received e-mail from Google with this:

To offset the lower pricing for storage, egress charges will be introduced at general network rates pricing when you create custom images or when you use custom images to create disks. These charges will be based on the source region, and only apply when the source and destination are in different regions.

Consider, for example, the following scenarios:

You create a 100 GB image in us-central1 from a Google Cloud Storage (GCS) bucket in europe-west1 - for each GB, you are charged the intercontinental egress rate of $0.12/GB, or $12 total.
You create a disk in the us-central1 region from an image in the us-central1 region - there are no network charges, because the source and destination are in the same location
If you do not specify a storage location, by default, your custom image is stored in the multi-region that is geographically closest to the location of your source region. Two regions australia-southeast1 and southamerica-east1 do not belong to a multi-region. Therefore, their default multi-region image storage location will incur network costs, unless you override the default when you create an image.

What do I need to do?
To prepare for the pricing changes, we advise you to use the image locations feature to find out where your existing images are located before December 10, 2019.

[yuri-sergiichuk]

@rickard-von-essen I've received the very same email and while I constantly spin up and shut down up to 200 GCE preemptible instances with custom images, it may be a significant cost for me to pay the for the network egress.

Is it a problem to use beta API?

[yuri-sergiichuk]

@rickard-von-essen looks like the storageLocations finally landed the v1 API.

[rickard-von-essen]

Is it a problem to use beta API?

One problem is that the beta API's are not covered by the SLA.

storageLocations finally landed the v1 API.

That's great!

[huihuiy02]

Does anyone work on this? I have a similar concern for the egress cost.

[SwampDragons]

No one is currently working on it, but PRs are always accepted :)

[lpxavi]

Hi,

I am very interested by this feature too.

Could be a feature request ?

@huihuiy02 did you find a workaround ?

tks,

[yuri-sergiichuk]

@lpxavi, you can use the workaround described in the issue itself:

A workaround that can be used right now:

1. create an image with packer;
2. create a new image from the one packer created using GCP Console UI and store it in a proper location (or using gcloud beta compute images create --source-image=<packer-created-image> --storage-location=<desired-location>).

[cpwc]

It's not official but if you guys need this feature can consider using the generated build from my PR :)

https://app.circleci.com/pipelines/github/hashicorp/packer/5295/workflows/97e2e9e0-4592-46b6-bac0-aa574ed87cad/jobs/55255/artifacts

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.