Atlantis: feat (GitHub): Inspect repo and PR contents without having to clone the repo

[timoguin]

GitHub has two API endpoints that should allow us to only clone repos when there are Terraform-related changes in the PR.

APIs

• The Contents API allows us to fetch the atlantis.yaml without cloning the repo
• The Pulls API allows us to list the files modified in a PR

Examples

This should only require two API calls after receiving a webhook.

Fetch the atlantis.yaml from the repo

GET /repos/example-owner/example-repo/atlantis.yaml

This returns a file object that contains the file contents as a base64-encoded string:

{
  "type": "file",
  "encoding": "base64",
  "size": 5362,
  "name": "README.md",
  "path": "README.md",
  "content": "encoded content ...",
  "sha": "3d21ec53a331a6f037a91c368710b99387d012c1",
  "url": "https://api.github.com/repos/octokit/octokit.rb/contents/README.md",
  "git_url": "https://api.github.com/repos/octokit/octokit.rb/git/blobs/3d21ec53a331a6f037a91c368710b99387d012c1",
  "html_url": "https://github.com/octokit/octokit.rb/blob/master/README.md",
  "download_url": "https://raw.githubusercontent.com/octokit/octokit.rb/master/README.md",
  "_links": {
    "git": "https://api.github.com/repos/octokit/octokit.rb/git/blobs/3d21ec53a331a6f037a91c368710b99387d012c1",
    "self": "https://api.github.com/repos/octokit/octokit.rb/contents/README.md",
    "html": "https://github.com/octokit/octokit.rb/blob/master/README.md"
  }
}

Get a list of files modified in a PR

GET /repos/example-owner/example-repo/pulls/100/files

This returns a list of file objects:

[
  {
    "sha": "bbcd538c8e72b8c175046e27cc8f907076331401",
    "filename": "file1.txt",
    "status": "added",
    "additions": 103,
    "deletions": 21,
    "changes": 124,
    "blob_url": "https://github.com/octocat/Hello-World/blob/6dcb09b5b57875f334f61aebed695e2e4193db5e/file1.txt",
    "raw_url": "https://github.com/octocat/Hello-World/raw/6dcb09b5b57875f334f61aebed695e2e4193db5e/file1.txt",
    "contents_url": "https://api.github.com/repos/octocat/Hello-World/contents/file1.txt?ref=6dcb09b5b57875f334f61aebed695e2e4193db5e",
    "patch": "@@ -132,7 +132,7 @@ module Test @@ -1000,7 +1000,7 @@ module Test"
  }
]

References

• Fetch a file from a repo: https://developer.github.com/v3/repos/contents/#get-the-readme
• List files modified in a PR: https://developer.github.com/v3/pulls/#list-pull-requests-files
• This is somewhat related to #966 in that we want to conditionally ignore certain repos and/or PRs.

[cket]

I think this would be useful for me as well. If we only care about Terraform-related changes would we need to inspect the atlantis.yaml file?

If not, I think this could be a straightforward change since it looks like Atlantis already uses the PR API to find the modified files prior to cloning the PR. I could take a shot at this, maybe making this an optional behavior we can enable with a flag.

[timoguin]

I think this would be useful for me as well. If we only care about Terraform-related changes would we need to inspect the atlantis.yaml file?

My main reason for needing to read the atlantis.yaml is because it can define different glob patterns to trigger plans. I often (and usually) want to trigger plans for other file types, as well as for relative paths outside the project root. Without inspecting the config, there's no way of knowing.

[lkysow]

@timoguin sorry but I'm missing the problem that you want to solve with this. Is it so non-terraform prs don't get atlantis triggered on them (https://github.com/runatlantis/atlantis/issues/221) or is to solve the blacklist problem because you're running the org webhook and some repos never need Atlantis?

[timoguin]

Hey @lkysow. We do use org-wide webhooks, but with thousands of repositories, the blacklist patterns often aren't sufficient. The main catalyst for this issue was a large repository (10GB+) with Terraform code in it. If a non-Terraform change is made, we don't want Atlantis to bother with cloning the repo, you know, because it's huge.

Edit: After looking at #221, it is indeed a similar problem we are trying to solve. I might even go so far as to call this issue a duplicate.

[lkysow]

we don't want Atlantis to bother with cloning the repo, you know, because it's huge.

Gotcha. Yeah I think we need to do the exploratory API calls that you've linked in the issue.

[aristocrates]

Something I've noticed that seems related to this is that PRs that get merged quickly enough after opening generate a message from atlantis saying "Plan Error" with a backtrace of the git clone failing:

Plan Error

running git clone --branch [branch name] --depth=1 --single-branch [repo remote location and local clone location]
error: unable to write file /home/atlantis/.atlantis/repos/[org/repo]/[PR number]/default/.git/objects/pack/pack-[hash].pack: No such file or directory
fatal: cannot store pack file
fatal: index-pack failed
: exit status 128

This can be confusing when the changes weren't terraform related.

[lkysow]

This will be available in the next release via the --skip-clone-no-changes flag. Note that this will only work if you have a repo atlantis.yaml file since without that file we try to auto-detect your terraform structure. Specifically, we check if a directory has main.tf and then decide that that is the root directory. Without cloning we can't do that. With the atlantis.yaml we don't do any auto-detection so we can just use its contents directly.