Go-github: Timeline Source Object is Incomplete

[ereyes01]

When attempting to use the Timeline feature, I noticed that the definition of the Source does not match what I get back from the Github API. For example, have a look at this issue in the docker repository (json formatting cleaned up by piping output to the jq command, this is optional):

> curl -H "Accept: application/vnd.github.mockingbird-preview"  https://api.github.com/repos/docker/docker/issues/29619/timeline | jq ""
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20120  100 20120    0     0  35562      0 --:--:-- --:--:-- --:--:-- 35547
[
  {
    "id": 901030061,
    "url": "https://api.github.com/repos/docker/docker/issues/events/901030061",
    "actor": {
      "login": "GordonTheTurtle",
      "id": 10620046,
      "avatar_url": "https://avatars.githubusercontent.com/u/10620046?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/GordonTheTurtle",
      "html_url": "https://github.com/GordonTheTurtle",
      "followers_url": "https://api.github.com/users/GordonTheTurtle/followers",
      "following_url": "https://api.github.com/users/GordonTheTurtle/following{/other_user}",
      "gists_url": "https://api.github.com/users/GordonTheTurtle/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/GordonTheTurtle/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/GordonTheTurtle/subscriptions",
      "organizations_url": "https://api.github.com/users/GordonTheTurtle/orgs",
      "repos_url": "https://api.github.com/users/GordonTheTurtle/repos",
      "events_url": "https://api.github.com/users/GordonTheTurtle/events{/privacy}",
      "received_events_url": "https://api.github.com/users/GordonTheTurtle/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "labeled",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2016-12-21T02:27:44Z",
    "label": {
      "name": "area/builder",
      "color": "fbca04"
    }
  },
  {
    "id": 901030063,
    "url": "https://api.github.com/repos/docker/docker/issues/events/901030063",
    "actor": {
      "login": "GordonTheTurtle",
      "id": 10620046,
      "avatar_url": "https://avatars.githubusercontent.com/u/10620046?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/GordonTheTurtle",
      "html_url": "https://github.com/GordonTheTurtle",
      "followers_url": "https://api.github.com/users/GordonTheTurtle/followers",
      "following_url": "https://api.github.com/users/GordonTheTurtle/following{/other_user}",
      "gists_url": "https://api.github.com/users/GordonTheTurtle/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/GordonTheTurtle/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/GordonTheTurtle/subscriptions",
      "organizations_url": "https://api.github.com/users/GordonTheTurtle/orgs",
      "repos_url": "https://api.github.com/users/GordonTheTurtle/repos",
      "events_url": "https://api.github.com/users/GordonTheTurtle/events{/privacy}",
      "received_events_url": "https://api.github.com/users/GordonTheTurtle/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "labeled",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2016-12-21T02:27:44Z",
    "label": {
      "name": "version/1.12",
      "color": "ededed"
    }
  },
  {
    "actor": {
      "login": "yongtang",
      "id": 6932348,
      "avatar_url": "https://avatars.githubusercontent.com/u/6932348?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/yongtang",
      "html_url": "https://github.com/yongtang",
      "followers_url": "https://api.github.com/users/yongtang/followers",
      "following_url": "https://api.github.com/users/yongtang/following{/other_user}",
      "gists_url": "https://api.github.com/users/yongtang/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/yongtang/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/yongtang/subscriptions",
      "organizations_url": "https://api.github.com/users/yongtang/orgs",
      "repos_url": "https://api.github.com/users/yongtang/repos",
      "events_url": "https://api.github.com/users/yongtang/events{/privacy}",
      "received_events_url": "https://api.github.com/users/yongtang/received_events",
      "type": "User",
      "site_admin": false
    },
    "created_at": "2016-12-21T06:26:57Z",
    "updated_at": "2016-12-21T06:26:57Z",
    "source": {
      "type": "issue",
      "issue": {
        "url": "https://api.github.com/repos/docker/docker/issues/29624",
        "repository_url": "https://api.github.com/repos/docker/docker",
        "labels_url": "https://api.github.com/repos/docker/docker/issues/29624/labels{/name}",
        "comments_url": "https://api.github.com/repos/docker/docker/issues/29624/comments",
        "events_url": "https://api.github.com/repos/docker/docker/issues/29624/events",
        "html_url": "https://github.com/docker/docker/pull/29624",
        "id": 196856577,
        "number": 29624,
        "title": "Sort the labels passed from `build --labels`",
        "user": {
          "login": "yongtang",
          "id": 6932348,
          "avatar_url": "https://avatars.githubusercontent.com/u/6932348?v=3",
          "gravatar_id": "",
          "url": "https://api.github.com/users/yongtang",
          "html_url": "https://github.com/yongtang",
          "followers_url": "https://api.github.com/users/yongtang/followers",
          "following_url": "https://api.github.com/users/yongtang/following{/other_user}",
          "gists_url": "https://api.github.com/users/yongtang/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/yongtang/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/yongtang/subscriptions",
          "organizations_url": "https://api.github.com/users/yongtang/orgs",
          "repos_url": "https://api.github.com/users/yongtang/repos",
          "events_url": "https://api.github.com/users/yongtang/events{/privacy}",
          "received_events_url": "https://api.github.com/users/yongtang/received_events",
          "type": "User",
          "site_admin": false
        },
        "labels": [
          {
            "id": 175375407,
            "url": "https://api.github.com/repos/docker/docker/labels/status/2-code-review",
            "name": "status/2-code-review",
            "color": "fad8c7",
            "default": false
          }
        ],
        "state": "closed",
        "locked": false,
        "assignee": null,
        "assignees": [],
        "milestone": {
          "url": "https://api.github.com/repos/docker/docker/milestones/60",
          "html_url": "https://github.com/docker/docker/milestone/60",
          "labels_url": "https://api.github.com/repos/docker/docker/milestones/60/labels",
          "id": 2126145,
          "number": 60,
          "title": "1.14.0",
          "description": null,
          "creator": {
            "login": "vieux",
            "id": 1032519,
            "avatar_url": "https://avatars.githubusercontent.com/u/1032519?v=3",
            "gravatar_id": "",
            "url": "https://api.github.com/users/vieux",
            "html_url": "https://github.com/vieux",
            "followers_url": "https://api.github.com/users/vieux/followers",
            "following_url": "https://api.github.com/users/vieux/following{/other_user}",
            "gists_url": "https://api.github.com/users/vieux/gists{/gist_id}",
            "starred_url": "https://api.github.com/users/vieux/starred{/owner}{/repo}",
            "subscriptions_url": "https://api.github.com/users/vieux/subscriptions",
            "organizations_url": "https://api.github.com/users/vieux/orgs",
            "repos_url": "https://api.github.com/users/vieux/repos",
            "events_url": "https://api.github.com/users/vieux/events{/privacy}",
            "received_events_url": "https://api.github.com/users/vieux/received_events",
            "type": "User",
            "site_admin": false
          },
          "open_issues": 18,
          "closed_issues": 224,
          "state": "open",
          "created_at": "2016-11-10T02:23:44Z",
          "updated_at": "2017-01-08T14:34:44Z",
          "due_on": null,
          "closed_at": null
        },
        "comments": 6,
        "created_at": "2016-12-21T06:26:57Z",
        "updated_at": "2016-12-29T23:09:55Z",
        "closed_at": "2016-12-29T23:09:54Z",
        "repository": {
          "id": 7691631,
          "name": "docker",
          "full_name": "docker/docker",
          "owner": {
            "login": "docker",
            "id": 5429470,
            "avatar_url": "https://avatars.githubusercontent.com/u/5429470?v=3",
            "gravatar_id": "",
            "url": "https://api.github.com/users/docker",
            "html_url": "https://github.com/docker",
            "followers_url": "https://api.github.com/users/docker/followers",
            "following_url": "https://api.github.com/users/docker/following{/other_user}",
            "gists_url": "https://api.github.com/users/docker/gists{/gist_id}",
            "starred_url": "https://api.github.com/users/docker/starred{/owner}{/repo}",
            "subscriptions_url": "https://api.github.com/users/docker/subscriptions",
            "organizations_url": "https://api.github.com/users/docker/orgs",
            "repos_url": "https://api.github.com/users/docker/repos",
            "events_url": "https://api.github.com/users/docker/events{/privacy}",
            "received_events_url": "https://api.github.com/users/docker/received_events",
            "type": "Organization",
            "site_admin": false
          },
          "private": false,
          "html_url": "https://github.com/docker/docker",
          "description": "Docker - the open-source application container engine",
          "fork": false,
          "url": "https://api.github.com/repos/docker/docker",
          "forks_url": "https://api.github.com/repos/docker/docker/forks",
          "keys_url": "https://api.github.com/repos/docker/docker/keys{/key_id}",
          "collaborators_url": "https://api.github.com/repos/docker/docker/collaborators{/collaborator}",
          "teams_url": "https://api.github.com/repos/docker/docker/teams",
          "hooks_url": "https://api.github.com/repos/docker/docker/hooks",
          "issue_events_url": "https://api.github.com/repos/docker/docker/issues/events{/number}",
          "events_url": "https://api.github.com/repos/docker/docker/events",
          "assignees_url": "https://api.github.com/repos/docker/docker/assignees{/user}",
          "branches_url": "https://api.github.com/repos/docker/docker/branches{/branch}",
          "tags_url": "https://api.github.com/repos/docker/docker/tags",
          "blobs_url": "https://api.github.com/repos/docker/docker/git/blobs{/sha}",
          "git_tags_url": "https://api.github.com/repos/docker/docker/git/tags{/sha}",
          "git_refs_url": "https://api.github.com/repos/docker/docker/git/refs{/sha}",
          "trees_url": "https://api.github.com/repos/docker/docker/git/trees{/sha}",
          "statuses_url": "https://api.github.com/repos/docker/docker/statuses/{sha}",
          "languages_url": "https://api.github.com/repos/docker/docker/languages",
          "stargazers_url": "https://api.github.com/repos/docker/docker/stargazers",
          "contributors_url": "https://api.github.com/repos/docker/docker/contributors",
          "subscribers_url": "https://api.github.com/repos/docker/docker/subscribers",
          "subscription_url": "https://api.github.com/repos/docker/docker/subscription",
          "commits_url": "https://api.github.com/repos/docker/docker/commits{/sha}",
          "git_commits_url": "https://api.github.com/repos/docker/docker/git/commits{/sha}",
          "comments_url": "https://api.github.com/repos/docker/docker/comments{/number}",
          "issue_comment_url": "https://api.github.com/repos/docker/docker/issues/comments{/number}",
          "contents_url": "https://api.github.com/repos/docker/docker/contents/{+path}",
          "compare_url": "https://api.github.com/repos/docker/docker/compare/{base}...{head}",
          "merges_url": "https://api.github.com/repos/docker/docker/merges",
          "archive_url": "https://api.github.com/repos/docker/docker/{archive_format}{/ref}",
          "downloads_url": "https://api.github.com/repos/docker/docker/downloads",
          "issues_url": "https://api.github.com/repos/docker/docker/issues{/number}",
          "pulls_url": "https://api.github.com/repos/docker/docker/pulls{/number}",
          "milestones_url": "https://api.github.com/repos/docker/docker/milestones{/number}",
          "notifications_url": "https://api.github.com/repos/docker/docker/notifications{?since,all,participating}",
          "labels_url": "https://api.github.com/repos/docker/docker/labels{/name}",
          "releases_url": "https://api.github.com/repos/docker/docker/releases{/id}",
          "deployments_url": "https://api.github.com/repos/docker/docker/deployments",
          "created_at": "2013-01-18T18:10:57Z",
          "updated_at": "2017-01-08T21:24:27Z",
          "pushed_at": "2017-01-08T14:34:43Z",
          "git_url": "git://github.com/docker/docker.git",
          "ssh_url": "[email protected]:docker/docker.git",
          "clone_url": "https://github.com/docker/docker.git",
          "svn_url": "https://github.com/docker/docker",
          "homepage": "http://www.docker.com",
          "size": 120296,
          "stargazers_count": 38490,
          "watchers_count": 38490,
          "language": "Go",
          "has_issues": true,
          "has_downloads": true,
          "has_wiki": true,
          "has_pages": false,
          "forks_count": 11484,
          "mirror_url": null,
          "open_issues_count": 2126,
          "forks": 11484,
          "open_issues": 2126,
          "watchers": 38490,
          "default_branch": "master"
        },
        "pull_request": {
          "url": "https://api.github.com/repos/docker/docker/pulls/29624",
          "html_url": "https://github.com/docker/docker/pull/29624",
          "diff_url": "https://github.com/docker/docker/pull/29624.diff",
          "patch_url": "https://github.com/docker/docker/pull/29624.patch"
        },
        "body": "**- What I did**\r\n\r\nThis fix tries to fix the issue in #29619 where labels passed from `build --labels` are not sorted.\r\nAs a result, if multiple labels have been passed, each `docker build --labels A=A --labels B=B --labels C=C` run will generate different layers.\r\n\r\n**- How I did it**\r\n\r\nThis fix fixes the issue by sort the Labels before they are concatenated to `LABEL ...`.\r\n\r\n**- How to verify it**\r\n\r\nAn integration test has been added to cover the changes\r\n\r\n**- Description for the changelog**\r\n\r\n**- A picture of a cute animal (not mandatory but encouraged)**\r\n\r\n![cute-kitten-865-10-400x250](https://cloud.githubusercontent.com/assets/6932348/21379497/49fb2592-c703-11e6-991c-3a4d61956bda.jpg)\r\n\r\nThis fix fixes #29619.\r\n\r\nSigned-off-by: Yong Tang <[email protected]>"
      }
    },
    "event": "cross-referenced"
  },
  {
    "url": "https://api.github.com/repos/docker/docker/issues/comments/268447299",
    "html_url": "https://github.com/docker/docker/issues/29619#issuecomment-268447299",
    "issue_url": "https://api.github.com/repos/docker/docker/issues/29619",
    "id": 268447299,
    "user": {
      "login": "yongtang",
      "id": 6932348,
      "avatar_url": "https://avatars.githubusercontent.com/u/6932348?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/yongtang",
      "html_url": "https://github.com/yongtang",
      "followers_url": "https://api.github.com/users/yongtang/followers",
      "following_url": "https://api.github.com/users/yongtang/following{/other_user}",
      "gists_url": "https://api.github.com/users/yongtang/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/yongtang/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/yongtang/subscriptions",
      "organizations_url": "https://api.github.com/users/yongtang/orgs",
      "repos_url": "https://api.github.com/users/yongtang/repos",
      "events_url": "https://api.github.com/users/yongtang/events{/privacy}",
      "received_events_url": "https://api.github.com/users/yongtang/received_events",
      "type": "User",
      "site_admin": false
    },
    "created_at": "2016-12-21T06:28:35Z",
    "updated_at": "2016-12-21T06:28:35Z",
    "body": "The `--labels` are processed in:\r\nhttps://github.com/docker/docker/blob/v1.13.0-rc4/builder/dockerfile/builder.go#L236-L246\r\nHowever, they are not sorted so it generates different layers for each `docker build` run.\r\n\r\nAdded a PR #29624 to address this issue.",
    "event": "commented",
    "actor": {
      "login": "yongtang",
      "id": 6932348,
      "avatar_url": "https://avatars.githubusercontent.com/u/6932348?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/yongtang",
      "html_url": "https://github.com/yongtang",
      "followers_url": "https://api.github.com/users/yongtang/followers",
      "following_url": "https://api.github.com/users/yongtang/following{/other_user}",
      "gists_url": "https://api.github.com/users/yongtang/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/yongtang/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/yongtang/subscriptions",
      "organizations_url": "https://api.github.com/users/yongtang/orgs",
      "repos_url": "https://api.github.com/users/yongtang/repos",
      "events_url": "https://api.github.com/users/yongtang/events{/privacy}",
      "received_events_url": "https://api.github.com/users/yongtang/received_events",
      "type": "User",
      "site_admin": false
    }
  },
  {
    "id": 902021335,
    "url": "https://api.github.com/repos/docker/docker/issues/events/902021335",
    "actor": {
      "login": "tonistiigi",
      "id": 585223,
      "avatar_url": "https://avatars.githubusercontent.com/u/585223?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/tonistiigi",
      "html_url": "https://github.com/tonistiigi",
      "followers_url": "https://api.github.com/users/tonistiigi/followers",
      "following_url": "https://api.github.com/users/tonistiigi/following{/other_user}",
      "gists_url": "https://api.github.com/users/tonistiigi/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/tonistiigi/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/tonistiigi/subscriptions",
      "organizations_url": "https://api.github.com/users/tonistiigi/orgs",
      "repos_url": "https://api.github.com/users/tonistiigi/repos",
      "events_url": "https://api.github.com/users/tonistiigi/events{/privacy}",
      "received_events_url": "https://api.github.com/users/tonistiigi/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "labeled",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2016-12-21T18:49:42Z",
    "label": {
      "name": "kind/bug",
      "color": "FF8C94"
    }
  },
  {
    "id": 907752135,
    "url": "https://api.github.com/repos/docker/docker/issues/events/907752135",
    "actor": {
      "login": "tonistiigi",
      "id": 585223,
      "avatar_url": "https://avatars.githubusercontent.com/u/585223?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/tonistiigi",
      "html_url": "https://github.com/tonistiigi",
      "followers_url": "https://api.github.com/users/tonistiigi/followers",
      "following_url": "https://api.github.com/users/tonistiigi/following{/other_user}",
      "gists_url": "https://api.github.com/users/tonistiigi/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/tonistiigi/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/tonistiigi/subscriptions",
      "organizations_url": "https://api.github.com/users/tonistiigi/orgs",
      "repos_url": "https://api.github.com/users/tonistiigi/repos",
      "events_url": "https://api.github.com/users/tonistiigi/events{/privacy}",
      "received_events_url": "https://api.github.com/users/tonistiigi/received_events",
      "type": "User",
      "site_admin": false
    },
    "event": "closed",
    "commit_id": null,
    "commit_url": null,
    "created_at": "2016-12-29T23:09:54Z"
  }
]

So, as you can see, the source object in this case basically just contains:

type Source struct {
    Type *string `json:"type,omitempty"`
    Issue *Issue `json:`issue,omitempty``
}

Where type is set to "issue". Currently, the Source struct contains:

// Source represents a reference's source.
type Source struct {
    ID    *int    `json:"id,omitempty"`
    URL   *string `json:"url,omitempty"`
    Actor *User   `json:"actor,omitempty"`
}

This makes sense, since it matches what the API docs say about the contents of source:

The `id`, `actor`, and `url` for the source of a reference from another issue. Only provided for `cross-referenced` events.

However, in this case, the behavior of the API doesn't match what the docs say (this is, after all, an unstable beta API). The observed behavior of the API is more useful IMO, because without it, the timeline feature gives no way to actually follow cross-referenced issues (like a pull request).

I have a fix that just adds the Type and Issue fields to the Source struct. I didn't remove the other fields, since they still match what's in the API docs. With the new fields added, I can get the information I need out of an issue timeline, as expected.

Thanks, Eddy

[dmitshur]

However, in this case, the behavior of the API doesn't match what the docs say (this is, after all, an unstable beta API). The observed behavior of the API is more useful IMO, because without it, the timeline feature gives no way to actually follow cross-referenced issues (like a pull request).

Indeed, so the real response doesn't line up with the documentation in this case. I've confirmed the JSON response you posted, it happens for me too.

Do you mind contacting GitHub support about it and inquire why we're seeing such a response that is counter to the docs?

In https://developer.github.com/changes/2016-05-23-timeline-preview-api/, they wrote:

During the preview period, we may change aspects of these API methods based on developer feedback. If we do, we will announce the changes here on the developer blog, but we will not provide any advance notice.

If you have any questions or feedback, please let us know!

But I don't see any updates regarding timeline API since that post, so contacting them is our best bet.

Once GitHub responds or fixes the documentation inconsistency on their end, it'll be easier for us to fix the issue correctly and completely on our end.

[ereyes01]

Thanks for the reply, I agree- we should get Github to clarify the discrepancy for posterity's sake.

I've sent them a request for help and pointed them at this issue.

[ereyes01]

Response today from Github:

Thanks so much for reporting this -- indeed, the documentation needs to be updated to match some recent changes we made to the preview in order to make that event more helpful and consistent with other events. I'll ask the team to update the documentation.

Hopefully the doc update will make the fields I observed "official" as well as clarify what others there may be.

[dmitshur]

Sounds good. Once we have that clarification, it might allow us to update #511 to actually remove some fields that are no longer relevant. Are you okay with waiting on that PR?

[ereyes01]

Yes, we can hold off on the PR until the docs make things clearer. In the meantime, I can switch to using my fork for now to unblock my other work. Thanks!

[dmitshur]

@ereyes01, have you heard anything else from GitHub since then?

It's been 22 days. I just checked the documentation at https://developer.github.com/v3/issues/timeline/ and I'm not seeing an update there. Do you think they're still working on it, or is a friendly ping warranted?

[ereyes01]

I'll send a friendly ping...

[ereyes01]

The support person said that the ticket to update the docs remains open but hasn't yet been prioritized / resolved. I asked if it could be due to further imminent API changes but he said he isn't aware of any.

[ereyes01]

Still no movement here. If I had to guess, they probably won't prioritize this until Timeline is close to being finalized and promoted out of beta.

[gmlewis]

Hi @tarebyte - this is our oldest outstanding PR and I'm hoping to clean these up. I just browsed through the history of this PR and was wondering if you could clarify the situation for us.

In a nutshell, https://developer.github.com/v3/issues/timeline/ makes no mention of a type or issue in the source field. Here's what it says:

source: The id, actor, and url for the source of a reference from another issue. Only provided for cross-referenced events.

and yet @ereyes01 has seen type and issue in the responses.

Can we go ahead with this PR, and can the documentation be updated, please, or shall we close this old issue?

[tarebyte]

@gmlewis I can confirm both type and issue can be part of the response.

I'll open an issue to get the documentation updated.

[gmlewis]

Thank you, @tarebyte! Moving forward with PR.