Go-github: Caching is not working properly with Repositories.GetContents

[m4ns0ur]

_Not sure if it's go-github issue or httpcache one._

I'm trying to cache Repositories.GetContents responses to avoid my rate limit's burning.

package main

import (
    "context"
    "fmt"

    "github.com/google/go-github/github"
    "github.com/gregjones/httpcache"
    "github.com/gregjones/httpcache/diskcache"
)

func main() {
    cl := github.NewClient(httpcache.NewTransport(diskcache.New(".")).Client())
    _, _, r, _ := cl.Repositories.GetContents(context.Background(), "google", "go-github", "LICENSE", nil)
    fmt.Println(r)
}

At first run (diskcache file is going to be created for the first time), it works

$ date && go run main.go 
Thu 23 Apr 2020 02:01:04 PM CEST
&{0xc00014c090 0 0 0 0  github.Rate{Limit:60, Remaining:5, Reset:github.Timestamp{2020-04-23 14:20:15 +0200 CEST}}}
...
$ date && go run main.go 
Thu 23 Apr 2020 02:01:27 PM CEST
&{0xc000114000 0 0 0 0  github.Rate{Limit:60, Remaining:5, Reset:github.Timestamp{2020-04-23 14:20:15 +0200 CEST}}}

But if you run it again, after 60s (Github API's max-age), it doesn't work anymore

$ date && go run main.go 
Thu 23 Apr 2020 02:02:17 PM CEST
&{0xc000112000 0 0 0 0  github.Rate{Limit:60, Remaining:4, Reset:github.Timestamp{2020-04-23 14:20:14 +0200 CEST}}}
...
$ date && go run main.go 
Thu 23 Apr 2020 02:02:19 PM CEST
&{0xc00018a000 0 0 0 0  github.Rate{Limit:60, Remaining:3, Reset:github.Timestamp{2020-04-23 14:20:14 +0200 CEST}}}

[gmlewis]

I see you already found #242 (where I probably would have put this issue).
I've never personally used httpcache and I see in #242 that others have had challenges with it.

I would suggest contacting the httpcache author and see what he recommends, then please report back here (or close this issue and report back in #242... either way is fine with me).

[m4ns0ur]

@gmlewis sure! let me try httpcache.
I'll keep this open since #242 could be different.

[m4ns0ur]

Update:
httpcache is lazy, and invalidates the cache after full http.body read. The issue is, we never get EOF here in code snippet even though go-github reads the full body.

At first place it was really confusing because it works occasionally! then I noticed, sometimes Github API server sends Transfer-encoding: chunked. Whenever we have chunked, it uses chunkedReader, gets EOF, and invalidates the cache. But without chunked, it uses bufio reader, and misses EOF.

An easy workaround for httpcache would be checking the length beside EOF here.

func (r *cachingReadCloser) Read(p []byte) (n int, err error) {
    n, err = r.R.Read(p)
    r.buf.Write(p[:n])
    if err == io.EOF || n < len(p) {
        r.OnEOF(bytes.NewReader(r.buf.Bytes()))
    }
    return n, err
}

But anyways we should check why bufio doesn't handle EOF.

[gmlewis]

Maybe you could submit a PR to the httpcache repo with the workaround?

Alternatively (or in addition :smile:), you could submit a PR to this repo with a unit test demonstrating the bufio EOF problem along with the fix (if I understand you correctly).

Third option - if you could put a tiny-as-possible test case into this issue that demonstrates the bufio EOF problem, I could start looking into a fix.

Thanks for the investigation.

[m4ns0ur]

The bufio issue could be in Go, not in go-github (I'm looking into it).
With fixing bufio there is no need to send any PR to httpcache.

[m4ns0ur]

It's not a Go issue because document says EOF is optional. I'm sending a PR to fix the issue in httpcache.

Since issue is happening inside JSON decoder (where we read the body), we cannot have a test/PR in this repo, sorry :)

[gmlewis]

OK, sounds good to me. Thanks, @m4ns0ur.
Closing issue.

[m4ns0ur]

For reference: https://github.com/gregjones/httpcache/pull/104

[gmlewis]

@m4ns0ur - would that PR you reference also resolve our #242?

[gmlewis]

If so, would you mind referencing your PR in that issue as well?

[m4ns0ur]

@gmlewis I don't think so.