Beego: memroy leak problem with static dir

[argentmoon]

GO 1.6.2 amd64
Beego 1.6.1

hi all,

I putted images in a directory, and set it as a static directory by "beego.SetStaticPath(..., ...)"

When requests for these images, memory seems leaks. Since the size of every image is just 100~200KB, but after thousands of requests, memory usage reached about 100M。

So I start memory pprof, and find the path of max memory usage :
1.net/http.(_conn).serve
2.net/http.serverHandler.ServerHTTP
3.github.com/astaxie/beego.(_ControllerRegister).ServerHTTP
4.github.com/astaxie/beego.seererStaticRouter
5.github.com/astaxie/beego.openFile
6.github.com/astaxie/beego/context.WriteFile
7.io.Copy
8.io.copyBuffer
9.github.com/astaxie/beego/context.(_nopRestWriter).Write
10.bytes.(_Buffer).Write
11.bytes.(*Buffer).grow
12.bytes.makeSlice

Well, something wrong in golang's bytes.makeSlice !

I have no choice but try to write a single controller to read and write these images, like this:
bts, err := ioutil.ReadFile(image path)
io.Copy(c.Ctx.ResponseWriter, bytes.NewBuffer(bts))

And It helps! Now, the memory usage performs quite normal.

Hope some helps for beego, thx.

[jelmerdereus]

I'm having a look at some relevant pieces from /context/acceptencoder.go

// WriteFile reads from file and writes to writer by the specific encoding(gzip/deflate)

func WriteFile(encoding string, writer io.Writer, file *os.File) (bool, string, error) {
    return writeLevel(encoding, writer, file, flate.BestCompression)
}

// writeLevel reads from reader,writes to writer by specific encoding and compress level
// the compress level is defined by deflate package

func writeLevel(encoding string, writer io.Writer, reader io.Reader, level int) (bool, string, error) {
    var outputWriter resetWriter
    var err error
    var ce = noneCompressEncoder

    if cf, ok := encoderMap[encoding]; ok {
        ce = cf
    }
    encoding = ce.name
    outputWriter = ce.encode(writer, level)
    defer ce.put(outputWriter, level)

    _, err = io.Copy(outputWriter, reader)
    if err != nil {
        return false, "", err
    }

    switch outputWriter.(type) {
    case io.WriteCloser:
        outputWriter.(io.WriteCloser).Close()
    }
    return encoding != "", encoding, nil
}

a noneCompressEncoder is an acceptEncoder

type acceptEncoder struct {
    name                    string
    levelEncode             func(int) resetWriter
    customCompressLevelPool *sync.Pool
    bestCompressionPool     *sync.Pool
}

noneCompressEncoder = acceptEncoder{"", nil, nil, nil}

and when defer ce.put(outputWriter, level) is called, the following function is called

`````` go
func (ac acceptEncoder) put(wr resetWriter, level int) {
if ac.customCompressLevelPool == nil || ac.bestCompressionPool == nil {
return
}
wr.Reset(nil)

//notice
//compressionLevel==BestCompression DOES NOT MATTER
//sync.Pool will not memory leak**

switch level {
case gzipCompressLevel:
    ac.customCompressLevelPool.Put(wr)
case flate.BestCompression:
    ac.bestCompressionPool.Put(wr)
}

}```
``````

[astaxie]

@JessonChan please check this issue

[youngsterxyf]

@jelmerdereus Do you request one image many times or many different images when you found memory leak? In Beego, static file will be loaded into memory to speed up next all request for that static file.

[argentmoon]

@youngsterxyf many different images

Did you mean it's just a buffer scheme?

[youngsterxyf]

@argentmoon yes.

In file staticfile.go

func openFile(filePath string, fi os.FileInfo, acceptEncoding string) (bool, string, *serveContentHolder, error) {
    mapKey := acceptEncoding + ":" + filePath
    mapLock.RLock()
    mapFile, _ := staticFileMap[mapKey]
    mapLock.RUnlock()
    if isOk(mapFile, fi) {
        return mapFile.encoding != "", mapFile.encoding, mapFile, nil
    }
    mapLock.Lock()
    defer mapLock.Unlock()
    if mapFile, _ = staticFileMap[mapKey]; !isOk(mapFile, fi) {
        file, err := os.Open(filePath)
        if err != nil {
            return false, "", nil, err
        }
        defer file.Close()
        var bufferWriter bytes.Buffer
        _, n, err := context.WriteFile(acceptEncoding, &bufferWriter, file)
        if err != nil {
            return false, "", nil, err
        }
        mapFile = &serveContentHolder{Reader: bytes.NewReader(bufferWriter.Bytes()), modTime: fi.ModTime(), size: int64(bufferWriter.Len()), encoding: n}
        staticFileMap[mapKey] = mapFile
    }

    return mapFile.encoding != "", mapFile.encoding, mapFile, nil
}

this staticFileMap is a memory cache.

[argentmoon]

@youngsterxyf thx a lot.

I think I used the static dir for a wrong place.

[youngsterxyf]

@astaxie Over

[richardweinberger]

Guys, your staticFileMap map is not a cache, it is a memory leak. Because the cache has no upper bound and never evicts entries.

So please make it very clear in your documentation that Beego's static files handler can only be used for trivial cases where you serve a few css/js files which can rest in memory for ever.
For anything else it will end up being a DoS vector.