Gogs: Cannot allocate memory when viewing repository

[floorish]

On v0.8.10 I get a 500 error whenever I want to open a repo with more than one file. I can access the other tabs (Issues/commits etc.), but the Code tab always results in an error.

[Macaron] Started GET /floorish/test for 77.*.*.*
2016/01/22 16:30:14 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/readme.md): fork/exec /usr/bin/git: cannot allocate memory
[Macaron] Completed /floorish/test 500 Internal Server Error in 88.956623ms

Just a simple test repository with a single readme.md works fine, but when I add another file I can't view the Code tab anymore.

Gogs works perfectly fine on v0.8.0 using the exact same config.

Actual memory usage looks fine as well:

[exmex]

Please test if it still happens in the latest 0.8.22 version.

[floorish]

There currently aren't any binaries available for that version right? I cannot compile it on the VPS:

$ go get -u -tags "sqlite" github.com/gogits/gogs
go build github.com/codegangsta/cli: /home/git/local/go/pkg/tool/linux_386/compile: fork/exec /home/git/local/go/pkg/tool/linux_386/compile: cannot allocate memory

And cross-compiling locally doesn't work either:

$ env GOOS=linux GOARCH=386 CGO_ENABLED=1 go build -v -tags "sqlite" 
# runtime/cgo
ld: unknown option: --build-id=none
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[unknwon]

@floorish how many files in the directory that fails?

[floorish]

When I add a second file to the repository, I can't open the code tab. When I remove the second file, commit and push with just 1 file it's fine.
So with 2+ files it fails on 0.8.10

[unknwon]

OK... how much memory do you have on VPS... I can't convert for bad math :sweat:

[floorish]

$ free -h shows 750MB, so not too much ;) (32bit Debian 7)

[unknwon]

750MB should be enough...

Gogs works perfectly fine on v0.8.0 using the exact same config.

Just to confirm you're sure about this?

[floorish]

100% looking at it right now with lots of files

[unknwon]

Thanks, got it...

[dakira]

I'm getting this from time to time, but it works after a reload. The log shows messages like these:

2016/01/29 01:26:17 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/tests): fork/exec /usr/bin/git: cannot allocate memory
2016/01/29 01:26:20 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/package.json): fork/exec /usr/bin/git: cannot allocate memory
2016/01/29 01:30:08 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/app): fork/exec /usr/bin/git: cannot allocate memory

There's more than enough memory available, though:

# free -m
             total       used       free     shared    buffers     cached
Mem:          8192       2347       5844        392          0       1455
-/+ buffers/cache:        892       7299
Swap:         4096          0       4096

[unknwon]

@dakira how many files in the directory that is having problem?

[dakira]

@Unknwon It's the root of the repo. 12 files, 8 folders.

[unknwon]

@dakira thanks, can you help confirm that 0.8.0 works fine?

[dakira]

@Unknwon I'm on 0.8.10. I'm not always getting this. When I hit reload all the time, I'm getting this in 1 out of 10 reloads.

If it helps I can set up a new installation with 0.8.0 and try with that.

[unknwon]

@dakira that would be best to try again 0.8.0, so we can be more sure about where the problem is.

[dakira]

Looks like I can't reproduce this with 0.8.0 while I can with 0.8.10 and 0.8.25 (by mercilessly hitting F5). Edit: I tried some more. This definately doesn't exist in 0.8.0.

[unknwon]

@dakira Got it, thank you!

[dakira]

Can confirm for develop, too.

[dakira]

This happens more often when there are more files and when Gogs has been running longer. It doesn't happen for a while after it has been restarted. Again with tons of memory available. Is there a workaround @Unknwon ?

[unknwon]

@dakira Are you using precompiled binary or installed from source code?

[dakira]

As I mentioned, I can also reproduce this in develop, compiled from source code (as of 7 days ago).

[floorish]

Just to confirm, on 0.8.25 (precompiled, 386) I get the same error.

I installed Gogs 0.8.25 (precompiled, amd64) on a different vps, there it works fine. They're totally different setups though, so hard to compare unfortunately.

[dakira]

@Unknwon as this is a regression, can I help somehow locating the problem if you can't reproduce yourself?

[ultimatedirty]

Same problem on 0.8.25.0129 (precompiled, amd64). Used Gogs Migration from freebsd/freebsd-ports and get those 500 errors in. Used Gogs Migration on gogits/gogs and its seems OK.

VPS: Ubuntu 14.04, 6 vCores, 8GB Ram

[dakira]

@ultimatedirty what exactly do you mean by Gogs Migration?

[ultimatedirty]

@dakira Top Menu Create -> New Migration

[dakira]

@ultimatedirty oh, okay. The error will appear for gogits/gogs if you hit F5 a couple of times. ;) Out of curiosity: what hoster are you using for your VPS?

[ultimatedirty]

@dakira Oh, I see it. My hoster is server4you.com

[unknwon]

Thanks @ultimatedirty , but the problem you encounter with is different from this one.

[NiklasRosenstein]

I've had the same issue yesterday. My one VPS has 2GB RAM and it couldn't handle the 7MB Repository with 1000 commits. Just as you described in the initial issue description, it showed a 500 Error when browsing the code, but I could visit any other page. The log also mentioned failure to allocate memory.

Now interestingly, the same repository runs smoothly with my other VPS which has 1GB of RAM only. I'm investigating

[dakira]

@ultimatedirty server4you uses Virtuozzo for their VPSs, right?

[unknwon]

@dakira just push some code to indicate this problem, please do go get -u github.com/gogits/git-module and recompile Gogs to help test!

[unknwon]

According to https://github.com/docker/docker/issues/8539#issuecomment-75291557 , the problem could be the Swap is 0.

Anyone can verify that by give some swap?

[floorish]

Well I can confirm that the server has no swap, but I can't add more swap (no permissions, possibly due to OpenVZ?). The other server does have swap and works perfectly.

[NiklasRosenstein]

Both of my servers don't have a swap, not the 2GB neither the 1GB one. As I mentioned above, the same repository "works" (ie. you can view the code) one the server with _less_ RAM. There is also a branch with only ~200 commits in the same repository (very outdated) and that _can_ be viewed on the 2GB server, so it seems like its not about the overall repository size, but the size of the history that needs to be viewed.

On a side note, the Git source code repsitory with ~40k commits is also too much to handle for my 1GB VPS. (who would have expected that.)

I'm no Linux expert, so I'm unsure for what differences of the two servers I could still look into. The 2GB server is a pretty cheap one though, I got it basically for free as an offer to students. Maybe they had some notes that you actually not always get 2 GB of RAM or whatever..

[unknwon]

https://try.gogs.io is running on DO $20 w/ 2GB RAM as well, but it has 2GM Swap.

[dakira]

@Unknwon I have the new build running now. I'm not sure if you already changed something but I can't reproduce the problem at the moment. I will test again with an older release if I still get the problem there.

[ultimatedirty]

FYI: I can't reproduce the problem, too. Updated to 0.8.39.0219. Now it's just horribly slow while browsing the repo. 6 vCores on 100% workload for about 8 minutes and it's still running.

[unknwon]

@ultimatedirty yes, limit 10 concurrent processes right now...

[NiklasRosenstein]

Did any optimizations on this happen in between 8.25 and 8.42 (of which the latter is currently on http://try.gogs.io) ? Unfortunately, I didn't test it on the Gogs Try server before the update. I've mirrored the Git source code here: https://try.gogs.io/niklas/git and it works. It's very slow, but it works (page load ~20s).

[unknwon]

@NiklasRosenstein yes, in the 0.8.42, Gogs no longer uses unlimited subprocesses, but limited to 10 pre time at max. Which is slow obviously, but does the job right.

[NiklasRosenstein]

@Unknwon Oh that is what your previous comment was about! Sorry. Do you think there is room for improving the performance now that the memory issue is solved? Of course fewer processes means slower results, but maybe there's something else that can be looked into.

PS: Is the number of maximum subprocesses configurable?

[unknwon]

@NiklasRosenstein yes, I'm thinking of that, also application cache layer could also improve performance a lot as well. (Gogs has zero cache for data right now, when we have cache, it should feel even faster for most of repo views)

[dakira]

@Unknwon awesome. Maybe something like russion doll caching would be interesting for Gogs.

[unknwon]

@dakira Mostly likely (simplest) would be LRU cache? At least for the first version I think.

[ddevault]

Can reproduce

2016/02/25 14:47:21 [...routers/repo/view.go:134 Home()] [E] GetCommitsInfo: GetCommitByPath (/arm.gpg): pipe2: too many open files

[dakira]

@SirCmpwn with which version (in the lower left of the screen). I have _"© 2016 Gogs Version: 0.8.44.0225"_ and don't experience this anymore.

[ddevault]

0.8.10.1217. I'll update and report back.

[ddevault]

Yep, fixed.

[floorish]

Not sure if this is supposed to be fixed, but I just tried binary 0.8.43.0223 and still same problem after adding 2-3 files (consistently):

[...routers/repo/view.go:136 Home()] [E] GetCommitsInfo: GetCommitByPath (/test3): fork/exec /usr/bin/git: cannot allocate memory

[unknwon]

@floorish it is supposed to at least reduce the possibility of occurrence, but we can't reproduce you situation right now... :sweat:

[floorish]

Ah right, got it. But don't worry about my situation, I have it installed on a different server and that works perfect :)

[unknwon]

@floorish that is a good news :D

[ghost]

I'm not allowed to install swap on my VPS and still have the described problems on 0.8.43.0223.
dakira said he doesn't experience it anymore on 0.8.44.0225.
Where can I get this version? I would like to install from binary. Can someone please upload it?

[unknwon]

@YugoCode the same error doesn't mean same cause, you need to first make sure your VPS has enough free memory.

In terms of this issue, there is no change between 0.8.43 and 0.8.44.

[ghost]

Thanks for answering :)
free -h shows:
total used free shared buffers cached
2.0G 225M 1.8G 68M 0B 165M

[unknwon]

@YugoCode thanks for the prove!

[NiklasRosenstein]

I just upgraded to 0.9.13.0318 and it seems to run well, though it feels very slow. When I go to the "Code" page to view the files on the master branch, the page loads in ~1800ms. However, when I go to view a single source file, it loads in ~400ms.

Is this difference in the page load time expected? Note that I do not know how Gogs actually gets the data from git. (Does it check out the whole working tree to get a list of the files?)

[unknwon]

@NiklasRosenstein https://github.com/gogits/gogs/issues/1518

[dakira]

This issue is back in 0.9.20.0404.

[Pastew]

Hi, I'm new to Gogs but I have the same error. I'm opening 10MB, one-commit newly pushed repo and I have 500 Internal Server Error in browser and in console:
[...routers/repo/view.go:136 Home()] [E] GetCommitsInfo: GetCommitByPath (/.idea): fork/exec /usr/bin/git: cannot allocate memory

Gogs version 0.9.13.0318

top output:

top - 14:45:28 up 49 min,  2 users,  load average: 0.04, 0.01, 0.00
Tasks:  40 total,   1 running,  39 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  1.1 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1048576 total,   308760 used,   739816 free,        0 buffers
KiB Swap:   524288 total,        0 used,   524288 free.   229948 cached Mem

[lucianolorenti]

I ran into the same problem with my server. Apparently it is related with the number of concurrent tasks performed in GetCommitsInfo of the Entries type: tree_entry.go..
I changed
taskChan := make(chan bool, 10)
to
taskChan := make(chan bool, 2)

and,for now, I can not cause the error again.

[bkcsoft]

@Unknwon Maybe the amount of goroutines for this should be configurable?
how hard would it be to do that ? (without passing it as an argument on
every single call... :unamused: )

[unknwon]

@bkcsoft yes, I think it is possible as a config option.

[y4my4my4m]

@Unknwon
Getting this too in 0.9.28.0527 almost 50% of the time.
Only getting this error viewing the root of the repository (which is owned by an organization)

Log

2016/06/29 14:24:19 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/app.js): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:20 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/node_modules): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:24 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:24:29 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:46 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/public): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:52 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/package.json): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:25:54 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/views): fork/exec /usr/bin/git: cannot allocate memory
2016/06/29 14:26:20 [...routers/repo/view.go:138 Home()] [E] GetCommitsInfo: GetCommitByPath (/public): fork/exec /usr/bin/git: cannot allocate memory

Memory:

              total        used        free      shared  buff/cache   available
Mem:           2.0G         44M        1.4G        162M        534M        1.7G
Swap:          4.0G          0B        4.0G

Any help?

I'm running on CentOS 7 and MySQL.

---

Edit:
taskChan := make(chan bool, 2)
seems to help, i was using centos7 packager, but now im building from scratch...just feels a bit unstable somehow.

Also noticed you've added
LimitMEMLOCK=infinity
LimitNOFILE=65535
To the gogs.service

Does this really work?

[unknwon]

@ubiko sorry, but really have no ideas how this problem is caused... maybe there are some limitation options we're not noticed because this is not happening to all machines (that's the hardest part for debugging).

[y4my4my4m]

@Unknwon

taskChan := make(chan bool, 2) is fixing it...but does it mean it's slower than if it was set at 10?

[unknwon]

@ubiko yes, that's right...

[bkcsoft]

This is most likely related to ulimit, there's a couple of issues about
that already, check them out :smiley:

[ghost]

@ubiko The line of code you posted is fixing the problem??? I don't care if it's slow, I need it to work at least. Where do I have to put the line? Or could you build a binary with this fixed line?

Thank you :)

[sjahreis]

@bkcsoft In my case it's not related to ulimit. No limitations configured...

Please add as config option!

[y4my4my4m]

@Unknown no worries, it's a pretty hard issue to figure out also. I gotta thank you for making Gogs before I'd complain. :+1:

@YugoCode Yes it is fixing it. It's more of a workaround than a fix though, credit goes to: @lucianolorenti (Higher up in the comments) . You can fix it by modifying tree_entry.go . You need to work from the source, you can't fix this if you're using a docker/packager. You need to build the app again.
Good luck.

[toudi]

I was also affected by this issue on the VPS. I am wondering whether maybe this has something to do with number of cores available? Because on my VPS I got this 500, but on my OrangePi gogs runs flawlessly and I don't have this error. The only difference seems to be in the architecture (VPS is i386 vs arm7 on the OrangePi) and the number of cores (OrangePi has 4 cores whereas my VPS has only one).

If this is false impression, please tell me and I'll remove my comment so that I don't introduce unnecesary mess.

Thank you so much for gogs!

[botboe]

As @sjahreis already mentioned, a config option would be awesome!

[botboe]

Something new here?

[y4my4my4m]

@toudi @Mad-Onion
I believe it has to do with available Swap memory on your VPS setup.
Either increase swap capacity on your VPS (if your host allows this) or your setup, you might be out of luck (as I was, decided to change environment)

[WolfgangKluge]

There's the pull request gogits/git-module#26 from 0xbaadf00d, but unfortunetely no progress since one month...

[dakira]

@WolfgangKluge This project seems to be dead. There is an extremely active fork here: https://github.com/go-gitea/gitea. I've been asked to wait for the 1.0 release before I switch over. But that should be soon.

[unknwon]

Please notify when this happens again.

[paradajz]

Happens to me as well. CentOS 6.8. Seemingly random behaviour. Latest Gogs. Let me know if you need more details.

[extraymond]

Happened in a docker container on google compute engine.
Hope this gets fixed, thx.