Gitea: Out of Memory trouble inside LXC

[mj-saunders]

• Gitea version (or commit ref): 1.10.2-1 (Arch package)

• Git version: 2.24.1 (inside container) - 2.11.0 (on host)

• Operating system: Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64
  (though gitea is running inside an Arch instance via lxc)

• Database (use [x]):

  • [x] PostgreSQL
  • [ ] MySQL
  • [ ] MSSQL
  • [ ] SQLite

• Can you reproduce the bug at https://try.gitea.io:

  • [ ] Yes (provide example URL)
  • [x] No
  • [x] Not relevant

• Log gist: https://gist.github.com/mj-saunders/6bf0b2ab0c652d7b948dd5fd09e99f00

Description

The instance has been running fine for around a week or two now.
First noticed this problem when members were trying to create new Wiki page - received 500 error. I'd then tried a few times on several repos, with same result.
Checked logs and found OOM issue.
vm.overcommit_memory = 0 (on host)
free -mh yields:

              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       158Mi       1.8Gi        42Mi       5.0Mi       1.8Gi
Swap:            0B          0B          0B

The instance still runs, but incidentally received the same OOM stacktrace when I ran:
gitea --version

If there is any more info I can provide, please ask.

[zeripath]

The problems had started before the wiki call:

2020/01/27 02:39:44 ...uters/repo/commit.go:272:Diff() [E] commit.GetBranchName: fork/exec /usr/bin/git: cannot allocate memory
2020/01/27 02:39:45 ...ces/mirror/mirror.go:37:readAddress() [E] remoteAddress: fork/exec /usr/bin/git: cannot allocate memory
2020/01/27 02:39:45 ...ces/mirror/mirror.go:37:readAddress() [E] remoteAddress: fork/exec /usr/bin/git: cannot allocate memory

What are the preceding logs like?

My suspicion is the Diff.

[mj-saunders]

@zeripath thanks for swift response.
Annoyingly that is the beginning of the log. The previous log is from the day before and has nothing connected to this.

Here's last line from previous log.gz and a snippet from the relevent time from journal:
https://gist.github.com/mj-saunders/63d6964e5d398869aaba521fe03beaf0

Are there any other places where I might find better info for you?

[zeripath]

I missed something very interesting in your comment:

The instance still runs, but incidentally received the same OOM stacktrace when I ran:
gitea --version

WTF?

[mj-saunders]

yeah, that caught me off-guard as well.

I have no idea how to start diagnosing this.

Just tried stopping the gitea service, and ran gitea --version again. Worked fine.
Started the service, tried again and got same stacktrace.

[zeripath]

Ah so, when you ran Gitea version and it got the oom the other Gitea was still running. Fine.

Ok have you got any logs from the latest oom ? We really want to see want was breaking before the oom you see.

I have two suspicions:

• You're diffing something big - the code that does that is badly written. I know how to do it but it needs a complete rewrite and I am still stuck getting the queue rearchitecture in.
• Git is doing a repack of a large repo or several repacks at the same time and you haven't set the appropriate controls on git to restrict the size.
• Too many things are happening at the same time - mirroring etc.

[guillep2k]

--version does almost nothing (as expected), but it does go through all the modules' init() function. @zeripath have you found a connection?

[zeripath]

Nope, except that there was no memory left to do the init at that point in time. So it was not really a problem with the version test more that the machine was totally broken

[mj-saunders]

Thank you again for all of the help.

Looks like the problem is elsewhere - specifically with a searx instance (https://asciimoo.github.io/searx) that's also on the server.
Not sure why I could not see where the memory was going.

It would seem that gitea simply was tipping things over the edge.
This morning I could not even attach the gitea lxc. Checked top and finally noticed where mem was disappearing.

I stopped searx and now gitea is behaving as expected.

[zeripath]

Cools, I'll close this then.