Gitea: Gitea growing memory usage when unused

[jmn]

• Gitea version (or commit ref): 1.0.0+133-gd2bb8ef5
• Operating system: Docker image, gitea/gitea@sha256:dd2212f37ca2f4db03f7018fad062adbe8801f1c96d77af9c10d245e97e0a99d
• Database (use [x]):
  
  
  
  • [ ] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [x] SQLite
  
  

Description

I am since yesterday measuring memory usage on Gitea running with no user activity and here is a graph showing memory usage and the memory usage is growing what I view as quite heavily over time. Perhaps all is in order, perhaps I have not configured something correctly.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[thibaultmeyer]

Do you have the same issue with another Database backend ?

[jmn]

Very good question, will try and check that and come back.

[metalmatze]

That pretty much looks like a memory leak. Would be interesting to start debugging with pprof, but one would have to include that themselves IIRC.

[jmn]

OK the problem seems to have been that I was just naively using the default configuration of sessions in the gitea.ini from the image gitea/gitea@sha256:dd2212f37ca2f4db03f7018fad062adbe8801f1c96d77af9c10d245e97e0a99d

and it turned out that sessions file storage very quickly grew out of hand because no SESSION_LIFE_TIME or GC_INTERVAL_TIME was set.

[session]
; Either "memory", "file", or "redis", default is "memory"
PROVIDER = file
PROVIDER_CONFIG = data/sessions
; Session GC time interval, default is 86400
; GC_INTERVAL_TIME = 86400
; Session life time, default is 86400
; SESSION_LIFE_TIME = 86400

[bkcsoft]

@metalmatze Would be fairly simple to add pprof-http-service just like with prometheus, though would obviously need to be locked down 😛

[metalmatze]

Yes. I now tried to find something on my local system, but I'm not able to.
Probably some config flag to turn on pprof listening on http://localhost:6060 would be nice, I guess.

[lunny]

@jmn so this is only a configuration problem?

[metalmatze]

I started to add pprof to gitea yesterday and will continue to make a PR today evening. I would really like you to check gitea with pprof after a day running or so. I will do the same.

[thibaultmeyer]

[OFF-TOPIC] Nice chart @jmn, what monitoring tools are your using?

[metalmatze]

The UI is definitely grafana. (github)

[jmn]

@jmn so this is only a configuration problem?

I think so yes. However, I think it would be good to try and provide default values and recommendations which try to prevent users from shooting themselves in the foot. E.g. not unlimited session lifetime with memory storage as default.

[lunny]

I will move this to v1.2 since no further information here.

[DrMegavolt]

It is not only the case for memory provider
If you set file provider. Gitea will start creating a ton of session files. RAM is fine in this case but it quickly takes all inodes available (df -i shows 100% )

[DrMegavolt]

And to clarify we have a lot of AdminAPI calls and access to git via ssh. Nobody really uses web admin. The question is why sessions are needed at all?

[bkcsoft]

@DrMegavolt That would be a separate issue (even though slightly related). Please create one so we can track them separetely :slightly_smiling_face:

[lunny]

Maybe related with #4450 ?

[lafriks]

@lunny I don't think that is related
@jmn is this still a problem in latest version

[lafriks]

I'm closing issue currently, please reopen if it is still a problem in latest version

[poVoq]

Running with Docker and Mysql I seem to have a similar issue, both on 1.9.3 and the "latest" image. Might have been there before, but I didn't notice.
I don't have a nice graphic for it, but within a few days the memory use of the container doubles.

[kfreidank]

Just installed built and installed 1.11.5 on Rasbian Pi3. All memory is immediately used. No sessions in process. Just one user account added, no repositories. Postgres on the backend.psql (PostgreSQL) 11.7 (Raspbian 11.7-0+deb10u1)
go version go1.11.6 linux/arm
Powered by Gitea Version: 1.11.5+11-ga854846f0

top - 13:20:46 up 5 days, 44 min, 1 user, load average: 0.00, 0.03, 0.00
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.6 sy, 0.0 ni, 98.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 926.1 total, 95.9 free, 200.9 used, 629.3 buff/cache
MiB Swap: 1872.0 total, 1846.2 free, 25.8 used. 628.0 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1689 couchdb 20 0 117156 30560 7892 S 1.3 3.2 195:35.94 beam.smp
19027 solomon 20 0 10292 2960 2460 R 1.0 0.3 0:01.71 top
24945 gitea 20 0 1001208 122304 40100 S 1.0 12.9 18:28.23 gitea <----------- took it all
21 root 20 0 0 0 0 S 0.3 0.0 0:11.15 ksoftirqd/2

[Flashlord]

I confirm gitea took 380Mb of memory without doing anything !

[fanlix]

330M mem usage for fresh install

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
12789 pi        20   0 1068880 331784  69200 S   0.7  17.5   0:14.17 /app/gitea/gitea web 

600M mem usage after first push.

• repo of 100 commits and total size 30M
• call go gc

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
12789 pi        20   0 1339744 599676  70492 S   1.3  31.6   0:47.65 /app/gitea/gitea web

admin-usage

System Status
Server Uptime
22 minutes, 33 seconds
Current Goroutines
112
Current Memory Usage
205 MiB
Total Memory Allocated
2.8 GiB
Memory Obtained
604 MiB
Pointer Lookup Times
0
Memory Allocations
3924943
Memory Frees
3083970
Current Heap Usage
205 MiB
Heap Memory Obtained
575 MiB
Heap Memory Idle
355 MiB
Heap Memory In Use
220 MiB
Heap Memory Released
123 MiB
Heap Objects
840973
Bootstrap Stack Usage
1.2 MiB
Stack Memory Obtained
1.2 MiB
MSpan Structures Usage
1.2 MiB
MSpan Structures Obtained
1.7 MiB
MCache Structures Usage
6.8 KiB
MCache Structures Obtained
16 KiB
Profiling Bucket Hash Table Obtained
1.6 MiB
GC Metadata Obtained
24 MiB
Other System Allocation Obtained
1.1 MiB
Next GC Recycle
401 MiB
Since Last GC Time
112.0s
Total GC Pause
0.0s
Last GC Pause
0.000s
GC Times
30

• pi4b 2Gb raspbian-64bit-2020.8
• gitea 1.14.0 sqlite

[zeripath]

If you really want to investigate:

ENABLE_PPROF = true in [server] and use:

go tool pprof http://localhost:6060/debug/pprof/heap

and it's web function to show where the memory is being used.

Then compare between the two.