$uname -srmo : Linux 5.6.7-1-ARCH aarch64 GNU/Linux
)[x]
):I run a private gitea instance with the systemd service of the archlinux package.
I am the only user on this instance. I have I think 6 quite small repos.
I locally created a new branch in a repo.
I pushed it by executing : git push origin {BranchName}
Gitea has been restarting every 20 seconds since, and generating tons of coredumps (following longer intervals are due to reboots and config tweaks that did not help) :
$ sudo coredumpctl
TIME PID UID GID SIG COREFILE EXE
Mon 2020-04-27 23:59:57 CEST 856 973 973 31 missing /usr/bin/gitea
Tue 2020-04-28 00:01:51 CEST 958 973 973 31 missing /usr/bin/gitea
Tue 2020-04-28 00:09:17 CEST 1139 973 973 31 missing /usr/bin/gitea
Tue 2020-04-28 00:25:34 CEST 1344 973 973 31 missing /usr/bin/gitea
Tue 2020-04-28 00:25:55 CEST 1375 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 00:58:50 CEST 1046 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 00:58:59 CEST 1074 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:07:54 CEST 1174 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:08:47 CEST 1212 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:10:03 CEST 1257 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:13:44 CEST 1288 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:13:54 CEST 1322 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:20:49 CEST 1384 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:21:38 CEST 1419 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:56:11 CEST 2937 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 01:59:21 CEST 2964 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 02:01:40 CEST 2994 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 02:02:17 CEST 3017 973 973 31 present /usr/bin/gitea
Tue 2020-04-28 02:02:29 CEST 3041 973 973 31 present /usr/bin/gitea
I tried to gdb the coredumps but I think its output is not very useful :
(sorry, this is the first time I ever launch gdb, I may not be very aware of what's useful or not. Saw it in another issue ...)
$ sudo coredumpctl gdb 3017
PID: 3017 (gitea)
UID: 973 (gitea)
GID: 973 (gitea)
Signal: 31 (SYS)
Timestamp: Tue 2020-04-28 01:58:01 CEST (29min ago)
Command Line: /usr/bin/gitea web -c /etc/gitea/app.ini
Executable: /usr/bin/gitea
Control Group: /system.slice/gitea.service
Unit: gitea.service
Slice: system.slice
Boot ID: b2c2901b20dd4532adcc64fe2f8f5ef3
Machine ID: e9924b3339eb455fbe8473f0fb3250c3
Hostname: rpi3
Storage: /var/lib/systemd/coredump/core.gitea.973.b2c2901b20dd4532adcc64fe2f8f5ef3.3017.1588031881000000000000.lz4
Message: Process 3017 (gitea) of user 973 dumped core.
Stack trace of thread 3033:
#0 0x0000aaaadb4ffcd0 n/a (gitea + 0x9e5cd0)
#1 0x0000aaaadb4fe224 n/a (gitea + 0x9e4224)
#2 0x0000aaaadb595fc4 n/a (gitea + 0xa7bfc4)
#3 0x0000aaaadb5a8d40 n/a (gitea + 0xa8ed40)
#4 0x0000aaaadb4cc1c4 n/a (gitea + 0x9b21c4)
#5 0x0000aaaadb597b70 n/a (gitea + 0xa7db70)
#6 0x0000aaaadb5a6054 n/a (gitea + 0xa8c054)
#7 0x0000aaaadb5a6710 n/a (gitea + 0xa8c710)
#8 0x0000aaaadb4b40a4 n/a (gitea + 0x99a0a4)
GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/gitea...
(No debugging symbols found in /usr/bin/gitea)
[New LWP 3033]
[New LWP 3019]
[New LWP 3020]
[New LWP 3023]
[New LWP 3022]
[New LWP 3021]
[New LWP 3024]
[New LWP 3017]
[New LWP 3025]
[New LWP 3026]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/bin/gitea web -c /etc/gitea/app.ini'.
Program terminated with signal SIGSYS, Bad system call.
#0 0x0000aaaadb4ffcd0 in ?? ()
[Current thread is 1 (Thread 0xffff6ffff1a0 (LWP 3033))]
(gdb) bt
#0 0x0000aaaadb4ffcd0 in ?? ()
#1 0x0000000000000007 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
I thinks something broke during the push command, and now gitea is trying to redo this at boot and constantly failing ?
If I can give you some more info, just ask !
Not relevant
Is the last line of the logs genuinely:
2020/04/28 01:59:25 ...er/issues/indexer.go:142:func2() [I] PID 3041: Initializing Issue Indexer: bleve
?
If so then you could try just deleting the indexer files - Gitea will just recreate them.
The other thing is to try to run git gc across your repos from the command line. You probably need to set your .gitconfig options for pack.packSizeLimit and related options.
Thanks for your quick answer !
Yes it is, and the log then reset and coredump etc...
2020/04/28 01:59:25 ...er/issues/indexer.go:142:func2() [I] PID 3041: Initializing Issue Indexer: bleve
is always the last log line.
I have the following :
# pwd
/var/lib/gitea/indexers
# ls -la
total 16
drwxr-x--- 4 gitea gitea 4096 May 18 2019 .
drwxr-xr-x 8 gitea gitea 4096 Apr 28 01:59 ..
drwx------ 2 gitea gitea 4096 May 18 2019 issues.bleve
drwxr-xr-x 2 gitea gitea 4096 Feb 13 01:56 issues.queue
Just to confirm that I understood correctly, you're telling me that I can delete them safely ?
I did both :
git gc
of all repos : no luckI join the journalctl message that shows the same error after both try :
Apr 28 03:18:30 rpi3 gitea[3726]: 2020/04/28 03:18:28 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column data db default is 'NULL', struct default is
Apr 28 03:18:30 rpi3 gitea[3726]: 2020/04/28 03:18:28 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column created_unix db default is NULL, struct default is
Apr 28 03:18:30 rpi3 gitea[3726]: 2020/04/28 03:18:28 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column updated_unix db default is NULL, struct default is
Apr 28 03:18:30 rpi3 gitea[3726]: 2020/04/28 03:18:28 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column expires_unix db default is NULL, struct default is
Apr 28 03:18:31 rpi3 gitea[3726]: 2020/04/28 03:18:31 ...er/issues/indexer.go:142:func2() [I] PID 3726: Initializing Issue Indexer: bleve
Apr 28 03:18:31 rpi3 audit[3726]: SECCOMP auid=4294967295 uid=973 gid=973 ses=4294967295 pid=3726 comm="gitea" exe="/usr/bin/gitea" sig=31 arch=c00000b7 syscall=163 compat=0 ip=0xaaaae602fcd0 code=0x80000000
Apr 28 03:18:31 rpi3 audit[3726]: ANOM_ABEND auid=4294967295 uid=973 gid=973 ses=4294967295 pid=3726 comm="gitea" exe="/usr/bin/gitea" sig=31 res=1
Apr 28 03:18:31 rpi3 kernel: audit: type=1326 audit(1588036711.068:1046): auid=4294967295 uid=973 gid=973 ses=4294967295 pid=3726 comm="gitea" exe="/usr/bin/gitea" sig=31 arch=c00000b7 syscall=163 compat=0 ip=0xaaaae602fcd0 code=0x800000>
Apr 28 03:18:31 rpi3 kernel: audit: type=1701 audit(1588036711.068:1047): auid=4294967295 uid=973 gid=973 ses=4294967295 pid=3726 comm="gitea" exe="/usr/bin/gitea" sig=31 res=1
Apr 28 03:18:31 rpi3 audit: AUDIT1334 prog-id=52 op=LOAD
Apr 28 03:18:31 rpi3 audit: AUDIT1334 prog-id=53 op=LOAD
Apr 28 03:18:31 rpi3 systemd[1]: Started Process Core Dump (PID 3747/UID 0).
Apr 28 03:18:31 rpi3 kernel: audit: type=1334 audit(1588036711.178:1048): prog-id=52 op=LOAD
Apr 28 03:18:31 rpi3 kernel: audit: type=1334 audit(1588036711.188:1049): prog-id=53 op=LOAD
Apr 28 03:18:31 rpi3 kernel: audit: type=1130 audit(1588036711.198:1050): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@20-3747-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=succ>
Apr 28 03:18:31 rpi3 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@20-3747-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
I'm currently trying to run gitea in gdb but I'm stuck with permission errors :
$sudo -u gitea gdb --args gitea web -c /etc/gitea/app.ini
... (same as system service log)
2020/04/28 14:55:42 routers/init.go:99:GlobalInit() [I] ORM engine initialization successful!
2020/04/28 14:55:42 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column data db default is 'NULL', struct default is
2020/04/28 14:55:42 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column created_unix db default is NULL, struct default is
2020/04/28 14:55:42 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column updated_unix db default is NULL, struct default is
2020/04/28 14:55:42 [email protected]/engine.go:1352:Sync2() [W] Table oauth2_session Column expires_unix db default is NULL, struct default is
2020/04/28 14:55:43 ...les/queue/setting.go:60:CreateQueue() [W] Unable to create queue for issue_indexer: mkdir data: permission denied
2020/04/28 14:55:43 ...les/queue/setting.go:61:CreateQueue() [W] Attempting to create wrapped queue
2020/04/28 14:55:43 ...les/queue/setting.go:60:CreateQueue() [W] Unable to create queue for task: mkdir data: permission denied
2020/04/28 14:55:43 ...les/queue/setting.go:61:CreateQueue() [W] Attempting to create wrapped queue
2020/04/28 14:55:43 ...er/issues/indexer.go:142:func2() [I] PID 5524: Initializing Issue Indexer: bleve
2020/04/28 14:55:43 routers/init.go:122:GlobalInit() [I] SQLite3 Supported
2020/04/28 14:55:43 ...eue/queue_wrapped.go:67:setInternal() [W] [Attempt: 1] Failed to create queue: level for cfg: [123 34 65 100 100 114 101 115 115 101 115 34 58 34 49 50 55 46 48 46 48 46 49 58 54 51 55 57 34 44 34 66 97 116 99 104 76 101 110 103 116 104 34 58 50 48 44 34 66 108 111 99 107 84 105 109 101 111 117 116 34 58 49 48 48 48 48 48 48 48 48 48 44 34 66 111 111 115 116 84 105 109 101 111 117 116 34 58 51 48 48 48 48 48 48 48 48 48 48 48 44 34 66 111 111 115 116 87 111 114 107 101 114 115 34 58 53 44 34 68 66 73 110 100 101 120 34 58 48 44 34 68 97 116 97 68 105 114 34 58 34 100 97 116 97 47 100 97 116 97 47 113 117 101 117 101 115 47 116 97 115 107 34 44 34 77 97 120 87 111 114 107 101 114 115 34 58 49 48 44 34 78 97 109 101 34 58 34 116 97 115 107 34 44 34 78 101 116 119 111 114 107 34 58 34 34 44 34 80 97 115 115 119 111 114 100 34 58 34 34 44 34 81 117 101 117 101 76 101 110 103 116 104 34 58 50 48 44 34 81 117 101 117 101 78 97 109 101 34 58 34 116 97 115 107 95 113 117 101 117 101 34 44 34 87 111 114 107 101 114 115 34 58 49 125] error: mkdir data: permission denied
2020/04/28 14:55:43 routers/init.go:46:checkRunMode() [I] Run Mode: Production
2020/04/28 14:55:43 ...er/issues/indexer.go:199:func3() [I] Issue Indexer Initialization took 13.938819ms
2020/04/28 14:55:43 ...eue/queue_wrapped.go:67:setInternal() [W] [Attempt: 1] Failed to create queue: level for cfg: [123 34 65 100 100 114 101 115 115 101 115 34 58 34 49 50 55 46 48 46 48 46 49 58 54 51 55 57 34 44 34 66 97 116 99 104 76 101 110 103 116 104 34 58 50 48 44 34 66 108 111 99 107 84 105 109 101 111 117 116 34 58 49 48 48 48 48 48 48 48 48 48 44 34 66 111 111 115 116 84 105 109 101 111 117 116 34 58 51 48 48 48 48 48 48 48 48 48 48 48 44 34 66 111 111 115 116 87 111 114 107 101 114 115 34 58 53 44 34 68 66 73 110 100 101 120 34 58 48 44 34 68 97 116 97 68 105 114 34 58 34 100 97 116 97 47 100 97 116 97 47 113 117 101 117 101 115 47 105 115 115 117 101 95 105 110 100 101 120 101 114 34 44 34 77 97 120 87 111 114 107 101 114 115 34 58 49 48 44 34 78 97 109 101 34 58 34 105 115 115 117 101 95 105 110 100 101 120 101 114 34 44 34 78 101 116 119 111 114 107 34 58 34 34 44 34 80 97 115 115 119 111 114 100 34 58 34 34 44 34 81 117 101 117 101 76 101 110 103 116 104 34 58 50 48 44 34 81 117 101 117 101 78 97 109 101 34 58 34 105 115 115 117 101 95 105 110 100 101 120 101 114 95 113 117 101 117 101 34 44 34 87 111 114 107 101 114 115 34 58 49 125] error: mkdir data: permission denied
2020/04/28 14:55:44 cmd/web.go:161:runWeb() [I] Listen: unix:///run/gitea/gitea.socket
2020/04/28 14:55:44 ...s/graceful/server.go:55:NewServer() [I] Starting new server: unix:/run/gitea/gitea.socket on PID: 5524
2020/04/28 14:55:44 ...s/graceful/server.go:79:ListenAndServe() [E] Unable to GetListener: listen unix /run/gitea/gitea.socket: bind: no such file or directory
2020/04/28 14:55:44 cmd/web.go:204:runWeb() [C] Failed to start server: listen unix /run/gitea/gitea.socket: bind: no such file or directory
2020/04/28 14:55:44 cmd/web.go:206:runWeb() [I] HTTP Listener: /run/gitea/gitea.socket Closed
Are those gdb related or is it what is really happening on my system ?
I'm changing the title as the multiple coredumps are due to the restart parameter of the gitea.service file, causing the restart of the application after failure.
2020/04/28 14:55:43 ...les/queue/setting.go:60:CreateQueue() [W] Unable to create queue for issue_indexer: mkdir data: permission denied
That's the hint. You need to allow Gitea to create the queues directories.
EDIT : Unrelated
Ok I managed to get it working by adding :
ReadWritePaths=/var/lib/gitea
in /usr/lib/systemd/system/gitea.service
What I don't get is why it is not working outside of systemd, when I launch the gitea app manually ?
```pwd
/var/lib/gitea
ls -l
total 20
srw-rw-rw- 1 gitea gitea 0 May 18 2019 0.0.0.0
drwxr-x--- 2 gitea gitea 4096 May 18 2019 attachments
drwxr-x--- 5 gitea gitea 4096 Feb 20 23:34 data
drwxr-x--- 3 gitea gitea 4096 Apr 28 14:36 indexers
drwxr-x--- 3 gitea gitea 4096 May 18 2019 repos
drwxr-x--- 2 gitea gitea 4096 May 18 2019 tmp
I reported the bug to archlinux as it is more related to systemd than to gitea.
https://bugs.archlinux.org/task/60669
Issue solved I guess ...
Huh, not that fixed though, because now I have a coredump everytime I push on this new branch. Pushing to master in other repos is OK.
r 29 04:31:33 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:31:33: Completed POST /api/internal/hook/post-receive/belette/socket 200 OK in 145.211576ms
Apr 29 04:31:33 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:31:33: Started POST /api/internal/ssh/5/update/12 for @
Apr 29 04:31:33 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:31:33: Completed POST /api/internal/ssh/5/update/12 200 OK in 13.630955ms
Apr 29 04:32:30 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:30: Started GET /api/internal/serv/command/5/belette/unicodeparser?mode=2&verb=git-receive-pack for @
Apr 29 04:32:30 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:30: Completed GET /api/internal/serv/command/5/belette/unicodeparser?mode=2&verb=git-receive-pack 200 OK in 16.187553ms
Apr 29 04:32:30 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:30: Started POST /api/internal/hook/pre-receive/belette/unicodeparser for @
Apr 29 04:32:30 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:30: Completed POST /api/internal/hook/pre-receive/belette/unicodeparser 200 OK in 6.447615ms
Apr 29 04:32:31 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:31: Started POST /api/internal/hook/post-receive/belette/unicodeparser for @
Apr 29 04:32:31 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:31: Completed POST /api/internal/hook/post-receive/belette/unicodeparser 200 OK in 107.111974ms
Apr 29 04:32:31 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:31: Started POST /api/internal/ssh/5/update/9 for @
Apr 29 04:32:31 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:32:31: Completed POST /api/internal/ssh/5/update/9 200 OK in 12.708982ms
Apr 29 04:33:06 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:33:06: Started GET /api/internal/serv/command/5/belette/minecraftmanager_server?mode=2&verb=git-receive-pack for @
Apr 29 04:33:06 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:33:06: Completed GET /api/internal/serv/command/5/belette/minecraftmanager_server?mode=2&verb=git-receive-pack 200 OK in 17.004788ms
Apr 29 04:33:06 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:33:06: Started POST /api/internal/hook/pre-receive/belette/minecraftmanager_server for @
Apr 29 04:33:06 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:33:06: Completed POST /api/internal/hook/pre-receive/belette/minecraftmanager_server 200 OK in 7.521984ms
Apr 29 04:33:07 rpi3 gitea[11384]: [Macaron] 2020-04-29 04:33:07: Started POST /api/internal/hook/post-receive/belette/minecraftmanager_server for @
Apr 29 04:33:52 rpi3 systemd[1]: gitea.service: Main process exited, code=killed, status=31/SYS
Apr 29 04:33:52 rpi3 systemd[1]: gitea.service: Failed with result 'signal'.
I'll do some more testing tomorrow (branch master in this repo, non-master branches in other repos)
Just in case, make sure you've got only one copy of gitea around, and hooks and authorized_keys point to it correctly (there's a site admin menu for rebuilding those).
I finally found it !!
Additional info :
First, after a coredump, running sudo -u gitea gdb --args gitea web -c /etc/gitea/app.ini
, running the debugger, stopping gdb and restarting the service works : the gitea instance starts successfully and the pushed commit is found in the gitea repo. However, it is not listed in the activity dashboard.
I created a test repository and :
So then I asked myself what the hell is the difference between my "coredump" repo and this one ??
The reason why this repo coredumped after commits is an invalid webhook : I hosted a while ago a drone instance on another raspberry and linked it to my gitea instance, but it took too much resources, so I temporarily disabled it. A invalid webhook was still enabled on this repository (with a red cross next to it in the {repo}/settings/hooks
page). After deleting it, no more coredumps happen. Why was it working in the master branch ? I don't know.
As the manual gitea
command seems to work, I think systemd hardening is wrong somewhere, regarding webhooks. I join my gitea systemd setup :
/etc/systemd/system/multi-user.target.wants/gitea.service
===========================================
[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=mysqld.service
After=postgresql.service
After=memcached.service
After=redis.service
[Service]
User=gitea
Group=gitea
Type=simple
WorkingDirectory=~
RuntimeDirectory=gitea
LogsDirectory=gitea
StateDirectory=gitea
Environment=USER=gitea HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea
ExecStart=/usr/bin/gitea web -c /etc/gitea/app.ini
Restart=always
RestartSec=2s
CapabilityBoundingSet=
NoNewPrivileges=True
PrivateUsers=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ProtectControlGroups=yes
ProtectKernelTunables=true
ProtectKernelModules=yes
ReadWritePaths=/etc/gitea/app.ini
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
/etc/systemd/system/gitea.service.d/override.conf
=====================================
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
PrivateUsers=false
Restart=no
ReadWritePaths=/etc/gitea/app.ini
ReadWritePaths=/var/lib/gitea
Do you have any idea on your end about why would this happen ?
I'm glad you pinpointed the problem and could have it fixed for you.
The webhook was invalid how? What kind of response was getting Gitea? An HTTP error? A timeout?
Gitea was receiving a 404 response from my non existing drone instance adress.
and in response to the 404 there is a core dump?!
@Ouack23 we really need to see the logs just prior to the core dump if there are any available.
@zeripath I joined the gitea logs in my original post and the system logs in this message.
No other log was available ...
This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.
This issue has been automatically closed because of inactivity. You can re-open it if needed.
Please see if https://github.com/go-gitea/gitea/issues/9792#issuecomment-691291999 solves the issue for you as well.