gitea doctor: panic: reflect: NumField of non-struct type

[b-ehlers]

• Gitea version (or commit ref): 1.11.6
• Git version: 2.20.1
• Operating system: Linux Debian 10.4 (Buster)
• Database (use [x]):
  
  
  
  • [ ] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [ ] MSSQL
  
  
  • [x] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io:
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [ ] No
  
  
  • [x] Not relevant
  
  
• Log gist:

Description

Running gitea doctor results in a panic, see below.

I haven't noticed any issues running gitea.

git@netcup:~/gitea$ ./gitea doctor all
[ 1 ] Check paths and basic configuration
- Configuration File Path:   '/home/git/gitea/custom/conf/app.ini'
- Repository Root Path:      '/home/git/gitea-repositories'
- Data Root Path:            '/home/git/gitea/data'
- Custom File Root Path:     '/home/git/gitea/custom'
- Work directory:            '/home/git/gitea'
- Log Root Path:             '/home/git/gitea/log'
OK.

[ 2 ] Check Database Version
OK.

[ 3 ] Check if OpenSSH authorized_keys file is up-to-date
OK.

[ 4 ] Check consistency of database
panic: reflect: NumField of non-struct type

goroutine 1 [running]:
reflect.(*rtype).NumField(0x1b6ea20, 0x1b6ea20)
    /usr/local/go/src/reflect/type.go:981 +0x5e
code.gitea.io/gitea/vendor/xorm.io/xorm.(*Engine).mapType(0xc000144b60, 0x1b6ea20, 0x32e2b40, 0x98, 0x1b6ea00, 0x8, 0x18)
    /go/src/code.gitea.io/gitea/vendor/xorm.io/xorm/engine.go:929 +0x2a4
code.gitea.io/gitea/vendor/xorm.io/xorm.(*Engine).autoMapType(0xc000144b60, 0x1b6ea20, 0x32e2b40, 0x98, 0x0, 0x0, 0x0)
    /go/src/code.gitea.io/gitea/vendor/xorm.io/xorm/engine.go:855 +0x193
code.gitea.io/gitea/vendor/xorm.io/xorm.(*Statement).setRefBean(0xc001a8ee18, 0x1b6ea20, 0x32e2b40, 0x3349960, 0x0)
    /go/src/code.gitea.io/gitea/vendor/xorm.io/xorm/statement.go:222 +0x69
code.gitea.io/gitea/vendor/xorm.io/xorm.(*Statement).genCountSQL(0xc001a8ee18, 0xc000b832d0, 0x1, 0x1, 0x1e978a7, 0xc, 0xc000b83190, 0xb93b59, 0xc001ba51a0, 0x1b6ea20, ...)
    /go/src/code.gitea.io/gitea/vendor/xorm.io/xorm/statement.go:992 +0x3f0
code.gitea.io/gitea/vendor/xorm.io/xorm.(*Session).Count(0xc001a8ee00, 0xc000b832d0, 0x1, 0x1, 0x0, 0x0, 0x0)
    /go/src/code.gitea.io/gitea/vendor/xorm.io/xorm/session_stats.go:24 +0x259
code.gitea.io/gitea/cmd.runDoctorCheckDBConsistency(0xc0000c7340, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/code.gitea.io/gitea/cmd/doctor.go:521 +0x26b
code.gitea.io/gitea/cmd.runDoctor(0xc0000c7340, 0x0, 0xc0000c71e0)
    /go/src/code.gitea.io/gitea/cmd/doctor.go:210 +0x9ca
code.gitea.io/gitea/vendor/github.com/urfave/cli.HandleAction(0x1bc5ee0, 0x2ea6b88, 0xc0000c7340, 0xc0003bd700, 0x0)
    /go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/app.go:490 +0xc8
code.gitea.io/gitea/vendor/github.com/urfave/cli.Command.Run(0x1e85db0, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1ea4b37, 0x11, 0x0, ...)
    /go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/command.go:210 +0x991
code.gitea.io/gitea/vendor/github.com/urfave/cli.(*App).Run(0xc00009b1e0, 0xc00003c180, 0x3, 0x3, 0x0, 0x0)
    /go/src/code.gitea.io/gitea/vendor/github.com/urfave/cli/app.go:255 +0x6ab
main.main()
    /go/src/code.gitea.io/gitea/main.go:110 +0x845
git@netcup:~/gitea$

[b-ehlers]

Update: When I correctly interpret the stack trace, doctor.go checks in line 521 the table tracked_time. In my gitea instance this table is empty.

git@netcup:~/gitea$ sqlite3 data/gitea.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> SELECT * FROM tracked_time LEFT JOIN issue ON tracked_time.issue_id=issue.id WHERE issue.id is NULL;
sqlite> SELECT * FROM tracked_time;
sqlite>

[6543]

Update: When I correctly interpret the stack trace, doctor.go checks in line 521 the table tracked_time. In my gitea instance this table is empty.

thanks this helps

I'll have a look at it

[6543]

OK this is an XORM bug in COUNT for sqlite

[6543]

tested this is no problem in master and v1.12 branch - so it is a old xorm bug

so the workaround should do it -> #11741

[Dessa]

after applying the PR to my 1.11.6 build the stacktrace now turned into this:

Error: Error 1052: Column 'id' in field list is ambiguous

im on mysql, not sqlite if that matters.

[lunny]

Since gitea doctor will not synchronize database, we cannot assume the table exist.

[b-ehlers]

Just tested the 1.11 branch after the PR being applied. The stacktrace issue is solved, but now I'm getting the same error as @Dessa (still using SQLite):

git@netcup:~/gitea$ ./gitea-1.11-linux-amd64 doctor all
[ 1 ] Check paths and basic configuration
- Configuration File Path:   '/home/git/gitea/custom/conf/app.ini'
- Repository Root Path:      '/home/git/gitea-repositories'
- Data Root Path:            '/home/git/gitea/data'
- Custom File Root Path:     '/home/git/gitea/custom'
- Work directory:            '/home/git/gitea'
- Log Root Path:             '/home/git/gitea/log'
OK.

[ 2 ] Check Database Version
OK.

[ 3 ] Check if OpenSSH authorized_keys file is up-to-date
OK.

[ 4 ] Check consistency of database
Error: ambiguous column name: id

[bendem]

OK this is an XORM bug in COUNT for sqlite

I have the exact same stack using postgresql so I'd say it's not a problem specific to sqlite.

gitea-local=# table tracked_time;
 id | issue_id | user_id | created_unix | time | deleted
----+----------+---------+--------------+------+---------
(0 rows)

[b-ehlers]

As the original bug is fixed with 1.11.7 I'm closing this.

If something is wrong with this fix, it's a new issue.

[hollie]

Hello,

I am running Gitea in docker, tried to upgrade to 1.12 from 1.11.6 some time ago. That failed (database issue) so I reverted back to 1.11.x and have been upgrading to latest 1.11 since then.

Right now I am running 1.11.8 and would like to upgrade to 1.12.x to get in line with the most recent releases.

However, when I run gitea doctor I encounter the error mentioned above.

From this ticket/thread it is not clear to me how to get rid of this error. Moreover because of the statement that the bug is fixed in 1.11.7. I don't think it is wise to try to upgrade to 1.12 as long as the error in the brew doctor output is present.

Can somebody please guide me in the right direction how to fix this? Searching on the project tickets and on Google does not turn up any useful results.

Thanks!
Lieven.

The gitea doctor output is:

[ 1 ] Check paths and basic configuration
- Configuration File Path:   '/data/gitea/conf/app.ini'
- Repository Root Path:      '/data/git/repositories'
- Data Root Path:            '/data/gitea'
- Custom File Root Path:     '/data/gitea'
- Work directory:            '/usr/local/bin'
- Log Root Path:             '/data/gitea/log'
OK.

[ 2 ] Check Database Version
OK.

[ 3 ] Check if OpenSSH authorized_keys file is up-to-date
Error: authorized_keys is out of date and should be regenerated with gitea admin regenerate keys

[ 4 ] Check consistency of database
Error: ambiguous column name: id

[zeripath]

Iirc you can just upgrade to 1.12 and run the doctor from there

[hollie]

Hello @zeripath thanks a lot for the feedback.

I attempted to update to 1.12.3 but alas, that did not work out as expected.

The container is restarting all the time, reporting Received signal 15: terminating. I searched the tickets further and found #10422 where you advise to remove the queues. I tried that and now the container is still crashing but with a different error:

gitea_1        | 2020/07/31 07:39:40 ...m.io/xorm/core/db.go:277:ExecContext() [I] [SQL] INSERT INTO `notice` (`type`,`description`,`created_unix`) VALUES (?, ?, ?) [2 Cron: Check all repository statistics has finished 1596181180] - 45.632014ms
gitea_1        | 2020/07/31 07:39:40 ...dexer/stats/queue.go:24:handle() [E] stats queue idexer.Index(117) failed: object not found
gitea_1        | 2020/07/31 07:39:40 ...m.io/xorm/core/db.go:154:QueryContext() [I] [SQL] SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` WHERE `id`=? LIMIT 1 [77] - 542.801µs
gitea_1        | 2020/07/31 07:39:40 ...m.io/xorm/core/db.go:154:QueryContext() [I] [SQL] SELECT `id`, `repo_id`, `commit_sha`, `indexer_type` FROM `repo_indexer_status` WHERE (`indexer_type` = ?) AND `repo_id`=? LIMIT 1 [1 77] - 28.031µs
gitea_1        | 2020/07/31 07:39:40 ...m.io/xorm/core/db.go:154:QueryContext() [I] [SQL] SELECT `pull_request`.`id` FROM `pull_request` WHERE (status=?) [1] - 45.192µs
gitea_1        | 2020/07/31 07:39:40 ...ueue_disk_channel.go:157:Run() [D] PersistableChannelUniqueQueue: pr_patch_checker Starting
gitea_1        | 2020/07/31 07:39:40 ...ue/queue_bytefifo.go:95:Run() [D] unique-level: pr_patch_checker-level Starting
gitea_1        | 2020/07/31 07:39:40 routers/init.go:162:GlobalInit() [I] SQLite3 Supported
gitea_1        | 2020/07/31 07:39:40 routers/init.go:50:checkRunMode() [I] Run Mode: Production
gitea_1        | 2020/07/31 07:39:40 ...ueue_disk_channel.go:137:Run() [D] PersistableChannelQueue: task Starting
gitea_1        | 2020/07/31 07:39:40 ...ue/queue_bytefifo.go:95:Run() [D] level: task-level Starting
gitea_1        | 2020/07/31 07:39:40 ...dexer/stats/queue.go:24:handle() [E] stats queue idexer.Index(77) failed: object not found
gitea_1        | 2020/07/31 07:39:40 ...m.io/xorm/core/db.go:154:QueryContext() [I] [SQL] SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` WHERE `id`=? LIMIT 1 [74] - 378.592µs

The weird thing is that there is what looks to be a typo missing the letter n in indexer in the error log:

[E] stats queue idexer.Index(117) failed: object not found

I do not know if this is the reason things go wrong.

I also cannot run gitea doctor because the container crashed before I have time to execute that command.

Do you have any further advice to get this fixed? As stated I am now trying to run gitea v1.12.3. My sanitised config file is

APP_NAME = Git server
RUN_USER = gogs
RUN_MODE = prod

[repository]
ROOT = /data/git/repositories
FORCE_PRIVATE = true

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
SSH_DOMAIN    = git.XXXXX.YYY
START_SSH_SERVER = false
HTTP_PORT     = <number>
ROOT_URL      = https://git.XXXXX.YYY/
DISABLE_SSH   = false
SSH_PORT      = <number>
OFFLINE_MODE  = false

[database]
HOST     = mysql:3306
PATH     = /data/gitea/gitea.db
DB_TYPE  = sqlite3
NAME     =
USER     =
PASSWD   =
SSL_MODE = disable

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER        = file

[picture]
AVATAR_UPLOAD_PATH      = /data/gitea/avatars
DISABLE_GRAVATAR        = false
ENABLE_FEDERATED_AVATAR = false

[attachment]
PATH = /data/gitea/attachments

[log]
ROOT_PATH = /data/gitea/log
MODE      = console
LEVEL     = debug

[mailer]
ENABLED = true
HOST    = <mailserver>
FROM    = <mailaddress>
USER    = <mailuser>
PASSWD  = <mailpass>

[service]
REGISTER_EMAIL_CONFIRM = true
ENABLE_NOTIFY_MAIL     = true
DISABLE_REGISTRATION   = true
ENABLE_CAPTCHA         = false
REQUIRE_SIGNIN_VIEW    = true

[security]
INSTALL_LOCK   = true
SECRET_KEY     = <mykey>
INTERNAL_TOKEN = <mytoken>
DISABLE_GIT_HOOKS = true

[oauth2]
JWT_SECRET = <mysecret>

[other]
SHOW_FOOTER_BRANDING = false
SHOW_FOOTER_VERSION = false

[api]
ENABLE_SWAGGER = false

; Operation timeout in seconds
[git.timeout]
DEFAULT = 360
MIGRATE = 600
MIRROR = 300
CLONE = 300
PULL = 300
GC = 120

The relevant docker compose section is:

  gitea:
    image: gitea/gitea:1.12
    environment:
      - USER=gogs
      - USER_UID=501
      - USER_GID=20
      - RUN_MODE=prod
      - DISABLE_REGISTRATION=true
      - REQUIRE_SIGNIN_VIEW=true
      - SSH_PORT=8400
    restart: always
    volumes:
      - /Volumes/Repositories/gitea-data:/data
    ports:
      - "<sshport:sshport>"
      - "<webport:webport>"
    security_opt:
      - no-new-privileges:true

Thank you,
Hollie.

[hollie]

Update: the reason of the crash was not the lines showing the error, but the fact that the container ran into the memory limit of 2GB that was in place on my system for the container during the migration process of the database from v1.11.8 to 1.12.3.
An increase of the memory for the container to 4 GB was required to let the migration process to complete correctly. I noticed a memory consumption of up to 3.5 GB during the migration, which returned to 1.4 GB after the migration was completed successfully.

Conclusion: the errors about the 'stats queue' were not related to the crash. The memory usage was.

Well, at least this resulted in a typo being fixed in the codebase ;-) Hopefully this will help other people who run into this issue so that they don't waste the better part of a day to find out what is going wrong with their gitea docker container.

A special thank to @tgerczei and @techknowlogick for helping me troubleshoot this. I owe you both a drink.

[zeripath]

So unfortunately that migration does use quite a lot of memory - it shouldn't require as much memory again.

Glad it's working now.