Gitea: Downgrading database version from '159' to '158' is not supported

[webworxshop]

• Gitea version (or commit ref): 1.14.0+dev-160-g30ce3731a (from latest docker image)
• Git version: 2.26.2
• Operating system: Official Docker image on an Ubuntu 18.04 host
• Database (use [x]):
  
  
  
  • [ ] PostgreSQL
  
  
  • [x] MySQL
  
  
  • [ ] MSSQL
  
  
  • [ ] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io:
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [x] No
  
  
• Log gist:

2020/11/07 13:50:22 cmd/web.go:115:runWeb() [I] Starting Gitea on PID: 14
2020/11/07 13:50:22 ...dules/setting/git.go:94:newGit() [I] Git Version: 2.26.2, Wire Protocol Version 2 Enabled
2020/11/07 13:50:22 cmd/web.go:160:runWeb() [I] Global init
2020/11/07 13:50:22 ...dules/setting/git.go:94:newGit() [I] Git Version: 2.26.2, Wire Protocol Version 2 Enabled
2020/11/07 13:50:22 routers/init.go:164:GlobalInit() [T] AppPath: /app/gitea/gitea
2020/11/07 13:50:22 routers/init.go:165:GlobalInit() [T] AppWorkPath: /app/gitea
2020/11/07 13:50:22 routers/init.go:166:GlobalInit() [T] Custom path: /data/gitea
2020/11/07 13:50:22 routers/init.go:167:GlobalInit() [T] Log path: /data/gitea/log
Downgrading database version from '159' to '158' is not supported and may result in loss of data integrity.
If you really know what you're doing, execute `UPDATE version SET version=158 WHERE id=1;`
Received signal 15; terminating.

Description

Since a routine update this morning (automated via Watchtower) my Gitea instance has been down. Looking at the logs shows that Gitea is repeatedly restarting with a message that downgrading the database is not supported (see logs above).

Any ideas? Is it safe to run the SQL statement given in the logs?

Thanks in advance.

[lunny]

execute UPDATE version SET version=158 WHERE id=1;

[davidnewhall]

Just want to pop in and mention that I began troubleshooting my Gitea Docker installation today. It died yesterday, also during an auto update, and has the same error message. Since I can safely say, _I do not know what I'm doing_ with Gitea's database, and I would really rather not compromise my data integrity, can someone explain why we are downgrading the database version after an update?

I think it uses sqlite3, so I suspect I need to find that db file and open it, then execute the provided command? And my installation wont blow up? Thanks!

[portsmouthcomputer]

I had the same problem with the downgrade on the database. This is how I solved it. I am using unRaid so all I know it is stored in your data folder

1. make a backup of the file in your gitea.db
2. copy the file gitea.db to your local computer
3. download a sqlite application that can view and edit. I used "DB Browser for SQLite"
4. open the local file gitea.db and look for where you can execute the command UPDATE version SET version=158 WHERE id=1;
5. save the changes
6. copy the file back to gitea.db
7. restart the docker container

This worked for me. Just make sure you have a backup just in case.

[davidnewhall]

Also using unRAID, and reasonably familiar with SQL. Appreciate the info, it's very helpful!

[davidnewhall]

Downgrading the version worked, but I'm still confused _why_ this was necessary. Will this be a common problem? Should I automate a solution?

This is what the fix looks like on unRAID. Ssh into your unraid server as root and cd into the gitea app data folder.

Linux 4.19.98-Unraid.
root@rzr:~# cd /mnt/disks/ssd/appdata/gitea/
root@rzr:/mnt/disks/ssd/appdata/gitea# ls
git/  gitea/  ssh/
root@rzr:/mnt/disks/ssd/appdata/gitea# cd gitea/
root@rzr:/mnt/disks/ssd/appdata/gitea/gitea# cp -r gitea.db gitea_bak.db
root@rzr:/mnt/disks/ssd/appdata/gitea/gitea# sqlite3 gitea.db
SQLite version 3.30.1 2019-10-10 20:19:45
Enter ".help" for usage hints.
sqlite> UPDATE version SET version=158 WHERE id=1;
sqlite> ^D

I was also seeing these errors and wondering if this is going to break ssh checkouts.

Could not load host certificate "/data/ssh/ssh_host_ed25519_cert": No such file or directory
Could not load host certificate "/data/ssh/ssh_host_rsa_cert": No such file or directory
Could not load host certificate "/data/ssh/ssh_host_ecdsa_cert": No such file or directory
Could not load host certificate "/data/ssh/ssh_host_dsa_cert": No such file or directory

~This was the fix:~ Never mind, the ownership changes back to root when the gitea container is started and the errors continue. I have not found a fix for these errors yet. I just noticed the error messages are actually missing files, so maybe this is OK. No idea really.

root@rzr:/mnt/disks/ssd/appdata/gitea/ssh# ls -la
total 32
drwx------ 2 root   root   226 Nov  1 21:40 ./
drwxrwxrwx 5 nobody users   41 Nov  1 23:25 ../
-rw------- 1 root   root  1381 Nov  1 21:40 ssh_host_dsa_key
-rw------- 1 root   root   607 Nov  1 21:40 ssh_host_dsa_key.pub
-rw------- 1 root   root   513 Nov  1 21:40 ssh_host_ecdsa_key
-rw------- 1 root   root   179 Nov  1 21:40 ssh_host_ecdsa_key.pub
-rw------- 1 root   root   411 Nov  1 21:40 ssh_host_ed25519_key
-rw------- 1 root   root    99 Nov  1 21:40 ssh_host_ed25519_key.pub
-rw------- 1 root   root  1823 Nov  1 21:40 ssh_host_rsa_key
-rw------- 1 root   root   399 Nov  1 21:40 ssh_host_rsa_key.pub
root@rzr:/mnt/disks/ssd/appdata/gitea/ssh# chown nobody . *

Hope this helps someone else!

[webworxshop]

I can confirm the fix also works for me, using MariaDB/MySQL. Here's what the fix looks like for me (with MariaDB also running under Docker):

$ docker exec -it apps_mariadb-gitea_1 mysql -u gitea -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 452
Server version: 10.5.7-MariaDB-1:10.5.7+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use gitea;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [gitea]> UPDATE version SET version=158 WHERE id=1;
Query OK, 1 row affected (0.003 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Thanks all!

[lunny]

I'm sorry to make the break which disturb you because of a revert to a wrong PR.
I think this is a rare accident and not a common problem.

[JamesMcTavish]

I originally had the restarting constantly with the Downgrading database version from '159' to '158' message.

I did the update to change the version to 158, restarted the container, and it is up and stable.

However now browsing Issues gives me a 500 Internal Server Error. The text of the issues is in the dashboard in the history, so I know they're still in the DB. Nothing else seems to be affected browsing around.

Logs only give me:

[Macaron] 2020-11-07 18:29:26: Started GET /paradigm/Budgeting/issues/36 for 10.10.0.26
[Macaron] 2020-11-07 18:29:26: Completed GET /paradigm/Budgeting/issues/36 500 Internal Server Error in 29.544341ms
[Macaron] 2020-11-07 18:29:26: Started GET /user/avatar/paradigm/-1 for 10.10.0.26
[Macaron] 2020-11-07 18:29:26: Completed GET /user/avatar/paradigm/-1 302 Found in 2.837103ms

Whenever I try to browse an issue.

What can I do to start troubleshooting this?

[lunny]

Could you test if all issues return the error or only some issues returned that?

[JamesMcTavish]

Every one I try returns this. Is there additional logging I could turn on?

[lunny]

You can change your app.ini and restart.

[database]
LOG_SQL  = true

[log]
LEVEL                = trace

[JamesMcTavish]

My bad. When I copied back into the container after updating the database, I forgot to reset the owner/group. Reset that and everything works now.

Thank you for your help.