Gitea: SQLite variable limit check for migrating PR reviews

[PeterSurda]

• Gitea version (or commit ref): 1.12.0+dev-222-gd26aee383
• Git version: 2.24.3
• Operating system: Ubuntu 18.04
• Database (use [x]):
  
  
  
  • [ ] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [ ] MSSQL
  
  
  • [x] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io:
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [ ] No
  
  
  • [ ] Not relevant
  
  
• Log gist: https://gist.github.com/PeterSurda/ee7291b12c7d98f8902e353d0c9cf2dc#file-gitea-log

Description

Trying to migrate a repository from github failed when migrating pull requests is requested. I am in the process of creating a new repo to demonstrate how to reproduce the issue. I'll post an update once it's available. The error I get is
Migrate repository from xxxxxxxxx failed: too many SQL variables

I think it happens if a PR review has too many comments. SQLite has a compile time limit of arguments, defaulting to 999. I don't think the value can be increased without recompiling the library.

I tried to find the location in the code that could be causing this, and my best guess is that it's here: https://github.com/go-gitea/gitea/blob/1f0b797ddc139c3241c9b9694f0666a28ab41d80/models/review.go#L461

As far as I can see, there is no check for the number of arguments. One row appears to have 31 columns in my case, so doing the insert in chunks of int(999/31) = 32 would fix it.

I'd try to create a PR myself but I'm not familiar with go (language, build system, tests, ...).
...

Screenshots

[PeterSurda]

This is the review that triggers it: https://github.com/Bitmessage/PyBitmessage/pull/1367#pullrequestreview-164175633

I'm trying to create a new minimal repo to reproduce it.

[guillep2k]

Can you check the log for errors? Look for entries containing [E].

[lunny]

We should batch insert every 100 records.

[PeterSurda]

I updated with gist of log. As you can see, the row contains 31 colums. To remain under the limit, there can be a max of 32 rows in one INSERT query of this type.

[PeterSurda]

I managed to create a small repo that triggers this: https://github.com/PeterSurda/gitea-sqlite-limit

[PeterSurda]

Workaround: https://gist.github.com/PeterSurda/fb2c3f8fc1b2bd1208fd691cbc302a0a

Works on both mentioned repos. It's the first code I ever wrote in golang so it may be crude.

[PeterSurda]

I confirm this fixed the issue for me, using the latest docker image from docker hub.