Gitea: Minio S3 storage not working for LFS or attachments

Created on 24 Oct 2020 · 12Comments · Source: go-gitea/gitea

Gitea version (or commit ref): 1.14.0+dev-69-gd2ad4dec6
Git version: 2.26.2
Operating system: Ubuntu 20

Using latest Docker image.

Database (use [x]):
- [x] PostgreSQL
- [ ] MySQL
- [ ] MSSQL
- [ ] SQLite
Log gist: https://gist.github.com/adamb70/0f58c68486d826cd4810792e1dad9b66

Description

I am trying to use S3 to host files in attachments and LFS using the minio storage type, but the files are instead being stored locally. I think this may be a problem of improper configuration, as there are no errors logged so it seems that Gitea doesn't even try to save the files to S3. The relevant parts of the ini file are below, any help is appreciated.

[server]
APP_DATA_PATH    = /data/gitea
DOMAIN           = git.mydomain.com
SSH_DOMAIN       = git.mydomain.com
HTTP_PORT        = 3000
ROOT_URL         = https://git.mydomain.com/
DISABLE_SSH      = false
SSH_PORT         = 22
SSH_LISTEN_PORT  = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET   = {secret}
OFFLINE_MODE     = false

[lfs]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /lfs
MINIO_USE_SSL = true

[attachment]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true

kinbug

Source

adamb70

Most helpful comment

Actually it wasn't!

The issue is that the toConfig string wasn't matching the Mappable interface because the config wasn't being passed as a pointer.

There was a further bug in the MapTo.

Sorry about this - I think I have it finally fixed now.

zeripath on 25 Oct 2020

🎉1 👍1

All 12 comments

Could you find storages logs when gitea start, like below

2020/10/24 09:16:20 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/attachments
2020/10/24 09:16:20 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/avatars
2020/10/24 09:16:20 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/repo-avatars
2020/10/24 09:16:20 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/lfs

lunny on 24 Oct 2020

I don't see anything about storage in the logs. Here is the startup log: https://gist.github.com/adamb70/5415b31be481db6fc325773c0c956c12

And here is the logging config in my ini file:

[log]
MODE                 = console
LEVEL                = trace
REDIRECT_MACARON_LOG = true
MACARON              = console
ROUTER               = console
ROOT_PATH            = /data/gitea/log

If it helps at all, here is my full gitea-app.ini file: https://gist.github.com/adamb70/963accf8c1e8b28f43c5a336f6262259

adamb70 on 24 Oct 2020

It seems there are something lost in your log?

lunny on 24 Oct 2020

I am starting Gitea up through docker-compose, so maybe some log info doesn't get sent through to the console?

I have tried again with setting log MODE=file, and the content in gitea.log is still the same as the console output so it doesn't seem like it's Docker related.

adamb70 on 24 Oct 2020

@adamb70 please update to at least g25b7766673. We added logging in from that point to help us diagnose what was going on with storage.

(Assuming you're not running gitea with --port - which you shouldn't anyway - just jump to the latest)

zeripath on 24 Oct 2020

Thanks for the new release @zeripath. The full log is as follows: https://gist.github.com/adamb70/5e315aa4bc8afde849e64033703e7ddb

The relevant storage part is here:

gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/attachments
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/git/lfs

It seems like it's still trying to use local storage, despite the app.ini file.

I am binding the ini file in the Docker container to a local file outside so it can be managed from outside the container. This seems to work fine to change any other settings in the ini file, it's just the storage I'm seeing the issue with.

docker-compose.yml

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    ...
    volumes:
      - gitea:/data
      - ./gitea-app.ini:/data/gitea/conf/app.ini  # <-----
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

adamb70 on 24 Oct 2020

Looks like the storage type isn't being set...

zeripath on 24 Oct 2020

OK it looks like there's a bug here:

https://github.com/go-gitea/gitea/blob/c5020cff3df0a7d77ff7a453a2007edc7e5c3667/modules/setting/storage.go#L49

I bet that this should actually default to typ instead of "".

in the meantime I think if you create a new ini section:

[storage.minio]
STORAGE_TYPE=minio

That would make the minio work

zeripath on 24 Oct 2020

Strangely I'm now getting the following error:

gitea      | 2020/10/24 22:34:07 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: minio
gitea      | 2020/10/24 22:34:07 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at : with base path
gitea      | 2020/10/24 22:34:07 routers/init.go:60:NewServices() [F] storage init failed: Endpoint:  does not follow ip address or domain name standards.

It seems like it is interpreting the endpoint URL as an empty string for some reason. Here's the line that I think is throwing the error https://github.com/minio/minio-go/blob/master/utils.go#L91 .

app.ini

[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /lfs

[attachment]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /attachment

[storage.minio]
STORAGE_TYPE=minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true

adamb70 on 24 Oct 2020

It seems like the config in NewMinioStorage() is set to null values. I've never used golang before, but is this line correct?

https://github.com/go-gitea/gitea/blob/25b7766673867d2a9ac32fde6cb0d719f04b2953/modules/storage/minio.go#L89

Some Googling suggests it should be config, ok := configInterface.(MinioStorageConfig).

adamb70 on 25 Oct 2020

nope that's not it.

The problem will be in modules/setting/storage.go

zeripath on 25 Oct 2020

Actually it wasn't!

The issue is that the toConfig string wasn't matching the Mappable interface because the config wasn't being passed as a pointer.

There was a further bug in the MapTo.

Sorry about this - I think I have it finally fixed now.