Gitea: Gitea cannot start with a read-only config

[captn3m0]

• Gitea version (or commit ref): 1.7.0-rc2
• Git version: NA
• Operating system: Arch Linux with Docker 18.09
• Database (use [x]):
  
  
  
  • [ ] PostgreSQL
  
  
  • [ ] MySQL
  
  
  • [ ] MSSQL
  
  
  • [x] SQLite
  
  
• Can you reproduce the bug at https://try.gitea.io: NA
  
  
  
  • [ ] Yes (provide example URL)
  
  
  • [ ] No
  
  
  • [ ] Not relevant
  
  
• Log gist:

2019/01/11 19:34:13 [...s/setting/setting.go:941 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied
2019/01/11 19:34:14 [...s/setting/setting.go:941 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied

Description

Running with the exact same configuration and on 1.6 latest docker image does not give this error.

The configuration file does have the following lines:

LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/gitea/lfs
LFS_JWT_SECRET   = ${lfs-jwt-secret}

[captn3m0]

The secret is exactly 32 characters encoded in base64, and it works perfectly if I switch the image to 1.6

[bkraul]

I think my issue is related. gitea/gitea:latest broke my install, apparently when it cannot migrate, because it is getting "permission denied", eventhough the app.ini file is owned by user git (as it should be). The log keeps showing this repeatedly.

2019/01/11 23:20:22 [T] Log path: /data/gitea/log
2019/01/11 23:20:23 [T] AppPath: /app/gitea/gitea
2019/01/11 23:20:23 [T] AppWorkPath: /app/gitea
2019/01/11 23:20:23 [T] Custom path: /data/gitea
2019/01/11 23:20:23 [T] Log path: /data/gitea/log
2019/01/11 23:20:24 [T] AppPath: /app/gitea/gitea
2019/01/11 23:20:24 [T] AppWorkPath: /app/gitea
2019/01/11 23:20:24 [T] Custom path: /data/gitea
2019/01/11 23:20:24 [T] Log path: /data/gitea/log

Everything works fine when I revert to tag 1.6.

EDIT: Take that back...on one of my installs, even with 1.6 the installation is hosed (same repeated messages as above). What is going on??

[captn3m0]

I'm currently running with LFS disabled and it does indeed work (as someone already mentioned): https://git.captnemo.in/nemo/nebula/commit/18164d175e38ea6383462b2eb870c82d21f84f2c

But if you are using LFS and have a secret already in your config, 1.7 breaks your config and overwrites it.

[bkraul]

In my case, setting LFS off does nothing. The install is still broken. Seems mine is unrelated in the sense that it doesn't relate to LFS. Opened #5708 .

[zeripath]

Hmm... is this still broken since this commit: https://github.com/go-gitea/gitea/commit/9e9d1b8f951c0c87af6bed71340b92ee22f5bb3b ?

[captn3m0]

Tested against :latest docker image (8d40032), and it seems to be fixed.

[zeripath]

Cool. Sorry about that.

We need to add an integration test that takes a Gitea db from the minimal version and a few choice other versions and migrates it. That way we'll know if there's an issue in migration quicker.

This would be a quick pr I think.

[MrFloppy]

Unfortunately :latest docker image from 14.01.2019 still is broken with the permission denied error on the app.ini for me. Reverting to 1.6 makes it run still fine.

Is there something like a "stable" channel?

[captn3m0]

1.6.3 is the last stable release. 1.7 is still in RC.

I double checked after your comment, and switching to :latest with LFS turned on broke my setup again. So I'm running with LFS disabled on LFS for now. (The container exited, but didn't have any logs so I need to investigate this a little bit more).

[bkraul]

Not sure what is happening with your setup, but I have LFS_START_SERVER = true in 2 of my installations and have it working OK on :latest (270fa6d). I can also see the app.ini file's permissions being set to the proper user on container start (changed them to a different user on the host and they were changed by the container after it starts) so this may not be an issue of permissions of that particular file. Of course, I could be wrong.

Please pardon my ignorance, but where is ${lfs-jwt-secret} coming from? Mine has an actual secret string.

[captn3m0]

It gets rendered in via terraform: https://git.captnemo.in/nemo/nebula/src/branch/master/gitea/data.tf#L13-L23, which uploads it as a file inside the container.

I'll try to switch LFS back on and see if I can get an error code or some logs out it tomorrow.

[bkraul]

Might I add that I am running the stack on my server with docker-compose as root. I do not know if this matters, but I have never tried any other way to do with docker-compose. Obviously, inside the container, the user that runs the app is the default (gitea 1000). Perhaps that might have something to do with this.

[captn3m0]

Okay, now my setup is borked in the same way as @bkraul

2019/01/21 16:14:46 [T] AppPath: /app/gitea/gitea
2019/01/21 16:14:46 [T] AppWorkPath: /app/gitea
2019/01/21 16:14:46 [T] Custom path: /data/gitea
2019/01/21 16:14:46 [T] Log path: /app/gitea/log
2019/01/21 16:14:46 [I] Gitea v8da5237 built with: bindata, sqlite, sqlite_unlock_notify

Keeps on repeating the same logs. Switching on debug logs does give some information:

2019/01/21 16:16:46 [T] AppPath: /app/gitea/gitea
2019/01/21 16:16:46 [T] AppWorkPath: /app/gitea
2019/01/21 16:16:46 [T] Custom path: /data/gitea
2019/01/21 16:16:46 [T] Log path: /app/gitea/log
2019/01/21 16:16:46 [I] Gitea v8da5237 built with: bindata, sqlite, sqlite_unlock_notify
2019/01/21 16:16:46 [I] Log Mode: Console(Debug)
2019/01/21 16:16:46 [I] XORM Log Mode: Console(Debug)
2019/01/21 16:16:46 [I] Cache Service Enabled
2019/01/21 16:16:46 [I] Session Service Enabled
2019/01/21 16:16:46 [I] Mail Service Enabled
2019/01/21 16:16:46 [I] Register Mail Service Enabled
2019/01/21 16:16:46 [I] Notify Mail Service Enabled
2019/01/21 16:16:46 [I] PING DATABASE sqlite3
2019/01/21 16:16:46 [...itea/routers/init.go:61 GlobalInit()] [E] Failed to initialize ORM engine: sync database struct error: Unknown col "is_empty" in index is_bare of table repository, columns [id owner_id lower_name name description website default_branch num_watches num_stars num_forks num_issues num_closed_issues num_pulls num_closed_pulls num_milestones num_closed_milestones is_private is_empty is_mirror is_fork fork_id size created_unix updated_unix is_fsck_enabled topics]

Edit: This looks like a known issue: #5759

Will close this issue once I can get past that.

[bkraul]

@captn3m0, workaround explained in #5681 . Basically, you connect to the MySQL/MariaDB DB and set the theme column in the user table for all the users as 'gitea', then reattempt to start. This, for me was a little inconvenient because I had it running in a stack where it wasn't publicly available, so I had to temporarily make it available to fix.

After setting that up, everything worked fine with the latest version.

[captn3m0]

My nemesis was #5759 (I made the theme change, but it doesn't help).

Finally fixed it by running all the sqlite commands in migrations/78 manually and bumping the version to 79.

[captn3m0]

docker logs gitea Jan 21 18:28:34 syslogd started: BusyBox v1.28.4 Jan 21 18:28:34 sshd[17]: Server listening on :: port 22. Jan 21 18:28:34 sshd[17]: Server listening on 0.0.0.0 port 22. 2019/01/21 18:28:34 [...s/setting/setting.go:947 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied

Turned back LFS_START_SERVER = true on :latest and it breaks still.

[zeripath]

Does your app.ini still have: LFS_JWT_SECRET = ${lfs-jwt-secret}?

Rather than an actual jws secret in it?

[captn3m0]

No it doesn't.

It gets rendered by Terraform, so gitea gets the actual secret:

grep LFS app.ini
LFS_START_SERVER = false
LFS_CONTENT_PATH = /data/gitea/lfs
LFS_JWT_SECRET   = mhicnuXXXXXXXXXXXXXX7qtma1

Right now set to false to keep the server running. The server is up at https://git.captnemo.in, happy to give you temporary admin access if you signup.

[zeripath]

So, in your testing if LFS is set to true, do you get anything in the logs?

I'm just asking so that it's easier to look for the misbehaving code.

[zeripath]

Ah sorry just noticed the above

[captn3m0]

I've switched to log level Trace, but it doesn't get me anything more.

docker logs gitea -f
Jan 22 07:31:17 syslogd started: BusyBox v1.28.4
Jan 22 07:31:17 sshd[17]: Server listening on :: port 22.
Jan 22 07:31:17 sshd[17]: Server listening on 0.0.0.0 port 22.
2019/01/22 07:31:17 [...s/setting/setting.go:947 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied
2019/01/22 07:31:18 [...s/setting/setting.go:947 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied

[typeless]

@captn3m0 how about docker exec my-gitea-container stat /my/custom/conf/app,ini

[zeripath]

Ok there's two places that error is printed:

• https://github.com/go-gitea/gitea/blob/1bb22b2b4777b47b05bf023d7737c03dc9560b09/modules/setting/setting.go#L947
• https://github.com/go-gitea/gitea/blob/1bb22b2b4777b47b05bf023d7737c03dc9560b09/modules/setting/setting.go#L986

Line 986 can only be reached if INTERNAL_SECRET is empty and stopping LFS would have no effect on that.

Line 947 is therefore the culprit. This can only be reached depending on the results of https://github.com/go-gitea/gitea/blob/1bb22b2b4777b47b05bf023d7737c03dc9560b09/modules/setting/setting.go#L923

n, err := base64.RawURLEncoding.Decode(LFS.JWTSecretBytes, []byte(LFS.JWTSecretBase64))

If err isn't nil or n isn't 32.

Ok are you sure that base64 URLDecoded your LFS value is 32 and that it's valid?

[captn3m0]

I'd gotten to the same root cause, and mentioned this above:

The secret is exactly 32 characters encoded in base64, and it works perfectly if I switch the image to 1.6

https://github.com/go-gitea/gitea/issues/5704#issuecomment-453633387

[captn3m0]

docker exec -it gitea stat /data/gitea/conf/app.ini
  File: /data/gitea/conf/app.ini
  Size: 7878        Blocks: 16         IO Block: 4096   regular file
Device: 37h/55d Inode: 6400700     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-01-22 07:33:03.000000000
Modify: 1970-01-01 00:00:00.000000000
Change: 2019-01-22 07:33:03.000000000

[zeripath]

Actually looking at that file I'm not sure that I see LFS.JWTSecretBase64 is set read/set anywhere...

[typeless]

The Dockerfile shows that gitea runs as the user git rather than root.
So, it's reasonable that it cannot write the file.

Edit: A possible cause is that the app.ini is mapped from the host machine which belongs to a particular user. But the UID/GID of the file does not exist in /etc/passwd.
I don't run Gitea with Docker, but I guess people who do that have an idiomatic way to deal with such problem?

@typeless he doesn't want Gitea to write to that file, and for some reason it is despite having a good config that should mean it doesn't need to write to the file.

Edit2: @zeripath Re-read your earlier comment and I can understand what you mean now.

[zeripath]

I'm not at a dev box. Are you able to diff modules/setting/setting.go between master and v1.6 and see if there are changes related to this.

[zeripath]

@typeless he doesn't want Gitea to write to that file, and for some reason it is despite having a good config that should mean it doesn't need to write to the file.

Primarily Gitea is writing to app.ini because it can't find a valid jwt token for LFS but there is one in the app.ini.

[zeripath]

OK, so I've looked at this again. Looking at modules/setting/setting.go:

sec = Cfg.Section("server")
if err = sec.MapTo(&LFS); err != nil {
    log.Fatal(4, "Failed to map LFS settings: %v", err)
}
LFS.ContentPath = sec.Key("LFS_CONTENT_PATH").MustString(filepath.Join(AppDataPath, "lfs"))
if !filepath.IsAbs(LFS.ContentPath) {
    LFS.ContentPath = filepath.Join(AppWorkPath, LFS.ContentPath)
}

LFS.HTTPAuthExpiry = sec.Key("LFS_HTTP_AUTH_EXPIRY").MustDuration(20 * time.Minute)

Now the err = sec.MapTo(&LFS) should sets the values of LFS including the JWTSecretBase64 and testing on master with a simple change it certainly does.

So I think we're back to the original:

n, err := base64.RawURLEncoding.Decode(LFS.JWTSecretBytes, []byte(LFS.JWTSecretBase64))

err is either not nil or n is not equal to 32

Basically your JWT secret is not correct. Now that code hasn't changed so I don't understand why it was working for you on 1.6

How can we work around this?

I suggest you regenerate a JWT Secret and see if that fixes your problem. If it does - then you can post your old JWT secret and we can test it or you can use the playground I've put in here to test your secret:

https://play.golang.org/p/GhUV1waCX6i

[captn3m0]

Thanks a lot for sticking with me on this. Looks like my secret is incorrect after all. I think I screwed up when I made the check last time and checked some other secret. I'll file a PR for the documentation, it needs to mention base64 (and how to generate a valid secret)

However, I'm unable to still generate a valid secret that decodes to 32 bytes.

I've tried openssl rand -base64 32 as well as openssl rand 32 | base64 -w 0 and neither seems to work correctly in the playground. My shell says 32 bytes for both:

openssl rand 32 | base64 -w 0|base64 -d | wc -c
32
openssl rand -base64 32 | base64 -d |wc -c
32

[zeripath]

It's not just base64 it's RawURLencoded base64

[captn3m0]

I tried generating it with go: https://play.golang.org/p/VeLnNm8piDY

Still can't get it to work.

[micw]

I still have the issue with "latest" which is <1 hour old.

I create the config as kubernetes ConfigMap, so it's read-only. In my logs I see lots of

2019/02/12 09:49:36 [...s/setting/setting.go:982 NewContext()] [E] Error saving generated JWT Secret to custom config: open /data/gitea/conf/app.ini: permission denied

The secret I set (for testing purpose, not my actual secret) is LFS_JWT_SECRET = ZbVOMOq1jKet3XH2UvIS1gTB4C-nVz2dJCIR37J61wQ

I have reviewed the current master code and tried to reproduce that: https://play.golang.org/p/RT4Sn4KOusH - on that test it seems to work properly.

[micw]

Update: Figured out that it was the "INTERNAL_TOKEN", not the LFS token it tried to write.

[micw]

I ended up in mounting the config at a different location and copy it on container startup.

[micw]

A possible option to solve this issue would be to put the secrets into a different file (secrets.ini).

[lunny]

After you fill all the necessary items, it should be read-only. Otherwise, you have to fill even database config items.

[nodiscc]

I hit the same problem while trying to deploy gitea with ansible with a read-only app.ini. It would be nice to document how to manually generate the secrets in the config file (using shell commands or python for example). As far as I can see the secrets that must be generated are:

• LFS_JWT_SECRET: 32 character key, encoded with unpadded base64. How to generate: ??? (Edit: not required)
• SECRET_KEY: 64 character key. How to generate: openssl rand -hex 64
• INTERNAL_TOKEN: ???

[lunny]

You can visit https://docs.gitea.io/en-us/command-line/ and find generate->secret command to do that.

[nodiscc]

Gitea is now running properly with a read-only config file for me. Indeed the solution was to generate INTERNAL_TOKEN using gitea generate secret INTERNAL_TOKEN. You can find my install procedure (using ansible) here

[lunny]

So let's close this issue. Please feel free to reopen it.

[nodiscc]

I submitted https://github.com/go-gitea/gitea/pull/6348 to document this a bit more.

[captn3m0]

A recent release enable oauth2, which means gitea will now try to generate a oauth JWT secret, and fail on the upgrade. Took me a while to figure out, because the error message doesn't say that the JWT is for oauth. (just error saving generating JWT secret).

Either disable oauth2 or generate a new JWT secret and put it in the oauth2 block for a fix.