Docker-gitlab: backup fail in 8.16.6

[gimler]

i did not have configure a aws or google backup.

8.16.4

Uploading backup archive to remote storage  ... skipped

8.16.6

Uploading backup archive to remote storage {{GCS_BACKUP_BUCKET}} ... rake aborted!
Fog::Service::NotFound: google has no storage service
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:18:in `rescue in new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:8:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/storage.rb:27:in `new'
/home/git/gitlab/lib/backup/manager.rb:172:in `connect_to_remote_directory'
/home/git/gitlab/lib/backup/manager.rb:50:in `upload'
/home/git/gitlab/lib/backup/manager.rb:37:in `block in pack'
/home/git/gitlab/lib/backup/manager.rb:20:in `chdir'
/home/git/gitlab/lib/backup/manager.rb:20:in `pack'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
NoMethodError: undefined method `to_sym' for {"GCS_BACKUP_ACCESS_KEY_ID"=>nil}:Hash
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:19:in `block in prepare_service_settings'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:17:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:17:in `prepare_service_settings'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:18:in `block in prepare_service_settings'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:17:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/utils.rb:17:in `prepare_service_settings'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/service.rb:267:in `handle_settings'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/service.rb:98:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-google-0.5.0/lib/fog/storage/google.rb:13:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:16:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/storage.rb:27:in `new'
/home/git/gitlab/lib/backup/manager.rb:172:in `connect_to_remote_directory'
/home/git/gitlab/lib/backup/manager.rb:50:in `upload'
/home/git/gitlab/lib/backup/manager.rb:37:in `block in pack'
/home/git/gitlab/lib/backup/manager.rb:20:in `chdir'
/home/git/gitlab/lib/backup/manager.rb:20:in `pack'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

[jostyee]

maybe there's sth missing w/ https://github.com/sameersbn/docker-gitlab/pull/1030

[r2evans]

What file is that failure logged in?

I have this 8.16.6 image but am seeing a different backup failure: the backup tar file (%s_%Y_%m_%d_gitlab_backup.tar) is fine, but all of its components are also in the backup directory.

For instance, before today, the process just creates a single tar-file:

-rw-------  1 1000 1000 106956800 Feb 22 01:00 1487754057_2017_02_22_gitlab_backup.tar

But today, I find this:

-rw-------  1 1000 1000 106967040 Feb 23 01:00 1487840434_2017_02_23_gitlab_backup.tar
-rw-------  1 1000 1000       146 Feb 23 01:00 artifacts.tar.gz
-rw-rw-r--  1 1000 1000       159 Feb 23 01:00 backup_information.yml
-rw-------  1 1000 1000       146 Feb 23 01:00 builds.tar.gz
drwxrwxr-x  2 1000 1000      4096 Feb 23 01:00 db/
-rw-------  1 1000 1000   3709913 Feb 23 01:00 lfs.tar.gz
drwx------  4 1000 1000      4096 Feb 23 01:00 repositories/
-rw-------  1 1000 1000   2640677 Feb 23 01:00 uploads.tar.gz

They are confirmed redundant (the tar-file is complete, all components are identical). It's easy enough to remove them manually, but this will quickly clutter things. I don't know what will happen if the files exist when the next backup is attempted tomorrow morning (I don't really want to find out, meaning I have another daily cleanup task until this is fixed).

(I'm happy to move this to a different issue if it's completely unrelated.)

[jostyee]

@r2evans I've met similar issue in 8.17, or even worse. No complete backup tar file generated, but only components.

[solidnerd]

Hey,
I'm a little bit confused now. Is this a problem our Image or does it come from the upstream? The first error could be from us but this now seems me as an upstream error.

[jostyee]

My case https://github.com/sameersbn/docker-gitlab/issues/1094#issuecomment-282649270

• run docker-compose run --rm gitlab app:rake gitlab:backup:create manually failed, GitLab tries to backup Pages which I haven't enabled:
  Dumping pages ... rake aborted! Errno::ENOENT: No such file or directory @ realpath_rec - /home/git/data/shared/pages /home/git/gitlab/lib/backup/files.rb:9:in `realpath' /home/git/gitlab/lib/backup/files.rb:9:in `initialize' /home/git/gitlab/lib/backup/pages.rb:6:in `initialize' /home/git/gitlab/lib/tasks/gitlab/backup.rake:171:in `new' /home/git/gitlab/lib/tasks/gitlab/backup.rake:171:in `block (4 levels) in <top (required)>' /home/git/gitlab/lib/tasks/gitlab/backup.rake:16:in `block (3 levels) in <top (required)>' Tasks: TOP => gitlab:backup:pages:create (See full trace by running task with --trace)

• add SKIP=pages to backup cmd and it woking now.
  Creating backup archive: 1488251513_2017_02_28_gitlab_backup.tar ... done

[ayapapa]

Hi, I encountered an issue same as @gimler mentioned, and also found reason.
I will make a pull request to resolve it.

[solidnerd]

It will be available on the latest tag and later in the 8.17.0 when it's finished. My personal eta for this is the end of the week.

[solidnerd]

If it's not fixed please reopen the issue.

[QuickJack]

Does this mean that backups are not working in 8.16.6? Will there be an 8.16.6-1 release?

[r2evans]

Backups are working, they're just leaving some artifacts behind (including artifacts.tar.gz, coincidentally). The full-size %s_%Y_%m_%d_gitlab_backup.tar is still legitimate, and does not appear to be affected by this buglet. (I wouldn't say it's a "bug" or "debilitating problem" with the backup, just incomplete cleanup.)

I go in periodically and clear the redundant files, but it's only for my OCD ... everything else seems to be working just fine.

(At least, this is the behavior in my install of gitlab-8.16.6.)

[QuickJack]

I have tested everything on a testing server but did not test manual backup creation. I have done the upgrade to 8.16.6 on the production server just now. After manually trying to create a backup I saw the shocking error message.

There was another strange thing. Most projects were displayed with [done] when creating the backup. But some where displayed with [skipped]. What does this mean?

I hope that you are right and backups could be restored in case of a problem.

[QuickJack]

And there seems to be another problem. I found 21 backups on my testing server. I guess that old backups are not deleted automatically.

[r2evans]

@QuickJack where are you seeing the [done] and [skipped] messages? I'm seeing neither, apologize if I'm being dim.

[QuickJack]

It is displayed as some kind of status message for each repository as it is beeing backed up during the backup creation process.

[QuickJack]

I have created an issue for the other problem (https://github.com/sameersbn/docker-gitlab/issues/1122). It seems that there are a lot of backup related problems in 8.16.6.

[QuickJack]

@r2evans : It may be this bug https://gitlab.com/gitlab-org/gitlab-ce/issues/28854

[QuickJack]

I have just tested 8.17.4. The [SKIPPED] message is displayed for empty repositories only. The issue seems to be fixed.