Docker-gitlab: Error on upgrade to 10.4

[Ilya-Kuchaev]

ArgumentError: key must be 32 bytes or longer
  from encryptor.rb:60:in `crypt'
  from encryptor.rb:36:in `encrypt'
  from attr_encrypted.rb:259:in `encrypt'
  from attr_encrypted.rb:337:in `encrypt'
  from attr_encrypted.rb:159:in `block (2 levels) in attr_encrypted'
  from attr_encrypted/adapters/active_record.rb:76:in `block in attr_encrypted'
  from active_record/attribute_assignment.rb:54:in `public_send'
  from active_record/attribute_assignment.rb:54:in `_assign_attribute'
  from active_record/attribute_assignment.rb:41:in `block in assign_attributes'
  from active_record/attribute_assignment.rb:35:in `each'
  from active_record/attribute_assignment.rb:35:in `assign_attributes'
  from attr_encrypted/adapters/active_record.rb:29:in `perform_attribute_assignment'
  from attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
  from active_record/core.rb:566:in `init_attributes'
  from active_record/core.rb:281:in `initialize'
  from active_record/inheritance.rb:61:in `new'
  from active_record/inheritance.rb:61:in `new'
  from active_record/reflection.rb:141:in `build_association'
  from active_record/associations/association.rb:250:in `build_record'
  from active_record/associations/singular_association.rb:29:in `build'
  from active_record/associations/builder/singular_association.rb:18:in `build_platform_kubernetes'
  from active_record/nested_attributes.rb:398:in `assign_nested_attributes_for_one_to_one_association'
  from active_record/nested_attributes.rb:343:in `platform_kubernetes_attributes='
  from active_record/attribute_assignment.rb:54:in `public_send'
  from active_record/attribute_assignment.rb:54:in `_assign_attribute'
  from active_record/attribute_assignment.rb:65:in `block in assign_nested_parameter_attributes'
  from active_record/attribute_assignment.rb:65:in `each'
  from active_record/attribute_assignment.rb:65:in `assign_nested_parameter_attributes'
  from active_record/attribute_assignment.rb:45:in `assign_attributes'
  from attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
  from attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
  from active_record/core.rb:566:in `init_attributes'
  from active_record/core.rb:281:in `initialize'
  from active_record/inheritance.rb:61:in `new'
  from active_record/inheritance.rb:61:in `new'
  from active_record/persistence.rb:33:in `create'
  from db/post_migrate/20171124104327_migrate_kubernetes_service_to_new_clusters_architectures.rb:122:in `block (2 levels) in up'
  from active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
  from active_record/relation/batches.rb:51:in `each'
  from active_record/relation/batches.rb:51:in `block in find_each'
  from active_record/relation/batches.rb:124:in `find_in_batches'
  from active_record/relation/batches.rb:50:in `find_each'
  from db/post_migrate/20171124104327_migrate_kubernetes_service_to_new_clusters_architectures.rb:121:in `block in up'
  from active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
  from active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
  from active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
  from active_record/transactions.rb:220:in `transaction'
  from db/post_migrate/20171124104327_migrate_kubernetes_service_to_new_clusters_architectures.rb:119:in `up'
  from active_record/migration.rb:611:in `exec_migration'
  from active_record/migration.rb:592:in `block (2 levels) in migrate'
  from benchmark.rb:293:in `measure'
  from active_record/migration.rb:591:in `block in migrate'
  from active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
  from active_record/migration.rb:590:in `migrate'
  from active_record/migration.rb:768:in `migrate'
  from active_record/migration.rb:1023:in `block in execute_migration_in_transaction'
  from active_record/migration.rb:1071:in `ddl_transaction'
  from active_record/migration.rb:1022:in `execute_migration_in_transaction'
  from active_record/migration.rb:984:in `block in migrate'
  from active_record/migration.rb:980:in `each'
  from active_record/migration.rb:980:in `migrate'
  from active_record/migration.rb:823:in `up'
  from active_record/migration.rb:801:in `migrate'
  from active_record/tasks/database_tasks.rb:139:in `migrate'
  from active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
  from rake/task.rb:251:in `block in execute'
  from rake/task.rb:251:in `each'
  from rake/task.rb:251:in `execute'
  from rake/task.rb:195:in `block in invoke_with_call_chain'
  from monitor.rb:214:in `mon_synchronize'
  from rake/task.rb:188:in `invoke_with_call_chain'
  from rake/task.rb:181:in `invoke'
  from rake/application.rb:160:in `invoke_task'
  from rake/application.rb:116:in `block (2 levels) in top_level'
  from rake/application.rb:116:in `each'
  from rake/application.rb:116:in `block in top_level'
  from rake/application.rb:125:in `run_with_threads'
  from rake/application.rb:110:in `top_level'
  from rake/application.rb:83:in `block in run'
  from rake/application.rb:186:in `standard_exception_handling'
  from rake/application.rb:80:in `run'
  from bundle/ruby/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
  from bundle/ruby/2.3.0/bin/rake:22:in `load'
  from bundle/ruby/2.3.0/bin/rake:22:in `<main>'

Is there any way to change the short db_key_base? I'm pretty sure that's the issue. I have the 10 digits one.

[Ilya-Kuchaev]

@sameersbn Any thoughts?

[compleatang]

For us, none of the 10.4.X images worked. They all threw in exactly the same manner.

[Ilya-Kuchaev]

Anyone?

[solidnerd]

@Ilya-Kuchaev It's a quite interessting problem because this should not work since the last 8 major version upgrades. The problem in here is that your encryption key is to short. It should be at least 32 digits. A better solution is to use 64 digits. Another problem could be the .secret file in your gitlab installation

The way how solve this is manually dececrypt the active records with the old digist and after that encrypt them with the longer digist. The last step is to use that variables in your docker-compose.yml

[compleatang]

@solidnerd or anyone else, is there anywhere I can look for how to perform the sequence you propose?

We use this image in k8s context rather than docker-compose.yml and it is not entirely clear to me what encryption key is throwing here. Is it the SSH host keys? Or another key?

[ldouchy]

Hi,
I'm exactly in the same situation as compleatang. Is there any information on how to update the key?

[ldouchy]

I managed to solve my issue and start a docker container in version 10.4.2 and even 10.4.4 after creating a 64 characters long secret for the GITLAB_SECRETS_DB_KEY_BASE parameter.

[Ilya-Kuchaev]

If somebody interested...

1. Create a dump of all project variables

#!/usr/bin/env ruby

require 'pry'
require 'gitlab'
require 'terminal-table'
require 'yaml'

Gitlab.endpoint = 'https://<gitlab url>/api/v4'
Gitlab.private_token = '<administrator token>'

projects = Gitlab.projects(order_by: 'id')

projects.auto_paginate do |project|
  begin 
    variables = Gitlab.variables(project.id)
    project_vars = {}
    variables.auto_paginate do |variable|
      project_vars[variable.key] = variable.value
    end
    unless project_vars.empty?
      puts project.name
      File.open("dump/#{project.id}.yml", 'w') { |f| f.write(YAML.dump(project_vars)) }
    end
  rescue
    next
  end
end

1. Upgrade GitLab to 10.4 with the new GITLAB_SECRETS_DB_KEY_BASE value
2. Delete all variables directly in postgres

delete from ci_variables;

1. Re-create variables from dump

#!/usr/bin/env ruby

require 'pry'
require 'gitlab'
require 'terminal-table'
require 'yaml'

Gitlab.endpoint = 'https://<gitlab url>/api/v4'
Gitlab.private_token = '<administrator token>'

Dir.chdir('dump')
Dir.glob('*.yml') do |item|
  variables = YAML.load(File.read(item))
  variables.each do |variable|
    puts File.basename(item, ".*")
    Gitlab.create_variable(File.basename(item, ".*"), variable[0], variable[1])
  end 
end

If you're using the Group based variables edit the scripts to get/create them

[compleatang]

Thank you @Ilya-Kuchaev your process worked well for me.