概要
状況を整理するためにメモ
(バグなのか、環境変数等が足りないせいなのかまだ判別がついていない状態です)
ちゃんと、バージョンアップする前にbackupを取らないとだめですね。。。
環境
ホストOS: Ubuntu 14.04
Docker: Docker version 1.12.3, build 6b644ec
docker compose:docker-compose version 1.8.1, build 878cff1
Dockerコンテナ:
- sameersbn/gitlab:8.16.6 //8.12.3からアップグレード
- sameersbn/redis:latest
- sameersbn/postgresql:9.5-1
直接は関係ないが以下のコンテナもいっしょに起動している
- fluentd/fluend
- kibana:4.6.2
- elasticsearch:2.4.1
症状
バックアップのRakeタスクを実行するとエラーが発生する
$ docker-compose run --rm gitlab app:rake gitlab:backup:create
1回目
$ docker-compose run --rm gitlab app:rake gitlab:backup:create
(メモ撮り忘れた)
fatal: Need a repository to create a bundle.
2回目(dbがバックアップできなくなりました)
$ docker-compose run --rm gitlab app:rake gitlab:backup:create
Creating network "docker_default" with the default driver
Creating docker_postgresql_1
Creating docker_elasticsearch_1
Creating docker_redis_1
Creating docker_fluentd_1
Initializing logdir...
Initializing datadir...
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database..
Configuring gitlab::redis
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::unicorn...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::lfs...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::backups...
Configuring gitlab::backups::schedule...
Configuring gitlab::registry...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Running raketask gitlab:backup:create...
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "projects" does not exist
LINE 5: WHERE a.attrelid = '"projects"'::regclass
^
: SELECT a.attname, format_type(a.atttypid, a.atttypmod),
pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '"projects"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:733:in `column_definitions'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:197:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/schema_cache.rb:43:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:93:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/acts-as-taggable-on-4.0.0/lib/acts_as_taggable_on/taggable/cache.rb:37:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:98:in `columns_hash'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:969:in `block in create_binds'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `partition'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `create_binds'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:954:in `build_where'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:584:in `where!'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:574:in `where'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:10:in `where'
/home/git/gitlab/app/models/project.rb:214:in `block in <class:Project>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (3 levels) in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:302:in `scoping'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (2 levels) in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `inject'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `block in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:127:in `evaluate_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:103:in `build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:33:in `default_scoped'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:28:in `all'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:9:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:74:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
PG::UndefinedTable: ERROR: relation "projects" does not exist
LINE 5: WHERE a.attrelid = '"projects"'::regclass
^
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:733:in `column_definitions'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:197:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/schema_cache.rb:43:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:93:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/acts-as-taggable-on-4.0.0/lib/acts_as_taggable_on/taggable/cache.rb:37:in `columns'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:98:in `columns_hash'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:969:in `block in create_binds'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `partition'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `create_binds'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:954:in `build_where'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:584:in `where!'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:574:in `where'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:10:in `where'
/home/git/gitlab/app/models/project.rb:214:in `block in <class:Project>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (3 levels) in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:302:in `scoping'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (2 levels) in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `inject'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `block in build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:127:in `evaluate_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:103:in `build_default_scope'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:33:in `default_scoped'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:28:in `all'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:9:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:74:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)
Githubみるとバックアップ関連のissueがいくつか上がっている
遭遇したエラーとは別なので直接関係はないかもだが。
- Rotation of backup archives not working in 8.16.6 · Issue #1122 · sameersbn/docker-gitlab
- backup fail in 8.16.6 · Issue #1094 · sameersbn/docker-gitlab
コメントには
https://github.com/sameersbn/docker-gitlab/issues/1094#issuecomment-286041125
I have created an issue for the other problem (#1122). It seems that there are a lot of backup related problems in 8.16.6.
とあるのでほかにもいくつか不具合があるのかも。
アップデートは可能だが、ダウングレードは単純にコンテナ差し替えるだけだとできないので注意
試しに、gitlab:8.15.4にダウングレードしようとすると、下記のエラーがでて起動しませんでした。
(おそらくバージョンアップ前のバックアップからrestoreしないとだめ)
gitlab_1 | ERROR:
gitlab_1 | Cannot downgrade from GitLab version 8.16.6 to 8.15.4.
gitlab_1 | Only upgrades are allowed. Please use sameersbn/gitlab:8.16.6 or higher.
gitlab_1 | Cannot continue. Aborting!