gitlabの/admin/runnerが500番を返す

環境

Ubuntu 20.04.3 LTS

Omnibus GitLab 14.5.0 (aptでインストール)

状態

• gitlab.example.com/admin/runner
• gitlab.example.com/gitlab-instance-*/Monitoring/-/pipelines

が500番を返す。

log

グローバルIPが表示されていた部分は<ip address>に置き換えています。

$ gitlab-ctl tail

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin/runners" for <ip address> at 2021-11-29 11:15:30 +0000
Processing by Admin::RunnersController#index as HTML
  Rendered layout layouts/admin.html.haml (Duration: 53.8ms | Allocations: 11026)
Completed 500 Internal Server Error in 78ms (ActiveRecord: 8.5ms | Elasticsearch: 0.0ms | Allocations: 14590)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/admin/runners","format":"html","controller":"Admin::RunnersController","action":"index","status":500,"time":"2021-11-28T09:12:30.743Z","params":[],"correlation_id":"01FNJX2EN68BVCSN056FGQS6N0","meta.user":"viktor
demo","meta.caller_id":"Admin::RunnersController#index","meta.remote_ip":"<ip address>","meta.feature_category":"runner","meta.client_id":"user/1","remote_ip":"<ip address>","user_id":1,"username":"viktordemo","ua":"Mozilla/5.0 (Window
s NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","request_urgency":"default","target_duration_s":1,"redis_calls":7,"redis_duration_s":0.002013,"redis_read_bytes":185,"redis_write_bytes":153
2,"redis_cache_calls":4,"redis_cache_duration_s":0.001106,"redis_cache_read_bytes":4,"redis_cache_write_bytes":683,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000907,"redis_shared_state_read_bytes":181,"redis_shared_st
ate_write_bytes":849,"db_count":16,"db_write_count":2,"db_cached_count":6,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":16,"db_primary_cached_count":6,"db_pri
mary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.011,"cpu_s":0.10079,"mem_objects":22264,"mem_bytes":2210336,"mem_mallocs":5688,"mem_total_bytes":3100896,"pid":7701,"exception.class"
:"ActionView::Template::Error","exception.message":"","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:112:in
 `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:1
25:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'","app/models/concerns/token_authenticatable_strategies/base.rb:44:in `reset_token!'","app/models/concerns/token_authenticatabl
e_strategies/base.rb:37:in `ensure_token!'","app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'","app/models/application_setting_implementation.rb:394:in `runners_registration_token'","lib/gitlab
/current_settings.rb:32:in `method_missing'","app/helpers/ci/runners_helper.rb:68:in `admin_runners_data_attributes'","app/views/admin/runners/index.html.haml:4","app/controllers/application_controller.rb:136:in `render'","ee/lib/gitlab/
ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'","app/controllers/application_controller.rb:504:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/c
ontrollers/application_controller.rb:495:in `set_session_storage'","lib/gitlab/i18n.rb:105:in `with_locale'","lib/gitlab/i18n.rb:111:in `with_user_locale'","app/controllers/application_controller.rb:489:in `set_locale'","app/controllers/
application_controller.rb:483:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib
/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50
:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `
call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.r
b:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/relea
se_env.rb:13:in `call'"],"db_duration_s":0.00847,"view_duration_s":0.0,"duration_s":0.07901}
==> /var/log/gitlab/gitlab-rails/production.log <==

ActionView::Template::Error ():
    1: - breadcrumb_title _('Runners')
    2: - page_title _('Runners')
    3:
    4: #js-admin-runners{ data: admin_runners_data_attributes }
                                                                             
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
app/models/concerns/token_authenticatable_strategies/base.rb:44:in `reset_token!'
app/models/concerns/token_authenticatable_strategies/base.rb:37:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'
app/models/application_setting_implementation.rb:394:in `runners_registration_token'
lib/gitlab/current_settings.rb:32:in `method_missing'
app/helpers/ci/runners_helper.rb:68:in `admin_runners_data_attributes'
app/views/admin/runners/index.html.haml:4
app/controllers/application_controller.rb:136:in `render'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:504:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:495:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:489:in `set_locale'
app/controllers/application_controller.rb:483:in `set_current_context'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

やったけど解決しなかったこと

gitlab-rails console

参考

$ gitlab-rails console
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
 GitLab:       14.5.0 (f8796c0836e) FOSS
 GitLab Shell: 13.22.0
 PostgreSQL:   12.7
--------------------------------------------------------------------------------
Loading production environment (Rails 6.1.4.1)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!

ApplicationSetting.current.reset_runners_registration_token!の時点でエラーになる。

gitlab-rails dbconsole

参考

$ gitlab-rails dbconsole

エラーが出てコンソール自体開けない。

解決した

クリーンインストール

$ gitlab-ctl stop
$ gitlab-ctl uninstall
$ gitlab-ctl cleanse
$ gitlab-ctl remove-accounts
$ apt remove gitlab-ce

$ EXTERNAL_URL="https://gitlab.example.com" apt install gitlab-ce

データもリセットされてしまうが、自分は問題なかったためこれで解決しました。

原因はいまいち分からんかった・・・

最初インストールするときに１回DB関連でエラーを出してたのが怪しいと思ってる
reconfigureしてエラーが出なくなったから大丈夫だと思ってたけど影響してるのかな？