CircleCI 2.0 移行時に直面した問題とその解決方法 for Rails Project

2018年8月31日を最後に、CircleCI 1.0が使えなくなることがアナウンスされました。

See: CircleCI 1.0 End of Life on August 31, 2018

社内向けに展開していたのですが、需要がありそうなので公開してみます。

ハマったポイント

1. rspec_junit_formatter がロードできないと言われる

$ #!/bin/bash -eo pipefail
bundle exec rspec --profile 10 \
                  --format RspecJunitFormatter \
                  --format progress \
                  --out test_results/rspec.xml \
                  $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)

bundler: failed to load command: rspec (/home/circleci/sbj-os/vendor/bundle/ruby/2.4.0/bin/rspec)
LoadError: cannot load such file -- rspec_junit_formatter

対応

Gemfile に sj26/rspec_junit_formatter を追加する。

公式ドキュメントを読めばちゃんと書いてあるのだけれど、CircleCI の設定を変える時に Gemfile を触るイメージがないのでハマりやすい。

If they succeed, it stores the test results using store_test_results so we can quickly see our build failures in the Test Summary section. This is why we added RspecJunitFormatter to our Gemfile.
Language Guide: Ruby - CircleCI

2. RSpecを並列で実施すると死ぬ

$ #!/bin/bash -eo pipefail
mkdir /tmp/test-results
TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
bundle exec rspec --format progress \
                  --format RspecJunitFormatter \
                  --out /tmp/test-results/rspec.xml \
                  "${TEST_FILES}"

bundler: failed to load command: rspec (/home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec)
NoMethodError: undefined method `captures' for nil:NilClass
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:124:in `parse_id'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1953:in `extract_location'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1892:in `block in get_files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `each'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `flat_map'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/flat_map.rb:7:in `flat_map'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1890:in `get_files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:928:in `files_to_run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1433:in `load_spec_files'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:100:in `setup'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:86:in `run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `load'
  /home/circleci/sbj_os/vendor/bundle/ruby/2.4.0/bin/rspec:23:in `<top (required)>'

対応

[10/11 追記]
調べた結果 Spec fails only on CI (details included) · Issue #2451 · rspec/rspec-core
に辿り着いた。コメントにある通り、下記の記述を削除することで対応した。

$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)

[12/22 追記]
公式ドキュメントのサンプルが修正されていた。下記の通り、shell 実行するように書き換えることで並列実行に成功。
Language Guide: Ruby - CircleCI

- - run:
-     name: run tests
-     command: |
-       mkdir /tmp/test-results
-       TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
-       bundle exec rspec --format progress \
-                         --format RspecJunitFormatter \
-                         --out /tmp/test-results/rspec.xml
+ - type: shell
+   command: |
+     bundle exec rspec --profile 10 \
+                       --format RspecJunitFormatter \
+                       --out test_results/rspec.xml \
+                       --format progress \
+                       $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)

3. Capybaraさんが特定のボタンを見つけられず、マウス操作に失敗する

Failure/Error: click_link 'カートに入れる'

Capybara::Poltergeist::MouseEventFailed:
  Firing a click at co-ordinates [990, 329.5] failed. Poltergeist detected another element with CSS selector 'html body.forSubdomain div.mainContents article div.container-fluid.os-container.os-sp-container div#dev-brand-site table.table.table-bordered.products tbody tr td' at this position. It may be overlapping the element you are trying to interact with. If you don't care about overlapping elements, try using node.trigger('click').
# spec/steps/order_steps.rb:69:in `block (2 levels) in <top (required)>'
# spec/steps/order_steps.rb:67:in `block in <top (required)>'
# spec/steps/order_steps.rb:63:in `block in <top (required)>'
# ./spec/features/cart.feature:10:in `run_step'
# ./spec/features/cart.feature:9:in `block (4 levels) in run_feature'
# ./spec/features/cart.feature:8:in `each'
# ./spec/features/cart.feature:8:in `block (3 levels) in run_feature'
# ./spec/features/cart.feature:10:in `カートに商品を投入する'
# ------------------
# --- Caused by: ---
# Capybara::Poltergeist::BrowserError:
#   There was an error inside the PhantomJS portion of Poltergeist. If this is the error returned, and not the cause of a more detailed error response, this is probably a bug, so please report it. 
#   
#   Poltergeist.MouseEventFailed: click
#   html body.forSubdomain div.mainContents article div.container-fluid.os-container.os-sp-container div#dev-brand-site table.table.table-bordered.products tbody tr td
#   {"x"=>990, "y"=>329.5}
#   spec/steps/order_steps.rb:69:in `block (2 levels) in <top (required)>'

対応

CircleCI上でCapybara, Poltergeist, PhantomJSを利用したテストがMouseEventFailed失敗す… - Sooey

4. 数回に一回、Capybaraさんが特定文言を見つけられない

Failure/Error: find('.alert-info', text: "added!", wait: 30)

Capybara::ElementNotFound:
  Unable to find css ".alert-info" with text "added!"
# spec/steps/order_steps.rb:72:in `block in <top (required)>'
# spec/steps/order_steps.rb:63:in `block in <top (required)>'
# ./spec/features/order.feature:750:in `run_step'
# ./spec/features/order.feature:749:in `block (4 levels) in run_feature'
# ./spec/features/order.feature:748:in `each'
# ./spec/features/order.feature:748:in `block (3 levels) in run_feature'
# ./spec/features/order.feature:750:in `カートに商品を投入する'

対応

NoRedInk/rspec-retry で retry するようにした。しばらくこれで様子を見る。

この話題に関しては別に記事を書いているので、興味があればこちらも参照して欲しい。

See: Capybara::ElementNotFound 回避に wait させるか retry させるか - Qiita