GitLab QA orchestratorはGitLab開発者が改修したコードにデグレがないとか、想定通りの挙動になっているかとか、GitLab開発のCI/CD専用ツールだと思っていましたか？
違うぜ！自分のインスタンスに対しても自動テストスイートが実行できるんだぞこれ！
GitLab QA orchestratorとは
GitLab QA orchestratorはGitLab開発チームが開発したGitLabプロダクトの自動テスト実行ツールです。
Omnibusパッケージとして作られていて、ブラウザー上でのEnd to Endテストが簡単に実行できるようにうなっています。
インストール方法
gitlab-qaというGemが公開されています。
gem install gitlab-qaでインストールするとgitlab-qaのコマンドがターミナルで使えるようになります。
特定のインスタンスに対する実行方法
さっそく自分のお気に入りのインスタンスに対して実行してみます。
> export GITLAB_USERNAME=root
> export GITLAB_PASSWORD=yay!
> gitlab-qa Test::Instance::Any EE:11.0.0-ee https://gitlab.bakaneco.io
Running test suite `Test::Instance` for gitlab-ee
Docker shell command: `docker run -t --rm --net=bridge --env GITLAB_USERNAME="$GITLAB_USERNAME" --env GITLAB_PASSWORD="$GITLAB_PASSWORD" --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /tmp/gitlab-qa/gitlab-qa-run-2018-07-19-21-26-44-f9ff9115/gitlab-ee-qa-a4990339:/home/qa/tmp:z --name gitlab-ee-qa-a4990339 gitlab/gitlab-ee-qa:11.0.0-ee Test::Instance https://gitlab.bakaneco.io`
Run options: include {:core=>true}
Randomized with seed 54509
merge request rebase
Switched to a new branch 'master'
Already on 'master'
Already on 'master'
rebases source branch of merge request
merge request squash commits
Switched to a new branch 'master'
Already on 'master'
Switched to a new branch 'qa-test-feature-d009f7cd6a985edb'
when squash commits is marked before merge (FAILED - 1)
HTML screenshot: file:///home/qa/tmp/screenshot_2018-07-19-12-30-41.766.html
Image screenshot: file:///home/qa/tmp/screenshot_2018-07-19-12-30-41.766.png
cloning code using a deploy key
Executing `docker pull gitlab/gitlab-runner:alpine`
alpine: Pulling from gitlab/gitlab-runner
911c6d0c7995: Pulling fs layer
aeb647805e91: Pulling fs layer
166817d40d50: Pulling fs layer
5e690cf98d1c: Pulling fs layer
768db1615c8a: Pulling fs layer
f08b1e2e86fa: Pulling fs layer
797cced447d5: Pulling fs layer
768db1615c8a: Waiting
f08b1e2e86fa: Waiting
797cced447d5: Waiting
5e690cf98d1c: Waiting
aeb647805e91: Download complete
911c6d0c7995: Verifying Checksum
911c6d0c7995: Download complete
911c6d0c7995: Pull complete
aeb647805e91: Pull complete
166817d40d50: Download complete
768db1615c8a: Download complete
797cced447d5: Download complete
166817d40d50: Pull complete
f08b1e2e86fa: Verifying Checksum
f08b1e2e86fa: Download complete
5e690cf98d1c: Verifying Checksum
5e690cf98d1c: Download complete
5e690cf98d1c: Pull complete
768db1615c8a: Pull complete
f08b1e2e86fa: Pull complete
797cced447d5: Pull complete
Digest: sha256:a8cab05a4555a65ccd31e4ad0e6971a86ae0bfb137fcad7e2656a44db54c7a7d
Status: Downloaded newer image for gitlab/gitlab-runner:alpine
Executing `docker network inspect test`
[]
Error: No such network: test
Executing `docker run -d --rm --entrypoint=/bin/sh --network bridge --name qa-runner-1532003444 -e CI_SERVER_URL=https://gitlab.bakaneco.io/ -e REGISTER_NON_INTERACTIVE=true -e REGISTRATION_TOKEN=EkF7peAqs9JC5joU1ZeV -e RUNNER_EXECUTOR=shell -e RUNNER_TAG_LIST=qa,docker -e RUNNER_NAME=qa-runner-1532003444 gitlab/gitlab-runner:ubuntu -c 'gitlab-runner register && gitlab-runner run' `
Unable to find image 'gitlab/gitlab-runner:ubuntu' locally
ubuntu: Pulling from gitlab/gitlab-runner
b234f539f7a1: Pulling fs layer
55172d420b43: Pulling fs layer
5ba5bbeb6b91: Pulling fs layer
43ae2841ad7a: Pulling fs layer
f6c9c6de4190: Pulling fs layer
705463cce042: Pulling fs layer
6aae22efe036: Pulling fs layer
d31a84bae2a6: Pulling fs layer
6b00830c5af7: Pulling fs layer
ad8ca5cf9cf9: Pulling fs layer
43ae2841ad7a: Waiting
705463cce042: Waiting
6aae22efe036: Waiting
d31a84bae2a6: Waiting
6b00830c5af7: Waiting
f6c9c6de4190: Waiting
ad8ca5cf9cf9: Waiting
55172d420b43: Verifying Checksum
55172d420b43: Download complete
5ba5bbeb6b91: Verifying Checksum
5ba5bbeb6b91: Download complete
f6c9c6de4190: Verifying Checksum
f6c9c6de4190: Download complete
43ae2841ad7a: Verifying Checksum
43ae2841ad7a: Download complete
b234f539f7a1: Verifying Checksum
b234f539f7a1: Download complete
d31a84bae2a6: Verifying Checksum
d31a84bae2a6: Download complete
b234f539f7a1: Pull complete
55172d420b43: Pull complete
5ba5bbeb6b91: Pull complete
43ae2841ad7a: Pull complete
f6c9c6de4190: Pull complete
6aae22efe036: Verifying Checksum
6aae22efe036: Download complete
705463cce042: Verifying Checksum
705463cce042: Download complete
ad8ca5cf9cf9: Verifying Checksum
ad8ca5cf9cf9: Download complete
705463cce042: Pull complete
6aae22efe036: Pull complete
d31a84bae2a6: Pull complete
6b00830c5af7: Verifying Checksum
6b00830c5af7: Download complete
6b00830c5af7: Pull complete
ad8ca5cf9cf9: Pull complete
Digest: sha256:cda817249a6620649ad2828b5cc28182078b109a789a8d3087df812c16da5945
Status: Downloaded newer image for gitlab/gitlab-runner:ubuntu
17af5fe91e05a74ed1cb8ab082d5e53eccde533666c7ec958140c164cd1b6bea
Executing `["ssh-keygen", "-t", "ed25519", "-b", "256", "-f", "/tmp/d20180719-23-1bn61n5/id_ed25519", "-N", ""]`
Generating public/private ed25519 key pair.
Your identification has been saved in /tmp/d20180719-23-1bn61n5/id_ed25519.
Your public key has been saved in /tmp/d20180719-23-1bn61n5/id_ed25519.pub.
The key fingerprint is:
SHA256:IrW6HQWz0hQiTI4cqDJlJE/rKg/qz+9huGER5Dn9iAA root@9f719f075c7d
The key's randomart image is:
+--[ED25519 256]--+
|E==.. . |
|+*Boo. . |
|o=+= .= |
|+.. ++o= |
|...oo.=.S |
| . o+ o |
|+ +.o. |
|ooo +o.. |
|o.o++o. |
+----[SHA256]-----+
Switched to a new branch 'DEPLOY_KEY_ed25519_256'
user sets up a deploy key with QA::Runtime::Key::ED25519() to clone code using pipelines
Executing `["ssh-keygen", "-t", "rsa", "-b", "8192", "-f", "/tmp/d20180719-23-o11920/id_rsa", "-N", ""]`
Generating public/private rsa key pair.
Your identification has been saved in /tmp/d20180719-23-o11920/id_rsa.
Your public key has been saved in /tmp/d20180719-23-o11920/id_rsa.pub.
The key fingerprint is:
SHA256:S/T3YP/lmmWhnWd/BNrXS26gXVyV8jvS1URIsBOTfeE root@9f719f075c7d
The key's randomart image is:
+---[RSA 8192]----+
| +=.++|
| .=+.+|
| . o oEo|
| . . ...+|
| S . +ooo=|
| . . o.*+B*|
| . o.O=O|
| . . @=|
| +.=|
+----[SHA256]-----+
Switched to a new branch 'DEPLOY_KEY_rsa_8192'
user sets up a deploy key with QA::Runtime::Key::RSA(8192) to clone code using pipelines
Executing `["ssh-keygen", "-t", "ecdsa", "-b", "521", "-f", "/tmp/d20180719-23-18otzsi/id_ecdsa", "-N", ""]`
Generating public/private ecdsa key pair.
Your identification has been saved in /tmp/d20180719-23-18otzsi/id_ecdsa.
Your public key has been saved in /tmp/d20180719-23-18otzsi/id_ecdsa.pub.
The key fingerprint is:
SHA256:/zjZnn9DIqHNgeugBGg1ezEzu/kv4xn4H5vncdM3Op4 root@9f719f075c7d
The key's randomart image is:
+---[ECDSA 521]---+
| |
| o = |
| o o * . |
| o o o . o |
|. o o S = o |
| +.. + + ... |
| ..o.o..+.ooo.|
| ..+o.*++o+.+|
| .+==++=Eo..|
+----[SHA256]-----+
Switched to a new branch 'DEPLOY_KEY_ecdsa_521'
user sets up a deploy key with QA::Runtime::Key::ECDSA(521) to clone code using pipelines
Executing `docker rm -f qa-runner-1532003444`
qa-runner-1532003444
creates issue
user creates issue
branch protection support
when developers and maintainers are allowed to push to a protected branch
Switched to a new branch 'master'
Already on 'master'
Switched to a new branch 'protected-branch'
user with push rights successfully pushes to the protected branch
when developers and maintainers are not allowed to push to a protected branch
Switched to a new branch 'master'
Already on 'master'
Switched to a new branch 'protected-branch'
user without push rights fails to push to the protected branch
API basics
user creates a project with a file and deletes them afterwards
clone code from the repository
with regular account over http
user performs a deep clone
user performs a shallow clone
CI/CD Pipelines
Executing `docker pull gitlab/gitlab-runner:alpine`
alpine: Pulling from gitlab/gitlab-runner
Digest: sha256:a8cab05a4555a65ccd31e4ad0e6971a86ae0bfb137fcad7e2656a44db54c7a7d
Status: Image is up to date for gitlab/gitlab-runner:alpine
Executing `docker network inspect test`
[]
Error: No such network: test
Executing `docker run -d --rm --entrypoint=/bin/sh --network bridge --name qa-runner-1532003812 -e CI_SERVER_URL=https://gitlab.bakaneco.io/ -e REGISTER_NON_INTERACTIVE=true -e REGISTRATION_TOKEN=4V28ZQoK8oFgtWrkbtAj -e RUNNER_EXECUTOR=shell -e RUNNER_TAG_LIST=qa,e2e -e RUNNER_NAME=qa-runner-1532003812 gitlab/gitlab-runner:alpine -c 'gitlab-runner register && gitlab-runner run' `
15a8986ed9c592f7942a340e8c22c7f7f11e838715550d8539e76b2889682dda
Executing `docker rm -f qa-runner-1532003812`
qa-runner-1532003812
user registers a new specific runner
Executing `docker pull gitlab/gitlab-runner:alpine`
alpine: Pulling from gitlab/gitlab-runner
Digest: sha256:a8cab05a4555a65ccd31e4ad0e6971a86ae0bfb137fcad7e2656a44db54c7a7d
Status: Image is up to date for gitlab/gitlab-runner:alpine
Executing `docker network inspect test`
[]
Error: No such network: test
Executing `docker run -d --rm --entrypoint=/bin/sh --network bridge --name qa-runner-1532003844 -e CI_SERVER_URL=https://gitlab.bakaneco.io/ -e REGISTER_NON_INTERACTIVE=true -e REGISTRATION_TOKEN=wx7oQohSyfqtDyVmHjUq -e RUNNER_EXECUTOR=shell -e RUNNER_TAG_LIST=qa,test -e RUNNER_NAME=qa-runner-1532003844 gitlab/gitlab-runner:alpine -c 'gitlab-runner register && gitlab-runner run' `
2128e7c7efc03642c198c19c76a1dbd06a4a9fa830389335d1e0a5e3db05d434
Switched to a new branch 'master'
Waiting for the runner to process the pipeline
Executing `docker rm -f qa-runner-1532003844`
qa-runner-1532003844
users creates a new pipeline
secret variables support
user adds a secret variable
push code to repository
with regular account over http
Switched to a new branch 'master'
user pushes code to the repository
API users
submit request with an invalid token
when authenticated
get list of users
submit request with an invalid user name
submit request with a valid user name
creates a merge request
Switched to a new branch 'master'
Already on 'master'
user creates a new merge request
create a new project
user creates a new project
deploy keys support
Executing `["ssh-keygen", "-t", "rsa", "-b", "4096", "-f", "/tmp/d20180719-23-o2khkk/id_rsa", "-N", ""]`
Generating public/private rsa key pair.
Your identification has been saved in /tmp/d20180719-23-o2khkk/id_rsa.
Your public key has been saved in /tmp/d20180719-23-o2khkk/id_rsa.pub.
The key fingerprint is:
SHA256:AahKXwCkgds+eYDGB6YiI4traFMuHPd35mli479g9XI root@9f719f075c7d
The key's randomart image is:
+---[RSA 4096]----+
|+o. .. |
|o+ .. . |
|=+... . |
|O++. . . |
|B=oo. S. |
|+.++. . . |
|o.=o. o . E |
|o* . ..=.++ |
|o o +.O=. |
+----[SHA256]-----+
user adds a deploy key
activity page
Switched to a new branch 'master'
push creates an event in the activity page
standard user login
user logs in using credentials
Failures:
1) merge request squash commits when squash commits is marked before merge
Failure/Error: expect(repository.commits.size).to eq 3
expected: 3
got: 4
(compared using ==)
# ./qa/specs/features/merge_request/squash_spec.rb:43:in `block (4 levels) in <module:QA>'
# ./qa/scenario/actable.rb:14:in `perform'
# ./qa/git/repository.rb:12:in `block (2 levels) in perform'
# ./qa/git/repository.rb:12:in `chdir'
# ./qa/git/repository.rb:12:in `block in perform'
# ./qa/git/repository.rb:11:in `perform'
# ./qa/specs/features/merge_request/squash_spec.rb:33:in `block (3 levels) in <module:QA>'
# ./qa/scenario/actable.rb:14:in `perform'
# ./qa/specs/features/merge_request/squash_spec.rb:27:in `block (2 levels) in <module:QA>'
# ./qa/specs/runner.rb:22:in `perform'
# ./qa/scenario/template.rb:7:in `block in perform'
# ./qa/scenario/template.rb:5:in `tap'
# ./qa/scenario/template.rb:5:in `perform'
# ./qa/scenario/test/instance.rb:22:in `perform'
# ./qa/scenario/template.rb:7:in `block in perform'
# ./qa/scenario/template.rb:5:in `tap'
# ./qa/scenario/template.rb:5:in `perform'
# ./qa/scenario/bootable.rb:14:in `launch!'
Top 10 slowest examples (586.88 seconds, 73.5% of total time):
merge request rebase rebases source branch of merge request
199.26 seconds ./qa/specs/features/merge_request/rebase_spec.rb:3
cloning code using a deploy key user sets up a deploy key with QA::Runtime::Key::RSA(8192) to clone code using pipelines
114.03 seconds ./qa/specs/features/project/deploy_key_clone_spec.rb:42
CI/CD Pipelines users creates a new pipeline
45.93 seconds ./qa/specs/features/project/pipelines_spec.rb:28
cloning code using a deploy key user sets up a deploy key with QA::Runtime::Key::ECDSA(521) to clone code using pipelines
40.48 seconds ./qa/specs/features/project/deploy_key_clone_spec.rb:42
cloning code using a deploy key user sets up a deploy key with QA::Runtime::Key::ED25519() to clone code using pipelines
39.02 seconds ./qa/specs/features/project/deploy_key_clone_spec.rb:42
merge request squash commits when squash commits is marked before merge
33.62 seconds ./qa/specs/features/merge_request/squash_spec.rb:3
branch protection support when developers and maintainers are allowed to push to a protected branch user with push rights successfully pushes to the protected branch
30.23 seconds ./qa/specs/features/repository/protected_branches_spec.rb:26
branch protection support when developers and maintainers are not allowed to push to a protected branch user without push rights fails to push to the protected branch
29.76 seconds ./qa/specs/features/repository/protected_branches_spec.rb:37
creates a merge request user creates a new merge request
29.53 seconds ./qa/specs/features/merge_request/create_spec.rb:3
deploy keys support user adds a deploy key
25.02 seconds ./qa/specs/features/project/add_deploy_key_spec.rb:3
Top 10 slowest example groups:
merge request rebase
199.26 seconds average (199.26 seconds / 1 example) ./qa/specs/features/merge_request/rebase_spec.rb:2
cloning code using a deploy key
83.52 seconds average (250.55 seconds / 3 examples) ./qa/specs/features/project/deploy_key_clone_spec.rb:4
CI/CD Pipelines
34.74 seconds average (69.48 seconds / 2 examples) ./qa/specs/features/project/pipelines_spec.rb:2
merge request squash commits
33.62 seconds average (33.62 seconds / 1 example) ./qa/specs/features/merge_request/squash_spec.rb:2
branch protection support
30 seconds average (59.99 seconds / 2 examples) ./qa/specs/features/repository/protected_branches_spec.rb:2
creates a merge request
29.53 seconds average (29.53 seconds / 1 example) ./qa/specs/features/merge_request/create_spec.rb:2
deploy keys support
25.02 seconds average (25.02 seconds / 1 example) ./qa/specs/features/project/add_deploy_key_spec.rb:2
push code to repository
20.32 seconds average (20.32 seconds / 1 example) ./qa/specs/features/repository/push_spec.rb:2
secret variables support
16.02 seconds average (16.02 seconds / 1 example) ./qa/specs/features/project/add_secret_variable_spec.rb:2
creates issue
15.59 seconds average (15.59 seconds / 1 example) ./qa/specs/features/project/create_issue_spec.rb:2
Finished in 13 minutes 19 seconds (files took 0.3984 seconds to load)
24 examples, 1 failure
Failed examples:
rspec ./qa/specs/features/merge_request/squash_spec.rb:3 # merge request squash commits when squash commits is marked before merge
Randomized with seed 54509
/Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/shellout.rb:31:in `block in execute!': Docker command `docker run -t --rm --net=bridge --env GITLAB_USERNAME="$GITLAB_USERNAME" --env GITLAB_PASSWORD="$GITLAB_PASSWORD" --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /tmp/gitlab-qa/gitlab-qa-run-2018-07-19-21-26-44-f9ff9115/gitlab-ee-qa-a4990339:/home/qa/tmp:z --name gitlab-ee-qa-a4990339 gitlab/gitlab-ee-qa:11.0.0-ee Test::Instance https://gitlab.bakaneco.io` failed! (Gitlab::QA::Docker::Shellout::StatusError)
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/2.4.0/open3.rb:205:in `popen_run'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/2.4.0/open3.rb:190:in `popen2e'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/shellout.rb:19:in `execute!'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/command.rb:36:in `execute!'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/engine.rb:22:in `block in run'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/engine.rb:16:in `tap'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/docker/engine.rb:16:in `run'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/component/specs.rb:24:in `perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:6:in `tap'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:6:in `perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/test/instance/any.rb:12:in `perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:6:in `tap'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/lib/gitlab/qa/scenario/template.rb:6:in `perform'
from /Users/jb/.rbenv/versions/2.4.2/lib/ruby/gems/2.4.0/gems/gitlab-qa-2.0.0/exe/gitlab-qa:7:in `<top (required)>'
from /Users/jb/.rbenv/versions/2.4.2/bin/gitlab-qa:23:in `load'
from /Users/jb/.rbenv/versions/2.4.2/bin/gitlab-qa:23:in `<main>'
動きとしては、GemからGitLab QAのDockerコンテナが実行されて対象インスタンスにアクセスし、headless chromeブラウザーでテストシナリオを実行するイメージです。
失敗したテストケースは/tmp/gitlab-qa配下に画像とHTMLが保存されて参照できるようになっています。
ちなみに自分の場合はmerge request squash commits when squash commits is marked before mergeが失敗しました。
新規インストール後、アップグレード後の緊張感よ、さらば！