XCTestCaseで作ったテストをxcodebuildで実行する方法 #iOS

CIにテストを実行するためにXcodeの場合、xcodebuildを用いて処理するように設定していくと思います。
その際、端末別やOS別のテストを１つのクラス内に作っているケースがあるとして、テストケースを改修するコストよりもコマンド側で個別に呼び出す方針を取ったとき、どう設定すれば良いのかまとめておく。

目的

CIに端末別とOS別にテストを行うジョブを構築
特定のXCTestCaseのテストをxcodebuildで実行させる （←本題）

環境

Mac
- macOS Mojave - version 10.14.6
Xcode
- version 11.2
CI
- Jenkins

準備

CI機となる共有PC（Mac miniなど）にXcodeをダウンロードし、１度Xcodeを起動させて初回の設定を済ませておく。
また今回は、CIに関することは割愛とする。

次のように$ instruments -s devicesまたは$ xcrun simctl listからどのシミュレータを扱うか確認しておく。
もし扱いたい機種やOSがdevicesに無い場合は、Xcodeからデバイスを作成すること。

$ instruments -s devices
CoreData: annotation:  Failed to load optimized model at path '/Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/Frameworks/InstrumentsPackaging.framework/Versions/A/Resources/XRPackageModel.momd/XRPackageModel 9.0.omo'
Known Devices:
gremitoのMacBook Pro [664E45D8-39FF-5D37-A42C-26FE51F10A9B]
Apple TV (13.2) [ADD78466-F544-4D33-9321-C81D4D0FC150] (Simulator)
Apple TV 4K (13.2) [C132945D-C5E9-40A6-915D-879E1B82BF2D] (Simulator)
Apple TV 4K (at 1080p) (13.2) [CF8827A4-3A31-48C9-9418-C31C8C9866EA] (Simulator)
Apple Watch Series 4 - 40mm (6.1) [2D3C75B6-07A6-4702-94C5-14A3CA12A818] (Simulator)
Apple Watch Series 4 - 44mm (6.1) [91BADC1D-4E20-44D3-A2A7-51E73693BC29] (Simulator)
iPad (7th generation) (13.2) [32489E3C-0348-477D-980A-11F0A0C27445] (Simulator)
iPad Air (3rd generation) (13.2) [53AA791E-5D2A-4989-8AAB-0564A1B10491] (Simulator)
iPad Pro (11-inch) (13.2) [7D8E1B5E-2286-475A-9406-17613AEC5998] (Simulator)
iPad Pro (12.9-inch) (3rd generation) (13.2) [602C1998-075C-4F53-8E9D-9D4233F26747] (Simulator)
iPad Pro (9.7-inch) (13.2) [C2B230EE-A01D-4E76-BA91-2214812D5047] (Simulator)
iPhone 11 (13.2) [E451C70D-4AA3-4C4B-A88C-E7A5EB1126A2] (Simulator)
iPhone 11 Pro (13.2) [B042F66E-2277-482D-97AE-9F2884EEDAE0] (Simulator)
iPhone 11 Pro (13.2) + Apple Watch Series 5 - 40mm (6.1) [32D82CF1-770B-4839-8F31-90C4CCCADCE7] (Simulator)
iPhone 11 Pro Max (13.2) [BF1E7BC6-A927-4C43-ADA7-8FEB2A658881] (Simulator)
iPhone 11 Pro Max (13.2) + Apple Watch Series 5 - 44mm (6.1) [26D86766-3435-42E9-B0D0-1F9F62DA7681] (Simulator)
iPhone 8 (13.2) [BDDE5F62-2721-41D5-ACA9-1EF6394627BC] (Simulator)
iPhone 8 Plus (13.2) [BD3AFEE1-975A-41DE-BE91-776D7E37F270] (Simulator)

$ xcrun simctl list
== Device Types ==
iPhone 4s (com.apple.CoreSimulator.SimDeviceType.iPhone-4s)
iPhone 5 (com.apple.CoreSimulator.SimDeviceType.iPhone-5)
iPhone 5s (com.apple.CoreSimulator.SimDeviceType.iPhone-5s)
iPhone 6 Plus (com.apple.CoreSimulator.SimDeviceType.iPhone-6-Plus)
iPhone 6 (com.apple.CoreSimulator.SimDeviceType.iPhone-6)
iPhone 6s (com.apple.CoreSimulator.SimDeviceType.iPhone-6s)
iPhone 6s Plus (com.apple.CoreSimulator.SimDeviceType.iPhone-6s-Plus)
iPhone SE (com.apple.CoreSimulator.SimDeviceType.iPhone-SE)
iPhone 7 (com.apple.CoreSimulator.SimDeviceType.iPhone-7)
iPhone 7 Plus (com.apple.CoreSimulator.SimDeviceType.iPhone-7-Plus)
iPhone 8 (com.apple.CoreSimulator.SimDeviceType.iPhone-8)
iPhone 8 Plus (com.apple.CoreSimulator.SimDeviceType.iPhone-8-Plus)
iPhone X (com.apple.CoreSimulator.SimDeviceType.iPhone-X)
iPhone Xs (com.apple.CoreSimulator.SimDeviceType.iPhone-XS)
iPhone Xs Max (com.apple.CoreSimulator.SimDeviceType.iPhone-XS-Max)
iPhone Xʀ (com.apple.CoreSimulator.SimDeviceType.iPhone-XR)
iPhone 11 (com.apple.CoreSimulator.SimDeviceType.iPhone-11)
iPhone 11 Pro (com.apple.CoreSimulator.SimDeviceType.iPhone-11-Pro)
iPhone 11 Pro Max (com.apple.CoreSimulator.SimDeviceType.iPhone-11-Pro-Max)
iPad 2 (com.apple.CoreSimulator.SimDeviceType.iPad-2)
iPad Retina (com.apple.CoreSimulator.SimDeviceType.iPad-Retina)
iPad Air (com.apple.CoreSimulator.SimDeviceType.iPad-Air)
iPad mini 2 (com.apple.CoreSimulator.SimDeviceType.iPad-mini-2)
iPad mini 3 (com.apple.CoreSimulator.SimDeviceType.iPad-mini-3)
iPad mini 4 (com.apple.CoreSimulator.SimDeviceType.iPad-mini-4)
iPad Air 2 (com.apple.CoreSimulator.SimDeviceType.iPad-Air-2)
iPad Pro (9.7-inch) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro--9-7-inch-)
iPad Pro (12.9-inch) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro)
iPad (5th generation) (com.apple.CoreSimulator.SimDeviceType.iPad--5th-generation-)
iPad Pro (12.9-inch) (2nd generation) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro--12-9-inch---2nd-generation-)
iPad Pro (10.5-inch) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro--10-5-inch-)
iPad (6th generation) (com.apple.CoreSimulator.SimDeviceType.iPad--6th-generation-)
iPad (7th generation) (com.apple.CoreSimulator.SimDeviceType.iPad--7th-generation-)
iPad Pro (11-inch) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro--11-inch-)
iPad Pro (12.9-inch) (3rd generation) (com.apple.CoreSimulator.SimDeviceType.iPad-Pro--12-9-inch---3rd-generation-)
iPad mini (5th generation) (com.apple.CoreSimulator.SimDeviceType.iPad-mini--5th-generation-)
iPad Air (3rd generation) (com.apple.CoreSimulator.SimDeviceType.iPad-Air--3rd-generation-)
Apple TV (com.apple.CoreSimulator.SimDeviceType.Apple-TV-1080p)
Apple TV 4K (com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-4K)
Apple TV 4K (at 1080p) (com.apple.CoreSimulator.SimDeviceType.Apple-TV-4K-1080p)
Apple Watch - 38mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-38mm)
Apple Watch - 42mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-42mm)
Apple Watch Series 2 - 38mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-2-38mm)
Apple Watch Series 2 - 42mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-2-42mm)
Apple Watch Series 3 - 38mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-3-38mm)
Apple Watch Series 3 - 42mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-3-42mm)
Apple Watch Series 4 - 40mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-4-40mm)
Apple Watch Series 4 - 44mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-4-44mm)
Apple Watch Series 5 - 40mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-5-40mm)
Apple Watch Series 5 - 44mm (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-5-44mm)
== Runtimes ==
iOS 13.2 (13.2 - 17B84) - com.apple.CoreSimulator.SimRuntime.iOS-13-2 
tvOS 13.2 (13.2 - 17K81) - com.apple.CoreSimulator.SimRuntime.tvOS-13-2 
watchOS 6.1 (6.1 - 17S83) - com.apple.CoreSimulator.SimRuntime.watchOS-6-1 
== Devices ==
-- iOS 13.2 --
    iPhone 8 (BDDE5F62-2721-41D5-ACA9-1EF6394627BC) (Shutdown) 
    iPhone 8 Plus (BD3AFEE1-975A-41DE-BE91-776D7E37F270) (Shutdown) 
    iPhone 11 (E451C70D-4AA3-4C4B-A88C-E7A5EB1126A2) (Shutdown) 
    iPhone 11 Pro (B042F66E-2277-482D-97AE-9F2884EEDAE0) (Shutdown) 
    iPhone 11 Pro Max (BF1E7BC6-A927-4C43-ADA7-8FEB2A658881) (Shutdown) 
    iPad Pro (9.7-inch) (C2B230EE-A01D-4E76-BA91-2214812D5047) (Shutdown) 
    iPad (7th generation) (32489E3C-0348-477D-980A-11F0A0C27445) (Shutdown) 
    iPad Pro (11-inch) (7D8E1B5E-2286-475A-9406-17613AEC5998) (Shutdown) 
    iPad Pro (12.9-inch) (3rd generation) (602C1998-075C-4F53-8E9D-9D4233F26747) (Shutdown) 
    iPad Air (3rd generation) (53AA791E-5D2A-4989-8AAB-0564A1B10491) (Shutdown) 
-- tvOS 13.2 --
    Apple TV (ADD78466-F544-4D33-9321-C81D4D0FC150) (Shutdown) 
    Apple TV 4K (C132945D-C5E9-40A6-915D-879E1B82BF2D) (Shutdown) 
    Apple TV 4K (at 1080p) (CF8827A4-3A31-48C9-9418-C31C8C9866EA) (Shutdown) 
-- watchOS 6.1 --
    Apple Watch Series 4 - 40mm (2D3C75B6-07A6-4702-94C5-14A3CA12A818) (Shutdown) 
    Apple Watch Series 4 - 44mm (91BADC1D-4E20-44D3-A2A7-51E73693BC29) (Shutdown) 
    Apple Watch Series 5 - 40mm (32D82CF1-770B-4839-8F31-90C4CCCADCE7) (Shutdown) 
    Apple Watch Series 5 - 44mm (26D86766-3435-42E9-B0D0-1F9F62DA7681) (Shutdown) 
== Device Pairs ==
8266A1DA-60F1-443E-A9C3-FBED1C9E94F7 (active, disconnected)
    Watch: Apple Watch Series 5 - 40mm (32D82CF1-770B-4839-8F31-90C4CCCADCE7) (Shutdown)
    Phone: iPhone 11 Pro (B042F66E-2277-482D-97AE-9F2884EEDAE0) (Shutdown)
9EA20D9C-4745-4A3B-A89A-22237520ADD1 (active, disconnected)
    Watch: Apple Watch Series 5 - 44mm (26D86766-3435-42E9-B0D0-1F9F62DA7681) (Shutdown)
    Phone: iPhone 11 Pro Max (BF1E7BC6-A927-4C43-ADA7-8FEB2A658881) (Shutdown)

設定

以下のようにCIに設定することで特定のテストが処理させるようになる。
念のためCIで動作して確認する前にターミナルで問題なく動作するか確認しておくこと。

# iPhone 11(iOS 13.2)のテスト
xcodebuild \
 -workspace iOSAppProject/iOSAppProject.xcodeproj/project.xcworkspace \
 -scheme iOSAppProject \
 -destination OS=13.2,name="iPhone 11" \
 -only-testing:iOSAppProjectTests/iPhoneTests/iOS12OrBelowTest \
 -configuration Debug build test

# iPhone X(iOS 12.1)のテスト
xcodebuild \
 -workspace iOSAppProject/iOSAppProject.xcodeproj/project.xcworkspace \
 -scheme iOSAppProject \
 -destination OS=12.1,name="iPhone 10" \
 -only-testing:iOSAppProjectTests/iPhoneTests/iOS13OrHigherTest \
 -configuration Debug build test

# iPad Pro (12.9-inch)(iOS 13.2)のテスト
xcodebuild \
 -workspace iOSAppProject/iOSAppProject.xcodeproj/project.xcworkspace \
 -scheme iOSAppProject \
 -destination OS=13.2,name="iPad Pro (12.9-inch)" \
 -only-testing:iOSAppProjectTests/iPadTests/iPadProTest \
 -configuration Debug build test

# iPad mini 2(iOS 12.4.3)のテスト
xcodebuild \
 -workspace iOSAppProject/iOSAppProject.xcodeproj/project.xcworkspace \
 -scheme iOSAppProject \
 -destination OS=12.4.3,name="iPad mini 2" \
 -only-testing:iOSAppProjectTests/iPadTests/iPadMiniTest \
 -configuration Debug build test

上記のように特定のオプションの設定を変えるだけなので、シェル対応で変数化すると良いでしょう。
端末やOSはシミュレータの確認で表示された内容を指定できます。

XCTestCaseで作ったテストをxcodebuildで実行する方法

目的

環境

準備

設定

参考記事