こんにちはフリーランスの永田です。年始からSwiftUI案件で稼働します。案件獲得のコツは技術を磨くです。
AVCaptureMultiCamSession
MovieSession
上記の最新APIは両画面同時撮影する機能を持ちます。このAPIを使用して11月から作成して、OSS化しました。
宜しければスターお願い致します。
機能
- 前面背面同時撮影
- インナー画面の位置の可変。画面域の可変
- 2画面同じ比率での撮影
- 縦画面、横画面の対応(右側、左側対応)
環境
- Xcode11.3
- Swift5.0
- iOS13~
動作の一例
全部の向きに対応しました。 pic.twitter.com/sZh9StPknR
— DaisukeNagata (@dbank0208) December 17, 2019
全部の向きに対応しました。 pic.twitter.com/5XyvygpJZi
— DaisukeNagata (@dbank0208) December 17, 2019
発見から、修正まで1時間ぐらい。
— DaisukeNagata (@dbank0208) December 14, 2019
パープルボタンを押した時に2画面表示の場合、撮影画面が2ミリぐらいズレる動作を修正しました。 pic.twitter.com/eqFSi6eZC1
iPhone X系で、プレビュー画面と録画画面を合わせました。 pic.twitter.com/I8a12ZxHIz
— DaisukeNagata (@dbank0208) December 9, 2019
ポイント
全てのOSS機能を言語化するとかなりの文字量になってしまい、何を見ているのか、プログラムのどこにいるのかなどを保ちながら理解するのが、困難になる場合があるので、ポイントを集約します。
プレビュー画面は自作で合わせる。
BothSidesMixer.swift
とBothSidesMixer.metal
でプレビュー画面を設定しています。
BothSidesMixer.swift
のこの部分です。
プログラム内のコメント、Fixed with memory measuresの部分を同じメソッドでやると高速ループの中に入り、高メモリーがさらに高くなるので、
フラグが変更する度に呼び出すようにして、メモリーが上昇するのを極力制御しています。
getMtlSize メソッド
CVPixelBuffer
を新たに作成する機能です。
mix メソッド
CVPixelBufferPoolCreatePixelBuffer
はCVPixelBuffer
を作成しています。
fullScreenTexture
は全画面
pipTexture
はインナー画面
outputTexture
はMTLTexture
でfullScreenTexture
とpipTexture
を表示するTextureです。
BothSidesMixer.metal
で書き込み処理をしています。
inputImage
は2画面同時比率を実現しています。
座標を合わせる技はこちら
CGAffineTransform
の合わせ技です。
let inputImage = CIImage(cvImageBuffer: fullScreenPixelBuffer, options: nil).transformed(by: CGAffineTransform(scaleX: 0.5, y: 0.5).translatedBy(x: CGFloat(fullScreenTexture.width/2), y: 0))
fullScreenTexture
にMTLTexture
を作成しています。
makeTextureFromCVPixelBuffer
はMTLTexture
を作成するロジックを用意しています。
guard let newfullScreenTexture = makeTextureFromCVPixelBuffer(pixelBuffer: pixelBuffer) else {
print("AVCaptureMultiCamViewModel_mix")
return nil
}
var parameters = MixerParameters(pipPosition: pipPosition, pipSize: pipSize)
でBothSidesMixer.metal
に渡すパラメータを用意します。
let pipPosition = SIMD2(Float(pipFrame.origin.x) * Float(fullScreenTexture.width),Float(pipFrame.origin.y) * Float(fullScreenTexture.height))
let pipSize = SIMD2(Float(pipFrame.size.width) * Float(pipTexture.width),Float(pipFrame.size.height) * Float(pipTexture.height))
var parameters = MixerParameters(pipPosition: pipPosition, pipSize: pipSize)
commandQueue
はMetalを使用して、GPUに書き込むプログラムになっています。
commandEncoder.setTexture
とMetalFileのreporterMixer
メソッドを見ると理解しやすいと思います。
引数が連動しています。
画面を作成しているメソッドの全体像です。
// Fixed with memory measures
func getMtlSize(mtl: MTLTexture, sameRatio: Bool) {
if sameRatio == true && pixelBuffer == nil {
let options = [
kCVPixelBufferCGImageCompatibilityKey as String: true,
kCVPixelBufferCGBitmapContextCompatibilityKey as String: true,
kCVPixelBufferIOSurfacePropertiesKey as String: [:]
] as [String : Any]
cvReturn = CVPixelBufferCreate(kCFAllocatorDefault,
Int(mtl.width),
Int(mtl.height),
kCVPixelFormatType_32BGRA,
options as CFDictionary,
&pixelBuffer)
}
}
func mix(fullScreenPixelBuffer: CVPixelBuffer,
pipPixelBuffer: CVPixelBuffer,
_ sameRatio: Bool) -> CVPixelBuffer? {
guard let outputPixelBufferPool = outputPixelBufferPool else { return nil }
var newPixelBuffer: CVPixelBuffer?
CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, outputPixelBufferPool, &newPixelBuffer)
let outputPixelBuffer = newPixelBuffer
let outputTexture = makeTextureFromCVPixelBuffer(pixelBuffer: outputPixelBuffer)
guard var fullScreenTexture = makeTextureFromCVPixelBuffer(pixelBuffer: fullScreenPixelBuffer) else { return nil}
guard let pipTexture = makeTextureFromCVPixelBuffer(pixelBuffer: pipPixelBuffer) else { return nil}
if sameRatio == true {
// Fixed with memory measures
getMtlSize(mtl: fullScreenTexture,sameRatio: sameRatio)
if cvReturn == kCVReturnSuccess {
guard let pixelBuffer = pixelBuffer else {
print("AVCaptureMultiCamViewModel_mix")
return nil
}
let ciContext = CIContext()
let inputImage = CIImage(cvImageBuffer: fullScreenPixelBuffer, options: nil).transformed(by: CGAffineTransform(scaleX: 0.5, y: 0.5).translatedBy(x: CGFloat(fullScreenTexture.width/2), y: 0))
let colorSpace = CGColorSpaceCreateDeviceRGB()
ciContext.render(inputImage, to: pixelBuffer, bounds: inputImage.extent, colorSpace: colorSpace)
guard let newfullScreenTexture = makeTextureFromCVPixelBuffer(pixelBuffer: pixelBuffer) else {
print("AVCaptureMultiCamViewModel_mix")
return nil
}
fullScreenTexture = newfullScreenTexture
}
}
let pipPosition = SIMD2(Float(pipFrame.origin.x) * Float(fullScreenTexture.width), Float(pipFrame.origin.y) * Float(fullScreenTexture.height))
let pipSize = SIMD2(Float(pipFrame.size.width) * Float(pipTexture.width), Float(pipFrame.size.height) * Float(pipTexture.height))
var parameters = MixerParameters(pipPosition: pipPosition, pipSize: pipSize)
guard let commandQueue = commandQueue,
let commandBuffer = commandQueue.makeCommandBuffer(),
let commandEncoder = commandBuffer.makeComputeCommandEncoder(),
let computePipelineState = computePipelineState else {
print("BothSidesMixer_computePipelineState")
if let textureCache = textureCache { CVMetalTextureCacheFlush(textureCache, 0) }
return nil
}
commandEncoder.setComputePipelineState(computePipelineState)
commandEncoder.setTexture(fullScreenTexture, index: 0)
commandEncoder.setTexture(pipTexture, index: 2)
commandEncoder.setTexture(outputTexture, index: 3)
commandEncoder.setBytes(UnsafeMutableRawPointer(¶meters), length: MemoryLayout<MixerParameters>.size, index: 0)
let width = computePipelineState.threadExecutionWidth
let height = computePipelineState.maxTotalThreadsPerThreadgroup / width
let threadsPerThreadgroup = MTLSizeMake(width, height, 1)
let threadgroupsPerGrid = MTLSize(width: (fullScreenTexture.width + width - 1) / width,
height: (fullScreenTexture.height + height - 1) / height,
depth: 1)
commandEncoder.dispatchThreadgroups(threadgroupsPerGrid, threadsPerThreadgroup: threadsPerThreadgroup)
commandEncoder.endEncoding()
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
return outputPixelBuffer
}
肝心のカメラの合成クラス
主な合成メソッドです。このconfigureBackCamera
とconfigureFrontCamera
の2つのメソッドは録画をする上で、必須でどうしてもメモリが高メモリになってしまいます。iPhonePro11で260MB前後です。
背面カメラの設定configureBackCamera
メソッド
前面カメラの設定configureFrontCamera
メソッド
音の設定configureMicrophone
メソッド
処理の内容はコード見て理解して下さい。それが一番わかりやすいです。
final class BothSidesMultiCamViewModel: NSObject {
var session = AVCaptureMultiCamSession()
var aModel : BothSidesMultiCamSessionModel?
var backCamera : AVCaptureDevice?
var backDeviceInput : AVCaptureDeviceInput?
var frontDeviceInput : AVCaptureDeviceInput?
private var microphoneDeviceInput : AVCaptureDeviceInput?
private let backCameraVideoDataOutput = AVCaptureVideoDataOutput()
private let frontCameraVideoDataOutput = AVCaptureVideoDataOutput()
private let backMicrophoneAudioDataOutput = AVCaptureAudioDataOutput()
private let frontMicrophoneAudioDataOutput = AVCaptureAudioDataOutput()
private let dataOutputQueue = DispatchQueue(label: "data output queue")
override init() {
aModel = BothSidesMultiCamSessionModel()
super.init()
dataSet()
}
// Flash
func pushFlash() {
do {
try backCamera?.lockForConfiguration()
switch backCamera?.torchMode {
case .off:
backCamera?.torchMode = AVCaptureDevice.TorchMode.on
case .on:
backCamera?.torchMode = AVCaptureDevice.TorchMode.off
default: break
}
backCamera?.unlockForConfiguration()
} catch {
print("not be used")
}
}
func dataSet() {
aModel?.dataOutput(backdataOutput: backCameraVideoDataOutput,
frontDataOutput: frontCameraVideoDataOutput,
backicrophoneDataOutput: backMicrophoneAudioDataOutput,
fronticrophoneDataOutput: frontMicrophoneAudioDataOutput)
}
func screenShot(call: @escaping() -> Void, orientation: UIInterfaceOrientation) { aModel?.screenShot(call: call, orientation: orientation) }
func changeDviceType() {
guard let backDeviceInput = backDeviceInput else {
print("AVCaptureMultiCamViewModel_session")
return
}
session.removeInput(backDeviceInput)
session.removeOutput(backCameraVideoDataOutput)
backCamera = nil
}
func configureBackCamera(_ backCameraVideoPreviewLayer: AVCaptureVideoPreviewLayer?,deviceType :AVCaptureDevice.DeviceType) {
session.beginConfiguration()
defer {
session.commitConfiguration()
}
backCamera = AVCaptureDevice.default(deviceType, for: .video, position: .back)
guard let backCamera = backCamera else {
print("BothSidesMultiCamViewModel_backCamera")
return
}
// Camera support is limited.
if deviceType == .builtInWideAngleCamera {
do {
try backCamera.lockForConfiguration()
backCamera.focusMode = .continuousAutoFocus
backCamera.unlockForConfiguration()
} catch {
print("not be used")
}
}
do {
backDeviceInput = try AVCaptureDeviceInput(device: backCamera)
guard let backCameraDeviceInput = backDeviceInput,
session.canAddInput(backCameraDeviceInput) else {
print("AVCaptureMultiCamViewModel_backCameraDeviceInput")
return
}
session.addInputWithNoConnections(backCameraDeviceInput)
} catch {
return
}
guard let backCameraDeviceInput = backDeviceInput,
let backCameraVideoPort = backCameraDeviceInput.ports(for: .video,
sourceDeviceType: backCamera.deviceType,
sourceDevicePosition: backCamera.position).first else {
print("AVCaptureMultiCamViewModel_backCameraVideoPort")
return
}
guard session.canAddOutput(backCameraVideoDataOutput) else {
print("AVCaptureMultiCamViewModel_session.canAddOutput")
return
}
session.addOutputWithNoConnections(backCameraVideoDataOutput)
backCameraVideoDataOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)]
backCameraVideoDataOutput.setSampleBufferDelegate(aModel, queue: dataOutputQueue)
let backCameraVideoDataOutputConnection = AVCaptureConnection(inputPorts: [backCameraVideoPort], output: backCameraVideoDataOutput)
guard session.canAddConnection(backCameraVideoDataOutputConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(backCameraVideoDataOutputConnection)
backCameraVideoDataOutputConnection.videoOrientation = .portrait
guard let backCameraVideoPreviewLayer = backCameraVideoPreviewLayer else {
print("AVCaptureMultiCamViewModel_backCameraVideoPreviewLayer")
return
}
let backCameraVideoPreviewLayerConnection = AVCaptureConnection(inputPort: backCameraVideoPort, videoPreviewLayer: backCameraVideoPreviewLayer)
guard session.canAddConnection(backCameraVideoPreviewLayerConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(backCameraVideoPreviewLayerConnection)
}
func configureFrontCamera(_ frontCameraVideoPreviewLayer: AVCaptureVideoPreviewLayer?, deviceType :AVCaptureDevice.DeviceType) {
session.beginConfiguration()
defer {
session.commitConfiguration()
}
guard let frontCamera = AVCaptureDevice.default(deviceType, for: .video, position: .front) else {
print("AVCaptureMultiCamViewModel_frontCamera")
return
}
do {
frontDeviceInput = try AVCaptureDeviceInput(device: frontCamera)
guard let frontCameraDeviceInput = frontDeviceInput,
session.canAddInput(frontCameraDeviceInput) else {
print("AVCaptureMultiCamViewModel_frontCameraDeviceInput")
return
}
session.addInputWithNoConnections(frontCameraDeviceInput)
} catch {
return
}
guard let frontCameraDeviceInput = frontDeviceInput,
let frontCameraVideoPort = frontCameraDeviceInput.ports(for: .video,
sourceDeviceType: frontCamera.deviceType,
sourceDevicePosition: frontCamera.position).first else {
print("AVCaptureMultiCamViewModel_frontCameraVideoPort")
return
}
guard session.canAddOutput(frontCameraVideoDataOutput) else {
print("AVCaptureMultiCamViewModel_session.canAddOutput")
return
}
session.addOutputWithNoConnections(frontCameraVideoDataOutput)
frontCameraVideoDataOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)]
frontCameraVideoDataOutput.setSampleBufferDelegate(aModel, queue: dataOutputQueue)
let frontCameraVideoDataOutputConnection = AVCaptureConnection(inputPorts: [frontCameraVideoPort], output: frontCameraVideoDataOutput)
guard session.canAddConnection(frontCameraVideoDataOutputConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(frontCameraVideoDataOutputConnection)
frontCameraVideoDataOutputConnection.videoOrientation = .portrait
frontCameraVideoDataOutputConnection.automaticallyAdjustsVideoMirroring = false
frontCameraVideoDataOutputConnection.isVideoMirrored = true
guard let frontCameraVideoPreviewLayer = frontCameraVideoPreviewLayer else {
print("AVCaptureMultiCamViewModel_frontCameraVideoPreviewLayer")
return
}
let frontCameraVideoPreviewLayerConnection = AVCaptureConnection(inputPort: frontCameraVideoPort, videoPreviewLayer: frontCameraVideoPreviewLayer)
guard session.canAddConnection(frontCameraVideoPreviewLayerConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(frontCameraVideoPreviewLayerConnection)
frontCameraVideoPreviewLayerConnection.automaticallyAdjustsVideoMirroring = false
frontCameraVideoPreviewLayerConnection.isVideoMirrored = true
}
func configureMicrophone() {
session.beginConfiguration()
defer {
session.commitConfiguration()
}
guard let microphone = AVCaptureDevice.default(for: .audio) else {
print("AVCaptureMultiCamViewModel_microphone")
return
}
do {
self.microphoneDeviceInput = try AVCaptureDeviceInput(device: microphone)
guard let microphoneDeviceInput = microphoneDeviceInput,
session.canAddInput(microphoneDeviceInput) else {
print("AVCaptureMultiCamViewModel_microphoneDeviceInput")
return
}
session.addInputWithNoConnections(microphoneDeviceInput)
} catch {
return
}
guard let microphoneDeviceInput = microphoneDeviceInput,
let backMicrophonePort = microphoneDeviceInput.ports(for: .audio,
sourceDeviceType: microphone.deviceType,
sourceDevicePosition: .back).first else {
print("AVCaptureMultiCamViewModel_microphoneDeviceInput")
return
}
guard let frontMicrophonePort = microphoneDeviceInput.ports(for: .audio,
sourceDeviceType: microphone.deviceType,
sourceDevicePosition: .front).first else {
print("AVCaptureMultiCamViewModel_frontMicrophonePort")
return
}
guard session.canAddOutput(backMicrophoneAudioDataOutput) else {
print("AVCaptureMultiCamViewModel_session.canAddOutput")
return
}
session.addOutputWithNoConnections(backMicrophoneAudioDataOutput)
backMicrophoneAudioDataOutput.setSampleBufferDelegate(aModel, queue: dataOutputQueue)
guard session.canAddOutput(frontMicrophoneAudioDataOutput) else {
print("AVCaptureMultiCamViewModel_session.canAddOutput")
return
}
session.addOutputWithNoConnections(frontMicrophoneAudioDataOutput)
frontMicrophoneAudioDataOutput.setSampleBufferDelegate(aModel, queue: dataOutputQueue)
let backMicrophoneAudioDataOutputConnection = AVCaptureConnection(inputPorts: [backMicrophonePort], output: backMicrophoneAudioDataOutput)
guard session.canAddConnection(backMicrophoneAudioDataOutputConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(backMicrophoneAudioDataOutputConnection)
let frontMicrophoneAudioDataOutputConnection = AVCaptureConnection(inputPorts: [frontMicrophonePort], output: frontMicrophoneAudioDataOutput)
guard session.canAddConnection(frontMicrophoneAudioDataOutputConnection) else {
print("AVCaptureMultiCamViewModel_session.canAddConnection")
return
}
session.addConnection(frontMicrophoneAudioDataOutputConnection)
}
}
以上、AVCaptureMultiCamSessionを使用して作成したOSSの紹介をさせていただきました。
来年も皆様、良い一年になりますように。
貴重なお時間、お読み下さいまして、誠にありがとうございます。