logo标签列表

如何同步AVPlayer和MTKView

iosswiftsynchronizationavplayermetal
5
我有一个项目，用户可以拍摄视频，然后添加滤镜或更改基本设置，例如亮度和对比度。为了实现这一点，我使用了BBMetalImage，它基本上将视频返回到MTKView中（在该项目中称为BBMetalView）。
一切工作得很好 - 我可以播放视频，添加所需的滤镜和效果，但没有音频。我向作者询问了此事，他建议使用AVPlayer（或AVAudioPlayer）。所以我这样做了。然而，视频和音频不同步。可能是因为最初的比特率不同，库的作者还提到，由于过滤器过程（这需要时间），帧速率也可能不同：

渲染视图FPS与实际速率不完全相同。 因为视频源输出帧经过过滤器处理， 过滤器处理时间是可变的。

首先，我将视频裁剪到所需的纵横比（4:5）。我使用AVVideoProfileLevelH264HighAutoLevel作为AVVideoProfileLevelKey将其保存到本地文件（480x600）。使用NextLevelSessionExporter，我的音频配置如下： AVEncoderBitRateKey: 128000，AVNumberOfChannelsKey: 2，AVSampleRateKey: 44100。
然后，BBMetalImage库接受此保存的音频文件，并提供MTKView（BBMetalView）以显示视频，允许我实时添加滤镜和效果。该设置看起来像这样：
self.metalView = BBMetalView(frame: CGRect(x: 0, y: self.view.center.y - ((UIScreen.main.bounds.width * 1.25) / 2), width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.width * 1.25))
self.view.addSubview(self.metalView)
self.videoSource = BBMetalVideoSource(url: outputURL)
self.videoSource.playWithVideoRate = true
self.videoSource.audioConsumer = self.metalAudio
self.videoSource.add(consumer: self.metalView)
self.videoSource.add(consumer: self.videoWriter)
self.audioItem = AVPlayerItem(url: outputURL)                            
self.audioPlayer = AVPlayer(playerItem: self.audioItem)
self.playerLayer = AVPlayerLayer(player: self.audioPlayer)
self.videoPreview.layer.addSublayer(self.playerLayer!)
self.playerLayer?.frame = CGRect(x: 0, y: 0, width: 0, height: 0)
self.playerLayer?.backgroundColor = UIColor.black.cgColor
self.startVideo()

startVideo() 的使用方法如下：

audioPlayer.seek(to: .zero)
audioPlayer.play()
videoSource.start(progress: { (frameTime) in
    print(frameTime)
}) { [weak self] (finish) in
guard let self = self else { return }
    self.startVideo()
}

由于使用了外部库，这些内容可能比较模糊。不过，我的问题很简单：有没有办法将MTKView与我的AVPlayer同步？这会帮助我很多，我相信Silence-GitHub也会将此功能添加到库中，以帮助其他用户。欢迎提出任何解决方案！

因此，为了让你有所了解——音频通常不是以“帧速率”录制的，而视频肯定有“帧速率”。这其中的诀窍在于MTKView可能会出现一些问题，这可能会影响frameRate。例如，假设由于某种原因，MTKView每15fps更新一次，我们不希望我们的音频以每15fps的速度录制。我们希望音频每“毫秒”录制一次。因此，简单的布局应该是线程1-尽力记录视频和线程2-记录音频，以便它没有等待MTKView的问题。 - impression7vx
啊，不对。也许我没有表达清楚我的流程。我在临时目录中记录并保存了一个视频文件 - 它具有正确的视频和音频轨道。这些使用NextLevelSessionExporter导出，并且当我将其保存到我的相机胶卷中时，它们都可以正常播放并同步。然而，问题出现在接下来的部分。我通过MetalView渲染这个原始视频文件，因为现在，我想允许用户添加滤镜和效果。所以我通过这个MetalView播放视频帧，并且我目前在我的应用程序中使用AVPlayer播放音频。 - PennyWise
好的，我明白了。所以，当你正常播放时，比如通过相机胶卷，一切都很好。但是，当你将两者分开，并尝试使用自定义的MTKView来播放视频和单独的AVPlayer来播放音频时，就会出现这种不协调。有没有特定的模式？视频总是比音频快/慢吗？ - impression7vx
我之所以这样问，是因为如果视频以随机速率播放 - 比如每帧以60 fps处理应用的滤镜，那么音频可能会正常播放。然而，想象一下，如果过滤导致MTKView以1 fps进行处理，但您想播放音频。如果视频以1 fps播放，您知道音频会听起来像什么吗？我也不想知道。因此，这取决于您是要尝试减慢/加快音频还是尝试加快过滤。 - impression7vx
请查看 https://pastebin.com/0v0VDSjU。我看到某种模式，其中视频帧的时间刻度为600，而音频的时间刻度为44100。这与NextLevelSessionExporter的输出匹配，我使用AVVideoAverageBitRateKey为6000000和AVSampleRateKey为44100将mp4文件保存到我的临时文件中。这看起来很有前途，对吗？ - 然而，这是当我将视频保存到mp4文件时。这不是MTKView的输出。我如何访问这些值？我能显示MTKView的帧率吗？ - PennyWise
显示剩余29条评论
3个回答
1
我将BBMetalVideoSource定制如下，然后它就能正常工作：
  1. Create a delegate in BBMetalVideoSource to get the current time of the audio player with which we want to sync

  2. In func private func processAsset(progress:, completion:), I replace this block of code if useVideoRate { //... } by: 

    if useVideoRate {
    if let playerTime = delegate.getAudioPlayerCurrentTime() {
        let diff = CMTimeGetSeconds(sampleFrameTime) - playerTime
        if diff > 0.0 {
            sleepTime = diff
            if sleepTime > 1.0 {
                sleepTime = 0.0
            }
            usleep(UInt32(1000000 * sleepTime))
        } else {
            sleepTime = 0
        }
    }
}
这段代码帮助我们解决了两个问题：1. 预览视频效果时没有声音，2. 将音频与视频同步。
您所说的“预览视频效果时没有声音”是什么意思？这是否意味着应用效果时音频停止播放，因为我想保持音频运行但与视频同步。 - PennyWise
@PennyWise BBMetalImage的视频示例存在两个问题：1.没有音频；2.如果我们使用AVPlayer来播放音频以解决问题1，则音频会不同步。我只想说上面的代码解决了问题2，当然也包括问题1。 - hoangdado
getAudioPlayerCurrentTime()是什么，或者它来自哪里？我知道它必须被设置，但是我该如何操作？返回AVPlayer.currentItem?.currentTime? - PennyWise
我收到了许多错误，可能是因为我以错误的方式解析时间。有关getAudioPlayerCurrentTime()的任何提示？ - PennyWise
兄弟，我不得不改了几个东西，但基本上这个方法起作用了。就目前而言，我的视频和音频是同步的。我永远感激你，非常感谢。哇。 - PennyWise
0

根据您的情况，您似乎需要尝试以下两种方法之一：

1） 尝试应用某种覆盖层，以获得所需的视频效果。我可以尝试这样做，但我个人没有这样做过。

2） 这需要更多的时间预先准备 - 即程序需要花费一些时间（取决于您的过滤方式，时间会有所不同）来重新创建具有所需效果的新视频。您可以尝试这个方法，看看是否适合您。

我使用了SO的一些源代码制作了自己的VideoCreator。

//Recreates a new video with applied filter
    public static func createFilteredVideo(asset: AVAsset, completionHandler: @escaping (_ asset: AVAsset) -> Void) {
        let url = (asset as? AVURLAsset)!.url
        let snapshot = url.videoSnapshot()
        guard let image = snapshot else { return }
        let fps = Int32(asset.tracks(withMediaType: .video)[0].nominalFrameRate)
        let writer = VideoCreator(fps: Int32(fps), width: image.size.width, height: image.size.height, audioSettings: nil)

        let timeScale = asset.duration.timescale
        let timeValue = asset.duration.value
        let frameTime = 1/Double(fps) * Double(timeScale)
        let numberOfImages = Int(Double(timeValue)/Double(frameTime))
        let queue = DispatchQueue(label: "com.queue.queue", qos: .utility)
        let composition = AVVideoComposition(asset: asset) { (request) in
            let source = request.sourceImage.clampedToExtent()
            //This is where you create your filter and get your filtered result. 
            //Here is an example
            let filter = CIFilter(name: "CIBlendWithMask")
            filter!.setValue(maskImage, forKey: "inputMaskImage")
            filter!.setValue(regCIImage, forKey: "inputImage")
            let filteredImage = filter!.outputImage.clamped(to: source.extent)
            request.finish(with: filteredImage, context: nil)
        }

        var i = 0
        getAudioFromURL(url: url) { (buffer) in
            writer.addAudio(audio: buffer, time: .zero)
            i == 0 ? writer.startCreatingVideo(initialBuffer: buffer, completion: {}) : nil
            i += 1
        }

        let group = DispatchGroup()
        for i in 0..<numberOfImages {
            group.enter()
            autoreleasepool {
                let time = CMTime(seconds: Double(Double(i) * frameTime / Double(timeScale)), preferredTimescale: timeScale)
                let image = url.videoSnapshot(time: time, composition: composition)
                queue.async {

                    writer.addImageAndAudio(image: image!, audio: nil, time: time.seconds)
                    group.leave()
                }
            }
        }
        group.notify(queue: queue) {
            writer.finishWriting()
            let url = writer.getURL()

            //Now create exporter to add audio then do completion handler
            completionHandler(AVAsset(url: url))

        }
    }

    static func getAudioFromURL(url: URL, completionHandlerPerBuffer: @escaping ((_ buffer:CMSampleBuffer) -> Void)) {
        let asset = AVURLAsset(url: url, options: [AVURLAssetPreferPreciseDurationAndTimingKey: NSNumber(value: true as Bool)])

        guard let assetTrack = asset.tracks(withMediaType: AVMediaType.audio).first else {
            fatalError("Couldn't load AVAssetTrack")
        }


        guard let reader = try? AVAssetReader(asset: asset)
            else {
                fatalError("Couldn't initialize the AVAssetReader")
        }
        reader.timeRange = CMTimeRange(start: .zero, duration: asset.duration)

        let outputSettingsDict: [String : Any] = [
            AVFormatIDKey: Int(kAudioFormatLinearPCM),
            AVLinearPCMBitDepthKey: 16,
            AVLinearPCMIsBigEndianKey: false,
            AVLinearPCMIsFloatKey: false,
            AVLinearPCMIsNonInterleaved: false
        ]
        let readerOutput = AVAssetReaderTrackOutput(track: assetTrack,
                                                    outputSettings: outputSettingsDict)
        readerOutput.alwaysCopiesSampleData = false
        reader.add(readerOutput)

        while reader.status == .reading {
            guard let readSampleBuffer = readerOutput.copyNextSampleBuffer() else { break }
            completionHandlerPerBuffer(readSampleBuffer)

        }
    }

extension URL {
    func videoSnapshot(time:CMTime? = nil, composition:AVVideoComposition? = nil) -> UIImage? {
        let asset = AVURLAsset(url: self)
        let generator = AVAssetImageGenerator(asset: asset)
        generator.appliesPreferredTrackTransform = true
        generator.requestedTimeToleranceBefore = .zero
        generator.requestedTimeToleranceAfter = .zero
        generator.videoComposition = composition

        let timestamp = time == nil ? CMTime(seconds: 1, preferredTimescale: 60) : time

        do {
            let imageRef = try generator.copyCGImage(at: timestamp!, actualTime: nil)
            return UIImage(cgImage: imageRef)
        }
        catch let error as NSError
        {
            print("Image generation failed with error \(error)")
            return nil
        }
    }
}

以下是视频创建器

//
//  VideoCreator.swift
//  AKPickerView-Swift
//
//  Created by Impression7vx on 7/16/19.
//

import UIKit

import AVFoundation
import UIKit
import Photos

@available(iOS 11.0, *)
public class VideoCreator: NSObject {

    private var settings:RenderSettings!
    private var imageAnimator:ImageAnimator!

    public override init() {
        self.settings = RenderSettings()
        self.imageAnimator = ImageAnimator(renderSettings: self.settings)
    }

    public convenience init(fps: Int32, width: CGFloat, height: CGFloat, audioSettings: [String:Any]?) {
        self.init()
        self.settings = RenderSettings(fps: fps, width: width, height: height)
        self.imageAnimator = ImageAnimator(renderSettings: self.settings, audioSettings: audioSettings)
    }

    public convenience init(width: CGFloat, height: CGFloat) {
        self.init()
        self.settings = RenderSettings(width: width, height: height)
        self.imageAnimator = ImageAnimator(renderSettings: self.settings)
    }

    func startCreatingVideo(initialBuffer: CMSampleBuffer?, completion: @escaping (() -> Void)) {
        self.imageAnimator.render(initialBuffer: initialBuffer) {
            completion()
        }
    }

    func finishWriting() {
        self.imageAnimator.isDone = true
    }

    func addImageAndAudio(image:UIImage, audio:CMSampleBuffer?, time:CFAbsoluteTime) {
        self.imageAnimator.addImageAndAudio(image: image, audio: audio, time: time)
    }

    func getURL() -> URL {
        return settings!.outputURL
    }

    func addAudio(audio: CMSampleBuffer, time: CMTime) {
        self.imageAnimator.videoWriter.addAudio(buffer: audio, time: time)
    }
}


@available(iOS 11.0, *)
public struct RenderSettings {

    var width: CGFloat = 1280
    var height: CGFloat = 720
    var fps: Int32 = 2   // 2 frames per second
    var avCodecKey = AVVideoCodecType.h264
    var videoFilename = "video"
    var videoFilenameExt = "mov"

    init() { }

    init(width: CGFloat, height: CGFloat) {
        self.width = width
        self.height = height
    }

    init(fps: Int32) {
        self.fps = fps
    }

    init(fps: Int32, width: CGFloat, height: CGFloat) {
        self.fps = fps
        self.width = width
        self.height = height
    }

    var size: CGSize {
        return CGSize(width: width, height: height)
    }

    var outputURL: URL {
        // Use the CachesDirectory so the rendered video file sticks around as long as we need it to.
        // Using the CachesDirectory ensures the file won't be included in a backup of the app.
        let fileManager = FileManager.default
        if let tmpDirURL = try? fileManager.url(for: .cachesDirectory, in: .userDomainMask, appropriateFor: nil, create: true) {
            return tmpDirURL.appendingPathComponent(videoFilename).appendingPathExtension(videoFilenameExt)
        }
        fatalError("URLForDirectory() failed")
    }
}

@available(iOS 11.0, *)
public class ImageAnimator {

    // Apple suggests a timescale of 600 because it's a multiple of standard video rates 24, 25, 30, 60 fps etc.
    static let kTimescale: Int32 = 600

    let settings: RenderSettings
    let videoWriter: VideoWriter
    var imagesAndAudio:SynchronizedArray<(UIImage, CMSampleBuffer?, CFAbsoluteTime)> = SynchronizedArray<(UIImage, CMSampleBuffer?, CFAbsoluteTime)>()
    var isDone:Bool = false
    let semaphore = DispatchSemaphore(value: 1)

    var frameNum = 0

    class func removeFileAtURL(fileURL: URL) {
        do {
            try FileManager.default.removeItem(atPath: fileURL.path)
        }
        catch _ as NSError {
            // Assume file doesn't exist.
        }
    }

    init(renderSettings: RenderSettings, audioSettings:[String:Any]? = nil) {
        settings = renderSettings
        videoWriter = VideoWriter(renderSettings: settings, audioSettings: audioSettings)
    }

    func addImageAndAudio(image: UIImage, audio: CMSampleBuffer?, time:CFAbsoluteTime) {
        self.imagesAndAudio.append((image, audio, time))
//        print("Adding to array -- \(self.imagesAndAudio.count)")
    }

    func render(initialBuffer: CMSampleBuffer?, completion: @escaping ()->Void) {

        // The VideoWriter will fail if a file exists at the URL, so clear it out first.
        ImageAnimator.removeFileAtURL(fileURL: settings.outputURL)

        videoWriter.start(initialBuffer: initialBuffer)
        videoWriter.render(appendPixelBuffers: appendPixelBuffers) {
            //ImageAnimator.saveToLibrary(self.settings.outputURL)
            completion()
        }

    }

    // This is the callback function for VideoWriter.render()
    func appendPixelBuffers(writer: VideoWriter) -> Bool {

        //Don't stop while images are NOT empty
        while !imagesAndAudio.isEmpty || !isDone {

            if(!imagesAndAudio.isEmpty) {
                let date = Date()

                if writer.isReadyForVideoData == false {
                    // Inform writer we have more buffers to write.
//                    print("Writer is not ready for more data")
                    return false
                }

                autoreleasepool {
                    //This should help but truly doesn't suffice - still need a mutex/lock
                    if(!imagesAndAudio.isEmpty) {
                        semaphore.wait() // requesting resource
                        let imageAndAudio = imagesAndAudio.first()!
                        let image = imageAndAudio.0
//                        let audio = imageAndAudio.1
                        let time = imageAndAudio.2
                        self.imagesAndAudio.removeAtIndex(index: 0)
                        semaphore.signal() // releasing resource
                        let presentationTime = CMTime(seconds: time, preferredTimescale: 600)

//                        if(audio != nil) { videoWriter.addAudio(buffer: audio!) }
                        let success = videoWriter.addImage(image: image, withPresentationTime: presentationTime)
                        if success == false {
                            fatalError("addImage() failed")
                        }
                        else {
//                            print("Added image @ frame \(frameNum) with presTime: \(presentationTime)")
                        }

                        frameNum += 1
                        let final = Date()
                        let timeDiff = final.timeIntervalSince(date)
//                        print("Time: \(timeDiff)")
                    }
                    else {
//                        print("Images was empty")
                    }
                }
            }
        }

        print("Done writing")
        // Inform writer all buffers have been written.
        return true
    }

}

@available(iOS 11.0, *)
public class VideoWriter {

    let renderSettings: RenderSettings
    var audioSettings: [String:Any]?
    var videoWriter: AVAssetWriter!
    var videoWriterInput: AVAssetWriterInput!
    var pixelBufferAdaptor: AVAssetWriterInputPixelBufferAdaptor!
    var audioWriterInput: AVAssetWriterInput!
    static var ci:Int = 0
    var initialTime:CMTime!

    var isReadyForVideoData: Bool {
        return (videoWriterInput == nil ? false : videoWriterInput!.isReadyForMoreMediaData )
    }

    var isReadyForAudioData: Bool {
        return (audioWriterInput == nil ? false : audioWriterInput!.isReadyForMoreMediaData)
    }

    class func pixelBufferFromImage(image: UIImage, pixelBufferPool: CVPixelBufferPool, size: CGSize, alpha:CGImageAlphaInfo) -> CVPixelBuffer? {

        var pixelBufferOut: CVPixelBuffer?

        let status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool, &pixelBufferOut)
        if status != kCVReturnSuccess {
            fatalError("CVPixelBufferPoolCreatePixelBuffer() failed")
        }

        let pixelBuffer = pixelBufferOut!

        CVPixelBufferLockBaseAddress(pixelBuffer, [])

        let data = CVPixelBufferGetBaseAddress(pixelBuffer)
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: data, width: Int(size.width), height: Int(size.height),
                                bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), space: rgbColorSpace, bitmapInfo: alpha.rawValue)

        context!.clear(CGRect(x: 0, y: 0, width: size.width, height: size.height))

        let horizontalRatio = size.width / image.size.width
        let verticalRatio = size.height / image.size.height
        //aspectRatio = max(horizontalRatio, verticalRatio) // ScaleAspectFill
        let aspectRatio = min(horizontalRatio, verticalRatio) // ScaleAspectFit

        let newSize = CGSize(width: image.size.width * aspectRatio, height: image.size.height * aspectRatio)

        let x = newSize.width < size.width ? (size.width - newSize.width) / 2 : 0
        let y = newSize.height < size.height ? (size.height - newSize.height) / 2 : 0

        let cgImage = image.cgImage != nil ? image.cgImage! : image.ciImage!.convertCIImageToCGImage()

        context!.draw(cgImage!, in: CGRect(x: x, y: y, width: newSize.width, height: newSize.height))

        CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
        return pixelBuffer
    }

    @available(iOS 11.0, *)
    init(renderSettings: RenderSettings, audioSettings:[String:Any]? = nil) {
        self.renderSettings = renderSettings
        self.audioSettings = audioSettings
    }

    func start(initialBuffer: CMSampleBuffer?) {

        let avOutputSettings: [String: AnyObject] = [
            AVVideoCodecKey: renderSettings.avCodecKey as AnyObject,
            AVVideoWidthKey: NSNumber(value: Float(renderSettings.width)),
            AVVideoHeightKey: NSNumber(value: Float(renderSettings.height))
        ]

        let avAudioSettings = audioSettings

        func createPixelBufferAdaptor() {
            let sourcePixelBufferAttributesDictionary = [
                kCVPixelBufferPixelFormatTypeKey as String: NSNumber(value: kCVPixelFormatType_32ARGB),
                kCVPixelBufferWidthKey as String: NSNumber(value: Float(renderSettings.width)),
                kCVPixelBufferHeightKey as String: NSNumber(value: Float(renderSettings.height))
            ]
            pixelBufferAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: videoWriterInput,
                                                                      sourcePixelBufferAttributes: sourcePixelBufferAttributesDictionary)
        }

        func createAssetWriter(outputURL: URL) -> AVAssetWriter {
            guard let assetWriter = try? AVAssetWriter(outputURL: outputURL, fileType: AVFileType.mov) else {
                fatalError("AVAssetWriter() failed")
            }

            guard assetWriter.canApply(outputSettings: avOutputSettings, forMediaType: AVMediaType.video) else {
                fatalError("canApplyOutputSettings() failed")
            }

            return assetWriter
        }

        videoWriter = createAssetWriter(outputURL: renderSettings.outputURL)
        videoWriterInput = AVAssetWriterInput(mediaType: AVMediaType.video, outputSettings: avOutputSettings)
//        if(audioSettings != nil) {
        audioWriterInput = AVAssetWriterInput(mediaType: .audio, outputSettings: nil)
        audioWriterInput.expectsMediaDataInRealTime = true
//        }

        if videoWriter.canAdd(videoWriterInput) {
            videoWriter.add(videoWriterInput)
        }
        else {
            fatalError("canAddInput() returned false")
        }

//        if(audioSettings != nil) {
            if videoWriter.canAdd(audioWriterInput) {
                videoWriter.add(audioWriterInput)
            }
            else {
                fatalError("canAddInput() returned false")
            }
//        }

        // The pixel buffer adaptor must be created before we start writing.
        createPixelBufferAdaptor()

        if videoWriter.startWriting() == false {
            fatalError("startWriting() failed")
        }


        self.initialTime = initialBuffer != nil ? CMSampleBufferGetPresentationTimeStamp(initialBuffer!) : CMTime.zero
        videoWriter.startSession(atSourceTime: self.initialTime)

        precondition(pixelBufferAdaptor.pixelBufferPool != nil, "nil pixelBufferPool")
    }

    func render(appendPixelBuffers: @escaping (VideoWriter)->Bool, completion: @escaping ()->Void) {

        precondition(videoWriter != nil, "Call start() to initialze the writer")

        let queue = DispatchQueue(__label: "mediaInputQueue", attr: nil)
        videoWriterInput.requestMediaDataWhenReady(on: queue) {
            let isFinished = appendPixelBuffers(self)
            if isFinished {
                self.videoWriterInput.markAsFinished()
                self.videoWriter.finishWriting() {
                    DispatchQueue.main.async {
                        print("Done Creating Video")
                        completion()
                    }
                }
            }
            else {
                // Fall through. The closure will be called again when the writer is ready.
            }
        }
    }

    func addAudio(buffer: CMSampleBuffer, time: CMTime) {
        if(isReadyForAudioData) {
            print("Writing audio \(VideoWriter.ci) of a time of \(CMSampleBufferGetPresentationTimeStamp(buffer))")
            let duration = CMSampleBufferGetDuration(buffer)
            let offsetBuffer = CMSampleBuffer.createSampleBuffer(fromSampleBuffer: buffer, withTimeOffset: time, duration: duration)
            if(offsetBuffer != nil) {
                print("Added audio")
                self.audioWriterInput.append(offsetBuffer!)
            }
            else {
                print("Not adding audio")
            }
        }

        VideoWriter.ci += 1
    }

    func addImage(image: UIImage, withPresentationTime presentationTime: CMTime) -> Bool {

        precondition(pixelBufferAdaptor != nil, "Call start() to initialze the writer")
        //1
        let pixelBuffer = VideoWriter.pixelBufferFromImage(image: image, pixelBufferPool: pixelBufferAdaptor.pixelBufferPool!, size: renderSettings.size, alpha: CGImageAlphaInfo.premultipliedFirst)!

        return pixelBufferAdaptor.append(pixelBuffer, withPresentationTime: presentationTime + self.initialTime)
    }
}
1
太疯狂了。我需要一些时间来仔细阅读这个全面的答案，但我相信我会学到很多东西。明晚或周六早上会深入研究它。非常感谢迄今为止所有的帮助和建议。 - PennyWise
0

我进一步研究了这个问题 - 虽然我可以更新我的答案，但我宁愿在新的领域开辟这个分支来分离这些想法。苹果公司表示，我们可以使用AVVideoComposition来“为了使用创建的视频组合进行播放，请从与组合源相同的资产创建一个AVPlayerItem对象，然后将组合分配给播放器项的videoComposition属性。要将组合导出到新的电影文件，请从相同的源资产创建一个AVAssetExportSession对象，然后将组合分配给导出会话的videoComposition属性。”。

https://developer.apple.com/documentation/avfoundation/avasynchronousciimagefilteringrequest

所以，你可以尝试使用AVPlayer播放原始URL，然后再尝试应用你的过滤器。

let filter = CIFilter(name: "CIGaussianBlur")!
let composition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in

    // Clamp to avoid blurring transparent pixels at the image edges
    let source = request.sourceImage.imageByClampingToExtent()
    filter.setValue(source, forKey: kCIInputImageKey)

    // Vary filter parameters based on video timing
    let seconds = CMTimeGetSeconds(request.compositionTime)
    filter.setValue(seconds * 10.0, forKey: kCIInputRadiusKey)

    // Crop the blurred output to the bounds of the original image
    let output = filter.outputImage!.imageByCroppingToRect(request.sourceImage.extent)

    // Provide the filter output to the composition
    request.finishWithImage(output, context: nil)
})

let asset = AVAsset(url: originalURL)
let item = AVPlayerItem(asset: asset)
item.videoComposition = composition
let player = AVPlayer(playerItem: item)

我相信你知道接下来该怎么做。这可能允许您对过滤进行“实时”处理。我能看到的潜在问题是，它会遇到与原始内容相同的问题，即每帧仍需要一定的时间来运行，从而导致音频和视频之间的延迟。但是，这种情况可能不会发生。如果您成功实现此功能，用户选择其过滤器后，您可以使用AVAssetExportSession导出特定的videoComposition。

如果需要帮助，请点击此处了解更多！

1
我还没有完全弄清楚，但是考虑到你迄今为止给了我很多启示，我接受了你的答案。 - PennyWise
如果需要进一步帮助，请告诉我。 - impression7vx
刚刚发现了另外一些有趣的值。这是我通过MTKView（使用videoSource）和AVPlayer播放的14秒视频在我的应用程序中录制并回放的打印输出：https://pastebin.com/4c1EwgyM - 注意音频的延迟开始。这个问题，我可能可以解决。还有其他我们可以从输出中学到的东西吗？ - PennyWise
这是有趣的事情。当我开始播放时，还没有应用任何过滤器。过滤器是由用户选择的，即使没有过滤器，原始视频通过MTKView渲染时也会有延迟。我甚至认为我上面的输出是没有应用过滤器的原始视频和音频。 - PennyWise
1
由于工作原因，我还没有能够实现fps的相关内容。但是，如果它不起作用，我可能会将其保留为现在的状态：音频和视频有一秒钟的延迟，音频比视频早结束一点，但在导出视频时一切正常。这是最基本的部分。已经花费了数小时来尝试修复它，但只是稍微改善了一点点。感觉这是一个活跃的问题，没有得到很好的解决。 - PennyWise
显示剩余8条评论
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接