音频CMSampleBuffer的深度复制

Question

我正在尝试创建由AVCaptureAudioDataOutputSampleBufferDelegate中的captureOutput返回的CMSampleBuffer的副本。

我的问题是，当我将从委托方法captureOutput:didOutputSampleBuffer:fromConnection:返回的帧保留在CFArray中一段时间后，它们就会被丢弃。

显然，我需要创建传入缓冲区的深层副本以进行进一步处理。 我也知道CMSampleBufferCreateCopy只会创建浅层副本。

有几个相关问题在SO上提出：

• 从CMSampleBuffer中提取数据以创建深度副本
• 创建CMSampleBuffer的副本在Swift中返回OSStatus -12743（无效媒体格式）
• CMImageBuffer或CVImageBuffer的深层副本

但是，它们都不能帮助我正确使用具有12个参数的CMSampleBufferCreate函数。

  CMSampleBufferRef copyBuffer;

  CMBlockBufferRef data = CMSampleBufferGetDataBuffer(sampleBuffer);
  CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer);
  CMItemCount itemCount = CMSampleBufferGetNumSamples(sampleBuffer);

  CMTime duration = CMSampleBufferGetDuration(sampleBuffer);
  CMTime presentationStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
  CMSampleTimingInfo timingInfo;
  timingInfo.duration = duration;
  timingInfo.presentationTimeStamp = presentationStamp;
  timingInfo.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer);


  size_t sampleSize = CMBlockBufferGetDataLength(data);
  CMBlockBufferRef sampleData;

  if (CMBlockBufferCopyDataBytes(data, 0, sampleSize, &sampleData) != kCMBlockBufferNoErr) {
    VLog(@"error during copying sample buffer");
  }

  // Here I tried data and sampleData CMBlockBuffer instance, but no success
  OSStatus status = CMSampleBufferCreate(kCFAllocatorDefault, data, isDataReady, nil, nil, formatDescription, itemCount, 1, &timingInfo, 1, &sampleSize, &copyBuffer);

  if (!self.sampleBufferArray)  {
    self.sampleBufferArray = CFArrayCreateMutable(NULL, 0, &kCFTypeArrayCallBacks);
    //EXC_BAD_ACCESS crash when trying to add sampleBuffer to the array
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  } else  {
    CFArrayAppendValue(self.sampleBufferArray, copyBuffer);
  }

如何深拷贝音频CMSampleBuffer？在您的答案中，可以使用任何语言（Swift/Objective-C）。

- Neil Galiaskarov

---

{btsdaf} - user8821870

你是否需要进行深拷贝？使用 CMSampleBufferCreateCopy 会发生什么？CMSampleBufferCopySampleBufferForRange 是否会给你一个深拷贝？你真的需要 CMSampleBuffer 进行进一步处理吗？如果你正在进行自己的处理，长度+指针可能更方便。 - Rhythmic Fistman

@RhythmicFistman 是的，很明显我需要进行深拷贝。如果我使用CMSampleBufferCreateCopy来复制样本，并在CFArray中保留复制的样本超过1秒钟，那么didOutputSampleBuffer将停止被调用。你可以通过这个问题轻松地重现它。我将使用CMSampleBufferCopySampleBufferForRange检查其行为，并向你更新。 - Neil Galiaskarov

啊，好的，保留缓冲区会阻止委托回调是非常重要的信息。你有以上代码的可运行版本链接吗？ - Rhythmic Fistman

Answer 1

这是我最终实现的可行解决方案。我将这个代码片段发送给了苹果开发技术支持，并询问他们是否正确地复制了输入样本缓冲区的方式。基本思路是复制 AudioBufferList ，然后创建一个 CMSampleBuffer 并将 AudioBufferList 设置为该样本。

AudioBufferList audioBufferList;
CMBlockBufferRef blockBuffer;
//Create an AudioBufferList containing the data from the CMSampleBuffer,
//and a CMBlockBuffer which references the data in that AudioBufferList.
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, NULL, &audioBufferList, sizeof(audioBufferList), NULL, NULL, 0, &blockBuffer);
NSUInteger size = sizeof(audioBufferList);
char buffer[size];

memcpy(buffer, &audioBufferList, size);
//This is the Audio data.
NSData *bufferData = [NSData dataWithBytes:buffer length:size];

const void *copyBufferData = [bufferData bytes];
copyBufferData = (char *)copyBufferData;

CMSampleBufferRef copyBuffer = NULL;
OSStatus status = -1;

/* Format Description */

AudioStreamBasicDescription audioFormat = *CMAudioFormatDescriptionGetStreamBasicDescription((CMAudioFormatDescriptionRef) CMSampleBufferGetFormatDescription(sampleBuffer));

CMFormatDescriptionRef format = NULL;
status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &audioFormat, 0, nil, 0, nil, nil, &format);

CMFormatDescriptionRef formatdes = NULL;
status = CMFormatDescriptionCreate(NULL, kCMMediaType_Audio, 'lpcm', NULL, &formatdes);
if (status != noErr)
{
  NSLog(@"Error in CMAudioFormatDescriptionCreator");
  CFRelease(blockBuffer);
  return;
}

/* Create sample Buffer */
CMItemCount framesCount = CMSampleBufferGetNumSamples(sampleBuffer);
CMSampleTimingInfo timing   = {.duration= CMTimeMake(1, 44100), .presentationTimeStamp= CMSampleBufferGetPresentationTimeStamp(sampleBuffer), .decodeTimeStamp= CMSampleBufferGetDecodeTimeStamp(sampleBuffer)};

status = CMSampleBufferCreate(kCFAllocatorDefault, nil , NO,nil,nil,format, framesCount, 1, &timing, 0, nil, &copyBuffer);

if( status != noErr) {
  NSLog(@"Error in CMSampleBufferCreate");
  CFRelease(blockBuffer);
  return;
}

/* Copy BufferList to Sample Buffer */
AudioBufferList receivedAudioBufferList;
memcpy(&receivedAudioBufferList, copyBufferData, sizeof(receivedAudioBufferList));

//Creates a CMBlockBuffer containing a copy of the data from the
//AudioBufferList.
status = CMSampleBufferSetDataBufferFromAudioBufferList(copyBuffer, kCFAllocatorDefault , kCFAllocatorDefault, 0, &receivedAudioBufferList);
if (status != noErr) {
  NSLog(@"Error in CMSampleBufferSetDataBufferFromAudioBufferList");
  CFRelease(blockBuffer);
  return;
}

代码级别支持答复：

这段代码看起来还不错（尽管您可能需要添加一些额外的错误检查）。我已经在一个实现了 AVCaptureAudioDataOutput 委托方法 captureOutput:didOutputSampleBuffer:fromConnection: 来捕获和记录音频的应用中成功测试过了。 当使用此深度复制代码时，我得到的捕获音频似乎与直接使用提供的示例缓冲区（不使用深度复制）时所得到的相同。

Apple 开发人员技术支持

- Neil Galiaskarov

---

这很棒，但我在处理视频样本时遇到了类似的问题（我需要保留样本缓冲区）。您有关于如何实现它的任何提示吗？ - Andy Hin

你好， 我正在开发一个项目，需要进行视频和音频的深度复制。在Swift中，对于视频samplebuffer的深度复制，我已经能够找到一个解决方案，但是对于音频却无法找到好的方法。 你是否有机会将你的代码转换成Swift？ 有了你的代码，一旦获得了samplebuffer的副本，你如何将其写入文件呢？ 谢谢 - user1523505

Answer 2

在 Swift 中找不到一个合适的答案。这里是一个扩展：

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        guard let formatDesc = CMSampleBufferGetFormatDescription(self),
              let data = try? self.dataBuffer?.dataBytes() else {
                  return nil
              }
        let nFrames = CMSampleBufferGetNumSamples(self)
        let pts = CMSampleBufferGetPresentationTimeStamp(self)
        let dataBuffer = data.withUnsafeBytes { (buffer) -> CMBlockBuffer? in
            var blockBuffer: CMBlockBuffer?
            let length: Int = data.count
            guard CMBlockBufferCreateWithMemoryBlock(
                allocator: kCFAllocatorDefault,
                memoryBlock: nil,
                blockLength: length,
                blockAllocator: nil,
                customBlockSource: nil,
                offsetToData: 0,
                dataLength: length,
                flags: 0,
                blockBufferOut: &blockBuffer) == noErr else {
                    print("Failed to create block")
                    return nil
                }
            guard CMBlockBufferReplaceDataBytes(
                with: buffer.baseAddress!,
                blockBuffer: blockBuffer!,
                offsetIntoDestination: 0,
                dataLength: length) == noErr else {
                    print("Failed to move bytes for block")
                    return nil
                }
            return blockBuffer
        }
        guard let dataBuffer = dataBuffer else {
            return nil
        }
        var newSampleBuffer: CMSampleBuffer?
        CMAudioSampleBufferCreateReadyWithPacketDescriptions(
            allocator: kCFAllocatorDefault,
            dataBuffer: dataBuffer,
            formatDescription: formatDesc,
            sampleCount: nFrames,
            presentationTimeStamp: pts,
            packetDescriptions: nil,
            sampleBufferOut: &newSampleBuffer
        )
        return newSampleBuffer
    }
}

- LLooggaann

---

我正在尝试使这个工作起来。我用let data = try? self.dataBuffer?.dataBytes()代替了let data = self.data。然后我检查了原始对象和深拷贝对象的描述输出，除了包含指向另一个4356字节块缓冲区的4096数据字节块缓冲区之外，一切都是相同的。在深拷贝的情况下，对第二个块缓冲区的引用不在那里。有什么提示吗？ - Jorge

Answer 3

之前发布的答案很棒！我已经在我的代码库中成功使用了LLooggaann的答案，并使用现代Swift API进行了重构。这可以使代码更短，如果出现问题还会抛出错误，并且减少了指针和内存管理的工作量。

对于所有需要此功能并希望使用更现代紧凑实现的未来开发人员：

extension CMSampleBuffer {
    struct InvalidAudioSampleBuffer: Swift.Error {}
    
    func deepCopyAudioSampleBuffer() throws -> CMSampleBuffer {
        guard let formatDescription, let dataBuffer else { throw InvalidAudioSampleBuffer() }

        let data = try dataBuffer.dataBytes()
        let dataBufferCopy = try data.withUnsafeBytes { buffer -> CMBlockBuffer in
            let blockBuffer = try CMBlockBuffer(length: data.count)
            try blockBuffer.replaceDataBytes(with: buffer)
            return blockBuffer
        }

        return try CMSampleBuffer(dataBuffer: dataBufferCopy,
                                  formatDescription: formatDescription,
                                  numSamples: numSamples,
                                  presentationTimeStamp: presentationTimeStamp,
                                  packetDescriptions: [])
    }
}

- Mac_Cain13

Answer 4

LLooggaann的解决方案更简单且效果很好，不过，如果有人感兴趣，我将原始解决方案迁移到了Swift 5.6：

extension CMSampleBuffer {
    func deepCopy() -> CMSampleBuffer? {
        var audioBufferList : AudioBufferList = AudioBufferList()
        var blockBuffer : CMBlockBuffer?

        let sizeOfAudioBufferList = MemoryLayout<AudioBufferList>.size
        
        //Create an AudioBufferList containing the data from the CMSampleBuffer.
        CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(self,
                                                                bufferListSizeNeededOut: nil,
                                                                bufferListOut: &audioBufferList,
                                                                bufferListSize: sizeOfAudioBufferList,
                                                                blockBufferAllocator: nil,
                                                                blockBufferMemoryAllocator: nil,
                                                                flags: 0,
                                                                blockBufferOut: &blockBuffer)

        guard audioBufferList.mNumberBuffers == 1 else { return nil }  //TODO: Make this generic for any number of buffers
        
        /* Deep copy the audio buffer */
        let audioBufferDataSize = Int(audioBufferList.mBuffers.mDataByteSize)
        let audioBuffer = audioBufferList.mBuffers
        let audioBufferDataCopyPointer = UnsafeMutableRawPointer.allocate(byteCount: audioBufferDataSize, alignment: 1)
                
        defer {
            audioBufferDataCopyPointer.deallocate()
        }
        
        memcpy(audioBufferDataCopyPointer, audioBufferList.mBuffers.mData, audioBufferDataSize)
        
        let copiedAudioBuffer = AudioBuffer(mNumberChannels: audioBuffer.mNumberChannels,
                                            mDataByteSize: audioBufferList.mBuffers.mDataByteSize,
                                            mData: audioBufferDataCopyPointer)
        
        /* Create a new audio buffer list with the deep copied audio buffer */
        var copiedAudioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: copiedAudioBuffer)

        /* Copy audio format description, to be used in the new sample buffer */
        guard let sampleBufferFormatDescription = CMSampleBufferGetFormatDescription(self) else { return nil }

        /* Create copy of timing for new sample buffer */
        var duration = CMSampleBufferGetDuration(self)
        duration.value /= Int64(numSamples)
        var timing = CMSampleTimingInfo(duration: duration,
                                        presentationTimeStamp: CMSampleBufferGetPresentationTimeStamp(self),
                                        decodeTimeStamp: CMSampleBufferGetDecodeTimeStamp(self))

        /* New sample buffer preparation, using the audio format description, and the timing information. */
        let sampleCount = CMSampleBufferGetNumSamples(self)
        var newSampleBuffer : CMSampleBuffer?

        guard CMSampleBufferCreate(allocator: kCFAllocatorDefault,
                                   dataBuffer: nil,
                                   dataReady: false,
                                   makeDataReadyCallback: nil,
                                   refcon: nil,
                                   formatDescription: sampleBufferFormatDescription,
                                   sampleCount: sampleCount,
                                   sampleTimingEntryCount: 1,
                                   sampleTimingArray: &timing,
                                   sampleSizeEntryCount: 0,
                                   sampleSizeArray: nil,
                                   sampleBufferOut: &newSampleBuffer) == noErr else { return nil }

        //Create a CMBlockBuffer containing a copy of the data from the AudioBufferList, add to new sample buffer.
        let status = CMSampleBufferSetDataBufferFromAudioBufferList(newSampleBuffer!,
                                                                    blockBufferAllocator: kCFAllocatorDefault,
                                                                    blockBufferMemoryAllocator: kCFAllocatorDefault,
                                                                    flags: 0,
                                                                    bufferList: &copiedAudioBufferList)
        
        guard status == noErr else { return nil }

        return newSampleBuffer
    }
}

- Jorge