如何在iOS中将kCVPixelFormatType_420YpCbCr8BiPlanarFullRange缓冲区转换为UIImage

Question

我试着在原帖中回答，但是SO不允许我这样做。希望有更高权限的人能把这个合并到原问题中。

好的，下面是更完整的答案。首先，设置捕获：

// Create capture session
self.captureSession = [[AVCaptureSession alloc] init];

[self.captureSession setSessionPreset:AVCaptureSessionPresetPhoto];

// Setup capture input
self.inputDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
AVCaptureDeviceInput *captureInput = [AVCaptureDeviceInput deviceInputWithDevice:self.inputDevice
                                                                           error:nil];
[self.captureSession addInput:captureInput];

// Setup video processing (capture output)
AVCaptureVideoDataOutput *captureOutput = [[AVCaptureVideoDataOutput alloc] init];
// Don't add frames to the queue if frames are already processing
captureOutput.alwaysDiscardsLateVideoFrames = YES;

// Create a serial queue to handle processing of frames
_videoQueue = dispatch_queue_create("cameraQueue", NULL);
[captureOutput setSampleBufferDelegate:self queue:_videoQueue];

// Set the video output to store frame in YUV
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;

NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange];
NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[captureOutput setVideoSettings:videoSettings];
[self.captureSession addOutput:captureOutput];

现在，关于代理/回调的实现：

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
   fromConnection:(AVCaptureConnection *)connection
{

// Create autorelease pool because we are not in the main_queue
@autoreleasepool {

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    //Lock the imagebuffer
    CVPixelBufferLockBaseAddress(imageBuffer,0);

    // Get information about the image
    uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);

    //    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);

    CVPlanarPixelBufferInfo_YCbCrBiPlanar *bufferInfo = (CVPlanarPixelBufferInfo_YCbCrBiPlanar *)baseAddress;

    // This just moved the pointer past the offset
    baseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);


    // convert the image
    _prefImageView.image = [self makeUIImage:baseAddress bufferInfo:bufferInfo width:width height:height bytesPerRow:bytesPerRow];

    // Update the display with the captured image for DEBUG purposes
    dispatch_async(dispatch_get_main_queue(), ^{
        [_myMainView.yUVImage setImage:_prefImageView.image];
    });        
}

最后，这是将YUV转换为UIImage的方法：

- (UIImage *)makeUIImage:(uint8_t *)inBaseAddress bufferInfo:(CVPlanarPixelBufferInfo_YCbCrBiPlanar *)inBufferInfo width:(size_t)inWidth height:(size_t)inHeight bytesPerRow:(size_t)inBytesPerRow {

NSUInteger yPitch = EndianU32_BtoN(inBufferInfo->componentInfoY.rowBytes);

uint8_t *rgbBuffer = (uint8_t *)malloc(inWidth * inHeight * 4);
uint8_t *yBuffer = (uint8_t *)inBaseAddress;
uint8_t val;
int bytesPerPixel = 4;

// for each byte in the input buffer, fill in the output buffer with four bytes
// the first byte is the Alpha channel, then the next three contain the same
// value of the input buffer
for(int y = 0; y < inHeight*inWidth; y++)
{
    val = yBuffer[y];
    // Alpha channel
    rgbBuffer[(y*bytesPerPixel)] = 0xff;

    // next three bytes same as input
    rgbBuffer[(y*bytesPerPixel)+1] = rgbBuffer[(y*bytesPerPixel)+2] =  rgbBuffer[y*bytesPerPixel+3] = val;
}

// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

CGContextRef context = CGBitmapContextCreate(rgbBuffer, yPitch, inHeight, 8,
                                             yPitch*bytesPerPixel, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedLast);

CGImageRef quartzImage = CGBitmapContextCreateImage(context);

CGContextRelease(context);
CGColorSpaceRelease(colorSpace);

UIImage *image = [UIImage imageWithCGImage:quartzImage];

CGImageRelease(quartzImage);
free(rgbBuffer);
return  image;
}

您还需要 #import "Endian.h"

请注意，对CGBitmapContextCreate的调用比我预期的要复杂得多。我对视频处理并不很精通，但这个调用让我困惑了一段时间。然后当它最终工作时，就像魔术一样。

- MichaelG

---

我刚刚花了最后两天的时间尝试将UIImage写入缓冲区以添加到视频中。我理解你的兴奋！ - Nicolas Manzini

@NicolasManzini 这个解决方案对你有用吗？我得到了 <Error>: copy_read_only: vm_copy failed: status 1. 看起来与 https://dev59.com/C3A75IYBdhLWcg3wT3L8 相关。 - Ryan Romanchuk

检查一下您的位图上下文大小，也许是这个问题。但我是用另一种方式 CGContextDrawImage(...)。 - Nicolas Manzini

你的YUVSP转ARGB的代码中，色度部分在哪里？我猜你只转换了亮度（Y）部分。 - thar_bun

Answer 1

背景信息：@Michaelg的版本只访问y缓冲区，因此您只能获得亮度而不是颜色。如果缓冲区中的间距和像素数量不匹配（由于某种原因在行末填充字节），则它还存在缓冲区溢出错误。这里发生的背景是一种平面图像格式，为亮度每像素分配一个字节，为颜色信息的4个像素分配2个字节。与连续存储在内存中不同，它们存储为“平面”，其中Y或亮度平面具有自己的内存块，CbCr或颜色平面也具有自己的内存块。 CbCr平面包括Y平面的1/4样本数（半高度和宽度），CbCr平面中的每个像素对应于Y平面中的2x2块。希望这些背景信息能够帮助您。

编辑：他的版本和我的旧版本都有可能超出缓冲区并且如果图像缓冲区中的行在每行末尾具有填充字节，则无法工作。此外，我的cbcr平面缓冲区未创建正确的偏移量。要正确执行此操作，您应始终使用核心视频功能，例如CVPixelBufferGetWidthOfPlane和CVPixelBufferGetBaseAddressOfPlane。这将确保您正确解释缓冲区，并且无论缓冲区是否具有标头以及指针数学是否出错，它都将起作用。您还应该使用Apple的函数中的行大小和缓冲区基地址。这些在以下文档中记录：https://developer.apple.com/library/prerelease/ios/documentation/QuartzCore/Reference/CVPixelBufferRef/index.html请注意，虽然此版本在某些情况下使用了Apple的函数和标头，但最好只使用Apple的函数。我将来可能会更新此内容，以完全不使用标头。

这将把kcvpixelformattype_420ypcbcr8biplanarfullrange缓冲区转换为UIImage，然后您可以使用它。

首先，设置捕获：

// Create capture session
self.captureSession = [[AVCaptureSession alloc] init];

[self.captureSession setSessionPreset:AVCaptureSessionPresetPhoto];

// Setup capture input
self.inputDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
AVCaptureDeviceInput *captureInput = [AVCaptureDeviceInput deviceInputWithDevice:self.inputDevice
                                                                           error:nil];
[self.captureSession addInput:captureInput];

// Setup video processing (capture output)
AVCaptureVideoDataOutput *captureOutput = [[AVCaptureVideoDataOutput alloc] init];
// Don't add frames to the queue if frames are already processing
captureOutput.alwaysDiscardsLateVideoFrames = YES;

// Create a serial queue to handle processing of frames
_videoQueue = dispatch_queue_create("cameraQueue", NULL);
[captureOutput setSampleBufferDelegate:self queue:_videoQueue];

// Set the video output to store frame in YUV
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;

NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange];
NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[captureOutput setVideoSettings:videoSettings];
[self.captureSession addOutput:captureOutput];

现在是委托/回调的实现：

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
   fromConnection:(AVCaptureConnection *)connection
{

// Create autorelease pool because we are not in the main_queue
@autoreleasepool {

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    //Lock the imagebuffer
    CVPixelBufferLockBaseAddress(imageBuffer,0);

    // Get information about the image
    uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);

    //    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);

    CVPlanarPixelBufferInfo_YCbCrBiPlanar *bufferInfo = (CVPlanarPixelBufferInfo_YCbCrBiPlanar *)baseAddress;
    //get the cbrbuffer base address
    uint8_t* cbrBuff = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1);
    // This just moved the pointer past the offset
    baseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);


    // convert the image
    _prefImageView.image = [self makeUIImage:baseAddress cBCrBuffer:cbrBuff bufferInfo:bufferInfo width:width height:height bytesPerRow:bytesPerRow];

    // Update the display with the captured image for DEBUG purposes
    dispatch_async(dispatch_get_main_queue(), ^{
        [_myMainView.yUVImage setImage:_prefImageView.image];
    });        
}

最后，这是从YUV转换为UIImage的方法。

- (UIImage *)makeUIImage:(uint8_t *)inBaseAddress cBCrBuffer:(uint8_t*)cbCrBuffer bufferInfo:(CVPlanarPixelBufferInfo_YCbCrBiPlanar *)inBufferInfo width:(size_t)inWidth height:(size_t)inHeight bytesPerRow:(size_t)inBytesPerRow {

     NSUInteger yPitch = EndianU32_BtoN(inBufferInfo->componentInfoY.rowBytes);
 NSUInteger cbCrOffset = EndianU32_BtoN(inBufferInfo->componentInfoCbCr.offset);
 uint8_t *rgbBuffer = (uint8_t *)malloc(inWidth * inHeight * 4);
 NSUInteger cbCrPitch = EndianU32_BtoN(inBufferInfo->componentInfoCbCr.rowBytes);
 uint8_t *yBuffer = (uint8_t *)inBaseAddress;
 //uint8_t *cbCrBuffer = inBaseAddress + cbCrOffset;
 uint8_t val;
 int bytesPerPixel = 4;

 for(int y = 0; y < inHeight; y++)
 {
 uint8_t *rgbBufferLine = &rgbBuffer[y * inWidth * bytesPerPixel];
 uint8_t *yBufferLine = &yBuffer[y * yPitch];
 uint8_t *cbCrBufferLine = &cbCrBuffer[(y >> 1) * cbCrPitch];

 for(int x = 0; x < inWidth; x++)
 {
 int16_t y = yBufferLine[x];
 int16_t cb = cbCrBufferLine[x & ~1] - 128; 
 int16_t cr = cbCrBufferLine[x | 1] - 128;

 uint8_t *rgbOutput = &rgbBufferLine[x*bytesPerPixel];

     int16_t r = (int16_t)roundf( y + cr *  1.4 );
     int16_t g = (int16_t)roundf( y + cb * -0.343 + cr * -0.711 );
     int16_t b = (int16_t)roundf( y + cb *  1.765);

 //ABGR
 rgbOutput[0] = 0xff;
     rgbOutput[1] = clamp(b);
     rgbOutput[2] = clamp(g);
     rgbOutput[3] = clamp(r);
 }
 }

 // Create a device-dependent RGB color space
 CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
 NSLog(@"ypitch:%lu inHeight:%zu bytesPerPixel:%d",(unsigned long)yPitch,inHeight,bytesPerPixel);
 NSLog(@"cbcrPitch:%lu",cbCrPitch);
 CGContextRef context = CGBitmapContextCreate(rgbBuffer, inWidth, inHeight, 8,
 inWidth*bytesPerPixel, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedLast);

 CGImageRef quartzImage = CGBitmapContextCreateImage(context);

 CGContextRelease(context);
 CGColorSpaceRelease(colorSpace);

 UIImage *image = [UIImage imageWithCGImage:quartzImage];

 CGImageRelease(quartzImage);
 free(rgbBuffer);
 return  image;
 }

您还需要导入#import "Endian.h"并定义#define clamp(a) (a>255?255:(a<0?0:a)); 请注意，调用CGBitmapContextCreate比我预期的要复杂得多。我对视频处理不是很熟悉，但这个调用让我困惑了一段时间。然后当它最终工作时，就像魔术一样。

- meursault334

---

如果您在AVCaptureConnection中更改了videoOrientation，则此代码将无法正常工作。有关更多信息，请参见此答案。 - Rodrigo Sieiro