如何在iOS 8中使用AVSampleBufferDisplayLayer和GStreamer来播放RTP H264流?

16

收到iOS 8中可供程序员使用HW-H264-Decoder的通知后,我现在想使用它。这里有一个关于“视频编码和解码的直接访问”的很好的介绍来自WWDC 2014。你可以看一下这里

基于那个案例1,我开始开发一个应用程序,它应该能够从GStreamer获取一个H264-RTP-UDP流,将其汇入“appsink”元素以直接访问NAL单元,并进行转换以创建CMSampleBuffers,然后我的AVSampleBufferDisplayLayer就可以显示它们了。

完成所有这些工作的有趣代码如下:

//
//  GStreamerBackend.m
// 

#import "GStreamerBackend.h"

NSString * const naluTypesStrings[] = {
    @"Unspecified (non-VCL)",
    @"Coded slice of a non-IDR picture (VCL)",
    @"Coded slice data partition A (VCL)",
    @"Coded slice data partition B (VCL)",
    @"Coded slice data partition C (VCL)",
    @"Coded slice of an IDR picture (VCL)",
    @"Supplemental enhancement information (SEI) (non-VCL)",
    @"Sequence parameter set (non-VCL)",
    @"Picture parameter set (non-VCL)",
    @"Access unit delimiter (non-VCL)",
    @"End of sequence (non-VCL)",
    @"End of stream (non-VCL)",
    @"Filler data (non-VCL)",
    @"Sequence parameter set extension (non-VCL)",
    @"Prefix NAL unit (non-VCL)",
    @"Subset sequence parameter set (non-VCL)",
    @"Reserved (non-VCL)",
    @"Reserved (non-VCL)",
    @"Reserved (non-VCL)",
    @"Coded slice of an auxiliary coded picture without partitioning (non-VCL)",
    @"Coded slice extension (non-VCL)",
    @"Coded slice extension for depth view components (non-VCL)",
    @"Reserved (non-VCL)",
    @"Reserved (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
    @"Unspecified (non-VCL)",
};


static GstFlowReturn new_sample(GstAppSink *sink, gpointer user_data)
{
    GStreamerBackend *backend = (__bridge GStreamerBackend *)(user_data);
    GstSample *sample = gst_app_sink_pull_sample(sink);
    GstBuffer *buffer = gst_sample_get_buffer(sample);
    GstMemory *memory = gst_buffer_get_all_memory(buffer);

    GstMapInfo info;
    gst_memory_map (memory, &info, GST_MAP_READ);

    int startCodeIndex = 0;
    for (int i = 0; i < 5; i++) {
        if (info.data[i] == 0x01) {
            startCodeIndex = i;
            break;
        }
    }
    int nalu_type = ((uint8_t)info.data[startCodeIndex + 1] & 0x1F);
    NSLog(@"NALU with Type \"%@\" received.", naluTypesStrings[nalu_type]);
    if(backend.searchForSPSAndPPS) {
        if (nalu_type == 7)
            backend.spsData = [NSData dataWithBytes:&(info.data[startCodeIndex + 1]) length: info.size - 4];

        if (nalu_type == 8)
            backend.ppsData = [NSData dataWithBytes:&(info.data[startCodeIndex + 1]) length: info.size - 4];

        if (backend.spsData != nil && backend.ppsData != nil) {
            const uint8_t* const parameterSetPointers[2] = { (const uint8_t*)[backend.spsData bytes], (const uint8_t*)[backend.ppsData bytes] };
            const size_t parameterSetSizes[2] = { [backend.spsData length], [backend.ppsData length] };

            CMVideoFormatDescriptionRef videoFormatDescr;
            OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, 4, &videoFormatDescr);
            [backend setVideoFormatDescr:videoFormatDescr];
            [backend setSearchForSPSAndPPS:false];
            NSLog(@"Found all data for CMVideoFormatDescription. Creation: %@.", (status == noErr) ? @"successfully." : @"failed.");
        }
    }
    if (nalu_type == 1 || nalu_type == 5) {
        CMBlockBufferRef videoBlock = NULL;
        OSStatus status = CMBlockBufferCreateWithMemoryBlock(NULL, info.data, info.size, kCFAllocatorNull, NULL, 0, info.size, 0, &videoBlock);
        NSLog(@"BlockBufferCreation: %@", (status == kCMBlockBufferNoErr) ? @"successfully." : @"failed.");
        const uint8_t sourceBytes[] = {(uint8_t)(info.size >> 24), (uint8_t)(info.size >> 16), (uint8_t)(info.size >> 8), (uint8_t)info.size};
        status = CMBlockBufferReplaceDataBytes(sourceBytes, videoBlock, 0, 4);
        NSLog(@"BlockBufferReplace: %@", (status == kCMBlockBufferNoErr) ? @"successfully." : @"failed.");

        CMSampleBufferRef sbRef = NULL;
        const size_t sampleSizeArray[] = {info.size};

        status = CMSampleBufferCreate(kCFAllocatorDefault, videoBlock, true, NULL, NULL, backend.videoFormatDescr, 1, 0, NULL, 1, sampleSizeArray, &sbRef);
        NSLog(@"SampleBufferCreate: %@", (status == noErr) ? @"successfully." : @"failed.");

        CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sbRef, YES);
        CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);
        CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);

        NSLog(@"Error: %@, Status:%@", backend.displayLayer.error, (backend.displayLayer.status == AVQueuedSampleBufferRenderingStatusUnknown)?@"unknown":((backend.displayLayer.status == AVQueuedSampleBufferRenderingStatusRendering)?@"rendering":@"failed"));
        dispatch_async(dispatch_get_main_queue(),^{
            [backend.displayLayer enqueueSampleBuffer:sbRef];
            [backend.displayLayer setNeedsDisplay];
        });

    }

    gst_memory_unmap(memory, &info);
    gst_memory_unref(memory);
    gst_buffer_unref(buffer);

    return GST_FLOW_OK;
}

@implementation GStreamerBackend

- (instancetype)init
{
    if (self = [super init]) {
        self.searchForSPSAndPPS = true;
        self.ppsData = nil;
        self.spsData = nil;
        self.displayLayer = [[AVSampleBufferDisplayLayer alloc] init];
        self.displayLayer.bounds = CGRectMake(0, 0, 300, 300);
        self.displayLayer.backgroundColor = [UIColor blackColor].CGColor;
        self.displayLayer.position = CGPointMake(500, 500);
        self.queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
        dispatch_async(self.queue, ^{
            [self app_function];
        });
    }
    return self;
}

- (void)start
{
    if(gst_element_set_state(self.pipeline, GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
        NSLog(@"Failed to set pipeline to playing");
    }
}

- (void)app_function
{
    GstElement *udpsrc, *rtphdepay, *capsfilter;
    GMainContext *context; /* GLib context used to run the main loop */
    GMainLoop *main_loop;  /* GLib main loop */


    context = g_main_context_new ();
    g_main_context_push_thread_default(context);

    g_set_application_name ("appsink");

    self.pipeline = gst_pipeline_new ("testpipe");

    udpsrc = gst_element_factory_make ("udpsrc", "udpsrc");
    GstCaps *caps = gst_caps_new_simple("application/x-rtp", "media", G_TYPE_STRING, "video", "clock-rate", G_TYPE_INT, 90000, "encoding-name", G_TYPE_STRING, "H264", NULL);
    g_object_set(udpsrc, "caps", caps, "port", 5000, NULL);
    gst_caps_unref(caps);
    rtphdepay = gst_element_factory_make("rtph264depay", "rtph264depay");
    capsfilter = gst_element_factory_make("capsfilter", "capsfilter");
    caps = gst_caps_new_simple("video/x-h264", "streamformat", G_TYPE_STRING, "byte-stream", "alignment", G_TYPE_STRING, "nal", NULL);
    g_object_set(capsfilter, "caps", caps, NULL);
    self.appsink = gst_element_factory_make ("appsink", "appsink");

    gst_bin_add_many (GST_BIN (self.pipeline), udpsrc, rtphdepay, capsfilter, self.appsink, NULL);

    if(!gst_element_link_many (udpsrc, rtphdepay, capsfilter, self.appsink, NULL)) {
        NSLog(@"Cannot link gstreamer elements");
        exit (1);
    }

    if(gst_element_set_state(self.pipeline, GST_STATE_READY) != GST_STATE_CHANGE_SUCCESS)
        NSLog(@"could not change to ready");

    GstAppSinkCallbacks callbacks = { NULL, NULL, new_sample,
        NULL, NULL};
    gst_app_sink_set_callbacks (GST_APP_SINK(self.appsink), &callbacks, (__bridge gpointer)(self), NULL);

    main_loop = g_main_loop_new (context, FALSE);
    g_main_loop_run (main_loop);


    /* Free resources */
    g_main_loop_unref (main_loop);
    main_loop = NULL;
    g_main_context_pop_thread_default(context);
    g_main_context_unref (context);
    gst_element_set_state (GST_ELEMENT (self.pipeline), GST_STATE_NULL);
    gst_object_unref (GST_OBJECT (self.pipeline));
}

@end

当我运行应用并开始流式传输到iOS设备时,我得到了什么:
NALU with Type "Sequence parameter set (non-VCL)" received.
NALU with Type "Picture parameter set   (non-VCL)" received.

Found all data for CMVideoFormatDescription. Creation: successfully..

NALU with Type "Coded slice of an IDR picture (VCL)" received.
BlockBufferCreation: successfully.
BlockBufferReplace: successfully.
SampleBufferCreate: successfully.
Error: (null), Status:unknown

NALU with Type "Coded slice of a non-IDR picture (VCL)" received.
BlockBufferCreation: successfully.
BlockBufferReplace: successfully.
SampleBufferCreate: successfully.
Error: (null), Status:rendering
[...] (repetition of the last 5 lines)

所以看起来解码工作正常,但我的问题是,我在AVSampleBufferDisplayLayer中看不到任何内容。可能与kCMSampleAttachmentKey_DisplayImmediately有关,但我已按照此处(请参阅“重要”注释)所说的设置了它。
欢迎提出任何想法;)

我已经快完成实现这个东西了,但是你为什么要检查起始码呢?难道这只在字节流中才有(如果它是通过TCP传输的话)?我认为如果是通过RTP(UDP)传输的话,由于已经分成了数据包,所以不再需要起始码。我学习这个过程中的所有知识都来自于这个RFC,它并没有提到需要查找起始码,因为它已经被分成了数据包。我知道你发布链接的视频中提到了起始码,但我一直很困惑它们之间的冲突。 - ddelnano
我不确定规格书上怎么说。但是因为我之前使用过GStreamer来获取流,并且尤其是指定NALU作为输出,GStreamer可以将它转换成任何本来不在UDP数据包中的东西。因此,即使UDP数据包中没有起始码,在GStreamer中添加起始码也是可以完成的。 - Zappel
我说的对吗,你的代码是在寻找起始码0x0001或0x000001?在你的服务器端流媒体时,你使用gstreamer作为命令行工具吗?如果是这样,你能给我展示一下你使用的命令吗? - ddelnano
既不是这个也不是那个。我的代码寻找_两者_。因此它可以检测到3字节和4字节的起始码。在服务器端,我的情况下是树莓派,我运行以下命令:raspivid -t 0 -h 720 -w 1280 -fps 45 -vf -hf -b 6500000 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay ! udpsink host=192.168.0.12 port=5000 - Zappel
2个回答

2

现在已经解决了。每个NALU的长度不包含长度头本身。所以,在使用我的sourceBytes之前,我必须从我的info.size中减去4。


嗨Zappel!我和我的团队过去几天一直在开发一个相当基本的AVSampleBufferDisplayLayer + AVAssetReader。我们取得了很大的进展,但卡在了一些问题上。如果可能的话,我们会很乐意得到帮助(即使需要支付)。有没有什么方式可以联系上你呢? - Roi Mulia

1
根据您的代码指示,我编写了一个程序,使用AVSampleBufferDisplayLayer对H.264流进行解码和显示。我使用了live555而不是GSStream来接收H.264 NAL单元。
不幸的是,我的应用程序只显示了几帧图像,之后就无法再显示任何图像了。您的应用程序是否遇到过同样的问题?

不,我的应用程序没有这些问题。您是否评估了CM函数的OSStatus返回值?如果是,它们的值是什么?给我们更多的输入来帮助您。也许您应该查看应用程序的线程安全性。首先尝试在主线程中实现它。 - Zappel

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接