logo标签列表

将 SceneKit 场景渲染到视频输出

iosmacosvideorenderingscenekit
13

作为一名主要从事高级/iOS开发的人员，我对使用SceneKit进行动画项目很感兴趣。

虽然显然SceneKit是为“实时”交互而设计的，但我已经玩了几个月了，我会发现如果能够将SKScene“渲染”成视频将非常有用。 目前，我一直在使用Quicktime的屏幕录制器来捕获视频输出，但（当然）这样做会导致帧速率下降。 是否有另一种选择可以使场景以其自己的速度呈现并输出为平滑的视频文件？

我知道这不太可能...只是想问问，以防我漏掉了一些更低级的东西！

你用了哪个选项？你能找到一些东西，也能像从移动相机节点拍摄的视频一样呈现吗？ - Crashalot
4个回答
8
你可以使用SCNRenderer将内容渲染到CGImage上，然后使用AVFoundation将CGImage添加到视频流中。
我编写了这个Swift扩展程序，可以用于渲染成CGImage。
public extension SCNRenderer {

    public func renderToImageSize(size: CGSize, floatComponents: Bool, atTime time: NSTimeInterval) -> CGImage? {

        var thumbnailCGImage: CGImage?

        let width = GLsizei(size.width), height = GLsizei(size.height)
        let samplesPerPixel = 4

        #if os(iOS)
            let oldGLContext = EAGLContext.currentContext()
            let glContext = unsafeBitCast(context, EAGLContext.self)

            EAGLContext.setCurrentContext(glContext)
            objc_sync_enter(glContext)
        #elseif os(OSX)
            let oldGLContext = CGLGetCurrentContext()
            let glContext = unsafeBitCast(context, CGLContextObj.self)

            CGLSetCurrentContext(glContext)
            CGLLockContext(glContext)
        #endif

        // set up the OpenGL buffers
        var thumbnailFramebuffer: GLuint = 0
        glGenFramebuffers(1, &thumbnailFramebuffer)
        glBindFramebuffer(GLenum(GL_FRAMEBUFFER), thumbnailFramebuffer); checkGLErrors()

        var colorRenderbuffer: GLuint = 0
        glGenRenderbuffers(1, &colorRenderbuffer)
        glBindRenderbuffer(GLenum(GL_RENDERBUFFER), colorRenderbuffer)
        if floatComponents {
            glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_RGBA16F), width, height)
        } else {
            glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_RGBA8), width, height)
        }
        glFramebufferRenderbuffer(GLenum(GL_FRAMEBUFFER), GLenum(GL_COLOR_ATTACHMENT0), GLenum(GL_RENDERBUFFER), colorRenderbuffer); checkGLErrors()

        var depthRenderbuffer: GLuint = 0
        glGenRenderbuffers(1, &depthRenderbuffer)
        glBindRenderbuffer(GLenum(GL_RENDERBUFFER), depthRenderbuffer)
        glRenderbufferStorage(GLenum(GL_RENDERBUFFER), GLenum(GL_DEPTH_COMPONENT24), width, height)
        glFramebufferRenderbuffer(GLenum(GL_FRAMEBUFFER), GLenum(GL_DEPTH_ATTACHMENT), GLenum(GL_RENDERBUFFER), depthRenderbuffer); checkGLErrors()

        let framebufferStatus = Int32(glCheckFramebufferStatus(GLenum(GL_FRAMEBUFFER)))
        assert(framebufferStatus == GL_FRAMEBUFFER_COMPLETE)
        if framebufferStatus != GL_FRAMEBUFFER_COMPLETE {
            return nil
        }

        // clear buffer
        glViewport(0, 0, width, height)
        glClear(GLbitfield(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)); checkGLErrors()

        // render
        renderAtTime(time); checkGLErrors()

        // create the image
        if floatComponents { // float components (16-bits of actual precision)

            // slurp bytes out of OpenGL
            typealias ComponentType = Float

            var imageRawBuffer = [ComponentType](count: Int(width * height) * samplesPerPixel * sizeof(ComponentType), repeatedValue: 0)
            glReadPixels(GLint(0), GLint(0), width, height, GLenum(GL_RGBA), GLenum(GL_FLOAT), &imageRawBuffer)

            // flip image vertically — OpenGL has a different 'up' than CoreGraphics
            let rowLength = Int(width) * samplesPerPixel
            for rowIndex in 0..<(Int(height) / 2) {
                let baseIndex = rowIndex * rowLength
                let destinationIndex = (Int(height) - 1 - rowIndex) * rowLength

                swap(&imageRawBuffer[baseIndex..<(baseIndex + rowLength)], &imageRawBuffer[destinationIndex..<(destinationIndex + rowLength)])
            }

            // make the CGImage
            var imageBuffer = vImage_Buffer(
                data: UnsafeMutablePointer<Float>(imageRawBuffer),
                height: vImagePixelCount(height),
                width: vImagePixelCount(width),
                rowBytes: Int(width) * sizeof(ComponentType) * samplesPerPixel)

            var format = vImage_CGImageFormat(
                bitsPerComponent: UInt32(sizeof(ComponentType) * 8),
                bitsPerPixel: UInt32(sizeof(ComponentType) * samplesPerPixel * 8),
                colorSpace: nil, // defaults to sRGB
                bitmapInfo: CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Little.rawValue | CGBitmapInfo.FloatComponents.rawValue),
                version: UInt32(0),
                decode: nil,
                renderingIntent: kCGRenderingIntentDefault)

            var error: vImage_Error = 0
            thumbnailCGImage = vImageCreateCGImageFromBuffer(&imageBuffer, &format, nil, nil, vImage_Flags(kvImagePrintDiagnosticsToConsole), &error)!.takeRetainedValue()

        } else { // byte components

            // slurp bytes out of OpenGL
            typealias ComponentType = UInt8

            var imageRawBuffer = [ComponentType](count: Int(width * height) * samplesPerPixel * sizeof(ComponentType), repeatedValue: 0)
            glReadPixels(GLint(0), GLint(0), width, height, GLenum(GL_RGBA), GLenum(GL_UNSIGNED_BYTE), &imageRawBuffer)

            // flip image vertically — OpenGL has a different 'up' than CoreGraphics
            let rowLength = Int(width) * samplesPerPixel
            for rowIndex in 0..<(Int(height) / 2) {
                let baseIndex = rowIndex * rowLength
                let destinationIndex = (Int(height) - 1 - rowIndex) * rowLength

                swap(&imageRawBuffer[baseIndex..<(baseIndex + rowLength)], &imageRawBuffer[destinationIndex..<(destinationIndex + rowLength)])
            }

            // make the CGImage
            var imageBuffer = vImage_Buffer(
                data: UnsafeMutablePointer<Float>(imageRawBuffer),
                height: vImagePixelCount(height),
                width: vImagePixelCount(width),
                rowBytes: Int(width) * sizeof(ComponentType) * samplesPerPixel)

            var format = vImage_CGImageFormat(
                bitsPerComponent: UInt32(sizeof(ComponentType) * 8),
                bitsPerPixel: UInt32(sizeof(ComponentType) * samplesPerPixel * 8),
                colorSpace: nil, // defaults to sRGB
                bitmapInfo: CGBitmapInfo(CGImageAlphaInfo.PremultipliedLast.rawValue | CGBitmapInfo.ByteOrder32Big.rawValue),
                version: UInt32(0),
                decode: nil,
                renderingIntent: kCGRenderingIntentDefault)

            var error: vImage_Error = 0
            thumbnailCGImage = vImageCreateCGImageFromBuffer(&imageBuffer, &format, nil, nil, vImage_Flags(kvImagePrintDiagnosticsToConsole), &error)!.takeRetainedValue()
        }

        #if os(iOS)
            objc_sync_exit(glContext)
            if oldGLContext != nil {
                EAGLContext.setCurrentContext(oldGLContext)
            }
        #elseif os(OSX)
            CGLUnlockContext(glContext)
            if oldGLContext != nil {
                CGLSetCurrentContext(oldGLContext)
            }
        #endif

        return thumbnailCGImage
    }
}


func checkGLErrors() {
    var glError: GLenum
    var hadError = false
    do {
        glError = glGetError()
        if glError != 0 {
            println(String(format: "OpenGL error %#x", glError))
            hadError = true
        }
    } while glError != 0
    assert(!hadError)
}
这看起来很棒！如果我们想要捕捉像从移动相机节点拍摄的视频，这个方案可行吗？换句话说，目标是模拟一个相机节点探索SCNView的视频。 - Crashalot
是的，这个渲染是从渲染器当前的场景.pointOfView节点中生成的，因此您可以在帧之间移动该节点和/或更改其SCNCamera以进行动画处理。 - Wil Shipley
谢谢@WilShipley！如果您不介意，我想通过电子邮件向您询问一些问题。什么是最好的联系方式？ - Crashalot
wjs at delicious-monster dot com： - Wil Shipley
4

** 这是使用Metal的SceneKit的答案。

** 警告：这可能不是App Store的适当方法。但它是有效的。

步骤1：使用swizzling将CAMetalLayer的nextDrawable方法替换为新方法。 在每个渲染循环中保存CAMetalDrawable。

extension CAMetalLayer {
  public static func setupSwizzling() {
    struct Static {
      static var token: dispatch_once_t = 0
    }

    dispatch_once(&Static.token) {
      let copiedOriginalSelector = #selector(CAMetalLayer.orginalNextDrawable)
      let originalSelector = #selector(CAMetalLayer.nextDrawable)
      let swizzledSelector = #selector(CAMetalLayer.newNextDrawable)

      let copiedOriginalMethod = class_getInstanceMethod(self, copiedOriginalSelector)
      let originalMethod = class_getInstanceMethod(self, originalSelector)
      let swizzledMethod = class_getInstanceMethod(self, swizzledSelector)

      let oldImp = method_getImplementation(originalMethod)
      method_setImplementation(copiedOriginalMethod, oldImp)
      method_exchangeImplementations(originalMethod, swizzledMethod)
    }
  }


  func newNextDrawable() -> CAMetalDrawable? {
    let drawable = orginalNextDrawable()
    // Save the drawable to any where you want
    AppManager.sharedInstance.currentSceneDrawable = drawable
    return drawable
  }

  func orginalNextDrawable() -> CAMetalDrawable? {
    // This is just a placeholder. Implementation will be replaced with nextDrawable.
    return nil
  }
}

步骤2： 在AppDelegate: didFinishLaunchingWithOptions中设置方法替换。
func application(application: UIApplication, didFinishLaunchingWithOptions launchOptions: [NSObject: AnyObject]?) -> Bool {
  CAMetalLayer.setupSwizzling()
  return true
}

第三步： 为您的SCNView的CAMetalLayer禁用framebufferOnly（以便调用MTLTexture的getBytes方法）。
if let metalLayer = scnView.layer as? CAMetalLayer {
  metalLayer.framebufferOnly = false
}

第四步： 在SCNView的代理（SCNSceneRendererDelegate）中，调整纹理。
func renderer(renderer: SCNSceneRenderer, didRenderScene scene: SCNScene, atTime time: NSTimeInterval) {
    if let texture = AppManager.sharedInstance.currentSceneDrawable?.texture where !texture.framebufferOnly {
      AppManager.sharedInstance.currentSceneDrawable = nil
      // Get image from texture
      let image = texture.toImage()
      // Use the image for video recording
    }
}

extension MTLTexture {
  func bytes() -> UnsafeMutablePointer<Void> {
    let width = self.width
    let height   = self.height
    let rowBytes = self.width * 4
    let p = malloc(width * height * 4) //Beware for memory leak
    self.getBytes(p, bytesPerRow: rowBytes, fromRegion: MTLRegionMake2D(0, 0, width, height), mipmapLevel: 0)
    return p
  }

  func toImage() -> UIImage? {
    var uiImage: UIImage?
    let p = bytes()
    let pColorSpace = CGColorSpaceCreateDeviceRGB()
    let rawBitmapInfo = CGImageAlphaInfo.NoneSkipFirst.rawValue | CGBitmapInfo.ByteOrder32Little.rawValue
    let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)

    let selftureSize = self.width * self.height * 4
    let rowBytes = self.width * 4
    let provider = CGDataProviderCreateWithData(nil, p, selftureSize, {_,_,_ in })!

    if let cgImage = CGImageCreate(self.width, self.height, 8, 32, rowBytes, pColorSpace, bitmapInfo, provider, nil, true, CGColorRenderingIntent.RenderingIntentDefault) {
      uiImage = UIImage(CGImage: cgImage)
    }
    return uiImage
  }

  func toImageAsJpeg(compressionQuality: CGFloat) -> UIImage? {
  }
}

第五步（可选）：您可能需要确认您获取的CAMetalLayer中的drawable是您想要的目标。（如果同时有多个CAMetalLayer）

它在我的某些设备上完美运行。但是在一些设备上，它会在SceneKit框架内崩溃，并显示消息“在RenderPassDescriptor中设置rendertargets或设置renderTargetWidth和renderTargetHeight”。但似乎你的应用程序Facemous能够生成正确的视频。你能帮我解决这个问题吗？我可以为你创建一个示例。 - Joker
1

实际上这很容易！以下是我在 SCNView 上如何做的伪代码：

int numberOfFrames = 300;
int currentFrame = 0;
int framesPerSecond = 30;

-(void) renderAFrame{
    [self renderAtTime:1/framesPerSecond];

    NSImage *frame = [self snapshot];

    // save the image with the frame number in the name such as f_001.png

    currentFrame++;

    if(currentFrame < numberOfFrames){
        [self renderAFrame];
    }

}

它将为您输出一系列图像，以每秒30帧的速度渲染，您可以在任何编辑软件中导入并转换为视频。
这个方案对你有用吗？我们尝试过类似的方法，但在帧之间改变场景（例如放置节点或移动相机）与截图/快照不同步，这意味着你可能会创建一个块，但它不会反映在下一个快照中。 - Crashalot
1
你可以使用SKVideoNode来实现这一点，将其放入SKScene中，并将其用作SCNode的SCMaterial.Diffuse.Content（希望这很清楚 ;)）
player = AVPlayer(URL: fileURL!)
let videoSpriteKitNodeLeft = SKVideoNode(AVPlayer: player)
let videoNodeLeft = SCNNode()
let spriteKitScene1 = SKScene(size: CGSize(width: 1280 * screenScale, height: 1280 * screenScale))
spriteKitScene1.shouldRasterize = true

videoNodeLeft.geometry = SCNSphere(radius: 30)
spriteKitScene1.scaleMode = .AspectFit
videoSpriteKitNodeLeft.position = CGPoint(
  x: spriteKitScene1.size.width / 2.0, y: spriteKitScene1.size.height / 2.0)
videoSpriteKitNodeLeft.size = spriteKitScene1.size

spriteKitScene1.addChild(videoSpriteKitNodeLeft)

videoNodeLeft.geometry?.firstMaterial?.diffuse.contents = spriteKitScene1
videoNodeLeft.geometry?.firstMaterial?.doubleSided = true

// Flip video upside down, so that it's shown in the right position
var transform = SCNMatrix4MakeRotation(Float(M_PI), 0.0, 0.0, 1.0)
transform = SCNMatrix4Translate(transform, 1.0, 1.0, 0.0)

videoNodeLeft.pivot = SCNMatrix4MakeRotation(Float(M_PI_2), 0.0, -1.0, 0.0)
videoNodeLeft.geometry?.firstMaterial?.diffuse.contentsTransform = transform

videoNodeLeft.position = SCNVector3(x: 0, y: 0, z: 0)

scene.rootNode.addChildNode(videoNodeLeft)

我从我的Github项目中提取了代码，用SceneKit制作了一个360度视频播放器，可以在3D球体内播放视频：https://github.com/Aralekk/simple360player_iOS/blob/master/simple360player/ViewController.swift 希望这能有所帮助！
Arthur
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接