logo标签列表

安卓ImageReader YUV 420 888 重复数据

androidopencv
4
我正在尝试使用Camera 2 API将从ImageReader收到的Image转换为OpenCV矩阵，并使用CameraBridgeViewBase中的deliverAndDrawFrame函数在屏幕上显示它。读取器的ImageFormat为YUV_420_888，据我所知，它具有每个像素的灰度值的Y平面和每4个像素中具有U / V的U平面。然而，当我尝试显示这个图像时，它似乎重复出现并且旋转了90度。下面的代码旨在将YUV数据放入OpenCV矩阵中（目前仅为灰度，不是rgba）：
/**
 * Takes an {@link Image} in the {@link ImageFormat#YUV_420_888} and puts it into a provided {@link Mat} in rgba format.
 *
 * @param yuvImage {@link Image} in the {@link ImageFormat#YUV_420_888} format.
 */
public static void yuv420888imageToRgbaMat(final Image yuvImage, final Mat rgbaMat) {

    final Image.Plane
            Yp     = yuvImage.getPlanes()[0],
            UandVp = yuvImage.getPlanes()[1];

    final ByteBuffer
            Ybb     = Yp    .getBuffer(),
            UandVbb = UandVp.getBuffer();

    Ybb    .get(mYdata    , 0, 480*640        );
    UandVbb.get(mUandVData, 0, 480*640 / 2 - 8);

    for (int i = 0; i < 640*480; i++) {
        for (int j = 0; j < 4; j++) {
            mRawRGBAFrameData[i + 640*480*j] = mYdata[i];
        }
        mRawRGBAFrameData[i*4  ] = mYdata[i];
        mRawRGBAFrameData[i*4+1] = mYdata[i];
        mRawRGBAFrameData[i*4+2] = mYdata[i];
        mRawRGBAFrameData[i*4+3] = -1;
    }
}

这是我为OpenCV框架编写的代码：
private class CameraFrame implements CvCameraViewFrame {

    private Mat mRgba;

    @Override
    public Mat gray() {

        return null;
    }

    @Override
    public Mat rgba() {

        mRgbaMat.put(0, 0, mRawRGBAFrameData);

        return mRgba;
    }

    public CameraFrame(final Mat rgba) {

        super();

        mRgba = rgba;
    }
}

接收并绘制帧的代码：
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {

    @Override
    public void onImageAvailable(ImageReader reader) {

        final Image yuvImage = reader.acquireLatestImage();

        yuv420888imageToRgbaMat(yuvImage, mRgbaMat);

        deliverAndDrawFrame(mFrame);

        yuvImage.close();
    }
};

这是制作图像读取器的代码：

mRgbaMat = new Mat(mFrameHeight, mFrameWidth, CvType.CV_8UC4);

mFrame = new CameraFrame(mRgbaMat);

mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 1);

mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);

AllocateCache();

这是数组的初始化:

protected static byte[] mRawRGBAFrameData = new byte[640*480*4], mYdata = new byte[640*480], mUandVData = new byte[640*480 / 2];

注意：mFrameWidth为480，mFrameHeight为640。奇怪的是，ImageReader的高度和宽度以及从中接收到的Image具有倒置的尺寸。
以下是使用上述代码的图像：https://istack.dev59.com/lcdzf.webp 这是在yuv420888imageToRgbaMat中使用此代码的图像： https://istack.dev59.com/T2MOI.webp 
for (int i = 0; i < 640*480; i++) {
    mRawRGBAFrameData[i] = mYdata[i];
}

我们可以看到数据在Y帧中重复出现，不知何故这给出了一个实际上好看的图像。
1个回答
2

对于任何遇到使用Camera 2 API与OpenCV一起使用的问题的人，我已经找到了解决方案。我发现的第一件事是ImageReader提供的ByteBuffer中存在填充，如果您不加考虑就会导致输出失真。我选择做的另一件事是创建自己的SurfaceView，并使用Bitmap在其上绘制，而不是使用CameraViewBase，目前效果非常好。OpenCV有一个函数Util.matToBitmap，它可以将BGR矩阵转换为Android Bitmap，因此这非常有用。我通过将ImageReader提供的前两个Image.Plane中的信息放入格式为YUV 420的OpenCV单通道矩阵中，并使用Imgproc.cvtColor和Imgproc.COLOR_YUV420p2BGR来获取BGR矩阵。需要知道的重要事情是图像的Y平面具有完整的像素，但第二个UV平面具有交错像素，每四个Y像素映射一个UV像素，因此UV平面的总长度是Y平面的一半。详见here。无论如何，以下是一些代码：
翻译：矩阵的初始化
m_BGRMat = new Mat(Constants.VISION_IMAGE_HEIGHT, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC3);
m_Yuv420FrameMat = new Mat(Constants.VISION_IMAGE_HEIGHT * 3 / 2, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC1);

每一帧：
// Convert image to YUV 420 matrix
ImageUtils.imageToMat(image, m_Yuv420FrameMat, m_RawFrameData, m_RawFrameRowData);
// Convert YUV matrix to BGR matrix
Imgproc.cvtColor(m_Yuv420FrameMat, m_BGRMat, Imgproc.COLOR_YUV420p2BGR);
// Flip width and height then mirror vertically
Core.transpose(m_BGRMat, m_BGRMat);
Core.flip(m_BGRMat, m_BGRMat, 0);
// Draw to Surface View
m_PreviewView.drawImageMat(m_BGRMat);

这是转换为YUV 420矩阵的结果：
/**
 * Takes an Android {@link Image} in the {@link ImageFormat#YUV_420_888} format and returns an OpenCV {@link Mat}.
 *
 * @param image {@link Image} in the {@link ImageFormat#YUV_420_888} format
 */
public static void imageToMat(final Image image, final Mat mat, byte[] data, byte[] rowData) {
    ByteBuffer buffer;
    int rowStride, pixelStride, width = image.getWidth(), height = image.getHeight(), offset = 0;
    Image.Plane[] planes = image.getPlanes();
    if (data == null || data.length != width * height) data = new byte[width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    if (rowData == null || rowData.length != planes[0].getRowStride()) rowData = new byte[planes[0].getRowStride()];
    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int
            w = (i == 0) ? width : width / 2,
            h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);
                // Advance buffer the remainder of the row stride, unless on the last row.
                // Otherwise, this will throw an IllegalArgumentException because the buffer
                // doesn't include the last padding.
                if (h - row != 1)
                    buffer.position(buffer.position() + rowStride - length);
                offset += length;
            } else {
                // On the last row only read the width of the image minus the pixel stride
                // plus one. Otherwise, this will throw a BufferUnderflowException because the
                // buffer doesn't include the last padding.
                if (h - row == 1)
                    buffer.get(rowData, 0, width - pixelStride + 1);
                else
                    buffer.get(rowData, 0, rowStride);
                for (int col = 0; col < w; col++)
                    data[offset++] = rowData[col * pixelStride];
            }
        }
    }
    mat.put(0, 0, data);
}

最后，绘画。
/**
 * Given an {@link Mat} that represents a BGR image, draw it on the surface canvas.
 * use the OpenCV helper function {@link Utils#matToBitmap(Mat, Bitmap)} to create a {@link Bitmap}.
 *
 * @param bgrMat BGR frame {@link Mat}
 */
public void drawImageMat(final Mat bgrMat) {
    if (m_HolderReady) {
        // Create bitmap from BGR matrix
        Utils.matToBitmap(bgrMat, m_Bitmap);
        // Obtain the canvas and draw the bitmap on top of it
        final SurfaceHolder holder = getHolder();
        final Canvas canvas = holder.lockCanvas();
        canvas.drawBitmap(m_Bitmap, null, new Rect(0, 0, m_HolderWidth, m_HolderHeight), null);
        holder.unlockCanvasAndPost(canvas);
    }
}

这种方法可以工作，但我想最好的方法是设置一个OpenGL渲染上下文，并编写一些简单的着色器来显示矩阵。
1
我认为这一行代码Core.flip(m_BGRMat, m_BGRMat, 0);存在问题，图像看起来是倒置的，所以我将其更改为Core.flip(m_BGRMat, m_BGRMat, 1);，同时将COLOR_YUV420p2BGR更改为COLOR_YUV420p2RGB以进行JPEG转换。 - sercanturkmen
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接