OpenCV / C++:如何有效地设置树莓派摄像头帧率

3
我最近开始使用OpenCV。虽然一开始我取得了很多进展,但是现在我快要因为看起来很简单的任务而失去理智:使用Raspberry Pi相机记录一个简单的视频。问题在于生成的视频似乎是快进的,并且原因似乎是在录制时没有写入所需帧的一半。
经过许多个小时的尝试编解码器和对我的代码进行时间测量以找到瓶颈,我现在发现问题似乎与OpenCV VideoCapture类有些相关,该类实例在我的代码中实际上提供的帧远少于预期。
所以我编写了一个简单的程序,在五秒钟内计算VideoCapture提供的帧数。将捕获属性设置为640x480x30fps可以正常工作,并提供约150个帧。但将其调整为1920x1080x30fps(根据规格是有效的相机模式并在其他应用程序中正常工作)最终只能在5秒内提供大约15个帧。
可能有非常明显的解决方案,但我完全无法想到。有人能帮帮我吗?谢谢!
#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <ctime>

float getElapsedCPUTime(std::clock_t begin){
    return float(clock() - begin)/CLOCKS_PER_SEC;
}

std::time_t getCurrentWallTime(){
    return std::time(nullptr);
}

int main (){ 
//  int cols(640);
//  int rows(480);
    int cols(1920);
    int rows(1080);
    cv::Mat currentFrame;

    // set capture properties
    cv::VideoCapture cap(0);
    cap.set(CV_CAP_PROP_FRAME_HEIGHT, rows);
    cap.set(CV_CAP_PROP_FRAME_WIDTH, cols);
    cap.set(cv::CAP_PROP_FPS, 30);
    cap.set(cv::CAP_PROP_FOURCC, 0x21);

    // control capture properties
    int rows_c(cap.get(CV_CAP_PROP_FRAME_HEIGHT));
    int cols_c(cap.get(CV_CAP_PROP_FRAME_WIDTH));
    int fps(cap.get(cv::CAP_PROP_FPS));
    std::cout << "rows: " << rows_c << ", cols " << cols_c << ", fps " << fps << ", CPS: " << CLOCKS_PER_SEC << std::endl;

    int cnt(0);
    std::time_t loopExecution_begin(getCurrentWallTime());
    while(1){
        std::string msg("");

        // capture frame
        std::clock_t capture_begin(clock());
        cap >> currentFrame;
        float time_for_capture = getElapsedCPUTime(capture_begin);
        ++cnt;

        // get elapsed wall time
        std::time_t loopRunTime = getCurrentWallTime() - loopExecution_begin;

        // output message
        msg += "#: " + std::to_string(cnt);
        msg += "\tTicks begin: " + std::to_string(capture_begin);
        msg += "\tCapturetime: " + std::to_string(time_for_capture) + "s";
        msg += "\tLoop Runtime: " + std::to_string(loopRunTime) + "s";
        std::cout << msg << std::endl;

        // break after 5s
        if (loopRunTime > 5.0) break;

    }   
}

编辑:这是输出结果:

rows: 1080, cols 1920, fps 30, CPS: 1000000
#: 1    Ticks begin: 362378 Capturetime: 0.055826s  Loop Runtime: 1s
#: 2    Ticks begin: 418543 Capturetime: 0.022631s  Loop Runtime: 1s
#: 3    Ticks begin: 441338 Capturetime: 0.022695s  Loop Runtime: 1s
#: 4    Ticks begin: 464196 Capturetime: 0.023302s  Loop Runtime: 2s
#: 5    Ticks begin: 487659 Capturetime: 0.022729s  Loop Runtime: 2s
#: 6    Ticks begin: 510551 Capturetime: 0.022631s  Loop Runtime: 2s
#: 7    Ticks begin: 533349 Capturetime: 0.022663s  Loop Runtime: 2s
#: 8    Ticks begin: 556176 Capturetime: 0.023194s  Loop Runtime: 3s
#: 9    Ticks begin: 579535 Capturetime: 0.022640s  Loop Runtime: 3s
#: 10   Ticks begin: 602337 Capturetime: 0.023267s  Loop Runtime: 3s
#: 11   Ticks begin: 625789 Capturetime: 0.022741s  Loop Runtime: 3s
#: 12   Ticks begin: 648694 Capturetime: 0.023210s  Loop Runtime: 3s
#: 13   Ticks begin: 672069 Capturetime: 0.022487s  Loop Runtime: 4s
#: 14   Ticks begin: 694721 Capturetime: 0.023162s  Loop Runtime: 4s
#: 15   Ticks begin: 718051 Capturetime: 0.022611s  Loop Runtime: 4s
#: 16   Ticks begin: 740822 Capturetime: 0.023602s  Loop Runtime: 4s
#: 17   Ticks begin: 764600 Capturetime: 0.022555s  Loop Runtime: 5s
#: 18   Ticks begin: 787321 Capturetime: 0.022532s  Loop Runtime: 5s
#: 19   Ticks begin: 810019 Capturetime: 0.022626s  Loop Runtime: 5s
#: 20   Ticks begin: 832813 Capturetime: 0.023161s  Loop Runtime: 5s
#: 21   Ticks begin: 856138 Capturetime: 0.022543s  Loop Runtime: 6s

你用的是哪个相机,它是通过CSI/USB连接的吗?你正在使用的FOUR_CC代码是什么,你从哪里获得的? - Mark Setchell
@MarkSetchell:我正在使用Raspberry Camera Module v2,它连接到树莓派板子上的一个独立端口。FOUR_CC 0x21 显然是 H264,我正在使用它来解决另一个问题,那就是 OpenCV 的 VideoWriter。我花了一些时间才发现文件扩展名需要与 FOUR_CC 相匹配。在寻找 FOUR_CC 到有效扩展名的映射时(您不巧正好有吗?),我偶然发现了这里的 SO 线程 和 0x21 FOUR_CC。 - TonySoprano
这没有意义。你的帧需要25毫秒才能获取,但每秒只能获取4个。如果每个帧需要25毫秒,那么你应该能够每秒获取40个。你的树莓派在每秒的其他900毫秒里在做什么?在每个帧的开始输出滴答声时间可能会很有启发性。 - Mark Setchell
@MarkSetchell:要是我知道就好了。我也在输出中加入了勾号,但是我并没有从中得到更多的信息。有什么想法吗? - TonySoprano
如果最后一帧从850,000个刻度开始,而第一帧从350,000个刻度开始,则整个过程需要500,000个刻度。如果您有1,000,000个CLOCKS_PER_SEC,那么整个过程只需要0.5秒?它运行了5秒吗?还是0.5秒? - Mark Setchell
根据我的手机秒表,运行了五秒钟。 - TonySoprano
1个回答

1

看起来您想从相机获取H.264流。

如果要对相机帧进行处理,OpenCV是无与伦比的选择,但我不一定会在需要获取H.264的任务中使用它,因为树莓派有一个硬件H.264编码器,我不确定在您的情况下是否使用了它。

要将H.264输出到文件或流式传输到其他地方,您可以使用标准的raspivid应用程序并将其输出导向所需位置,或者如果需要更多控制,则可以获取其源代码并进行修改。

我有点懒,就采用了前者。下面的代码将H.264流式传输到TCP套接字。如果出现拥塞(即数据无法快速写入套接字),代码将关闭套接字,此后我的客户端会重新连接。这种方法可能不是非常优雅,但我已经使用了一段时间,似乎效果还不错。

#include "CameraRelay.h"
#include <arpa/inet.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
#include <sys/select.h>
#include <poll.h>
#include <sys/ioctl.h>
#include <fcntl.h>
#include <thread>
#include <mutex>
#include <memory>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>

#define READ 0
#define WRITE 1

pid_t popen2(const char *command, char * const args[], int *infp, int *outfp) {
  int p_stdin[2], p_stdout[2];
  pid_t pid;

  if (pipe(p_stdin) != 0 || pipe(p_stdout) != 0)
    return -1;

  pid = fork();

  if (pid < 0)
    return pid;
  else if (pid == 0)
  {
    close(p_stdin[WRITE]);
    dup2(p_stdin[READ], READ);
    close(p_stdout[READ]);
    dup2(p_stdout[WRITE], WRITE);

    execv(command, args);
    perror("execl");
    exit(1);
  }

  if (infp == NULL)
    close(p_stdin[WRITE]);
  else
    *infp = p_stdin[WRITE];

  if (outfp == NULL)
    close(p_stdout[READ]);
  else
    *outfp = p_stdout[READ];

  return pid;
}

namespace CameraRelay {
  static std::unique_ptr<std::thread> s_thread;
  static std::mutex s_startStopMutex;
  static bool s_bRun;
  int s_listensock = -1;

  void RelayThread() {
    signal(SIGPIPE, SIG_IGN);
    s_listensock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    if (s_listensock < 0) {
      perror("socket() for camera relay failed");
      return;
    }

    int on = 1;
    setsockopt(s_listensock, SOL_SOCKET, SO_REUSEADDR, (char *)&on, sizeof(on));
    ioctl(s_listensock, FIONBIO, (char *)&on);

    unsigned int bufsz = 1024; // small buffer to control latency
    setsockopt(s_listensock, SOL_SOCKET, SO_SNDBUF, (void *)&bufsz, sizeof(bufsz));

    struct sockaddr_in sa;
    memset((char *)&sa, 0, sizeof(sa));
    sa.sin_family = AF_INET;
    sa.sin_addr.s_addr = htonl(INADDR_ANY);
    sa.sin_port = htons(7124);
    if (bind(s_listensock, (struct sockaddr *)&sa, sizeof(struct sockaddr)) < 0) {
      perror("bind() for camera relay failed");
      return;
    }
    bufsz = 1024; // small buffer to control latency
    setsockopt(s_listensock, SOL_SOCKET, SO_SNDBUF, (void *)&bufsz, sizeof(bufsz));
    if (listen(s_listensock, 1)) {
      perror("listen() for camera relay failed");
      return;
    }
    while (s_bRun) {
      sockaddr_in sa;
      memset((char *)&sa, 0, sizeof(sa));
      socklen_t len = sizeof(sa);
      struct pollfd pfd;
      memset(&pfd, 0, sizeof(pfd));
      pfd.fd = s_listensock;
      pfd.events = POLLIN;
      int ret = poll(&pfd, 1, 500);
      if (ret > 0) {        
        int sock = accept(s_listensock, (struct sockaddr *)&sa, &len);
        unsigned int bufsz = 16384; // small buffer to control latency
        setsockopt(sock, SOL_SOCKET, SO_SNDBUF, (void *)&bufsz, sizeof(bufsz));
        fcntl(sock, F_SETFL, fcntl(sock, F_GETFL, 0) | O_NONBLOCK);
        printf("camera relay socket got new connection.\n");
        char buffer[512];
        std::string cmd = "/usr/bin/raspivid";
        const char *args[] = { "/usr/bin/raspivid", "-o", "-", "-n" , "-s", "-t" ,"0", "-fl", "-hf", "-b", "5000000", "-w", "1280", "-h",  "720", "-fps", "25", nullptr };
        int pin = 0;
        int pout = 0;
        pid_t pid = popen2(cmd.c_str(), const_cast<char **>(args), &pin, &pout);
        if (pid <= 0) {
          perror("popen() failed!");
          break;
        }
        fcntl(pout, F_SETPIPE_SZ, 1024); // small buffer to control latency
        bool bFailed = false;
        int nWait = 0;
        while (s_bRun && !bFailed) {
          struct pollfd pfd2[2];
          memset(&pfd2, 0, sizeof(pfd2));
          pfd2[0].fd = sock;
          pfd2[0].events = POLLIN;
          pfd2[1].fd = pout;
          pfd2[1].events = POLLIN;
          poll(pfd2, 2, 500);

          if (pfd2[0].revents) {
            char rcv;
            if (recv(sock, &rcv, 1, 0)) {
            }
          }

          if (pfd2[1].revents) {
            int nRead = read(pout, buffer, sizeof(buffer));
            if (nRead <= 0) {
              break;
            }
            int nSent = 0;
            while (nSent < nRead) {
              int ret = send(sock, buffer + nSent, nRead - nSent, 0);
              if ((ret < 0 && errno != EWOULDBLOCK) || !s_bRun) {
                perror("camera relay send() failed");
                bFailed = true;
                break;
              }
              if (!nSent && ret == nRead) {
                nWait = 0;
              }
              nSent += ret;
              if (nSent < nRead) {
                memset(&pfd, 0, sizeof(pfd));
                pfd.fd = sock;
                pfd.events = POLLOUT;                
                poll(&pfd, 1, 500);
                nWait++;
                if (nWait > 2) {
                  bFailed = true;
                  break;
                }
              } 
            }
          }
        }
        if (pout > 0)
          close(pout);
        if (pin > 0)
          close(pin);
        if (pid > 0) {                    
          printf("Killing raspivid child process %d with SIGKILL...\n", pid);
          kill(pid, SIGINT);
          sleep(1);
          kill(pid, SIGKILL);
          int status = 0;
          printf("Waiting for process to end...\n");
          waitpid(pid, &status, 0);
          printf("Killed raspivid child process\n");
        }
        if (sock > 0)
          close(sock);        
      }
    }    
  }

  bool Open() {
    if (s_thread)
      return false;
    system("killall raspivid");
    s_bRun = true;
    s_thread = std::unique_ptr<std::thread>(new std::thread(RelayThread));
    return true;
  }

  void Close() {
     if (!s_thread)
      return;
    s_bRun = false;
    s_thread->join();
  }
};

这是一个非常有趣的方法!在接下来的两天里我无法处理它,但之后我会试一试。至于编解码器,我并不局限于h264,只是它是我能够让OpenCV VideoWriter工作的唯一编解码器(请参见上面的评论)。是否有任何关于哪些编解码器与OpenCV兼容的见解或文档? - TonySoprano
H.264是Pi唯一能够在硬件上进行编码和解码的压缩格式,因此如果您想存储或流传压缩视频数据,H.264是最佳选择。 - Sami Sallinen

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接