logo标签列表

将FFmpeg转码结果流式传输到S3

amazon-s3ffmpeg
18

我想使用FFMPEG转码一个大文件并直接将结果存储在AWS S3中。这将在一个具有有限tmp空间的AWS Lambda内完成，因此我无法在第二步中将转码结果本地存储然后上传到S3。我没有足够的tmp空间。因此，我希望直接将FFMPEG输出存储在S3上。

因此，我创建了一个允许'PUT'的S3预签名URL：

var outputPath = s3Client.GetPreSignedURL(new Amazon.S3.Model.GetPreSignedUrlRequest
{
    BucketName = "my-bucket",
    Expires = DateTime.UtcNow.AddMinutes(5),
    Key = "output.mp3",
    Verb = HttpVerb.PUT,
});

然后我使用生成的预签名URL调用了FFmpeg：

ffmpeg -i C:\input.wav -y -vn -ar 44100 -ac 2 -ab 192k -f mp3 https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D

FFMPEG返回1的退出代码，并带有以下输出：

ffmpeg version N-93120-ga84af760b8 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20190212
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 47.100 / 58. 47.100
  libavformat    58. 26.101 / 58. 26.101
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'C:\input.wav':
  Duration: 00:04:16.72, bitrate: 3072 kb/s
    Stream #0:0: Audio: pcm_s32le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s32, 3072 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s32le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D':
  Metadata:
    TSSE            : Lavf58.26.101
    Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s32p, 192 kb/s
    Metadata:
      encoder         : Lavc58.47.100 libmp3lame
size=     577kB time=00:00:24.58 bitrate= 192.2kbits/s speed=49.1x    
size=    1109kB time=00:00:47.28 bitrate= 192.1kbits/s speed=47.2x    
[tls @ 000001d73d786b00] Error in the push function.
av_interleaved_write_frame(): I/O error
Error writing trailer of https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D: I/O error
size=    1143kB time=00:00:48.77 bitrate= 192.0kbits/s speed=  47x    
video:0kB audio:1144kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[tls @ 000001d73d786b00] The specified session has been invalidated for some reason.
[tls @ 000001d73d786b00] Error in the pull function.
[https @ 000001d73d784fc0] URL read error:  -5
Conversion failed!

如您所见，我遇到了URL读取错误。这对我来说有些令人惊讶，因为我想要将输出写入该URL而不是读取它。

有人知道如何直接将我的FFMPEG输出存储到S3而不必先在本地存储吗？

编辑1 然后我尝试使用-method PUT参数并使用http而不是https来消除TLS的影响。以下是在使用-v trace选项运行ffmpeg时获得的输出。

ffmpeg version N-93120-ga84af760b8 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8.2.1 (GCC) 20190212
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 26.100 / 56. 26.100
  libavcodec     58. 47.100 / 58. 47.100
  libavformat    58. 26.101 / 58. 26.101
  libavdevice    58.  6.101 / 58.  6.101
  libavfilter     7. 48.100 /  7. 48.100
  libswscale      5.  4.100 /  5.  4.100
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Splitting the commandline.
Reading option '-i' ... matched as input url with argument 'C:\input.wav'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-vn' ... matched as option 'vn' (disable video) with argument '1'.
Reading option '-ar' ... matched as option 'ar' (set audio sampling rate (in Hz)) with argument '44100'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
Reading option '-ab' ... matched as option 'ab' (audio bitrate (please use -b:a)) with argument '192k'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'mp3'.
Reading option '-method' ... matched as AVOption 'method' with argument 'PUT'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'trace'.
Reading option 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument trace.
Successfully parsed a group of options.
Parsing a group of options: input url C:\input.wav.
Successfully parsed a group of options.
Opening an input file: C:\input.wav.
[NULL @ 000001fb37abb180] Opening 'C:\input.wav' for reading
[file @ 000001fb37abc180] Setting default whitelist 'file,crypto'
Probing wav score:99 size:2048
[wav @ 000001fb37abb180] Format wav probed with size=2048 and score=99
[wav @ 000001fb37abb180] Before avformat_find_stream_info() pos: 54 bytes read:65590 seeks:1 nb_streams:1
[wav @ 000001fb37abb180] parser not found for codec pcm_s32le, packets or times may be invalid.
    Last message repeated 1 times
[wav @ 000001fb37abb180] All info found
[wav @ 000001fb37abb180] stream 0: start_time: -192153584101141.156 duration: 256.716
[wav @ 000001fb37abb180] format: start_time: -9223372036854.775 duration: 256.716 bitrate=3072 kb/s
[wav @ 000001fb37abb180] After avformat_find_stream_info() pos: 204854 bytes read:294966 seeks:1 frames:50
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'C:\input.wav':
  Duration: 00:04:16.72, bitrate: 3072 kb/s
    Stream #0:0, 50, 1/48000: Audio: pcm_s32le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s32, 3072 kb/s
Successfully opened the file.
Parsing a group of options: output url https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D.
Applying option vn (disable video) with argument 1.
Applying option ar (set audio sampling rate (in Hz)) with argument 44100.
Applying option ac (set number of audio channels) with argument 2.
Applying option ab (audio bitrate (please use -b:a)) with argument 192k.
Applying option f (force format) with argument mp3.
Successfully parsed a group of options.
Opening an output file: https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D.
[http @ 000001fb37b15140] Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
[tcp @ 000001fb37b16c80] Original list of addresses:
[tcp @ 000001fb37b16c80] Address 52.216.8.203 port 80
[tcp @ 000001fb37b16c80] Interleaved list of addresses:
[tcp @ 000001fb37b16c80] Address 52.216.8.203 port 80
[tcp @ 000001fb37b16c80] Starting connection attempt to 52.216.8.203 port 80
[tcp @ 000001fb37b16c80] Successfully connected to 52.216.8.203 port 80
[http @ 000001fb37b15140] request: PUT /output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D HTTP/1.1
Transfer-Encoding: chunked
User-Agent: Lavf/58.26.101
Accept: */*
Connection: close
Host: landr-distribution-reportsdev-mb.s3.amazonaws.com
Icy-MetaData: 1
Successfully opened the file.
Stream mapping:
  Stream #0:0 -> #0:0 (pcm_s32le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
detected 8 logical cores
[graph_0_in_0_0 @ 000001fb37b21080] Setting 'time_base' to value '1/48000'
[graph_0_in_0_0 @ 000001fb37b21080] Setting 'sample_rate' to value '48000'
[graph_0_in_0_0 @ 000001fb37b21080] Setting 'sample_fmt' to value 's32'
[graph_0_in_0_0 @ 000001fb37b21080] Setting 'channel_layout' to value '0x3'
[graph_0_in_0_0 @ 000001fb37b21080] tb:1/48000 samplefmt:s32 samplerate:48000 chlayout:0x3
[format_out_0_0 @ 000001fb37b22cc0] Setting 'sample_fmts' to value 's32p|fltp|s16p'
[format_out_0_0 @ 000001fb37b22cc0] Setting 'sample_rates' to value '44100'
[format_out_0_0 @ 000001fb37b22cc0] Setting 'channel_layouts' to value '0x3'
[format_out_0_0 @ 000001fb37b22cc0] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_0'
[AVFilterGraph @ 000001fb37b0d940] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed
[auto_resampler_0 @ 000001fb37b251c0] picking s32p out of 3 ref:s32
[auto_resampler_0 @ 000001fb37b251c0] [SWR @ 000001fb37b252c0] Using fltp internally between filters
[auto_resampler_0 @ 000001fb37b251c0] ch:2 chl:stereo fmt:s32 r:48000Hz -> ch:2 chl:stereo fmt:s32p r:44100Hz
Output #0, mp3, to 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D':
  Metadata:
    TSSE            : Lavf58.26.101
    Stream #0:0, 0, 1/44100: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s32p, delay 1105, 192 kb/s
    Metadata:
      encoder         : Lavc58.47.100 libmp3lame
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
    Last message repeated 6 times
size=     649kB time=00:00:27.66 bitrate= 192.2kbits/s speed=55.3x    
size=    1207kB time=00:00:51.48 bitrate= 192.1kbits/s speed=51.5x    
av_interleaved_write_frame(): Unknown error
No more output streams to write to, finishing.
[libmp3lame @ 000001fb37b147c0] Trying to remove 47 more samples than there are in the queue
Error writing trailer of https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D: Error number -10054 occurred
size=    1251kB time=00:00:53.39 bitrate= 192.0kbits/s speed=51.5x    
video:0kB audio:1252kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (C:\input.wav):
  Input stream #0:0 (audio): 5014 packets read (20537344 bytes); 5014 frames decoded (2567168 samples); 
  Total: 5014 packets (20537344 bytes) demuxed
Output file #0 (https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D):
  Output stream #0:0 (audio): 2047 frames encoded (2358144 samples); 2045 packets muxed (1282089 bytes); 
  Total: 2045 packets (1282089 bytes) muxed
5014 frames successfully decoded, 0 decoding errors
[AVIOContext @ 000001fb37b1f440] Statistics: 0 seeks, 2046 writeouts
[http @ 000001fb37b15140] URL read error:  -10054
[AVIOContext @ 000001fb37ac4400] Statistics: 20611126 bytes read, 1 seeks
Conversion failed!

看起来它能够连接到我的S3预签名URL，但我仍然遇到了Error writing trailer错误以及一个URL read error。

@MattiasWadman，很抱歉我不太了解TLS。为确保预签名URL与HTTP PUT正常工作，我使用curl进行了尝试（curl --upload-file <some_file> <the_presigned_url>），文件已成功上传。您是否有任何指导，为什么curl可以使用此URL而ffmpeg却不能？ - mabead
啊哈，也许你需要添加“-method PUT”吗？似乎ffmpeg默认为“POST”。我猜S3可能还需要更多的东西，例如带有正确文件名的多部分等等？ - Mattias Wadman
2
请注意，输出到不可寻址的格式可能会对ffmpeg的输出产生一些影响。例如，某些格式将无法获得正确的长度头等，因为它无法回溯并更改它。 - Mattias Wadman
你尝试过非 HTTPS 吗？同时尝试使用 -v trace 运行并将日志发布在某个地方（记得删除敏感信息！） - Mattias Wadman
stream = fs.createWriteStream(fs.stream); ffcmd = Ffmpeg(input)...strmout = ffcmd.pipe(); strmout.on('data', function(chunk) { }) - Robert Rowntree
显示剩余5条评论
3个回答
16

我使用像@mabead在他的回答中提到的ffmpeg管道访问协议，一切都很正常。 实际上，我通过URL定位文件，似乎也可以工作。.mp4会导致一些问题，因为您需要能够在编码完成后返回输出的开头以编写标头。 添加-movflags frag_keyframe+empty_moov解决了我的用例问题。 希望这段代码有所帮助：

ffmpeg -i https://notreal-bucket.s3-us-west-1.amazonaws.com/video/video.mp4 -f mp4 -movflags frag_keyframe+empty_moov pipe:1 | aws s3 cp - s3://notreal-bucket/video/output.mp4

FFmpeg文档 - 管道

1
我正在使用Python的ffmpeg，但是出现了这个错误：无法找到适合'|'的输出格式。 - Osama Bin Saleem
13

AWS CLI实际上有一个功能可以完全按照@mabead上面描述的做法。由于CLI默认情况下未在lambda中安装，因此您需要将其包含在内，可能作为一种“层”，但如果您已经安装了ffmpeg，则显然知道如何执行此操作。

基本上，它看起来像这样（不包括ffmpeg选项）:

aws s3 cp s3://source-bucket/source.mp4 - | ffmpeg -i - -f matroska - | aws s3 cp - s3://dest-bucket/output.mkv

在CLI和ffmpeg命令中，您可以将破折号（“-”）作为源或文件名的一部分。因此，在这种情况下，我们正在从S3读取到STDOUT，将其传输到ffmpeg STDIN，将ffmpeg输出写入STDOUT，将其传输到S3目标。

通常我只处理视频文件，所以我对纯音频没有太多经验，你可以试试。我注意到某些容器格式不适用于此输出。例如，如果我尝试将其写入S3中的mp4文件，则会看到以下错误：

muxer does not support non seekable output Could not write header for
output file #0 (incorrect codec parameters ?): Invalid argument Error
initializing output stream 0:0 --

我认为这可能是与无法使用最终编码的结果更新标题的评论相同的问题。您将不得不看看mp3会发生什么。

你提到你通常使用视频文件，但上述方法不起作用，能否提供你正在使用的示例？我尝试了上述方法，像你所说，FFMPEG会抛出一些错误。 - Mike
@JasonK 有没有缩放图像的命令？我需要相当于 ffmpeg -i in.png -vf scale=640:480 out.png 的命令。 - V.Khakhil
11
由于目标是从S3获取一串字节并将其输出到S3，因此不必使用ffmpeg的HTTP功能。作为一个可从stdin接收输入并输出到stdout/stderr的命令行工具，使用这些功能比尝试让ffmpeg处理HTTP读写更加简单。您只需要将一个HTTP流（从S3读取）连接到ffmpeg的stdin，然后将其stdout连接到另一个流（写入到S3）。有关ffmpeg管道的更多信息，请参见此处。
最简单的实现如下：
var s3Client = new AmazonS3Client(RegionEndpoint.USEast1);

var startInfo = new ProcessStartInfo
{
    FileName = "ffmpeg",
    Arguments = $"-i pipe:0 -y -vn -ar 44100 -ab 192k -f mp3 pipe:1",
    CreateNoWindow = true,
    RedirectStandardInput = false,
    RedirectStandardOutput = false,
    UseShellExecute = false,
    RedirectStandardInput = true,
    RedirectStandardOutput = true,
};

using (var process = new Process { StartInfo = startInfo })
{
    // Get a stream to an object stored on S3.
    var s3InputObject = await s3Client.GetObjectAsync(new GetObjectRequest
    {
        BucketName = "my-bucket",
        Key = "input.wav",
    });

    process.Start();

    // Store the output of ffmpeg directly on S3 in a background thread
    // since I don't 'await'.
    var uploadTask = s3Client.PutObjectAsync(new PutObjectRequest
    {
        BucketName = "my-bucket",
        Key = "output.wav",
        InputStream = process.StandardOutput.BaseStream,
    });

    // Feed the S3 input stream into ffmpeg
    await s3Object.ResponseStream.CopyToAsync(process.StandardInput.BaseStream);
    process.StandardInput.Close();

    // Wait for ffmpeg to be done
    await uploadTask;

    process.WaitForExit();
}

这段代码展示了如何将ffmpeg的输入/输出进行连接。

不幸的是，这段代码无法正常工作。调用PutObjectAsync会抛出一个异常，说无法确定内容长度。是的，这是真的，S3只允许上传已知大小的文件，我们不能使用PutObjectAsync，因为我们不知道ffmpeg的输出有多大。

解决这个问题的方法是使用S3分块上传。所以，不直接将ffmpeg的输出传输到S3，而是将其写入内存缓冲区（比如25 MB），这样缓冲区不会太大（这样就不会消耗运行此代码的AWS Lambda的所有内存）。当缓冲区满时，使用分块上传将缓冲区上传到S3。然后，一旦ffmpeg完成转码输入文件，您可以获取当前内存缓冲区中剩余的内容，将最后一个缓冲区上传到S3，然后简单地调用CompleteMultipartUpload。这将取出所有的25MB部分并将它们合并成一个文件。

这就是全部。通过这种策略，可以在S3中读取文件，对其进行转码，并将其实时存储在S3中，而不存储任何东西。因此，在使用极少量的内存和几乎没有磁盘空间的AWS Lambda中转码大文件是可能的。

这已经成功实现了。我将尝试看看是否可以共享这段代码。

警告：正如评论中所提到的，如果我们流式传输ffmpeg的输出或者让ffmpeg自己写入本地文件，得到的结果并不完全相同。当写入到本地文件时，ffmpeg有能力在完成转码后回到文件开头。然后它可以使用一些转码结果更新文件元数据。我不知道没有更新元数据的影响是什么。

谢谢你，这非常有帮助。你提到可能会分享这段代码，这是你能做到的吗？ - Sonic Motion
也在等待这个！ - Agey
这将取决于我的老板。一旦确定，我会通知您。请注意，这可能要等几个月的时间。 - mabead
你能分享这个例子吗？ - Oddiesea
自从创建了这篇文章以来，AWS增加了将EFS驱动器挂载到lambda的可能性。因此，临时空间中512 MB的限制已经消失。因此，我们不再走FFMPEG输入/输出流的复杂路径，而是采用更简单的方法：1）lambda下载S3输入文件并将其存储在与EFS链接的路径中；2）lambda在tmp EFS文件夹中运行FFMPEG，并将结果也存储在tmp EFS文件夹中；3）lambda上传在上一步生成的输出文件到S3。虽然这不是最有效的方法，但非常可靠且易于实现/理解。 - mabead
@mabead，您能否分享一下如何将FFmpeg的输出存储到EFS存储中？根据您的步骤，我已经将EFS与Lambda挂载了。 - Parth Shah
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接