批处理文件上传到S3

3

我需要开发一个批处理文件,可以将多个小文本文件(4KB-100KB)发送到S3存储桶中的文件夹。在http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash 中灵感来自bash脚本,我开发了下面列出的批处理脚本。该脚本使用以下依赖项:

它运行得很好;但是,偶尔会有一个或多个文件无法上传,我会收到“HTTP/1.1 403 Forbidden / SignatureDoesNotMatch.” 的提示。这似乎是一个签名问题,可能是创建签名时出现的问题。行为没有明显的规律。为了测试,我正在使用8个文本文件进行上传。我可以多次运行,每次都可以上传所有文件,但下一次脚本只会上传其中5、6或7个文件。我不知道为什么会发生这种情况。希望能得到建议。

@echo off
setlocal enableDelayedExpansion

for /f %%f in ('dir /b *.txt') do (

REM Set date/time variables
for /f "tokens=1-4 delims=/ " %%a in ('date /t') do set dow=%%a&&set day=%%c&& set month=%%b&&set year=%%d
for /f "tokens=1-4 delims=:,. " %%h in ('echo %time%') do set hour=%%h&set min=%%i&set sec1=%%j&set sec2=%%k

REM Obtain three letter month value for DateValue HTTP Header
IF !month! == 01 set mname=Jan
IF !month! == 02 set mname=Feb
IF !month! == 03 set mname=Mar
IF !month! == 04 set mname=Apr
IF !month! == 05 set mname=May
IF !month! == 06 set mname=Jun
IF !month! == 07 set mname=Jul
IF !month! == 08 set mname=Aug
IF !month! == 09 set mname=Sep
IF !month! == 10 set mname=Oct
IF !month! == 11 set mname=Nov
IF !month! == 12 set mname=Dec

REM Set time as a 4 digit value if leading zero does not exist
for /f "tokens=1" %%u in ('echo %time%') do set t=%%u
IF "!hour:~1,1!"==":" set t=0!hour!

REM Obtain the ActiveBias value and convert to decimal 
for /f "tokens=3" %%a in ('reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v ActiveTimeBias ^| grep -i "ActiveTimeBias"') do set /a abias=%%a

REM Set the + or - sign variable to reflect the timezone offset
IF "!abias:~0,1!"=="-" (set si=+) ELSE (set si=-)
for /f "tokens=1 delims=-" %%t in ('echo !abias!') do set tzc=%%t

REM Calculate to obtain floating points (decimal values)
set /a tzd=100*!tzc!/60

REM Calculate the active bias to obtain the hour
set /a tze=!tzc!/60

REM Set the minutes based on the result of the floating point calculation
IF "!tzd!"=="0" (set en=00 && set si=)
IF "!tzd:~1!"=="00" (set en=00) ELSE IF "!tzd:~2!"=="00" (set en=00 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="50" (set en=30) ELSE IF "!tzd:~2!"=="50" (set en=30 && set tz=!tzd:~0,2!) 
IF "!tzd:~1!"=="75" (set en=45) ELSE IF "!tzd:~2!"=="75" (set en=45 && set tz=!tzd:~0,2!) 

REM Adding a 0 to the beginning of a single digit hour value
IF !tze! LSS 10 (set tz=0!tze!) 

REM Set the date/timestamp to meet required format
set dateValue=!dow!, !day! !mname! !year! !hour!:!min!:!sec1! !si!!tz!!en!

REM Preparing the HTTP header field
set file=%%f
set bucket=yourbucketname
set resource=/!bucket!/upload/!file!
set contentType=text/plain

REM You MUST have two returns after set NL=^
setlocal enableDelayedExpansion
set NL=^


set stringToSign=PUT!NL!!NL!!contentType!!NL!!dateValue!!NL!!resource!
<nul set /p ".=!stringToSign!" > put.tmp

set S3KEY="Your S3 Key Here"
set S3SECRET="Your S3 Secrect Key Here"

for /f "tokens=*" %%a in ('type put.tmp ^| openssl sha1 -hmac !S3SECRET! -binary ^| b64 -e') do set signature=%%a

REM Sending the data
curl -vvv --no-alpn --http2 -1 -S -X PUT -T "!file!" -H "Host: !bucket!.s3.amazonaws.com" -H "Date: !dateValue!" -H "Content-Type: !contentType!" -H "Authorization: AWS !S3KEY!:!signature!" "https://!bucket!.s3.amazonaws.com/upload/!file!"

REM Reset Variables
set "day="
set "month="
set "year="
set "retry="
set "dow="
set "hour="
set "min="
set "sec1="
set "sec2="
set "dateValue="
set "file="
set "mname="
set "bucket="
set "resource="
set "contentType="
set "stringToSign="
set "S3KEY="
set "S3SECRET="
set "t="

del put.tmp

)
1个回答

5
为什么不使用带有显式S3支持的Windows命令行客户端?无论如何,您的批处理文件都不能没有第三方依赖项。
亚马逊有自己的aws-cli

如果您需要更轻量级的解决方案,请使用WinSCP:

winscp.com /log=s3.log /command ^
    "open s3://%S3KEY%:%S3SECRET%@s3.amazonaws.com/" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

你需要对凭证中的特殊字符进行URL编码。WinSCP GUI可以为你生成一个S3脚本模板,就像上面的一个。
另外,自从WinSCP 5.19以来,你可以使用-username-password开关,它们不需要任何编码。
winscp.com /log=s3.log /command ^
    "open s3://s3.amazonaws.com/ -username=%S3KEY% -password=%S3SECRET%" ^
    "put *.txt /%BUCKET%/" ^
    "exit"

(我是 WinSCP 的作者)


2
它必须在环境中的每个端点上运行。据我回忆,如果我们使用带有所有依赖项的aws cli,部署包将会过大。你的例子正是我过去在使用ssh收集数据时所做的。我不知道WinSCP具有S3功能。这将成为我未来的首选。 - user2917347

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接