我想从API下载一个文件,然后直接上传(流式传输)到S3。
我的本地下载代码(完美运行)如下:
import requests
import datetime
import os
headers = {'Authorization': 'apikey THISISHIDDEN'}
baseURL = 'https://api.test.au/busschedule/'
target_path = datetime.datetime.now().strftime('%Y-%m-%d schedule') + '.zip'
response = requests.get(baseURL, stream=True, headers=headers)
handle = open(target_path, "wb")
for chunk in response.iter_content(chunk_size=512):
if chunk: # filter out keep-alive new chunks
handle.write(chunk)
handle.close()
我尝试将文件下载并传输至S3(但未成功):
# import requests
import datetime
import os
import boto3
import botocore.vendored.requests.packages.urllib3 as urllib3
# Get environment variables from serverless.yml
bucket = "bucket"
s3folder = "schedules"
# Set standard script parameters
headers = {'Authorization': 'apikey THISISHIDDEN'}
baseURL = 'https://api.test.au/busschedule/'
def run(event, context):
s3 = boto3.client('s3')
datetimestamp = datetime.datetime.today().strftime('%Y%m%dT%H%M%S')
filename = datetimestamp + " bus schedule.zip"
key = s3folder + '/' + filename # your desired s3 path or filename
http = urllib3.PoolManager()
s3.upload_fileobj(http.request('GET', baseURL,
headers=headers, preload_content=False),
bucket, key)
def main():
run({},{})
if __name__ == "__main__":
exit(main())
我收到的CloudWatch返回的错误是:
InsecureRequestWarning: Unverified HTTPS request is being made. Timeout after 300.10s.
编辑:该lambda函数的超时时间为300秒;但这应该足以下载文件(6mb)。本地下载只需要10秒左右。有没有更好的方法来处理这个问题?
os.environ['URL']
。这里的代码格式不正确,有一个无效的字符串文字。顺便说一下,这段代码格式不正确,你有可能漏掉了什么? - Ronyis