有一个使用AWS Glue的无服务器解决方案!(我差点因为找不到答案而放弃了)
这个解决方案分为两个部分:
- 一个Lambda函数,由S3触发上传ZIP文件并创建GlueJobRun-将S3对象键传递给Glue。
- 一个Glue Job,可以在内存中解压缩文件并上传回S3。
请查看下面的代码,该代码可以解压缩ZIP文件并将其内容放回到同一个存储桶中(可配置)。
如果有帮助,请点赞 :)
Lambda脚本(python3),调用名为YourGlueJob的Glue Job
import boto3
import urllib.parse
glue = boto3.client('glue')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
print(key)
try:
newJobRun = glue.start_job_run(
JobName = 'YourGlueJob',
Arguments = {
'--bucket':bucket,
'--key':key,
}
)
print("Successfully created unzip job")
return key
except Exception as e:
print(e)
print('Error starting unzip job for' + key)
raise e
AWS Glue作业脚本以解压缩文件
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME','bucket','key'],)
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
import boto3
import zipfile
import io
from contextlib import closing
s3 = boto3.client('s3')
s3r = boto3.resource('s3')
bucket = args["bucket"]
key = args["key"]
obj = s3r.Object(
bucket_name=bucket,
key=key
)
buffer = io.BytesIO(obj.get()["Body"].read())
z = zipfile.ZipFile(buffer)
list = z.namelist()
for filerr in list:
print(filerr)
y=z.open(filerr)
arcname = key + filerr
x = io.BytesIO(y.read())
s3.upload_fileobj(x, bucket, arcname)
y.close()
print(list)
job.commit()