我想从AWS下载特定文件,已经有了文件URL列表。使用CLI只能使用--recursive命令下载存储桶中的所有文件,但我只想下载我的列表中的文件。您有任何解决方法吗?
aws s3 cp s3://BUCKET/ folder --exclude "*" --include "2018-02-06*" --recursive
既然您已经有了一个包含S3 URL的文件(例如file.list
),如下所示-
s3://bucket/file1
s3://bucket/file2
while read -r line;do aws s3 cp "$line" .;done < test.list
import boto3
import botocore
import os
# Initialize the S3 client
aws_access_key_id = 'AWSACESSKEY'
aws_secret_access_key = 'AWSSECRETACESSKEY'
s3 = boto3.client('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
# List of image object keys you want to download
image_keys_to_download = [
"sta10/maths/chapter10.pdf",
"sta11/science/biology/chapter08.pdf",
"data/sta10/class11/scientists/pythagoras"
]
# Destination directory where the images will be downloaded
destination_base_directory = "~/s3-download-class"
# Loop through the list of image keys and download each image while preserving folder structure
for image_key in image_keys_to_download:
try:
# Extract the folder structure from the image key
folder_structure = os.path.dirname(image_key)
# Create the destination directory including the folder structure
destination_directory = os.path.join(destination_base_directory, folder_structure)
# Make sure the destination directory exists
os.makedirs(destination_directory, exist_ok=True)
# Get the object from S3 and save it locally while preserving folder structure
local_file_path = os.path.join(destination_directory, os.path.basename(image_key))
s3.download_file('class-10-data', image_key, local_file_path) #Add your S3 Bucket name here.
print(f"Downloaded {image_key}")
except botocore.exceptions.NoCredentialsError:
print("AWS credentials not found. Make sure you have configured your credentials.")
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
print(f"Image {image_key} not found in the S3 bucket.")
else:
print(f"Error downloading {image_key}: {e}")
print("Download process completed.")
--include
参数,每个参数指定一个要下载的文件。但是,我认为这并不一定比逐个下载更快,因为此功能需要 AWS CLI 首先扫描现有文件(以适应通配符功能),而复制特定文件只需要一个 API 调用。 - John Rotenstein