最近,我需要尽快用Python实现一个程序,将存储在Amazon EC2中的文件上传到S3。这些文件的大小为30KB。
我尝试了一些解决方案,包括使用多线程、多进程和协程。以下是我在Amazon EC2上进行性能测试的结果。
3600(文件数)* 30K(文件大小)~~ 105M(总共) --->
**5.5s [ 4 process + 100 coroutine ]**
10s [ 200 coroutine ]
14s [ 10 threads ]
以下是代码:
用于多线程
def mput(i, client, files):
for f in files:
if hash(f) % NTHREAD == i:
put(client, os.path.join(DATA_DIR, f))
def test_multithreading():
client = connect_to_s3_sevice()
files = os.listdir(DATA_DIR)
ths = [threading.Thread(target=mput, args=(i, client, files)) for i in range(NTHREAD)]
for th in ths:
th.daemon = True
th.start()
for th in ths:
th.join()
关于协程
client = connect_to_s3_sevice()
pool = eventlet.GreenPool(int(sys.argv[2]))
xput = functools.partial(put, client)
files = os.listdir(DATA_DIR)
for f in files:
pool.spawn_n(xput, os.path.join(DATA_DIR, f))
pool.waitall()
针对多进程和协程的处理
def pproc(i):
client = connect_to_s3_sevice()
files = os.listdir(DATA_DIR)
pool = eventlet.GreenPool(100)
xput = functools.partial(put, client)
for f in files:
if hash(f) % NPROCESS == i:
pool.spawn_n(xput, os.path.join(DATA_DIR, f))
pool.waitall()
def test_multiproc():
procs = [multiprocessing.Process(target=pproc, args=(i, )) for i in range(NPROCESS)]
for p in procs:
p.daemon = True
p.start()
for p in procs:
p.join()
机器的配置为Ubuntu 14.04, 2个CPU (2.50GHz), 4G内存
最高速度约为19Mb/s (105 / 5.5)。 总体而言,速度太慢了。有什么方法可以加速吗? 是否可以使用无栈python使其更快?