当我编写一种备份应用程序时,我在Windows上对文件复制性能进行了评估。
我有几个问题,想知道您的意见。
谢谢!
卢卡斯。
问题:
为什么复制10 GiB文件的性能比复制1 GiB文件慢得多?
为什么shutil.copyfile很慢?
为什么win32file.CopyFileEx很慢? 这可能是因为标志win32file.COPY_FILE_RESTARTABLE吗? 但是,它不接受int 1000作为标志(COPY_FILE_NO_BUFFERING), 这是推荐用于大文件的: http://msdn.microsoft.com/en-us/library/aa363852%28VS.85%29.aspx
使用空ProgressRoutine似乎对于根本不使用ProgressRoutine没有影响。
是否有另一种替代方法可以更好地执行文件复制并获取进度更新?
一个1 GiB和一个10 GiB文件的结果:
test_file_size 1082.1 MiB 10216.7 MiB
METHOD SPEED SPEED
robocopy.exe 111.0 MiB/s 75.4 MiB/s
cmd.exe /c copy 95.5 MiB/s 60.5 MiB/s
shutil.copyfile 51.0 MiB/s 29.4 MiB/s
win32api.CopyFile 104.8 MiB/s 74.2 MiB/s
win32file.CopyFile 108.2 MiB/s 73.4 MiB/s
win32file.CopyFileEx A 14.0 MiB/s 13.8 MiB/s
win32file.CopyFileEx B 14.6 MiB/s 14.9 MiB/s
测试环境:
Python:
ActivePython 2.7.0.2 (ActiveState Software Inc.) based on
Python 2.7 (r27:82500, Aug 23 2010, 17:17:51) [MSC v.1500 64 bit (AMD64)] on win32
source = mounted network drive
source_os = Windows Server 2008 x64
destination = local drive
destination_os = Windows Server 2008 R2 x64
注意:
'robocopy.exe' and 'cmd.exe /c copy' were run using subprocess.call()
win32file.CopyFileEx A (不使用 ProgressRoutine):
此函数用于在Windows系统中复制文件,它可以在不使用 ProgressRoutine 的情况下进行操作。
def Win32_CopyFileEx_NoProgress( ExistingFileName, NewFileName):
win32file.CopyFileEx(
ExistingFileName, # PyUNICODE | File to be copied
NewFileName, # PyUNICODE | Place to which it will be copied
None, # CopyProgressRoutine | A python function that receives progress updates, can be None
Data = None, # object | An arbitrary object to be passed to the callback function
Cancel = False, # boolean | Pass True to cancel a restartable copy that was previously interrupted
CopyFlags = win32file.COPY_FILE_RESTARTABLE, # int | Combination of COPY_FILE_* flags
Transaction = None # PyHANDLE | Handle to a transaction as returned by win32transaction::CreateTransaction
)
win32file.CopyFileEx B(使用空ProgressRoutine):
def Win32_CopyFileEx( ExistingFileName, NewFileName):
win32file.CopyFileEx(
ExistingFileName, # PyUNICODE | File to be copied
NewFileName, # PyUNICODE | Place to which it will be copied
Win32_CopyFileEx_ProgressRoutine, # CopyProgressRoutine | A python function that receives progress updates, can be None
Data = None, # object | An arbitrary object to be passed to the callback function
Cancel = False, # boolean | Pass True to cancel a restartable copy that was previously interrupted
CopyFlags = win32file.COPY_FILE_RESTARTABLE, # int | Combination of COPY_FILE_* flags
Transaction = None # PyHANDLE | Handle to a transaction as returned by win32transaction::CreateTransaction
)
def Win32_CopyFileEx_ProgressRoutine(
TotalFileSize,
TotalBytesTransferred,
StreamSize,
StreamBytesTransferred,
StreamNumber,
CallbackReason, # CALLBACK_CHUNK_FINISHED or CALLBACK_STREAM_SWITCH
SourceFile,
DestinationFile,
Data): # Description
return win32file.PROGRESS_CONTINUE # return of any win32file.PROGRESS_* constant