一个文件包含10000行,每行有一个条目。我需要分批次(小块)地处理该文件。
file = open("data.txt", "r")
data = file.readlines()
file.close()
total_count = len(data) # equals to ~10000 or less
max_batch = 50 # loop through 'data' with 50 entries at max in each loop.
for i in range(total_count):
batch = data[i:i+50] # first 50 entries
result = process_data(batch) # some time consuming processing on 50 entries
if result == True:
# add to DB that 50 entries are processed successfully!
else:
return 0 # quit the operation
# later start again from the point it failed.
# say 51st or 2560th or 9950th entry
如何在下一次循环中选择第51到100个条目以此类推?
如果操作不成功并在中途停止,则需要根据数据库记录仅从失败的批次开始循环。
我无法编写正确的逻辑。我应该保留两个列表吗?还是其他什么方法?
range(0, total_count, 50)
- Shane