我想使用queryset迭代器来迭代一个大数据集。Django提供了iterator()
来实现这一点,但是每次迭代都会访问数据库。我找到了以下代码来分块迭代 -
def queryset_iterator(queryset, chunksize=1000):
'''''
Iterate over a Django Queryset ordered by the primary key
This method loads a maximum of chunksize (default: 1000) rows in it's
memory at the same time while django normally would load all rows in it's
memory. Using the iterator() method only causes it to not preload all the
classes.
Note that the implementation of the iterator
does not support ordered query sets.
'''
pk = 0
last_pk = queryset.order_by('-pk').values_list('pk', flat=True).first()
if last_pk is not None:
queryset = queryset.order_by('pk')
while pk < last_pk:
for row in queryset.filter(pk__gt=pk)[:chunksize]:
pk = row.pk
yield row
gc.collect()
这适用于无序的查询集。是否有解决方案/变通方法可以在排序的查询集上执行此操作?