你可以通过读取文本文件开头的一小块行来推断数据的dtypes。一旦有了这些,你可以创建一个
可调整大小的HDF5数据集并迭代地将文本文件中的行块写入其中。
下面是一个生成器,它将文本文件中连续的行块作为numpy数组产生:
import numpy as np
import warnings
def iter_genfromtxt(path, chunksize=100, **kwargs):
"""Yields consecutive chunks of rows from a text file as numpy arrays.
Args:
path: Path to the text file.
chunksize: Maximum number of rows to yield at a time.
**kwargs: Additional keyword arguments are passed to `np.genfromtxt`,
with the exception of `skip_footer` which is unsupported.
Yields:
A sequence of `np.ndarray`s with a maximum row dimension of `chunksize`.
"""
names = kwargs.pop('names', None)
max_rows = kwargs.pop('max_rows', None)
skip_header = kwargs.pop('skip_header', kwargs.pop('skiprows', 0))
if kwargs.pop('skip_footer', None) is not None:
warnings.warn('`skip_footer` will be ignored')
with open(path, 'rb') as f:
chunk = np.genfromtxt(f, max_rows=chunksize, skip_header=skip_header,
names=names, **kwargs)
kwargs.update({'dtype':chunk.dtype})
while len(chunk):
yield chunk[:max_rows]
if max_rows is not None:
max_rows -= len(chunk)
if max_rows <= 0:
raise StopIteration
chunk = np.genfromtxt(f, max_rows=chunksize, **kwargs)
现在假设我们有一个包含以下内容的
.csv
文件:
strings,ints,floats
a,1,0.1256290043
b,2,0.0071402451
c,3,0.2551627907
d,4,0.7958570533
e,5,0.8968247722
f,6,0.7291124437
g,7,0.4196829806
h,8,0.398944394
i,9,0.8718244087
j,10,0.67605461
k,11,0.7105670336
l,12,0.6341504091
m,13,0.1324232855
n,14,0.7062503808
o,15,0.1915132527
p,16,0.4140093777
q,17,0.1458217602
r,18,0.1183596433
s,19,0.0014556247
t,20,0.1649811301
我们可以每次读取5行数据,并将结果数组写入可调整大小的数据集中:
import h5py
gen = iter_genfromtxt('/tmp/test.csv', chunksize=5, delimiter=',', names=True,
dtype=None)
chunk = next(gen)
dtype = chunk.dtype
row_count = chunk.shape[0]
with h5py.File('/tmp/test.h5', 'w') as f:
maxshape = (None,) + chunk.shape[1:]
dset = f.create_dataset('data', shape=chunk.shape, maxshape=maxshape,
chunks=chunk.shape, dtype=chunk.dtype)
dset[:] = chunk
for chunk in gen:
dset.resize(row_count + chunk.shape[0], axis=0)
dset[row_count:] = chunk
row_count += chunk.shape[0]
输出:
with h5py.File('/tmp/test.h5', 'r') as f:
print(repr(f['data'][:]))
对于您的数据集,您可能需要使用更大的块大小。