我正在尝试通过并行化textacy来加速处理大量文本列表的过程。当我使用multiprocessing中的Pool时,生成的textacy语料库为空。我不确定问题出在我使用textacy还是多进程编程模型上?以下是说明我的问题的示例:
import spacy
import textacy
from multiprocessing import Pool
texts_dict={
"key1":"First text 1."
,"key2":"Second text 2."
,"key3":"Third text 3."
,"key4":"Fourth text 4."
}
model=spacy.load('en_core_web_lg')
# this works
corpus = textacy.corpus.Corpus(lang=model)
corpus.add(tuple([value, {'key':key}],) for key,value in texts_dict.items())
print(corpus) # prints Corpus(4 docs, 8 tokens)
print([doc for doc in corpus])
# now the same thing with a worker pool returns empty corpus
corpus2 = textacy.corpus.Corpus(lang=model)
pool = Pool(processes=2)
pool.map( corpus2.add, (tuple([value, {'key':key}],) for key,value in texts_dict.items()) )
print(corpus2) # prints Corpus(0 docs, 0 tokens)
print([doc for doc in corpus2])
# to make sure we get the right data into corpus.add
pool.map( print, (tuple([value, {'key':key}],) for key,value in texts_dict.items()) )
Textacy基于spaCy。SpaCy不支持多线程,但据说可以在多个进程中运行。
https://github.com/explosion/spaCy/issues/2075根据@constt的建议https://dev59.com/Prbna4cB1Zd3GeqPg8oK#58317741,将结果收集到语料库中适用于数据集的大小最多为n_docs = 10273 n_sentences = 302510 n_tokens = 2053129。
对于更大的数据集(16000个文档3MM标记),我会得到以下错误:
result_corpus=corpus.get()
File "<string>", line 2, in get
File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/managers.py", line 283, in serve_client
send(msg)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
我会调查,但如果您有直接的解决方案 - 非常感谢!