我想制作一个程序,由两部分组成:其中一部分是接收数据,另一部分是将其写入文件。我认为如果我可以使用2个线程(可能是2个CPU核心)单独执行任务会更好。我发现了这个链接:https://numba.pydata.org/numba-doc/dev/user/jit.html#compilation-options,它允许您释放GIL。我想知道它是否适合我的目的,并且是否可以采用它来完成这种工作。这是我尝试过的内容:
import threading
import time
import os
import queue
import numba
import numpy as np
condition = threading.Condition()
q_text = queue.Queue()
#@numba.jit(nopython=True, nogil=True)
def consumer():
t = threading.currentThread()
with condition:
while True:
str_test = q_text.get()
with open('hello.txt', 'a') as f:
f.write(str_test)
condition.wait()
def sender():
with condition:
condition.notifyAll()
def add_q(arr="hi\n"):
q_text.put(arr)
sender()
c1 = threading.Thread(name='c1', target=consumer)
c1.start()
add_q()
没有使用 numba
时,它可以正常工作。但是当我将其应用于 consumer
时,它会给我一个错误:
Exception in thread c1:
Traceback (most recent call last):
File "d:\python36-32\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "d:\python36-32\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "d:\python36-32\lib\site-packages\numba\dispatcher.py", line 368, in _compile_for_args
raise e
File "d:\python36-32\lib\site-packages\numba\dispatcher.py", line 325, in _compile_for_args
return self.compile(tuple(argtypes))
File "d:\python36-32\lib\site-packages\numba\dispatcher.py", line 653, in compile
cres = self._compiler.compile(args, return_type)
File "d:\python36-32\lib\site-packages\numba\dispatcher.py", line 83, in compile
pipeline_class=self.pipeline_class)
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 873, in compile_extra
return pipeline.compile_extra(func)
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 367, in compile_extra
return self._compile_bytecode()
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 804, in _compile_bytecode
return self._compile_core()
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 791, in _compile_core
res = pm.run(self.status)
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 253, in run
raise patched_exception
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 245, in run
stage()
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 381, in stage_analyze_bytecode
func_ir = translate_stage(self.func_id, self.bc)
File "d:\python36-32\lib\site-packages\numba\compiler.py", line 937, in translate_stage
return interp.interpret(bytecode)
File "d:\python36-32\lib\site-packages\numba\interpreter.py", line 92, in interpret
self.cfa.run()
File "d:\python36-32\lib\site-packages\numba\controlflow.py", line 515, in run
assert not inst.is_jump, inst
AssertionError: Failed at nopython (analyzing bytecode)
SETUP_WITH(arg=60, lineno=17)
如果我从 consumer
中排除 condition(threading.Condition)
,就没有错误,这可能是因为JIT不解释它的原因?我想知道是否可以采用 numba
来实现这种目的,以及如何解决这个问题(如果可能的话)。
subprocess
而不是thread
吗? - John Anderson