动态导入 Python 模块和 Numba

Question

我有一个模块和一个模块加载器。该模块使用了numba。

当我使用importlib进行动态加载后调用该模块的函数时，会出现多个错误。 当我使用static导入并调用该函数时，一切正常。

当我不使用numba时，静态和动态导入都能正常工作。

以下是一个简单的示例来重现这个问题：

模块 mymod.py：

from numba import jit

@jit(nopython=True)
def process():
    a = 5
    res = a + 5
    return res

可靠的模块加载器：runb.py（静态导入）

import mymod

def main():
    res = mymod.process()
    print(res)

if __name__=="__main__":
    main()

模块加载器出现问题：run.py（动态导入）

from pathlib import Path
import importlib.util

def main():
    module_path = "./mymod.py"
    spec = importlib.util.spec_from_file_location(Path(module_path).name, module_path) 
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)

    res = module.process()

    print(res)

if __name__=="__main__":
    main()

软件环境：

• Ubuntu 19.04
• Python 3.7.5
• pip 18.1
• Numba 0.51.2
• llvmlite 0.34.0

文件夹结构：

• dumb/venv
• dumb/mymod.py
• dumb/run.py
• dumb/runb.py

我正在解决这个问题，但在互联网上没有找到任何有用的信息。

以下是错误信息：

Traceback (most recent call last):
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/environment.py", line 21, in from_fndesc
    return cls._memo[fndesc.env_name]
  File "/usr/lib/python3.7/weakref.py", line 137, in __getitem__
    o = self.data[key]()
KeyError: '_ZN08NumbaEnv5mymod2py11process$241E'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/funcdesc.py", line 93, in lookup_module
    return sys.modules[self.modname]
KeyError: 'mymod.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 16, in <module>
    main()
  File "run.py", line 10, in main
    res = module.process()
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 434, in _compile_for_args
    raise e
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 367, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 819, in compile
    cres = self._compiler.compile(args, return_type)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 78, in compile
    status, retval = self._compile_cached(args, return_type)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 92, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/dispatcher.py", line 110, in _compile_core
    pipeline_class=self.pipeline_class)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler.py", line 627, in compile_extra
    return pipeline.compile_extra(func)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler.py", line 363, in compile_extra
    return self._compile_bytecode()
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler.py", line 425, in _compile_bytecode
    return self._compile_core()
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler.py", line 405, in _compile_core
    raise e
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler.py", line 396, in _compile_core
    pm.run(self.state)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 341, in run
    raise patched_exception
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 332, in run
    self._runPass(idx, pass_inst, state)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 291, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/compiler_machinery.py", line 264, in check
    mangled = func(compiler_state)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/typed_passes.py", line 442, in run_pass
    NativeLowering().run_pass(state)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/typed_passes.py", line 369, in run_pass
    metadata=metadata)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/lowering.py", line 37, in __init__
    self.env = Environment.from_fndesc(self.fndesc)
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/environment.py", line 23, in from_fndesc
    inst = cls(fndesc.lookup_globals())
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/funcdesc.py", line 81, in lookup_globals
    return self.global_dict or self.lookup_module().__dict__
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/funcdesc.py", line 96, in lookup_module
    f"can't compile {self.qualname}: "
ModuleNotFoundError: can't compile process: import of module mymod.py failed

- SamoloH

---

我认为spec_from_file_location(name, path)的名称应该是mymod，而不是mymod.py。 - Niklas Mertsch

我已经将该行替换为：spec = importlib.util.spec_from_file_location("mymod", module_path)但问题仍然存在 :/ - SamoloH

你仍然收到 KeyError: 'mymod.py' 错误吗？ - Niklas Mertsch

你解决了吗？ - João P C Bertoldo

Answer 1

我找不到防止第一个异常发生的方法。但是第二个异常可以被处理。

Traceback (most recent call last):
  File "mypath/dumb/venv/lib/python3.7/site-packages/numba/core/funcdesc.py", line 93, in lookup_module
    return sys.modules[self.modname]
KeyError: 'mymod.py'

很容易修复。

将模块添加到提到的sys.modules字典中，以便在exec_module(module)调用期间通过其文件名访问它。

from pathlib import Path
import importlib.util
import sys

def main():
    module_path = "./mymod.py"
    spec = importlib.util.spec_from_file_location(Path(module_path).name, module_path) 
    module = importlib.util.module_from_spec(spec)
    sys.modules[spec.name] = module
    spec.loader.exec_module(module)

    res = module.process()

    print(res)

if __name__=="__main__":
    main()

输出

10

- Michael Szczesny