在Windows上使用MKL编译NumPy时出现DLL加载失败的问题

8

我一直在尝试使用MSVC编译器和Intel MKL在Windows 10上从源代码编译NumPy。

我的系统是Windows 10.0.18363,使用Microsoft Visual Studio 2019(16.8.4)和Intel MKL 2017.8.275。

我使用最新的Python和NumPy进行了最小化的设置,成功地复现了这个问题。

  1. Download latest Python (3.9.1) and latest NumPy (1.20.1) source.

  2. Open a VS command prompt, unpack Python source, build with PCbuild\build.bat

  3. Run mklvars.bat intel64 to get the right environment variables set.

  4. Add the Intel compilers (needed for ifort) to PATH:

    set PATH=C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64;%PATH%
    
  5. Create a virtual env, copy a few files from the Python build and activate the virtual env:

    copy Python\PCbuild\amd64\python39.dll venv\Scripts
    copy Python\PC\pyconfig.h venv\Include
    
  6. Build NumPy from source and install: pip install . -v

  7. Try to import NumPy: python -c "import numpy"

错误信息

Traceback (most recent call last):
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
    from . import multiarray
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
    from . import overrides
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\__init__.py", line 145, in <module>
    from . import core
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "C:\cygwin\home\user\numpy_clean_env\venv\Scripts\python.exe"
  * The NumPy version is: "1.20.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.

其他信息

由于编译返回结果,MKL库会被选中。

FOUND:
        libraries = ['mkl_rt']
        library_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib\\intel64']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib']

我尝试使用Dependencies(Dependency Walker的更新版本)分析DLL解析,但似乎MKL DLL加载正常。有一些DLL显示未正确加载,但据我所知,这是由于检查软件与Windows API集的限制引起的,并非实际问题,因此我认为系统设置正确。
以下是在分析_multiarray_umath.pyd时导致Dependencies中出现“NOT_FOUND”的独特DLL列表:
EMCLIENT.dll (NOT_FOUND) :
HvsiFileTrust.dll (NOT_FOUND) :
UpdateAPI.dll (NOT_FOUND) :
api-ms-win-core-comm-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-psapi-ansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-psapi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-stringansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-versionansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-versionansi-l1-1-1.dll (NOT_FOUND) :
api-ms-win-core-winrt-string-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-wow64-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-wow64-l1-1-1.dll (NOT_FOUND) :
api-ms-win-core-xstate-l2-1-0.dll (NOT_FOUND) :
api-ms-win-core-xstate-l2-1-1.dll (NOT_FOUND) :
api-ms-win-coreui-secruntime-l1-1-0.dll (NOT_FOUND) :
ext-ms-mf-pal-l2-1-0.dll (NOT_FOUND) :
ext-ms-onecore-appmodel-emclient-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-defaultdiscovery-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-orientation-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-shellchromeapi-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-deployment-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-usercontext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-viewscalefactor-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-audiocore-pal-l1-2-0.dll (NOT_FOUND) :
ext-ms-win-com-suspendresiliency-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-core-winrt-remote-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-dwmapi-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-dxcore-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-networking-wlanstorage-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-2.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-3.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-4.dll (NOT_FOUND) :
ext-ms-win-ras-tapi32-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-ro-typeresolution-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-rtcore-minuser-input-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-rtcore-minuser-private-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-security-capauthz-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-security-chambers-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-knownfolderext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-shell32-l1-2-0.dll (NOT_FOUND) :
ext-ms-win-shell-shlwapi-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-tabbedtitlebar-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell32-shellcom-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell32-shellfolders-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ui-viewmanagement-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-uiacore-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-wer-xbox-l1-1-0.dll (NOT_FOUND) :
ext-ms-win32-subsystem-query-l1-1-0.dll (NOT_FOUND) :
ext-ms-windowscore-deviceinfo-l1-1-0.dll (NOT_FOUND) :

我也尝试按照这篇指南的步骤操作,并且可以确认os.path.exists('_multiarray_umath.pyd')返回True,但是ctypes.WinDLL('_multiarray_umath.pyd')无法加载 DLL 文件(或其依赖项之一)。

可工作的示例

与上述步骤相同,但不需要执行mklvars.bat,因此没有链接到英特尔 MKL。

以前,这个设置曾经在 Python 3.6、VS2015 和类似版本的英特尔 MKL 上运行过。我可能漏掉了什么,欢迎任何建议。


BLAS库的问题是什么,这是“常规”的NumPy使用的吗?所有(递归地)所需的*.dll文件都存在吗(并且构建为正确的体系结构)?我注意到一些Cygwin*路径,你是否混合了工具链?在之前可行的设置中,版本是否相同? - CristiFati
@CristiFati 在我的情况下,Intel MKL 是必需的,以提高速度和稳定性。64位 DLL 可在预期路径上获得,并且它们在依赖项中显示为正确链接(mkl_rt 已列出并且路径存在)。Cygwin 路径只是因为“非最小”设置在 Cygwin 中,但在这种情况下,我只是加载了 VS2019 x64 命令提示符。可行的设置具有某些工具的不同版本(Python 3.6.2 > 3.9.1,VS2015 > VS2019);新的设置以相同方式失败,其中包括 NumPy 1.16、1.17 和 1.20,以及 Intel MKL 2017.2、2017.8 和 2021。 - cubanpit
你可能想要查看这个链接:https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy。 - CristiFati
谢谢提供链接,我可以尝试使用预编译版本,但是我使用MKL来构建其他组件,它们都必须一起发布。我认为两次发布MKL会占用太多空间,因此我需要检查是否有其他可行的方法。 - cubanpit
我还注意到该链接中有一个非常有限的(Python版本,NumPy版本)组合集,这可能会给其余的构建系统带来进一步的问题,从源代码编译可以提供更多的灵活性。 - cubanpit
1个回答

2

下面是一个在Numpy邮件列表 (链接) 上得到的建议解决了问题。

Python >= 3.8将不再使用PATH来解析扩展模块的依赖项。 在导入numpy之前,在所有脚本中使用os.add_dll_directory(mkl_bin_path) https://docs.python.org/3/library/os.html#os.add_dll_directory 或添加该调用到numpy包目录中的_distributor_init.py文件中。


这就是全部了吗?我想知道为什么我错过了它:https://dev59.com/brfna4cB1Zd3GeqPoTbL。 - CristiFati

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接