Monday, October 20, 2014

NumPy in virtualenv on Windows-x64 with wheels

[UPDATE 2014-11-17] I am about to eat my own words, because Carl Kleffner has provided openBLAS dynamic libraries and headers that you can use to build NumPy>=1.91. The DLL is compiled using his static mingw-w64 GCC toolchain. From his site download 2014-08-28_OpenBLAS-0.2.12dev_PR440.7z and extract it into numpy/core folder of the NumPy-1.9.1 distribution. Then follow the directions on the OpenBLAS users Google Group. A wheel is my dropbox. You can check it's config in a Python interpreter by importing NumPy and calling numpy.__config__.show(). If you have nose, you can also run its tests by calling numpy.test(). Thanks Carl and Xianyi, this is an important milestone in making powerful numerical software free and open-sourced.

I want you to repeat after me.
"You will never be able to build NumPy on Windows x64 without the Intel Compiler Suite."
OK, was that so hard? You may have briefly entertained pipe dreams of building NumPy with OpenBLAS binaries, but then reality sank in after NumPy fails to load with this sad traceback.
Traceback (most recent call last):
  File "", line 1, in 
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\__init__.py", line 170, in 
    from . import add_newdocs
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\add_newdocs.py", line 13, in 
    from numpy.lib import add_newdoc
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\lib\__init__.py", line 18, in 
    from .polynomial import *
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\lib\polynomial.py", line 19, in 
    from numpy.linalg import eigvals, lstsq, inv
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\linalg\__init__.py", line 51, in 
    from .linalg import *
  File "c:\.virtualenvs\openblas\lib\site-packages\numpy\linalg\linalg.py", line 29, in 
    from numpy.linalg import lapack_lite, _umath_linalg
ImportError: DLL load failed: The specified procedure could not be found
Everyone said, "use dependency walker," which you did, and it told you that ABI incompatibilities meant that NumPy had no idea how to call your libopenblas.dll functions, even though they were exported and supported by libgfortran-3.dll and libgcc_s_seh-1.dll and the right versions of msvcr90.dll which you found buried in winsxs.

You got briefly excited when you saw that Carl Kleffner is actually working on a solution to this by introducing a static GCC toolchain to compile NumPy with OpenBLAS (*), but then you had a sudden insight. Hasn't Christoph Gohlke already compiled these Python libraries using Intel's Math Kernel Library (MKL)? Why yes he has! And can't you convert a bdist_wininst to a wheel? Of course you can!
$ wheel convert numpy‑MKL‑1.8.2.win‑amd64‑py2.7.exe
Now you can just pip install the wheels in your virtualenvs.
(venv)
$ pip install -U numpy-1.8.2-cp27-none-win_amd64.whl
Get your wheels here: Thanks Christoph (and Carl and Xianyi)!

(* See UPDATE at the top of this post for more info on Carl Kleffners OpenBLAS version of NumPy.)
Fork me on GitHub