Context Navigation

With the recent release of python 2.6, it is possible again to build python extensions using the freely available MS tools. Python 2.6 is built with VS 2008, for every architecture (x86, amd64 and ia64, not that the later really matters).

This page list the issues to build numpy and scipy with the recent MS toolchains:

Visual Studio

Support for manifests when running tests at the configuration stage has been added to mingw in numpy.distutils

TODO: I would be nice to support cross-compilation from windows 32 bits -> 64 bits

MSVC 9 runtime

The msvc9.0 runtime is not well supported by mingw, it seems. There is a bash script in trunk/tools/win32/misc/msvc9 to build an up to date import library (should work on both 32 and 64 bits mingw).

64 bits

VS 2005 support for cross compilation: not in the express edition ? VS 2008 has support for 64 bits, but not the expression version (the PSDK seems to be the only free way to get 64 bits targetting compilers).

Mingw-w64

mingw: no official 64 bits support. There is the mingw-w64 project, though: it seemed murky from a legal POV (still true ?).

Building the toolchain

To build numpy and scipy, we need C, C++ and Fortran, plus a very recent mingw-w64 runtime, and since we don't support cross compilation, we need native toolchain, This means building the toolchain by ourselves. I have gathered some makefiles to make this relatively painless and reproductible (can build the native toolchain from mac os x and linux - building gcc on windows is a royal PITA and extremely slow):

http://github.com/cournape/cross-mingw-w64/tree/master

For just C support, gcc 4.3.3 + binutils 2.19 + mingw-w64 trunk works (gmp patch is needed for cross compilation, it is included in the git project). For complete support (C++/C/Fortran), only gcc 4.4 works, but it requires the native toolchain to be build with no optimization.

Building a mingw dll callable from MS-built main

Say we have a hello world:

hello.h

#ifndef _HELLO_
#define _HELLO_
void __declspec(dllexport) hello(const char* str);
#endif _HELLO_

hello.c

#include <stdio.h>
#include "hello.h"

void __declspec(dllexport) hello(const char* str)
{
print("%s\n", str);
}

main.c

#include "hello.h"

int main()
{
    hello("hello\n");
    return 0;
}

Build as a dll with mingw:

gcc -shared hello.c -o hello.dll

Now, to link to a dll, an import library (.lib) is needed by MS compiler. Using gcc (actually ld) --out-implib DOES NOT WORK (it will crash any binary accessing the binary); this is not specific to mingw-w64 BTW, import library built by gcc are often buggy on mingw32 too. But we can output the .def file instead:

gcc -shared hello.c -o hello.dll -WL,--output-def,hello.def

And then, generate the import lib from MS tool:

lib /MACHINE:AMD64 /DEF:hello.def

Finally:

cl /MD main.c hello.lib

Which works !

python extension

Now the real deal: building a python extension with mingw-w64, importable by official python (built with MS compiler).

hellomodule.c

#include <Python.h>

static PyObject* print_hello(PyObject*, PyObject *);

static PyMethodDef HelloMethods[] = {
    {"hello",  print_hello, METH_VARARGS,
    "Hello World for python C extension."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static PyObject *
print_hello(PyObject *self, PyObject *args)
{
    printf("Hello from C\n");

    return Py_None;
}

PyMODINIT_FUNC
init_hello(void)
{
    (void) Py_InitModule("_hello", HelloMethods);
}

The first problem is that on windows, python for amd64 (2.6 at least) requires the MS_WIN64 to be defined, and the later is only defined when using MS compiler. So you should define by hand:

gcc -I C:\Python26\include hellomodule.c -c -DMS_WIN64

This define should be used ALL THE TIME. In particular, without it, python will assume that Py_intptr_t is 4 bytes, which obviously won't work very well on a 64 bits arch. Hopefully, link will fail without this define: Py_InitModule4 not found -> python on windows x64 uses Py_InitModule4_64, which is only available if MS_WIN64 is defined. So it should not go undetected.

The second problem is to deal with the import library. There is no libpython26.a in the official python amd64 bits distribution, so we need to build our own. To get the list of export symbols, use this simple script:

import re 
import subprocess
        
START = re.compile(r'\[Ordinal/Name Pointer\] Table')                                     
TABLE = re.compile(r'^\s+\[([\s*\d*)\] (\w*)')                               
        
def dump_table(dllname):
    st = subprocess.Popen(['objdump.exe', '-p', dllname])                                 
    return st.stdout.readlines()
        
def generate_def(dllname, deffile):
    dump = dump_table(dllname)
    for i,x in enumerate(dump):
        if TABLE.match(x):
            break
        
    else:
        raise ValueError("Symbol table not found")                                        
        
    syms = []
    for j in range(i, len(dump)): 
        m = table.match(lines[j]) 
        if m:
            syms.append((int(m.group(1).strip()), m.group(2)))                            
        else:
            break
        
    d = open(deffile, 'w') 
    d.write('LIBRARY        %s\n' % dllname)                                              
    d.write(';CODE          PRELOAD MOVEABLE DISCARDABLE\n')                              
    d.write(';DATA          PRELOAD SINGLE\n')                                            
    d.write('\nEXPORTS\n')                                                                
    for s in syms:
        d.write('@%d    %s\n' % (s[0], s[1]))                                             
    d.close()

dllname = 'python26.dll'
defname = 'python26.def'
generate_def(dllname, defname)

This script dumps the symbol tables of the dll, parses it to get the public table (list of functions exported by the dll), and output the table into a .def file. It may not work if the dll is stripped, I am not sure. Then you can generate the import library for python from the .def file alone:

dlltool -d python26.def -l libpython26.a

You can then link your extension:

gcc hello.o libpython26.a -o hello.pyd

Build numpy with mingw-w64

Problems from mingw:

Missing msvcr90 import lib in ingw-w64: use the script
long double problem: sizeof(long double) == 16 with mingw, 8 with MS. Setting SIZEOF_LONG_DOUBLE to 8 does not work (my guess is that it crashes when calling into mingwrt functions), and letting it to 16 does not work either for some other code.

Problems from python:

support from distutils in numpy-mingw-w64 branch: mostly done, needs cleaning though.

Current state:

build completes, but the produced numpy crashes unit tests (for long double code), only sometimes (if it does not crash, the whole test suite passes).

Building scipy

Scipy can be built with a few code changes in scipy/special + a few minor changes in cephes.

Download in other formats:

Plain Text