Building binary wheel for Python 2.7/3.4 on Windows WITH PYBIND11
Background
On C, ABI is preserved. This means that you can link gcc/clang .o with msvc .o files.
(Note that this does not apply to C++.)
It means that (also because Python itself does not use C++) you can build binary wheel for Python 2.7/3.4 on Windows with Pybind11.
Building module with msvc-unsupported features on Python 3.5+
Let's start from Python 3.5+ but you need msvc-unsupported features.
In my example python-slz, https://github.com/wtarreau/libslz/blob/v1.2.0/src/slz.c requires inline assembler. So it has to be compiled using gcc/clang.
Declare ext_module
At very first, you have to declare ext_module as usual Extension task. For this phase, there are a lot of documentation floating on the Internet.
ext_modules = [
Pybind11Extension(
name="slz",
sources=["src/pyslz.cpp"],
extra_objects=[],
extra_compile_args=['-O2'],
extra_link_args=['-s'],
),
]
Declare build_ext extending class
You then declare a build_ext extending class (build_ext_hook) and pass cmdclass={"build_ext": build_ext_hook} to setup().
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
build_ext.build_extension(self, ext)
Add platform condition
Now let's implement the build_extension (Note that src/pyslz.cpp is already set in ext.sources).
We first introduce Windows conditional, which can be achieved by platform.system().
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
pass
else:
ext.sources.append('src/libslz/src/slz.c')
build_ext.build_extension(self, ext)
Add gcc subprocess
As we know src/libslz/src/slz.c is not compilable by MSVC, we need to compile it on gcc and pass .o to extra_objects.
Also, gcc adds ___chkstk_ms function dependency. It can be resolved by assembling https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/x86_64/chkstk.S 1.
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
# -DPRECOMPUTE_TABLES=1 should be added because otherwise __attribute__((constructor)) code is inserted
subprocess.check_call(['gcc', '-c', '-DPRECOMPUTE_TABLES=1', '-o', 'slz.o', '-O2', 'src/libslz/src/slz.c'])
subprocess.check_call(['gcc', '-c', '-o', 'chkstk.o', 'src/chkstk.S'])
ext.extra_objects.extend(['slz.o', 'chkstk.o'])
else:
ext.sources.append('src/libslz/src/slz.c')
build_ext.build_extension(self, ext)
Determine 32bit or 64bit
Finally we want to compile both 32bit and 64bit, which can be checked by sys.maxsize:
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
if sys.maxsize < 1<<32:
msiz = '-m32'
else:
msiz = '-m64'
subprocess.check_call(['gcc', msiz, '-c', '-DPRECOMPUTE_TABLES=1', '-o', 'slz.o', '-O2', 'src/libslz/src/slz.c'])
subprocess.check_call(['gcc', msiz, '-c', '-o', 'chkstk.o', 'src/chkstk.S'])
ext.extra_objects.extend(['slz.o', 'chkstk.o'])
else:
ext.sources.append('src/libslz/src/slz.c')
build_ext.build_extension(self, ext)
Now this module can be compiled on Windows although some part is incompatible with MSVC.
(This state is the same as https://github.com/cielavenir/python-slz/tree/48c7c420e8620db27e141d6ca91a076035e8fdf5)
Building module on Python 2.7/3.4
According to https://wiki.python.org/moin/WindowsCompilers, MSVC 2015+ is used only on Python 3.5+.
So for Python 2.7/3.4 we need to go a little further because src/pyslz.cpp needs to be compiled by MSVC 2015+ but it is not used by setuptools. Actually we compile it using clang.
Switch to clang
For some compatibility, you need to use clang, not gcc (see appendix below for detail).
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
if sys.maxsize < 1<<32:
msiz = '-m32'
else:
msiz = '-m64'
subprocess.check_call(['clang', msiz, '-c', '-DPRECOMPUTE_TABLES=1', '-o', 'slz.o', '-O2', 'src/libslz/src/slz.c'])
subprocess.check_call(['clang', msiz, '-c', '-o', 'chkstk.o', 'src/chkstk.S'])
ext.extra_objects.extend(['slz.o', 'chkstk.o'])
else:
ext.sources.append('src/libslz/src/slz.c')
build_ext.build_extension(self, ext)
Add python binding compilation
Now you need to move sources=["src/pyslz.cpp"] logic inside build_ext_hook.
Also there are a few points:
- Need to pass
-DHAVE_UINTPTR_T=1as it is not written in pyconfig.h. - Need to add include paths:
sysconfig.get_paths()['include'] sysconfig.get_paths()['platinclude'] pybind11.get_include(), which is mentioned in https://github.com/pybind/pybind11/blob/v2.9.1/pybind11/__main__.py#L11 . - 64bit build requires
-DMS_WIN64=1cf https://stackoverflow.com/a/9673051 - Need to
#define _hypot hypotin pybind11 library cf https://stackoverflow.com/a/65694764
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
if sys.maxsize < 1<<32:
msiz = '-m32'
plat = 'win32'
win64flags = []
else:
msiz = '-m64'
plat = 'win-amd64'
win64flags = ['-DMS_WIN64=1']
subprocess.check_call(['clang', msiz, '-c', '-DPRECOMPUTE_TABLES=1', '-o', 'slz.o', '-O2', 'src/libslz/src/slz.c'])
subprocess.check_call(['clang', msiz, '-c', '-o', 'chkstk.o', 'src/chkstk.S'])
if sys.version_info < (3,5):
import sysconfig
import pybind11
subprocess.check_call(['clang++', msiz, '-c', '-o', 'pyslz.o', '-O2',
'-DHAVE_UINTPTR_T=1',
'-I', sysconfig.get_paths()['include'],
'-I', sysconfig.get_paths()['platinclude'],
'-I', pybind11.get_include(),
'src/pyslz.cpp']+win64flags)
ext.extra_objects.append('pyslz.o')
else:
ext.sources.append('src/pyslz.cpp')
ext.extra_objects.extend(['slz.o', 'chkstk.o'])
else:
ext.sources.extend(['src/pyslz.cpp', 'src/libslz/src/slz.c'])
build_ext.build_extension(self, ext)
Linking
You noticed that you have to link pyslz.o using clang as C++ runtime and compiler_rt (runtime like libgcc) must be linked and specifying them is difficult on MSVC.
- the path to put pyd can be computed as
'build/lib.%s-%d.%d/%s.pyd'%(plat, sys.hexversion // 16777216, sys.hexversion // 65536 % 256, ext.name.replace('.', '/')). -
pythonNN.libimport library path can be computed as below:
Also remember that after linking you need to return without delegating to setuptools's build_ext.
libname = 'python%d%d.lib'%(sys.hexversion // 16777216, sys.hexversion // 65536 % 256)
# https://stackoverflow.com/a/48360354/2641271
d = Distribution()
b = d.get_command_class('build_ext')(d)
b.finalize_options()
libpath = next(join(dir, libname) for dir in b.library_dirs if isfile(join(dir, libname)))
class build_ext_hook(build_ext, object):
def build_extension(self, ext):
if platform.system() == 'Windows':
if sys.maxsize < 1<<32:
msiz = '-m32'
plat = 'win32'
win64flags = []
else:
msiz = '-m64'
plat = 'win-amd64'
win64flags = ['-DMS_WIN64=1']
subprocess.check_call(['clang', msiz, '-c', '-DPRECOMPUTE_TABLES=1', '-o', 'slz.o', '-O2', 'src/libslz/src/slz.c'])
subprocess.check_call(['clang', msiz, '-c', '-o', 'chkstk.o', 'src/chkstk.S'])
if sys.version_info < (3,5):
import sysconfig
import pybind11
subprocess.check_call(['clang++', msiz, '-c', '-o', 'pyslz.o', '-O2',
'-DHAVE_UINTPTR_T=1',
'-I', sysconfig.get_paths()['include'],
'-I', sysconfig.get_paths()['platinclude'],
'-I', pybind11.get_include(),
'src/pyslz.cpp']+win64flags)
ext.extra_objects.append('pyslz.o')
if True:
ext.extra_objects.extend(['slz.o'])
pydpath = 'build/lib.%s-%d.%d/%s.pyd'%(plat, sys.hexversion // 16777216, sys.hexversion // 65536 % 256, ext.name.replace('.', '/'))
subprocess.check_call(['mkdir', '-p', dirname(pydpath)])
libname = 'python%d%d.lib'%(sys.hexversion // 16777216, sys.hexversion // 65536 % 256)
# https://stackoverflow.com/a/48360354/2641271
d = Distribution()
b = d.get_command_class('build_ext')(d)
b.finalize_options()
libpath = next(join(dir, libname) for dir in b.library_dirs if isfile(join(dir, libname)))
subprocess.check_call([
'clang++', msiz, '-shared', '-o', pydpath,
]+ext.extra_objects+[libpath])
return
else:
ext.sources.append('src/pyslz.cpp')
ext.extra_objects.extend(['slz.o', 'chkstk.o'])
else:
ext.sources.extend(['src/pyslz.cpp', 'src/libslz/src/slz.c'])
build_ext.build_extension(self, ext)
Now you got Python 2.7 wheel for Windows even though Python 2.7's MSVC version is incompatible with Pybind11 
Appendix
Can we keep gcc?
Technically yes.
- For 32bit, you need to download Install mingw-w64-i686-gcc using this yaml. Unlike clang,
gcc -m32will work only for compilation, not linking.
- name: "Install mingw-w64-i686-gcc"
if: ${{ matrix.os == 'windows-latest' && matrix.architecture == 'x86' }}
uses: msys2/setup-msys2@v2
with:
msystem: MINGW32
path-type: inherit
install: mingw-w64-i686-gcc
- You need to add link option:
'-static-libstdc++', '-static-libgcc', '-Wl,-Bstatic,--whole-archive', '-lwinpthread', '-Wl,--no-whole-archive,-Bdynamic'
However this will result in very huge binary.
-
For __divdi3 etc, check https://github.com/glitchub/arith64/blob/master/arith64.c as well. ↩