Building Python x64 on Windows 7 with SDK 7.0

[UPDATE 2015-07-02] Check out Python Bootstrap a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.

I know I was just ranting about the inability to distribute Python27 w/o admin rights, but surprise! This is a piece of cake, thanks to the whoever the amazing Python developers are who maintain the PCbuild and external buildbot tools for Windows. There are also a few sites out there that have similar information on building Python for windows, but honestly everything you need is in PCbuild/readme.txt. Read it, then read it again. Seriously. Also check out the Python docs developer's guide. Hmm, thinking of setting up an AppVeyor buildbot for this.

OK, let's do this:

1. Get Python and install it on your system. You may need a working binary to bootstrap the amd64 build.
2. Get a working version of Microsoft SDK for Windows 7 (7.0). AFAIK Visual Studio 2013 Express Desktop or Community editions include both SDK 7.0 and 7.1, so alternately install that. Make sure that you include the redistributables in when installing the SDK because you will need them to distribute your Python build. See upgrade to vs2013 for fixes to some issues you may encounter especially if you have some other VC components already installed.
3. Get a working svn binary for windows and put it on your path. I use CollabNet SubVersion commandline binaries. The easiest way to get and build all of the external libraries (bzip2, sqlite3, tk/tcl, etc.) is to use the Tools/buildbot batch scripts which call svn.exe.
4. Either download the gzipped source tarball from python.org or clone the tag v2.7.10 of the cpython mercurial repository. Archives of the Hg repo are also conveniently available from the repo viewer.
5. Read the PCbuild/readme.txt again
6. Open the SDK command shell from the Startmenu. Change the target to Release x86, the default is Debug x64, by typing the following:

C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /Release /x86

7. Change to the directory where the source tarball is extracted.
8. Patch the Tools/buildbot/externals batch script exactly as described in the PCbuild readme. I added the release build immediately after the debug fields.

if not exist tcltk\bin\tcl85.dll (
    @rem all and install need to be separate invocations, otherwise nmakehlp is not found on install
    cd tcl-8.5.15.0\win
    nmake -f makefile.vc INSTALLDIR=..\..\tcltk clean all
    nmake -f makefile.vc INSTALLDIR=..\..\tcltk install
    cd ..\..
)

if not exist tcltk\bin\tk85.dll (
    cd tk-8.5.15.0\win
    nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 clean
    nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 all
    nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 install
    cd ..\..
)

if not exist tcltk\lib\tix8.4.3\tix84.dll (
    cd tix-8.4.3.5\win
    nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk clean
    nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all
    nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk install
    cd ..\..
)

9. From the archive root (Python-2.7.9) call the externals batch script. It will copy and build all of the externals from svn.python.org in a folder called externals/.
10. Now cd to PCbuild and call build.bat. Voila, python.exe for x86.
11. Almost there. go back up to the root of the extracted tarball and rename externals to externals-x86.
12. Change the target to Release x64 by typing the following:

C:\Users\myname\downloads\Python-2.7.9>setenv /Release /x64

13. Set an environment variable HOST_PYTHON=C:\Python27\python.exe. You may not need this at all or you might be able to use the 32-bit version just built.
14. Patch the buildbot externals-amd64 batch script just like the x86 script.

if not exist tcltk64\bin\tcl85.dll (
    cd tcl-8.5.15.0\win
    nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all
    nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 install
    cd ..\..
)

if not exist tcltk64\bin\tk85.dll (
    cd tk-8.5.15.0\win
    nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 clean
    nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 all
    nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 install
    cd ..\..
)

if not exist tcltk64\lib\tix8.4.3\tix84.dll (
    cd tix-8.4.3.5\win
    nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 clean
    nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 all
    nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 install
    cd ..\..
)

15. From the archive root (Python-2.7.9) call the externals-amd64 batch script
16. Finally cd back to PCbuild and call build.bat -p x64. Voila, python.exe for x64.
17. Add externals/tcltk to your path and run the tests

C:\Users\myname\downloads\Python-2.7.9\PCbuild>set PATH=C:\Users\myname\downloads\Python-2.7.9\externals;%PATH%
C:\Users\myname\downloads\Python-2.7.9\PCbuild>rt

18. Copy the VC runtime from Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86\Microsoft.VC90.CRT\to PCbuild folder, and Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\amd64\Microsoft.VC90.CRT\to PCbuild\amd64 folder.
19. To distribute create a similar file structure for both archtypes and copy the files into the folders

Python27
|
+-python.exe
|
+-pythonw.exe
|
+-python27.dll
|
+-msvcr90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-msvcp90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-msvcm90.dll <- from VC/redist/MICROSOFT.VC90.CRT
|
+-MICROSOFT.VC90.CRT.manifest <- from VC/redist/MICROSOFT.VC90.CRT
|
+-DLLs <- all externals/tcltk/bin, PCbuild/*.dll & PCbuild/*.pyd files
|         & PC/py.ico, PC/pyc.ico & PC/pycon.ico.
|
+-Lib <- same as source archive except all *.pyc files, all plat-* folders
|        & ensurepip folder
|
+-libs <- all PCbuild/*.lib files
|
+-include <- same as source archive + PC/pyconfig.h
|
+-tcl <- everything in tcltk/lib
|
+-Scripts <- PCbuild/idle.bat
|
+-Doc <- set %PYTHON%=python.exe and build Sphinx docs with Sphinx if you have it

Note: wish85.exe and tclsh85.exe won't work with this Python installed file structure although it will work in the externals bin folder because they look for the tcl85.dll in ../lib. Also note that idle.bat needs some fixin. And also it's very important to know that most executables in Scripts created by installer packages have the path to python.exe hardwired, IE: they are initially not portable, however check out this blog for a few quick tricks to fix them.

You can download my x64 build and x86 build from dropbox. Congratulations!

Python issue 22516: Administrator rights required for local installation

[UPDATE 2015-07-02] Check out Python Bootstrap a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.

Windows Installer won't - even when using "just for me"option

This issue starts simple enough, but then unravels to reveal some very interesting insights. Evidently creating a Python for Windows installer that does not depend on administrator rights is not as easy as it seems. The question comes down to what we really need. Steve Dover from Microsoft breaks it down like this:

1. Python as application
This is when someone wants to use python to write scripts, do analysis, etc. Python is an application on their windows machine just like MATLAB or Excel. This could be installed per-user and possible without administrative rights.
2. Python as library
This version of Python could be used to embed Python in an application. Something similar to what pyexe and other python-freezing packages do. This could be a zip file.
3. Python as-if system
This version would be installed on a system, possibly by system admins, in a custom windows build, similar to the way that Python is integrated into Mac and Linux. It would require admin rights, and could be used by system admins to add custom functionality to the corporate OS.

Let me say 1st off that I think that #3 is absurd. I can't imagine Windows system admins ever using Python in this way. Most of them have never even heard of Python. And there is an entire .NET infrastructure to do exactly this. Why would you use Python instead of C#? Windows will never, ever be like Linux. It is not a POSIX environment, it does not use GNU tools and it does not need Python.

I think that #1 and #2 could serve the same purpose and should really be the only option available. Users who want to use Python on Windows should unzip the Python Windows distro to their System Root (ie: C:\Python27) and use it. No admin rights required. End of story. It should contain the msvcrt90 redistributable and all libraries it needs. There should be no other dependencies.

There is also a 0th option - do not distribute a binary for Windows at all. Let Windows users build from source themselves, or recommend alternate distribution, which Python.org already does on its alternative distribution page.. This is what Ruby does, and perhaps it's the best way to satisfy everyone. But the fact that official Python is available for windows is a very nice thing. Althoght Enthought and ActiveState have been around for a long time, they are private and could go out of business. Nevertheless, this does seem to be the path people are taking.

Anaconda, from Continuum Analytics, a relative newcomer, founded by Peter Wang and Travis Oliphant formerly from Enthought, seems to have become, almost overnight, the most popular source of Python on windows. It's baby sister, Miniconda, is less know but merely installs the conda package/environment manager and Python-2.7 which can be used to install more Python version and packages, whereas Anaconda preinstalls most sci-data packages. The only major concern for me with Anaconda is that it is closed source. Is it the python.org version built out of the box?WinPython on the other hand is open source on github and offers both 32 and 64 bit versions that do not require admin rights to instal. Enthought is also closed source and PortablePython only offers an older out of date 32bit version. There is also PythonXY but for me it seemed buggy.

Not sure what the future of Python on Windows will look like. If you are interested in shaping that future, I suggest you contact one of the Python devs and let them know what your use case is.

Upgrade to Visual Studio 2013 Express

Welcome to the future. It's nice of you to join us. Have you been limping along with very old outdated C/C++/C# toolsets? Still using Visual Studio 2010? 2008? Afraid to uninstall them for fear you will lose the ability to compile your projects. Let's take care of that right now, don't worry about a thing. In about 1-2 hours, you will be happily in the future, enjoying the modern conveniences of Visual Studio 2013. It's very nice here. Won't you join us?

1. Remove Visual Studio 2010 SP1 and run the uninstall utility

Name	Size	Version
Microsoft Visual Studio 2010 Service Pack 1	75.9 MB	10.0.40219
Microsoft Visual C++ 2010 Express - ENU		10.0.40219
Microsoft Visual C# 2010 Express - ENU		10.0.40219
Microsoft Visual Studio 2010 Express Prerequisites x64 - ENU	21.6 MB	10.0.40219

You can follow the instructions I posted in the 3/13/2015 update to Download sites for old, free MS compilers and IDEs but the system restore point has dubious value. In fact it didn't help at all. When I was in trouble I found myself reinstalling the application then removing it again in the correct order. The key here is to uninstall SP1 first, then use Stewart Heath's VS2010 uninstall utility in default mode. If you find yourself in trouble, reinstall VS2010 and VS2010-SP1. If you need installers I have them here in my dropbox.

2. Remove Visual Studio 2008 SP1.

Name	Size	Version
Microsoft Visual C++ 2008 Express Edition with SP1 - ENU

Same here, make sure you have an installer. The web installer in my dropbox still works surprisingly. Any trouble, reinstall and uninstall again.

3. Remove both SDKs for Windows 7 and .NET Frameworks 3.5 and 4.0

Name	Size	Version
Microsoft SDK for Windows 7 (7.1)		7.0.7600.16385.40715
Microsoft SDK for Windows 7 (7.0)		7.1.7600.0.30514

If you have any issues with this, look at the last entry in the log. There's a View Log button when setup fails next to the Finish button. A part of the SDK that the installer is looking for may be missing. Search for the keyword "fail" and "unknown source". If you find an unknown source in the log file, download and extract the ISO from the SDK archives page and run the installer for the missing component. Any archive client will work. I use 7-zip 9-22beta. For SDK 7.0 I had to reinstall Intellidocs before I could completely remove the SDK. And for SDK 7.1 I had to install the Windows Performance Toolkit to remove the SDK completely. Only the ISO will work here, not the redistributables.

Also beware of the Microsoft Install/Uninstall Fixit it doesn't actually do anything but clean your registry. It removed both SDKs but then wouldn't let me reinstall them, all of the files were still in C:\Program Files\Microsoft SDKs\Windows\v7.x all that was different was that they were not in the add/remove programs control panel.

4. Remove everything else

Name	Size	Version
Microsoft Document Explorer 2008
Microsoft Help Viewer 1.1	3.97 MB	1.1.40219
Microsoft SQL Server 2008 R2 Management Objects	12.4 MB	10.50.1750.9
Microsoft SQL Server Compact 3.5 SP2 ENU	3.39 MB	3.5.8080.0
Microsoft SQL Server Compact 3.5 SP2 x64 ENU	4.50 MB	3.5.8080.0
Microsoft SQL Server System CLR Types	930 KB	10.50.1750.9
Application Verifier (x64)	55.3 MB	4.1.1078
Debugging Tools for Windows (x64)	39.8 MB	6.12.2.633
Microsoft Visual Studio 2008 Remote Debugger light (x64) - ENU
Microsoft Visual Studio 2010 ADO.NET Entity Framework Tools	34.2 MB	10.0.40219
Microsoft Visual Studio 2010 Tools for Office Runtime (x64)		10.0.50903
Microsoft Windows Performance Toolkit	26.1 MB	4.8.0
Microsoft Windows SDK for Visual Studio 2008 Headers and Libraries	114 MB	6.1.5288.17011
Microsoft Windows SDK for Visual Studio 2008 SP1 Express Tools for .NET Framework - enu	4.41 MB	3.5.30729
Microsoft Windows SDK for Visual Studio 2008 SP1 Express Tools for Win32	2.61 MB	6.1.5295.17011

As you can see there is a lot of detritus left behind.

5. Remove the 2008 & 2010 C++ compilers and the Visual C++ 2010 SP1 redistributables.

Name	Size	Version
Microsoft Visual C++ Compilers 2008 Standard Edition - enu - x64	127 MB	9.0.30729
Microsoft Visual C++ Compilers 2008 Standard Edition - enu - x86	321 MB	9.0.30729
Microsoft Visual C++ Compilers 2010 SP1 Standard - x64	206 MB	10.0.40219
Microsoft Visual C++ Compilers 2010 SP1 Standard - x86	613 MB	10.0.40219
Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219	6.86 MB	10.0.40219
Microsoft Visual C++ 2010 x86 Redistributable - 10.0.40219	5.44 MB	10.0.40219

These will be reinstalled later. You can not install Windows SDK for Windows 7 (7.1) with NET 4.0 Framework if you already have the Visual C++ 2010 SP1 redistributable installed or you will get the dreaded error 5100.

6. Reinstall both SDKs

Use the web installers linked from the SDK archives page.

7. Install C++ compiler for Python 2.7 and patch vcvarsall.bat

The standalone Python compiler will install the VS2008 (VC90) compilers and headers as well as the vcvarsall.bat batch file that sets environment variables necessary to build Python packages on the fly using pip and setuptools>=6.0. However to build packages using distutils, i.e.: python setup.py build you will need to patch vcvarsall.bat in your C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC directory. To do this copy the vcvarsall.txt file that the SDK created as vcvarsall.bat, then patch it with the Gist in my post. i.e.: patch vcvarsall.bat vcvarsall.bat.patch in bash after downloading and extracting the Gist.

8. Reinstall the Visual C++ 2010 redistributables for x64 and x86
9. Install the Microsoft Visual C++ 2010 SP1 compiler update for Windows SDK 7.1
10. Install VS2013 Express with Update 4 or the free VS2013 Community edition.

The Express edition only has VB, C# and C/C++ compilers, and does not allow extensions, whereas the community edition has everything, but is restricted for commercial use in large corporations.

Voila!

sqlite in MATLAB

It turns out that MATLAB has sqlite builtin

• https://code.google.com/p/sqlite4java/
• http://almworks.com/sqlite4java/javadoc/index.html
• https://www.sqlite.org/c3ref/intro.html

% get or create a new database
db = com.almworks.sqlite4java.SQLiteConnection(java.io.File('sample.db'))
db.open % open database

% create a table called “person” with 2 columns, name and id
db.exec('create table person (id integer, name string)')

% add rows to “person” table
db.exec('insert into person values(1, "leo")')
db.exec('insert into person values(2, "yui")') 
db.dispose % dispose of db handle

% optionally close and reopen database to see it persists
db = com.almworks.sqlite4java.SQLiteConnection(java.io.File('sample.db'))
db.open

% create a prepared statement with ? wildcard
st = db.prepare('select * from person where id>?')
st.bind(1,0) % bind 1st ? wildcard to any number greater than 0

% binding the prepared statment also works for strings
% st = db.prepare('select * from person where name>=?')
% st.bind(1,'') % bind 1st ? wildcard
% note: all string are greater than or equal to ''

% step through matching rows
while st.step
  % returning the data type from the desired column
  st.columnInt(0) % get IDs from column 0
  st.columnString(1) % get name from column 1
end

% disposed of used up statement container
st.dispose
st.isDisposed

% ditto for db connection
db.dispose
db.isDisposed

% output
ans = 1
ans = leo
ans = 2
ans = yui

Although IMO xerial’s jdbc driver (with sqlite included) is much easier

• https://bitbucket.org/xerial/sqlite-jdbc
• http://docs.oracle.com/javase/tutorial/jdbc/index.html
• http://docs.oracle.com/javase/7/docs/technotes/guides/jdbc/

% https://bitbucket.org/xerial/sqlite-jdbc/wiki/Usage 
javaaddpath('C:\Users\mmikofski\Documents\MATLAB\sqlite\sqlite-jdbc-3.8.7.jar')
d = org.sqlite.JDBC
p = java.util.Properties()
c = d.createConnection('jdbc:sqlite:sample.db',p) % named file

% optional connections
% c = d.createConnection('jdbc:sqlite:C:/full/path/to/sample.db',p) % full path
% c = d.createConnection('jdbc:sqlite::memory:',p) % memory db
% c = d.createConnection('jdbc:sqlite:',p) % default
s = c.createStatement() % create a statement

% create a table, insert rows, etc.
% s.executeUpdate('create table person (id integer, name string)');
% s.executeUpdate('insert into person values(1, "leo")');
% s.executeUpdate('insert into person values(2, "yui")');

% execute query, get id and name
rs = s.executeQuery('select * from person')
while rs.next
    rs.getString('id')
    rs.getString('name')
end
c.close % close connection
c.isClosed

% output
ans = 1
ans = leo
ans = 2
ans = yui

Bootstrap & Syntax Highlighting

Bootstrap is probably the hottest web framework out there right now, but try using it with the extremely popular SyntaxHightlighter by Alex Gorbatchev, which I wrote about in syntax sensation. There are at least two issues:

1. Y scrollbars appear for no reason and
2. there is a conflict between Bootstrap's and SyntaxHighlighter's .container class.

Enter our latest contenders:

• Prism and
• highlight.js.

These are both extremely light and fast but offer somewhat more quality than google pretify which I also mentioned in syntax sensation. One thing I will say about both of these is that looking at the resulting DOM, highlight.js prepends hljs- to all of its classes, almost like a namespace, so that it's unlikely that there will ever be conflicts with any other plugin. highlight.js has many languages and styles while Prism has many extra plugins like line-numbers that you can add to your build from their download page. Finally both of these new syntax highlighter's conform to the <pre><code class="language-blah"></code></pre> style that evidently is the standard for putting code into HTML documents. Who knew? SyntaxHighlighter only uses <pre class="brush: blah"></pre> which is non-standard, I guess. Nit-pick much?

Check out for yourself how Bootstrap interacts with each syntax highlighter in the iframe below, or click the link to open in a new tab. The option menu on the right side of the navbar lets you choose which syntax highlighter to see. The template is Bootstrap's theme example which you can return to by clicking the brand on the left side of the navbar. Since highlight.hs comes with 49 styles, you can peruse them from the dropdown menu. Let me know if you find anything amiss anywhere. Of course you will see the extra y-scrollbar in the SyntaxHightlighter rendition.

Bootstrap & Syntax Highlighting 3-Way