Fix vcvarsall.bat to install Python-2.7 x64 extensions with v90 instead of sdk7

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

Previously in order to install Python-2.7 packages with extensions on my 64-bit Windows-7 machine, I have been using the Windows SDK-7 shell and setting the target and some environmental variables manually. What a drag! Then a few days ago, I discovered that the reason vcvarsall.bat has been complaining that it can't find my x64 compilers, even tho they are right there in my Visual Studio 2008 VC folder is because the path the the batch file that sets up the environmental variables for the x64 compiler was wrong. In fact all of the paths were wrong except for vcvars32.bat.

I posted a patch in this Gist:

Now to install packages that include extensions, EG: Dulwich, I just use pip install -U dulwich and easy peasy!

Sphinx with NumPyDoc and Consolidated Fields

[UPDATE 2014-01-31] This is a major update - Sphinx-1.3 now packages Napoleon, allowing you to use Google or NumPy style documentation and have them produce Sphinx formatted documentation.

Sphinx documentation is awesome as is, although IMHO the unformatted docstring is not easy to read (EG: using help(my_fun) to get help on my_fun will show the Sphinx ReST roles and directives). A couple of cool tweeks are consolidated fields and the NumPyDoc extension. For contrast I've also included the Google recommended Python documentation style for unformatted docstrings.

Here are some examples of Sphinx documentation using consolidated fields, Google-style and NumPyDoc without Napoleon.

Here are some examples of Sphinx documentation using the Napoleon extension with Google and NumPy style docs.

XLRD vs OPENPYXL, Round II

[UPDATE 2013-12-02] The major issue discussed in this post, RE: charts not read, worksheets skipped and out of order, was resolved and pulled into the latest release 1.7.0 as well as many other bug fixes. With this latest version, I think that OpenPyXL can be considered the dominant OOXML (post 2007) Excel reader and writer. Note that OpenPyXL is the default Excel reader for Pandas the rapidly growing Python data analysis toolset.

This is a continuation of the previous post on reading Excel from Python. Uh, I might have called it too early! XLRD pulls ahead, but will it win the bout? Read on ...

Reading the contents

Assume we have a sample Excel spreadsheet with 3 worksheets and 2 charts on sheets. In OpenPyXL, you load the workbook, but right away you notice something wrong with the sheet names. Where's 'Sheet3'?

>>> wb_openpyxl = load_workbook(sample)
>>> wb_openpyxl.get_sheet_names()
['Sheet1',
 'Sheet2',
 'Sheet3']

Loading the sheets with XLRD get's it right.

>>> wb_xlrd = open_workbook(sample)
>>> wb_xlrd.sheet_names()
[u'Sheet1',
 u'Sheet2',
 u'Sheet3']

XLRD returns the sheets in the same order as they are visible in the actual spreadsheet, and omits the charts which don't actually contain any data. Unfortunately OpenPyXL can't tell charts from sheets just yet, and is actually naming some of the sheets incorrectly after the charts.

'Sheet1' --> 'Sheet1'
'Sheet2' --> 'Sheet2'
'ChartA' --> 'Sheet3'

This would be OK, since all of the sheets are there, and you could use the sheets' indices, but if you don't only know their names and not the order, then this is an issue. It has been reported in issues #179, #165 and #209. Unfortunately, this issues affects the optimized reader as well. I sent a pull request with a proposed fix for it that has already been merged with master. This issue was resolved and pulled into the current release, OpenPyXL-1.7.0.

Reading Cells

OpenPyXL can use the Excel format, EG: 'A3' or by row & column.

>>> ws1_openpyxl = wb_openpyxl.get_sheet_by_name('Sheet1')
>>> ws1_openpyxl.cell('A3').value

XLRD only reads cells by (row, column).

>>> ws1_xlrd = wb_xlrd.get_sheet_by_name('Sheet1')
>>> ws1_xlrd.cell_value(2, 0)

Both can let you slice the data, but OpenPyXL also allows ranges using Excel format.

>>> ws1_openpyxl.range('A1:C2')

Weird thing about the optimized reader in OpenPyXL, is that it only allows reading sheet contents using the iter_rows() function, which in a way defeats the purpose of the optimized reader, since you have to read in all of the columns in each row!

>>> all_rows = [r for r in ws1_openpyxl.iter_rows()]

The Winner

I think XLRD wins this round, because even though its documentation is sparse, it's not rocket science, and it get's the worksheets, relatively quickly, and more importantly correctly! The screw up with the charts is kind of a non-starter for OpenPyXL.

And another thing occurred to me during Round II. XLRD can open any Excel spreadsheet dating back to like 1995, but OpenPyXL is only for Excel 2007 and newer, which if you didn't know is a zipped XML file.

Finally, even though XLRD doesn't let you use the easy Excel cell reference notation, it is generally faster. And the iter_rows() limitation for the optimized reader in OpenPyXL is a bit annoying, since you're forced to read in many columns that you might not have wanted to read!

spyderlib split

[UPDATE 2013-06-03] Spyderlib-2.2.0 has been released and all of the issues mentioned in this post have been resolved! How's that for support! Thanks spyderlib team for being so quick and responsive!

In addition, pdb debugging is now integrated into Spyder with a swank breakpoints window and debug toolbar with standard step, step-in, return & continue buttons. Either it's always been there and I just haven't noticed, or it is something new. So nice! This really makes Spyder a top notch IDE for python and a serious rival to PyDev.

Spyder is the IDE that comes packaged with Python (x, y), and for a community lead project it is a very impressive achievement. For those migrating from MATLAB, they will appreciate the variable explorer, the built in command window, directory and project browser and help.

If only the debugger was better, but it does link to winpdb, which will be more familiar to most MATLAB users than pdb CLI variants. (And now in Spyderlib-2.2.0, it is; see 2013-06-03 update!)

The trouble is that it consistently has little annoying bugs, which in some cases I think are really show stoppers unfortunately. So I recommend that if you are going to rely on Spyder, download the Python (x, y) distribution or the new Enthought Canopy distribution, because they are tested and frozen at releases that are reliable. (True there have been some bugs, but the responsiveness of the Spyderlib developers is fast and courteous. All of my issues have been addressed, and the release cycle is blazingly fast! And so sorry, but Enthought Canopy is **not** free for x64, but Syderlib-2.2.0 has a installer for Windows x64.)

The latest version of Spyder is 2.1.13.1 and like the version before it, there are at least 2 bugs, which intermittantly effect Windows users. The first is the userconfig.py, which will crash the program as soon as it is started, rendering Spyder useless. The fix is simple, apply the patch in comment #7 from issue 1086. The other bug that is not as bad only affects if you try to start Spyder from a BASH console like msysgit Git Bash  or MSYS. For some bizarre reason the shebang for the Spyder script is the WinPython binary, instead of the traditional:
    #! /usr/bin/env python
So just change it to the traditional python shebang and you can now use the script to start spyder from bash.

See updates above. Both these bugs (issue 1086, 1242 &1331) have been resolved in Spyderlib-2.2.0!

Also if you want to use git in Spyder, then apply this patch too, although all it does is start either gitk or git-gui, which is fine and actually better than nothing. (The patch is unnecessary! Just add

C:\Program Files (x86)\Git\cmd\ on Windows x64

or

C:\Program Files\Git\cmd\ on Windows x86

and gitk or git-gui will open from Spyder just fine! The msysgit developers have placed native win32 versions of gitk and git-gui in this folder. Thanks!)

The -mno-cygwin option has been deprecated for years

[UPDATE: 2015-03-19] Yay! Issue #12641 was closed in September-2013, and the latest releases of Python>=2.7.6 no longer have it.

So true. And there's even a bug filed in Python tracker. I also talked about it in this post. Here's how they deal with it in Mercurial (a.k.a. Hg).

# the -mno-cygwin option has been deprecated for years
Mingw32CCompiler = cygwinccompiler.Mingw32CCompiler
class HackedMingw32CCompiler(cygwinccompiler.Mingw32CCompiler):
    def __init__(self, *args, **kwargs):
        Mingw32CCompiler.__init__(self, *args, **kwargs)
        for i in 'compiler compiler_so linker_exe linker_so'.split():
            try:
                getattr(self, i).remove('-mno-cygwin')
            except ValueError:
                passcygwinccompiler.Mingw32CCompiler = HackedMingw32CCompiler

Console 2 by bozho

[UPDATE 2015-04-27]: After trying ConEmu by Maximus5, there is no turning back. ConEmu is so obviously superior, it has full 256-color xterm support, properly resizes windows like normal windows when maximized and as if that wasn't enough, comes with all of the most common shells like git bash and various windows compiler environments preconfigured, including admin versions. Seriously! And I'm not alone; other Console2 users have professed their conversion to ConEmu as well.

I just went to Maximus5 GitHub release page for ConEmu, downloaded the 7z archive of the most current release and unzipped it into C:\ConEmu and awesome console emulator installed. Make a short cut of ConEmu (either 32 or 64 bit versions executable to your desktop then start - first time it asks for some simple setup questions, like where to store config file, what shells to use (select from preconfigured) and then starts. It just works. Better than Windows CMD, mintty, rxvt, powershell and yest even the revered Console2. Thanks Bozho, but it's time to move on.

The console terminal for MS Windows kicks ass compared to mintty, rxvt, cmd or powershell.

Why? Let me enumerate:

1. Resizable windows
2. Tabs
3. Customizable shells.
4. Transparent background setting.
5. Customizable copy/paste commands.

I could go on and on. This is like 7zip, just get it and start being more productive right now!

Fedora 17 open-vm-tools

[UPDATE 2013-04-24]

This blog post here is now obsolete as open-vm-tools-9.2.2 with Fedora specific scripts is now available (via yum) from Fedora for both Fedora-17, Fedora-18 and soon to be Fedora-19 too.

Also a new stable open-vm-tools (open-vm-tools-9.2.3-1031360) has been released today 2013-04-24. A pseudo list of fixes is on Dmitry Torokhov sf.net page.

[UPDATE 2013-04-08]

Starting in Fedora 18, procps has been replaced by procps-ng. I used a soft symlink to point to libprocps: $ sudo ln -s /usr/lib/libprocps.so /usr/lib/libproc.so You will need to install the dev packages from Fedora.

Also you will have to build using CFLAGS=-Wno-deprecated-declarations to avoid errors from deprecated glib code.

AFAIK neither Fedora 16 nor 17 come with open-vm-tools, the open-source version of vmware-tools provided for guest systems by vmware. Ubuntu and OpenSUSE both have this essential tool in their repositories. In their instructions specifically for Fedora 16/17, vmware says to use the version of vmware-tools that comes with vmware-player, however, if your host machine is too old to run the latest version, then you are SOL because you can only update vmware-tools by updating vmware-player, which is a catch-22. Perhaps fortunately now, you can download current version of vmware-tools if you can figure out which version to use, because only red-hat enterprise is listed.

Fortunately you can build open-vm-tools from source, just make sure you have the dev files from the following packages:

• gtk
• pam
• ... <need to finish this list>

You can install them using `sudo yum install <package>-dev`. As you are building open-vm-tools you will be prompted which dev-packages are missing, so just install and restart.

To build you follow the familiar autotools pattern: `./configure`, `make` and `make install` which will automatically put it in /usr/local/... which is probably the best place for it. I don't recommend ever installing anything into /usr/bin/ or /usr/lib/ because they will probably be overwritten on any updates/upgrades and distribution updates/upgrades. The other option would be either /opt or ~/opt but that is usually reserved for precompiled binaries that are ready to be installed, and placing an add-on software package in ~/opt will only allow the user who installs it to use it, which might be your intent - but I digress.

I had no problems installing the dev packages or building open-vm-tools for my Fedora 17 vm, but when I restarted, the guest tools service isn't running! Of course, because I need to edit my /etc/init.d folder to add open-vm-tools as a service. I'm still working on that part, there are some clues on Fedora's init scripts man page and I've also copied the init.d scripts from Ubuntu's and openSUSE's open-vm-tool scripts. And of course there is also vmware-config-tools.pl which should also provide some more clues. One thought I had was to download one of the newer vmware-tools tarballs for RHEL and see what's in that version vmware-config-tools.pl script.

I was able to start the service manually by executing vmware-user-suid-wrapper but it complains that some of the modules haven't been loaded before logging in. You can load the modules manually too, but the init.d scripts are the right way to go. Even without the modules I can get drag and drop and auto-screen resizing to work just by running vmtoolsd or the aforementioned vmware-user.suid-wrapper. I can also mount my host shared folders using vmware-mounter. If anyone wants to help out, then we will have a very nice packaged version of open-vm-tools for Fedora.

The best online help I've found for open-vm-tools and vmware-tools for linux guests actually has been on the arch-linux site. According to the first few paragraphs you can enable open-vm-tools at boot by using `systemctl start vmtoolsd` and `systemctl enable vmtoolsd`. Trying either of these commands returns `vmtoolsd.service not found`. Bummer. Googling gives these fedora links:

• Systemd
• SysVinit to Systemd Cheatsheet
• Chapter 7. Services and Daemons

The last of these 3 links seems to confirm that indeed systemctl will automatically add the service for the next boot. However I don't think this takes care of all of the scripts, there are some in /etc/sysconfig. Anyway, it's obvious that you have to have the /etc/init.d or rc.d and /sysconfig scripts to start and enable the service. The vmware-config-tools.pl file from my version of vmware was too old to be relevant. There are rpm's of more current vmware-config-tools, so maybe I'll have a look in there.

dulwich porcelain

dulwich is an pure python implementation of git plumbing. It comes with a bin/scripts folder but it is incomplete - although I'm sure the developers would welcome additions. I have been working on a package called dulwich-porcelain, which facilitates adding features like checkout, fetch, push and status. Uh ... it's still beta, but contributions are welcome.

I also started a fork of dulwich with binpatches, porcelain and android branches. Android was actually the reason I origin started working on this. With just three commands it's easy to use your android device to retrieve your repos from a remote server anywhere you have 3g.

~$ dulwich clone <remote> <repo-name>

~$ cd <repo-name>

~/<repo-name>$ fetch_refs

~/<repo-name>$ checkout

n.b.: the functions also work as Python commands.

 You also need sl4a and py4a and some ssh authentication such as dropbear which is built into Cyanogenmod 7, openSSH is now used on CM9. Paramiko would be another option. I should make a post about this, but basically, I created an .ssh folder on my sdcard and created keys there. Cyanogenmod has a dropbear wiki and there is also a blogpost by TK, but both refer to creating an ssh server to communicate with your phone from a remote device, not creating keys to communicate with a remote device from your phone. Something like this should work.

~$ mkdir .ssh

~$ cd .ssh

~/.ssh$ dropbearkey -t rsa -f id_rsa

~/.ssh$ dropbearkey -y -f id_rsa > id_rsa_pub

Now you need to copy the public key, id_rsa_pub, to your remote Git server. You also need to make a small change to dulwich SSHvendor in client.py to make dropbear work. You can find the relevant changes in the android branch of my dulwich fork; basically append this class ...

class DropbearSSHVendor(object):
    def connect_ssh(self, host, command, username=None, port=None):
        import subprocess
        #FIXME: This has no way to deal with passwords..
        args = ['ssh', '-i', '/mnt/sdcard/.ssh/id_rsa']
        if port is not None:
            args.extend(['-p', str(port)])
        if username is not None:
            host = '%s@%s' % (username, host)
        args.append(host)
        proc = subprocess.Popen(args + command,
                                stdin=subprocess.PIPE,
                                stdout=subprocess.PIPE)
        return SubprocessWrapper(proc)

to the client.py file and then set get_ssh_vendor to the new vendor instead of the default one.

# Can be overridden by users
# get_ssh_vendor = SSHVendor
get_ssh_vendor = DropbearSSHVendor

Known hosts also got created automatically by dropbear in my .ssh folder on my sdcard. Dropbear doesn't let you use passwords, which is fine because dulwich has no way to deal with them. You could use keyring to get around that.

Now you can copy, send via email/bluetooth/wifi and/or edit your repo files on-the-go without needing to use a public computer or worry about ssh keys or passwords.

Download sites for old, free MS compilers and IDEs

[UPDATE 2015-03-13] I do not recommend installing VS2010 Express (or Professional). This program is very difficult to remove and has become obsolete, with VS2013. If you are stuck with this on your computer you may be able to remove it by following these steps.

1. Make sure you have copies of the VS2010 SP1 installer. The web installer is fine, but the iso may come in useful. Don't bother creating a manual system restore point. Even when you roll back changes, the issues that cause the majority of problems are not resolved. However I guess it can't hurt either. If you really want to cover your bases, the best thing would be to make a recovery disk and an disk image. This way you can roll back changes if they go awry.
2. If you have installed VS2010 SP1, you must remove this first. You should be able to remove it from add/remove programs in the control panel. If not try
• C:\ProgramData\VS\vs2010sp1\SetupCache\setup.exe /uninstall /force
• Download the installer and run VS10sp1-KB983509.exe /uninstall /force
• Download and extract the iso image and run setup.exe /uninstall /force
• Reinstall SP1 and then try uninstalling it from add/remove programs in control panel. You may need to reinstall VS2010 VC10 or VCS10 before SP1.
3. Remove VS2010 Tools for Office Runtime and VS2010 ADO.NET Entity Framework. You may need to remove these before SP1.
4. Use the VS2010 uninstall utility documented first here then later here. Make sure you use the options in the post /full to remove everything!

Note this took many iterations to actually get all of VS2010 off of my system, so be prepared for frustration and irritation, but persevere and it can be done. Just make sure you have copies of the installer so that you can reinstall and try to remove everything in the correct order again! You may have an issue removing or reinstalling Windows SDKs. If so see Upgrading to Visual Studio 2013.

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

[UPDATE 2013-10-03, 2014-04-16] Sad to report that clicking on #1 MS Visual Studio 2008 Express states that it has been retired. Oh, yes! VS2008 Express didn't allow targeting x64 platforms, without some serious hoop-jumping, but VS2010 Express does, and VS2010 also lets you select the V90 toolset, so building shared objects for Python-2.7 is no problem. Also, SDK7 has the exact same compilers as V90, and it is still supported so they are more current. Ditto for SDK7.1 & V100. Oh no! Does this mean we finally have to switch to Python 3? If you really want VS2008, the web installer does however still work. Grab it from my dropbox.

1. Microsoft Visual Studio 2008 Express Editions with SP1: This one is necessary for python 2.7.3 extensions. You can't get it from the Visual Studio page anymore, now that they've moved on to 2012, but download links for vcsetup.exe which install MSVC90 Express are in the Microsoft Download Center, as are vcssetup (C#), vbsetup.exe (vbs) and vwsetup.exe (web). You also get the opportunity to install SQL Server 2008 and Silverlight SDK. Visual Studio 2008 Express has been discontinued, and is obsolete because Visual Studio 2010 Express includes the V90 toolset. See update note at top of page. Not sure if Visual Studio 2008 SP1 download will install for free, but it may. VS2008 is not required to build Python-2.7 extensions, instead use SDK7 which replaces the v90 compilers.
2. Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1: You cannot build 64-bit applications with the free version of VC90 unless you have Windows SDK 7. Note this *not* 7.1, which is VC100, also note that .NET Framework 3.5 is part of Windows 7, so you don't have to download it separately, and of course make sure that you get all updates before installing. Specifically you must have SP1. The paths in vcvarsall.bat are incorrect, you will need to fix them to use pip with v90. Otherwise use sdk7 shell and follow the directions in the post on installing Python x64 extensions with pip. 
3. Visual Studio 2010 Express is still available from the main VS site. They have 2012 and 2010, and all of the flavors.
4. Microsoft SDK 7.1 has some issues with VC100. Basically you should follow this procedures:
1. Visual Studio 2010 RTM
2. Windows SDK 7.1
3. Visual Studio 2010 SP1
4. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1
See these posts:
Setup & Install by Heath Stewart
Visual C++ Team Blog
FIX: Visual C++ compilers are removed ...

5. Microsoft Windows SDK for Windows 7 and .NET Framework 4
Beware here, you might run into this error, which is very confusing, the installer quits with the message "Installation of the “Microsoft Windows SDKfor Windows 7” product has reported the following error: Please refer to Samples\Setup\HTML\ConfigDetails.htm document for further information." Totally useless, but further examination of the log file or googling around you might see the real source of the error, it's not SP1 but the redistributable that is too new: "C:\Program Files\Microsoft SDKs\Windows\v7.1\Setup\SFX\vcredist_x64.exe installation failed with return code 5100". The solution is to remove the redistributables and then reinstall them later. See this knowledge base article:
Windows SDK Fails to Install with Return Code 5100

6. Upgrade to Microsoft Visual Studio 2010 SP1. See this link for what's in SP1.
   Description of Visual Studio 2010 Service Pack 1
7. Download the update after you follow the procedures here:
   Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1

Python x64 Package Extensions with pip, MSVC 2008 Express & SDK 7

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

[UPDATED 2012-02-04, 2014-04-18] You can skip this if you fix your vcvarsall.bat to point at the correct paths.

If you try `user@machine ~$ pip install package` from a bash shell or even `C:\Users\user>C:\Python27\Scripts\pip.exe install package` on a 64-bit MS Windows machine with the 64-bit version of Python installed, then you are likely to get a traceback from `distutils.msvc9ccompiler.py` that `vcvarsall.bat` only returned `path` but it was also looking for `lib`, `libpath` and `includes`. Oh well.

Because you know that your version of Python 2.7 was compiled with MS Visual C++ 2008 (aka VC90), you might try to use pip in the VC90 command prompt, since it loads the variables for you, but then you will get the following linker error:

fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'

In a virgin cmd.exe shell you might try to run vcvarsall.bat amd64 to see if it works, but then you will see that it doesn't, when you get this discouraging stifling message:

The specified configuration type is missing.  The tools for the configuration might not be installed.

Then you might put this into Google and eventually you will find that MVSC 2008 Express doesn't support 64-bit compilation without Windows SDK 7 (see this post to find out where you can download this). Luckily, that has it's own nifty environment, run it from the start menu and change to 64-bit compilers and release mode by typing ...

> setenv /x64 /release

... and watching the screen font color change from yellow to green. Fun! Distutils would also like you to tell it not to bother looking around for vcvarsall, and just use the SDK environment by setting two environment variables:

> set DISTUTILS_USE_SDK=1
> set MSSdk=1

Also, you will need to have both Python27 and Python27\Scripts on your path before you run the SDK 7 CMD shell, or distutils will still fail. Unfortunately you can't add this by editing the path after starting the shell. The easiest way to do this is to right click on the My Computer icon on the desktop, select the Advanced tab and then click the Environment Variables button near the bottom. This will bring up the Environment Variables window. Add a new user variable for your personal profile by clicking the New button in the upper pane, and then type PATH in the Variable name: box and C:\Python27;C:\Python27\Scripts in the Variable value: box. Note the semicolon in between the 2 paths, if you've never edited your path before, this is the path separator on MS Windows. Then click OK, OK and OK. The changes are immediate for new CMD shells. Your local PATH is appended to the end of the system PATH.

Now try `pip install package` again and if you're lucky, voila, pip says, "Sucessfully installed package". Thanks pip! I tried this with dulwich, and it installed no problem.

There is some nifty info in the Python distutils docs and the Cython folks also have some nice 64-bit extensions help too.

How to remove noexec flag from your sdcard

Run mount -o remount, rw /mnt/sdcard/. This will also work on any external storage,  Even an internal mmc card. It won't survive a restart. You could try to modify your vold fstab file. Or add the commands to your .bashrc or profile and reverse it in .bashlogout. You may need to use /storage/sdcard0 or sdcard1 depending what ROM you are running.


Note this will not work on a fuse file system, which unfortunately is the new norm on many new smartphones (EG: Motorola RAZR HD).

Cygwin, MinGW, MSYS, GnuWin32, GNU Utilities for Win32 and now MinGW-w64

Introduction

GNU on Windows has a sordid past. Not really, but sordid sounds so much more interesting than complicated. So here's how I understand it, without any emotion, this is really just a navigational exercise.

Disclaimer

What was just said and what I am about to say is entirely opinion and not based on fact. I hope that I do not offend anyone, but I am sure that I inevitably will, as much of my opinion is based on hearsay. In that case I apologize, and please remember that the purpose of this blog is purely selfish; it is a note to myself to remind me of what took me so long to finally understand. So instead of getting angry with me, perhaps post a comment and disabuse me of my still as yet incomplete knowledge.

Cygwin

In the beginning there was Cygwin. Their tagline is "Get that Linux feeling - on Windows!" This does a lot to explain exactly what Cygwin is, Linux emulated on Windows. What this means in practice is that Cygwin applications use the cygwin.dll, a common runtime that must be linked to any applications that run on Cygwin, and consequently any applicaton compiled with Cygwin gcc. So essentially, when you use Cygwin you are more or less stuck in Cygwin. True, you could distribute the cygwin.dll with your application, so it would appear to be native, but it wouldn't be truly native. By native here I mean that it runs on one of the windows runtimes like the win32api or msvcrt. The Windows binary of GNU nano is an example of an application compiled using Cygwin that requires cygwin.dll to run. The probable downside of using this layer between the Windows API and your application may be potentially slower speed and some limits in its features.

MinGW

From Cygwin was born MinGW. I don't know if this is actually true, but that is what I've heard. Either way it doesn't really matter. The essential thing to realize here is this.

MinGW creates native applications for Windows using native libraries.

This means exactly what it sounds like, applications compiled using MinGW's compilers (gcc, g++, gfortran, etc.) will run on any Windows machine natively without any runtime other than the Windows API or mscvrt. Now it might be fun to speculate about some internal strife or a passionate drive of a group of individuals with some bold ideal, but that's irrelevant to this fundamental difference between the Cygwin and MinGW.

MSYS

MSYS may seem like a variation on Cygwin, except it is really meant to be used as an environment for running shell scripts, similar to bash (the Bourne Again SHell). It's true that if you compile an application using the MSYS version of gcc, then you will need to link it to the msys-1.0.dll, so in a sense it is the same as Cygwin. The difference is really in a state of mind. MSYS is presumably part of MinGW, which stands for Minimalist GNU for Windows. What that translates to is that MinGW and MSYS contain only the most essential tools required for developing native applications for Windows using the GNU open source suite of compilers, whereas Cygwin aims to provide Windows users every application available to Linux. In Cygwin you will find Python, GTK, nano, Ruby, Git, even some games I think, whereas in MinGW you will only find libraries most often used in common code. However there is a huge chunk of open source code that is compiled using autotools and depending on shell scripting, which I think is where MSYS gets involved. MSYS makes it easier to do this. True autotools, make and shell script functions have been ported as native win32 applications, but since they are only used as temporary tools to get to the finished product (a natively compiled win32 application) and the results don't depend on any of those temporary tools, why go through the effort of duplicating what Cygwin has already done so nicely?

GnuWin32 and GNU Utilities for Windows

If you were just reading the section above, then continuing on, GnuWin32 and the now defunct GNU Utilities for Windows seem to do precisely that. Both of these generally utilize MinGW to port GNU applications that were native to Linux or Cygwin to run natively on Windows. And in fact many open source projects now try to configure options for some Windows compiler, be it MinGW, MSVC or BCC (really, I haven't seen this too often although apparently it is still out there, wow they even have Delphi!). Some projects are optionally opting for CMake and alternative to Autotools. OK, so the key distinction here is that these are both compiled using MinGW and are basically an alternate to MSYS, if you prefer to continue to use the native Windows CMD console or something like Powershell.

MinGW-w64

MinGW-w64 is a fork of MinGW that allows you to cross compile code from one machine to another. For example, say you want to make 64-bit code, but you are on a 32-bit machine or a Linux box, or using Cygwin, then you would use MinGW-w64. The "w64" comes from it's ability to compile 64-bit code, which is not currently available (AFAIK) with MinGW (sometimes called mingw32). There are alternate distributions of both MinGW and MinGW-w64 provided by TDM. Also there are other cross-compilers, such as mxe.cc, but it is only for *nix (Linux/Unix). Coincidentally MinGW are MinGW-w64 both available as cross-compilers in most Linux distro repositories. Ubuntu, Fedora and Suse all have it. MacPorts, Fink and Homebrew may have them too. Same probably goes for mxe.cc. And who knows there are probably more cross-compilers out there as well.

Epilogue

Ah. So glad to get that all down, finally. So the moral of the story is this.

If you want to use native Windows apps compiled using GNU gcc use MinGW, but do not polute your toolchain with any MSYS or Cygwin libraries or you will be sorry.

Building numpy, scipy, matplotlib and PIL in virtualev on both windows and linux

This really should be split onto several posts. And I want to send a patch to matplotlib to make it easier to build in windows.

1. You need several dev files, most of which come standard on Linux and mingw-users. Still for PIL you new Tk.h and tcl.h, zlib, libjpg and ... Note on windows, you need the same version of Tck/Tk as in Python, which is probably 8.5.2. Look at `init.tcl` in `C:\Python27\tcl\tcl8.5` and it will tell you exactly which package is required.
2. On Linux, look for the corresponding dev packages, which may not be installed. On Ubuntu they are in the Ubuntu Software Center. In particular you will want the ATLAS, BLAS and LAPACK dev packages for Numpy and Scipy.
3. On Windows, in some cases, I untarred/unzipped the downloads to edit the setup.py or other relevant file, then zipped it back up and used pip. A few packages took command line options which I passed through pip using --install-options=" --my-option='blah blah blah' " as an example.
4. I added a pydistutils.cfg file to my home directory with [build] compiler=mingw32 which worked well for pip except with pyzmq, which I had to edit, because it makes its own compiler. So I edited it to make a mingw32 compiler, instead of the default which on Windows is msvcr90. In general I had to edit cygwinccompiler in distutils to remove the "-mno-cygwin" gcc option for the Mingw32Ccompiler class, which is no longer accepted for most gcc versions.
5. On Linux this was all very easy, as long as I had the correct dev packages, but be prepared on Windows, this took a lot of work.

[POSTSCRIPT 2014-08-01]

This post is seriously out of date, and it was a bit vague to begin with. I don't remember what steps I took to get these tools running on windows anymore, and a lot has changed, most significantly I have switched to 64-bit, I have found that more python packages can be built with microsoft compilers and openBLAS has been released which greatly simplifies building these packages as BLAS and LAPACK are already compiled and optimized.

• Consider using Windows SDK-7 for Python-2.7 after fixing vcvarsall.bat instead of mingw32. Windows SDK-7 replaces VC90 (aka V90) and comes with both 32-bit ad 64-bit compilers.
• MinGW only works on 32-bit Windows systems so an alternative I started using recently is Win-Builds which uses mingw-w64 compilers which can cross compile both 32-bit and 64-bit applications. Win-Builds works in MSYS, Cygwin or the native CMD shells, although I don't know how you would use autotools in a Windows CMD shell.
• Unfortunately win-builds doesn't have gfortran, so either stick with mingw-w64 or build it, ugh!
• Try using one of the new openBLAS binaries, maybe everything can be done with VC90 instead of using gcc.

building dbus-python on windows with mingw

dbus-python Build Instructions for Windows (x86) with MinGW

Revision history

1. r0.0.20120706 - First draft
2. r0.1.20120718 - Clean up markdown and post to poquitopicante.blogspot.com
3. r1.0.20120720 - Use autolauch
   • Need --with-dbus-session-bus-default-address=autolaunch: for dbus to work out-of-the-box and to keep dbus-python from throwing exception.
   • Also If using git repo, use ./autogen.sh instead of ./configure. Make sure you have autotools installed in your MinGW toolchain.
4. r1.1.20120723 - Correct GTK path
   • Remove dbus-glib examples Makefile patch; it's not needed if path to GTK runtime bin is correct, i.e.: no quotes in MS-Windows paths!
   • This was not an issue with dbus-binding-tool.exe, #52322 in bugzilla. There are no problems for dbus-glib to create binding "glue" when in glib-server mode if path to GTK bin folder is correctly.
   • setenv.cmd has an issue in which the quotation marks are in the wrong place, causing the quotes to be copied into the path, which is a big no-no in MS-Windows, because it causes inconsistent results. E.g. dbus-binding-tool.exe could be built, but not run.

Overview

These are instructions for building dbus-python, a binding to dbus on a MS-Windows x86 system using MinGW.

Set up MinGW build environment

1. Download and install most recent version of mingw-get-inst from sourceforge, current version is 20120426. This only needs to be done once because afterwards you can just run ...
   

   $ mingw-get update
   $ mingw-get install mingw-get
   

   ... to get the latest version. During the install, which is an Inno-Setup GUI, select the option to install the "C compiler" and "MSYS Basic System" but leave other default install options. It should install MinGW in C:\mingw and MSYS in C:\mingw\msys\1.0. It doesn't change your path or add any environmental variables, but it does add a shortcut to the MSYS bash shell on your desktop and start menu, and register mingw-get-inst so you can un-install it later. There is lots of rich material on MinGW and MSYS on the Internet. Also take a look at Configuring and Using mingw-get.
   
2. Start the MSYS shell, from the start menu, and install (or update) the developers toolkit. If you need to upgrade a component, use mingw-get upgrade <component>.
   

   $ mingw-get update
   $ mingw-get install gcc
   $ mingw-get install g++
   $ mingw-get install msys-make
   $ mingw-get install mingw32-make
   $ mingw-get install mingw-developer-toolkit
   

3. While still in the MSYS shell, install expat.
   

   $ mingw-get install expat
   $ mingw-get install libexpat
   $ mingw-get install msys-expat
   $ mingw-get install msys-libexpat
   

4. You also need libz and zlib.
   

   $ mingw-get install libz
   $ mingw-get install zlib
   $ mingw-get install msys-zlib
   

5. You may also need or want the following:
   
   • GNU patch, wget, tar, libxml2 and possibly others (libxslt and libffi). If possible install them from MinGW/MSYS with mingw-get. Use mingw-get list <package> to search for packages.
     

     $ mingw-get install msys-patch
     $ mingw-get install msys-wget
     $ mingw-get install msys-tar
     $ mingw-get install msys-libxml2
     

   • lib2xml and libxslt are also available from xmlsoft as windows binaries. If use them, copy the files into the matching folders in /local, e.g. bin files go in /local/bin.
     
   • zlib is also available from zlib, there are links to binaries, but it is easy to build by following the directions on Fragrant Memories, however, the generated library zlib1.dll may conflict with your Intel Wifi controller or something else. The MinGW/MSYS libraries are libz-1.dll and msys-z.dll. The files zlib1.dll, libexpat-1.dll and libxml2-2.dll may actually already be in your GTK runtime bin folder.
     $ which zlib1.dll /c/Program Files/Intel/WiFi/bin/zlib1.dll
     
   • libffi is available here and on Github and installation instructions are here.
     

Python

Download and install Python (2.7.3) with the Python Windows Installer.

PyGTK or GTK+

1. Download and install either the PyGTK all-in-one installer for your Python version (e.g. 2.7) or the GTK-2.0 all-in-one installer. If you're determined to build GLib from source see this post on Bootstrapping GLIB with MinGW.
   
2. Fix the setenv.bat by either removing the quotes entirely from the SET PATH statements, or correcting them as in the patch below.
   

   --- a/python27/lib/site-packages/gtk-2.0/runtime/bin/setenv.cmd
   +++ b/python27/lib/site-packages/gtk-2.0/runtime/bin/setenv.cmd
   @@ -23,14 +23,14 @@
    call:abspath PYTHON_PKGCONFIG "%PYTHON_ROOT%\Lib\pkgconfig\"
   
    if defined PATH (
   -    set PATH="%PYTHON_ROOT%;%PYTHON_SCRIPTS%;%RUNTIME_BIN%;%PATH%"
   +    set "PATH=%PYTHON_ROOT%;%PYTHON_SCRIPTS%;%RUNTIME_BIN%;%PATH%"
    ) else (
   -    set PATH="%PYTHON_ROOT%;%PYTHON_SCRIPTS%;%RUNTIME_BIN%"
   +    set "PATH=%PYTHON_ROOT%;%PYTHON_SCRIPTS%;%RUNTIME_BIN%"
    )
    if defined PKG_CONFIG_PATH (
   -    set PKG_CONFIG_PATH="%PYTHON_PKGCONFIG%;%RUNTIME_PKGCONFIG%;%PKGCONFIG_PATH%"
   +    set "PKG_CONFIG_PATH=%PYTHON_PKGCONFIG%;%RUNTIME_PKGCONFIG%;%PKGCONFIG_PATH%"
    ) else (
   -    set PKG_CONFIG_PATH="%PYTHON_PKGCONFIG%;%RUNTIME_PKGCONFIG%"
   +    set "PKG_CONFIG_PATH=%PYTHON_PKGCONFIG%;%RUNTIME_PKGCONFIG%"
    )
   
    cls
   

3. After correcting the setenv.cmd script as above, make a GTK bash shell environment by copying the MSYS/MinGW shell icon on the desktop (or making a shortcut to c:\mingw\msys\1.0\msys.bat) , right-clicking, selecting Properties, editing the Target: to be C:\WINDOWS\system32\cmd.exe /C ""C:\Python27\Lib\site-packages\gtk-2.0\runtime\bin\setenv.cmd"&&"C:\MinGW\msys\1.0\msys.bat" and renaming it to GTK MinGW shell. If the icon gets screwed up, select Properties again, Change Icon and point it to the same location as the original MSYS/MinGW shell shortcut's icon path. Note you will still have to point PKG_CONFIG to the GTK runtime pkgconfig.exe and add both /usr/local/lib/pkgconfig, /mingw/lib/pkgconfig and /usr/lib/pkgconfig to your PKG_CONFIG_PATH in your ./configure commands, but this should add all of the other GTK and PKG_CONFIG paths and environmental variables you'll need, especially for dbus-glib. An alternative is to simply add the GTK runtime to the path of your bash shell by editing (or creating) your ~/.bashrc file.
   

   $ export GTK_BASEPATH="/c/Python27/Lib/site-packages/gtk-2.0/runtime/"
   $ export PATH=$GTK_BASEPATH/bin:$PATH
   

   Another alternative is to start the PyGTK/GTK Command Prompt which executes setenv.cmd, creating several environmental variables in that shell such as PYTHON_ROOT, PYTHON_PKGCONFIG, RUNTIME_PKGCONFIG, PKG_CONFIG_PATH and RUNTIME_BIN, thereby making sure that your GTK environment is correctly configured. Then type c:\mingw\msys\1.0\msys.bat to start a MSYS/MinGW shell which will also set up your MinGW/MSYS environment. Note: you may have already added GTK to your path during installation, depending on what options you selected.
   The library libgio-2.0-0.dll and other binaries are used by dbus-binding-tool.exe and others, so you will need to use this shell, or make the GTK runtime bin folder available, whenever you wan to use dbus-glib.
   

Build DBus

1. Download the tarball or clone the git repository a folder in C:\, e.g. C:\DBus. Note: If you clone the repo, then instead of ./configure you will need to use ./autogen.sh which requires autotools. Use mingw-get to install autoconf, automake, m4 and libtool for both MSYS and MinGW. Also you may want to checkout the latest stable tag into your own private branch git checkout -b <myBranch> dbus-1.6.4.
   

   $ mingw-get install autoconf
   $ mingw-get install automake
   $ mingw-get install libtool
   $ mingw-get install msys-autoconf
   $ mingw-get install msys-automake
   $ mingw-get install msys-m4
   $ mingw-get install msys-libtool
   

2. Apply the MemoryBarrier macro patch. The location of this patch is debatable, and there have been some mailing list posts, tickets, forks and suggeted patches [1, 2, 3, 4, 5, 6, 7, 8]. I applied it to dbus\dbus-sysdeps-win.c which is where the macro is used. I also added a prototype in the header so that gcc won't warn me about it. I have no idea whether this works as intended, or even what the intent was, so use it at your own risk. It is identical to the macro used by mingw-w64 in winnt.h. The other options are to get rid of the MemoryBarrier() call or use a different compiler such as MSVC or MinGW-w64.
   

   MemoryBarrier.patch for dbus-sysdeps-win.c

   --- a/dbus/dbus-sysdeps-win.c
   +++ b/dbus/dbus-sysdeps-win.c
   @@ -54,6 +54,13 @@
    #include <windows.h>
    #include <ws2tcpip.h>
    #include <wincrypt.h>
   +
   +__CRT_INLINE VOID MemoryBarrier(VOID)
   +{
   +  LONG Barrier = 0;
   +  __asm__ __volatile__("xchgl %%eax,%0 "
   +    :"=r" (Barrier));
   +}
   
    /* Declarations missing in mingw's headers */
    extern BOOL WINAPI ConvertStringSidToSidA (LPCSTR  StringSid, PSID *Sid);
   

   MemoryBarrierHeader.patch for dbus-sysdeps-win.h

   --- a/dbus/dbus-sysdeps-win.h
   +++ b/dbus/dbus-sysdeps-win.h
   @@ -38,6 +38,8 @@
   
    #define DBUS_CONSOLE_DIR "/var/run/console/"
   
   +
   +void MemoryBarrier(void);
   
    void _dbus_win_set_errno (int err);
    const char* _dbus_win_error_from_last_error (void);
   

   You can use git apply --ignore-space-change --ignore-whitespace <path/to/MemoryBarrier.patch> followed by commit -am "<your message>" or patch <path/to/dbus-sysdeps-win.c> <path/to/MemoryBarrier.patch> if you are using the tarball.
   
3. configure, make and install
   

   $ ./configure --with-dbus-session-bus-default-address=autolaunch: PYTHON=/c/Python27/python PYTHON_INCLUDES=-I/c/Python27/include/ PYTHON_LIBS='-L/c/Python27/Lib -lpython27' PKG_CONFIG=/c/Python27/Lib/site-packages/gtk-2.0/runtime/bin/pkg-config.exe PKG_CONFIG_PATH=/c/Python27/Lib/pkgconfig:/c/Python27/Lib/site-packages/gtk-2.0/runtime/lib/pkgconfig:/c/mingw/msys/1.0/local/lib/pkgconfig:/c/mingw/msys/1.0/lib/pkgconfig:/c/mingw/lib/pkgconfig am_cv_python_pythondir='${prefix}/Lib/site-packages' am_cv_python_pyexecdir='${exec_prefix}/Lib/site-packages'
   $ make
   $ make install
   

4. Check that it works by opening a daemon, and monitoring it using two MSYS shells.
   

   MSYS shell with daemon

   $ dbus-daemon --session
   

   MSYS shell with monitor

   $ dbus-monitor
   signal sender=org.freedesktop.DBus -> dest=:1.0 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired   string ":1.0"
   method call sender=:1.0 -> dest=org.freedesktop.DBus serial=3 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch   string "eavesdrop=true,type='method_call'"
   method call sender=:1.0 -> dest=org.freedesktop.DBus serial=4 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch   string "eavesdrop=true,type='method_return'"
   method call sender=:1.0 -> dest=org.freedesktop.DBus serial=5 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch   string "eavesdrop=true,type='error'"
   

   You should see dbus-daemon and dbus-monitor as processes in ms-windows task manager. Use ctrl-c to kill the processes in each MSYS window.
   

Buld dbus-glib

1. Download the tarball or clone the git repository on your hardrive, e.g. c:\dbus-glib. If you are using the git repo, checkout the latest tag into your own branch, and then use ./autogen.sh instead of ./configure. Also you may need to add --disable-gtk-doc as a configure option. I did not, but I used the tarball.
   
2. confgure, make and install
   

   $ ./configure PYTHON=/c/Python27/python PYTHON_INCLUDES=-I/c/Python27/include/ PYTHON_LIBS='-L/c/Python27/Lib -lpython27' PKG_CONFIG=/c/Python27/Lib/site-packages/gtk-2.0/runtime/bin/pkg-config.exe PKG_CONFIG_PATH=/c/Python27/Lib/pkgconfig:/c/Python27/Lib/site-packages/gtk-2.0/runtime/lib/pkgconfig:/c/mingw/msys/1.0/local/lib/pkgconfig:/c/mingw/msys/1.0/lib/pkgconfig:/c/mingw/lib/pkgconfig am_cv_python_pythondir='${prefix}/Lib/site-packages' am_cv_python_pyexecdir='${exec_prefix}/Lib/site-packages'
   $ make
   $ make install
   

Build dbus-python, finally!

1. Download the wip-windows tarball or clone the git repository and checkout 2012-07-04 branch on your hardrive, e.g. c:\dbus-python. Again, if using Git, you should use ./autogen.sh instead of ./configure, with the same arguments.
   
2. configure, make and install
   

   $ ./configure PYTHON=/c/Python27/python PYTHON_INCLUDES=-I/c/Python27/include/ PYTHON_LIBS='-L/c/Python27/Lib -lpython27' PKG_CONFIG=/c/Python27/Lib/site-packages/gtk-2.0/runtime/bin/pkg-config.exe PKG_CONFIG_PATH=/c/Python27/Lib/pkgconfig:/c/Python27/Lib/site-packages/gtk-2.0/runtime/lib/pkgconfig:/c/mingw/msys/1.0/local/lib/pkgconfig:/c/mingw/msys/1.0/lib/pkgconfig:/c/mingw/lib/pkgconfig am_cv_python_pythondir='${prefix}/Lib/site-packages' am_cv_python_pyexecdir='${exec_prefix}/Lib/site-packages'
   

3. Check that it works by starting Python by adding your MSYS environment site-packages folder to your Python path, starting Python and import dbus.
   

   $ export PYTHONPATH="c:\\mingw\\msys\\1.0\\local\\lib\\site-packages"
   $ python
   Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import dbus
   >>> session_bus = dbus.SessionBus()
   

4. You now have a working dbus-python module. Yay!
   

Links

1. https://bitbucket.org/bradpitcher/mingw-cross-env/src/5f91fb4e0199/src/dbus-1-fixes.patch
2. https://sourceforge.net/tracker/?func=detail&atid=102435&aid=3420424&group_id=2435
3. http://msdn.microsoft.com/en-us/library/ms684208%28VS.85%29.aspx
4. https://bugs.freedesktop.org/show_bug.cgi?id=41423
5. https://sourceforge.net/mailarchive/message.php?msg_id=24260705
6. http://lists.gnu.org/archive/html/mingw-cross-env-list/2011-09/msg00033.html
7. https://github.com/mxe/mxe/commit/d51ad53b664859b38c3cee61f18c4e8e8810fcef
8. https://bitbucket.org/vog/mingw-cross-env/changeset/8bfc6d601bb1

This zip file contains an install script, patches, generated binaries, libraries and doc files.

Do not use it for evil.

Generated using markdown_py 2.2.0

PyDev internal error "loading bundles"

This was caused by some bad updates that were installed. Uninstall PyDev and reinstall it, but unclick the use all websites to find updates radio button. This is what they say on the PyDev issues forum somewhere, I'll look for the link.

Note, this isn't my image, but mine was nearly the same.

Dueling desktops

Update 2012-07-04. Happy Independence Day! I have posted a screenshot of PC-BSD's build of FreeBSD 9.0 - the Isotope edition below. Enjoy!

I've been playing around with virtual machines and different Linux distros and desktops. Pretty fun stuff. I know there are many websites out there that already cover this extensively. For example here's a discourse on the pros and cons of LXDE and Xfce, two popular "lightweight" desktops that are supported by almost every distro, and even BSD. My Geek Opinions - A Geek's Guide to Technology: LXDE vs Xfce. And here's a comparison of the two most popular heavyweights, GNOME (Ubuntu-Unity flavored) and KDE.

But this blog is realy just for me, and when I was deciding what distro to download and what desktop to use, I don't think the differences between distros and more so between desktops, were all that clear to me. So now that I understand it a little better, I don't want to forget it.

First of all I think it's super important to understand the difference between desktops and distros. I laugh a little about it now, but I actually thought Lubuntu, Kubuntu and Xubuntu were different distros. I didn't know that Linux and BSD could swap desktops. I'd only used machines with a terminal and a single desktop. I suppose in undergrad (1990's) I had the opportunity to use X Windows but that was just to display graphics like plots from MATLAB, and was not remotely like Windows 3.1 or the Macintosh OS the most popular graphic desktops at the time.

Suffice it to say that Linux and BSD unlike Windows and Mac, offer a choice of desktops. By far the most popular desktop is by GNOME, and by default the major Linux distribution all default to GNOME. However alternate desktops, such as KDE, LXDE and Xfce, will run on almost any Linux or BSD distribution. In fact, if you're so inclined, you could just use the command line and X Windows to run graphics programs like an internet browser. Wikipedia has a comparison of desktop environments. Wow, that's a pretty cool list - must check out Qt-Razor; there's a Fedora Spin in the works.

So I installed what I think are the most popular Linux distros as VMs: Fedora16, OpenSUSE12 and Ubuntu 12.04LTS (aka Precise Pangolin). I also installed FreeBSD a direct descendant of Unix. Of course there are other Unix-like systems, but I have a a 2-year old and a full time job so there are limits to how much time I can waste. On each system I chose a different desktop - Fedora got LXDE, OpenSUSE got GNOME, Ubuntu got Xfce and FreeBSD got KDE.

Installation
All four systems are a snap to install. All downloads are about 600MB. Fedora and OpenSUSE come as live ISO's that you start, and then have a app to install to the hard drive. Both also have a downloadable DVD image (about 3GB to 5GB) that can be used to install the system directly. Ubuntu on the other hand has a CD ISO that gives you the choice to install Ubuntu or run live. FreeBSD only has ISO for hard disk installation. I did not love having to run the live versions of Fedora and OpenSUSE to start the installation process, since I already knew I would install, so that was annoying. Also, since they run as live CDs, they require at least 512MB of RAM, which was an issue since I was installing them as VMs on pretty old machines, so RAM for me was scarce. VirtualBox couldn't handle it, but an older version of VMware Player was up to the challenge. Fedora and Ubuntu installations were a breeze, but there were some glitches in the graphics for Open SUSE, which may have been hardware related. I have a separate post on FreeBSD, but essentially it uses terminal style graphics, and installs as root, instead of an administrator, who can obtain root privileges.

Software Repository Packages
I installed some of my favorite tools: Numpy/SciPy, pip, virtualenv, virtualenvwrapper, Requests, Meld and Geany. Almost all were supported on every platform directly from the package repository, but at different releases. You can search Fedora, OpenSUSE and Ubuntu repositories online.

Screenshots
You can download and install Xubuntu, which is Ubuntu with the Xfce desktop.

Fedora has a spin for LXDE that you can download and install.

Open SUSE 12 only comes with GNOME or KDE, but there are older version that have LXDE and Xfce.

PC-BSD with FreeBSD 9.0

(Updated 2012-07-04) So far PC-BSD is like FreeBSD but with a nice desktop. The default is LXDE, but you can download XFCE, and others too, I think. I was too lazy to configure X11 correctly, and PC-BSD is a much easier. I was very impressed with the install procedure. One other nice feature is the edition of PBI's, which is like Ubuntu Software Center.

Conclusion

My favorite is still Ubuntu with Unity (GNOME3), but I don't have enough experience to discriminate between the other Linux distros. As far as desktops, I did not like KDE, even on FreeBSD. If I was going to re-install Ubuntu on my oldest laptop, I would use Xcfe, but Unity 2D seems to work just fine.

Package Predicament Part 3: Python, the final installment

[WARNING: Outdated Material] This post is over a year old, and consequently the ideas, opinions and facts may no longer be relevant or accurate (assuming there were actually facts in this post). Please proceed with caution. You have been warned.

So it turns out that there is a long history between Python and Linux (*). Debian has an official Python Package Policy. This is because Python is an integral part of Linux; Linux libraries depend on Python and Python packages. If you go moving them around, you will break your Linux installation.

One of the main differences you will see are the /lib/site-packages folder is renamed /lib/dist-packages on Linux (* just Ubuntu, see CORRECTION), and the existence of /usr/share/pyshared and /usr/lib/pymodules on Linux as well. I'm not going to pretend I understand everything that's going on, but in a nutshell these are Python modules that Linux is using, for something.

Lucky for you, distribute, setuptools, distutils and pip are tailored for Ubuntu so that they will install packages in the correct locations, and also know where to find them. So in general this supports the argument that you should look for your Python packages from the distro repo, especially pip, distribute, setuptools, distutil, and virtualenv. If you install these from your distro repo, you should be OK (**). And above all, "for the love of Guido," use pip, never easy_install!

[CORRECTION: 2012-04-11] (*) After several adventures with VMs (Fedora16, OpenSUSE 12 and even FreeBSD), I have done some experimentation, and some of these Python-peculiarities are only on Ubuntu/Debian distros. For example, there is no dist-packages folder on either OpenSUSE or Fedora. They both use the traditional Python file structure and name conventions such as site-packages. On Fedora it looks like your packages will go in /usr/lib/Python2.7/site-packages, whether you install them with $ sudo yum install python-package or $ sudo pip-python package. Note: on Fedora pip is pip-python, unless you are using it from a virtualenv, then it's just pip. There's nothing in the /usr/local/lib folder on Fedora remotely related to Python, nor /usr/local/share. I didn't get a chance to look for pyshared or pymodules, which are both in /usr/lib on Ubuntu 11.10, but I did notice that there is a lib-dynload in /python2.7.

So what does this all mean? Well, I tried to install numpy with pip in a virtualenv on Fedora, and it still failed miserably (***), see Package Predicament, Part 2. I've research it a little, there are some old bug reports, and several SO questions, but no good answer. I did not try to install it in the base system, but I'm believe it would fail there, just like it did on my Oneiric Ocelot. I think it's a problem with pip, not virtualenv, and I guess the Numpy setup.py or egg. BTW: it also fails on Windows, no surprise; where's my libgcc? Sorry I don't have the complete MinGW installation, only MsysGit. But of course I can install it using the *.exe all-in-one isntaller from Numpy/Scipy website just fine.

[UPDATE: 2013-01-16] (***) Duh, I needed the dev packages, _obviously_. Do yum/apt-get/zypper install libatlas-dev, ... and make sure you have gfortran. Numpy, SciPy and Matplotlib all build fine with pip in a virtualenv, once you have the proper dev files. Now Windows is an entirely different story, but it can also be done.

[UPDATE: 2012-04-17] (**) You can seriously f*** up your sh** if you start messing around with your distro's version of distribute, distutils or setuptools. For example, Unity's Software Center depends on these to install packages. If you screw up your version of setuptools, Software Center won't even open. My advice is to (1) use virtualenv for any package that you need that differs from your repo's version. For example python-requests is version 0.5 in Oneiric Ocelot, but newest version of Requests is (as of now) 11.1, so you should create a virtualenv and install it there. (2) Only pip Python packages that are pure Python, that are not in your repo, and do not let them install dependencies that are already installed by your repo. For example, Requests now requires chardet >=1.0.0, which because Ubuntu's version of chardet is named oddly 2.0.1-2 causes pip to replace it with the exact same code. Probably fine, but not smooth. (3) If your desired package has dependencies that already exist in the repo, then use virtualenv and install them there. (4) Put a simlink to packages that require compiled code, such as Numpy in your virtualenv site-packages folder.

Virtually vapid

[UPDATED 2016-02-23] Now I'm swinging back to VirtualBox on both systems as the overall King of VM. Unfortunately VMWare Player 7 on my Windows machine was freezing up randomly and at one point just refused to work. It turns out that there have been a lot of changes with VirtualBox in the last 4 years, and now not only does VirutalBox include hardware virtualization but it also has valid certs for Windows. Previously there were some scary messages along the lines of, "use at your own risk," which obviously goes without saying so ... why were they saying it? Anyway, the newest VirtualBox 5 is super fast and so far stable. In addition to these obvious performance perks, the fact that VirtualBox is really free, as in FOSS, means that there are tons of great features like cloning and snapshots, and the license allows commercial use with no restrictions. The VMWare player restricts VMWare Player to non-commercial use only, and it was a free lite version of the more powerful VMWare Workstation, so lite on features. So, for now, I'm really happy!

I did it. I installed a Windows VM on Ubuntu using VirtualBox and Ubuntu VM on Windows using VMware 3.1.5 (the PC was too old for 4.0.2). VirtualBox was a breeze and fast, but VMware and Ubuntu took forever (*). You have to install VMware Tools (**) which takes like a million years. It's probably my ancient laptop and the out-of-date VM, but that was the best I could do for now. What's the point? There is no point, it's just cool.
P.S. 99% of problems are solved by cycling vm, shut it down; not sleep.
P.P.S. "user$ sudo ./vmware-install.pl"  (**) since you need root access and your extracted vmware-tools folder is not on your path. "user$ sudo ./bin/vmware-uninstall.pl" to remove.
Update (2012-03-21): (**) Preferred method is to use open-vm-tools from package manager since it updates with your Linux distro. Use apt-get, Synaptic or Ubuntu Software Center. In other words do not use VMware Tools.
Update (2012-03-22): After installing your guest on VirtualBox VM, install guest-additions, to get video, mouse and other drivers. If you're on Ubuntu, use the iso image in the Software Center.
Update (2012-03-23): (*) OK, I've changed my opinion a little bit, after spending an embarrassing number of hours updating from XP to SP3 for the guest VM on the Linux host. What I'm getting at is that installing the guest Linux VM was a lot faster, even though the initial XP install was pretty fast, getting the updates is painfully slow. This is a good time to make a bullet list of lessons learned:

1. VMware >3 only works on machines with Hardware Assisted Virtualization like Intel VT-x.
2. VirtualBox works on anything.
3. If installing a guest Linux VM using VMware, do not download VMware Tools from the internet. Instead use package manager (apt-get, Synaptic or Ubuntu Software Center) to install open-vm-tools package.
4. If installing VirtualBox on Linux host, use package manager and install virtualbox-qt and virtualbox-guest-additions-iso (which is the VirtualBox equivalent of VMware Tools).
5. For best performance, install on a system with at least 2GB of RAM and >2 core processor. Although it will work on a single core with 1GB, you noticeably see performance suffer, and you make experience system hangs or BSOD.
6. If installing a guest Windows XP VM, turn off automatic updates and instead use Windows XP Service Pack 2 Network Installation Package for IT Professionals and Developers followed by Windows XP Service Pack 3 Network Installation Package for IT Professionals and Developers. Otherwise you will spend a very long time waiting for downloads and installing them.
7. Do not overtax your host machine during critical guest installations, e.g. installing OS. Think of rooting your phone - although the stakes are not as high, you want everything to go alright so you don't have to repeat the entire process on a new VM. It's also a good idea to kill your screensaver/lock/sleep feature on your host during crucial installs, otherwise your hard-drive might turn off.
8. After installing guest OS, don't forget to run either open-vm-tools for VMware from the package manager or virtual-box-guest-additions-iso for VirtualBox from the devices menus option on your VirtualBox VM after you've downloaded it package manager.