Friday, December 28, 2012

more msvc express cheats

To build the INSTALL project you can use devenv which is really just a shortcut to msvc.

Visual Studio C++ 2008 Express (MSVC90)


On express devenv is renamed vcexpress, but does it have the same functionality? Here is the help.

vcexpress  [solutionfile | projectfile | anyfile.ext]  [switches]

The first argument for devenv is usually a solution file or project file.
You can also use any other file as the first argument if you want to have the
file open automatically in an editor. When you enter a project file, the IDE
looks for an .sln file with the same base name as the project file in the
parent directory for the project file. If no such .sln file exists, then the
IDE looks for a single .sln file that references the project. If no such single
.sln file exists, then the IDE creates an unsaved solution with a default .sln
file name that has the same base name as the project file.

Command line builds:
devenv solutionfile.sln /build [ solutionconfig ] [ /project projectnameorfile [ /projectconfig name ] ]
Available command line switches:

/Log Logs IDE activity to the specified file for troubleshooting.
/ResetSettings Restores the IDE's default settings, optionally resets to
the specified VSSettings file.
/SafeMode Launches the IDE in safe mode loading minimal windows.

Product-specific switches:

/debugexe Open the specified executable to be debugged. The
remainder of the command line is passed to this
executable as its arguments.
/useenv Use PATH, INCLUDE, LIBPATH, and LIB environment variables
instead of IDE paths for VC++ builds.

To attach the debugger from the command line, use:
VsJITDebugger.exe -p <pid>


VCBUILD is also available, but I don't know exactly what the difference is.

C:\Program Files\Microsoft Visual Studio 9.0\VC>vcbuild /?
Microsoft (R) Visual C++ Project Builder - Command Line Version 9.00.30729
Copyright (C) Microsoft Corporation. All rights reserved.

Usage: vcbuild [options] [project|solution] [config|$ALL]

/clean (/c)     Clean build outputs only
/error:<str>    Prefix to add to error lines on output to stderr
/errfile:<file> Log all errors to the file specified
/htmllog:<file> Logs output to html file specified
                (default: $(IntDir)\BuildLog.htm)
/implib         Makes an import library for a DLL configuration (does not link)
/info:<str>     Prefix to add to information lines on output to stdout
/link (/l)      Performs a link without building sources
/logcommands    Prints commands and response files to screen
/logfile:<file> Log all output and build information to the file specified
/override:<file>Override the project settings with the settings in the given
                property sheet file

/M<number>      Specifies the number of concurrent builds to run, if possible
/msbuild:<opt>  Pass <opt> to msbuild.exe

/nocolor        Do not output error and warning messages in color
/nohtmllog      Do not write an html build log file

/noimplib       Does not generate an import library.
                NOTE: this option overrides /implib.
/nologo         Suppress version and copyright message
/nondefmsbuild  Do not use the copy of msbuild.exe located in the .NET
                Frameworks installation
/platform:<str> Build only configurations for the given platform
/implibobjs:<s> Additional dependencies for the librarian
/rebuild (/r)   Clean build outputs and perform a build
/forcelink      Forces a link without building sources
/showenv        Show environment in the html build log
/time           Times the build from start to finish
/upgrade        Upgrades the project file to the latest format supported
                NOTE: the upgrade switch does not perform a build
                NOTE: this option is ignored for solution files
/useenv (/u)    Use environment variables for INCLUDE and LIB paths
/overrideRefVer When upgrading, assume .NET Framework Version 3.5 for any assemb
ly references.
                Requires /upgrade switch.
/wrnfile:<file> Log all warnings to the file specified
/warning:<str>  Prefix to add to warning lines on output to stdout
@<file>         Read options from the specified response file

Default Behavior:
If no project is specified, and there is only a single .vcproj file in the
directory, that project will be built.
If no configuration is specified, and the VCBUILD_DEFAULT_CFG environment
variable is set, the configuration it specifies will be built. If it is not
set, then all configurations will be built.

The default action is to build the specified configurations without cleaning.

Options will also be read from the VCBUILD_DEFAULT_OPTIONS environment

Visual Studio C++ 2010 Express (MSVC100)

vcexpress = devenv

On express devenv is renamed vcexpress, but does it have the same functionality? Here is the help.

vcexpress  [solutionfile | projectfile | anyfile.ext]  [switches]

The first argument for vcexpress is usually a solution file or project file.
You can also use any other file as the first argument if you want to have the
file open automatically in an editor. When you enter a project file, the IDE
looks for an .sln file with the same base name as the project file in the
parent directory for the project file. If no such .sln file exists, then the
IDE looks for a single .sln file that references the project. If no such single
.sln file exists, then the IDE creates an unsaved solution with a default .sln
file name that has the same base name as the project file.

Command line builds:
vcexpress solutionfile.sln /build [ solutionconfig ] [ /project projectnameorfile [ /projectconfig name ] ]
Available command line switches:

/Log Logs IDE activity to the specified file for troubleshooting.
/ResetSettings Restores the IDE's default settings, optionally resets to
the specified VSSettings file.
/SafeMode Launches the IDE in safe mode loading minimal windows.

Product-specific switches:

/debugexe Open the specified executable to be debugged. The
remainder of the command line is passed to this
executable as its arguments.
/useenv Use PATH, INCLUDE, LIBPATH, and LIB environment variables
instead of IDE paths for VC++ builds.

To attach the debugger from the command line, use:
VsJITDebugger.exe -p <pid>


On MSVC100, VCBUILD seems to be renamed to MSBUILD, which is what actually runs even on mcvs90, when you run VCBUILD anyway.

Microsoft (R) Build Engine Version 4.0.30319.1
[Microsoft .NET Framework, Version 4.0.30319.269]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Syntax:              MSBuild.exe [options] [project file]

Description:         Builds the specified targets in the project file. If
                     a project file is not specified, MSBuild searches the
                     current working directory for a file that has a file
                     extension that ends in "proj" and uses that file.


  /target:<targets>  Build these targets in this project. Use a semicolon or a
                     comma to separate multiple targets, or specify each
                     target separately. (Short form: /t)

  /property:<n>=<v>  Set or override these project-level properties. <n> is
                     the property name, and <v> is the property value. Use a
                     semicolon or a comma to separate multiple properties, or
                     specify each property separately. (Short form: /p)

  /maxcpucount[:n]   Specifies the maximum number of concurrent processes to
                     build with. If the switch is not used, the default
                     value used is 1. If the switch is used without a value
                     MSBuild will use up to the number of processors on the
                     computer. (Short form: /m[:n])

                     The version of the MSBuild Toolset (tasks, targets, etc.)
                     to use during build. This version will override the
                     versions specified by individual projects. (Short form:

  /verbosity:<level> Display this amount of information in the event log.
                     The available verbosity levels are: q[uiet], m[inimal],
                     n[ormal], d[etailed], and diag[nostic]. (Short form: /v)

                     Parameters to console logger. (Short form: /clp)
                     The available parameters are:
                        PerformanceSummary--Show time spent in tasks, targets
                            and projects.
                        Summary--Show error and warning summary at the end.
                        NoSummary--Don't show error and warning summary at the
                        ErrorsOnly--Show only errors.
                        WarningsOnly--Show only warnings.
                        NoItemAndPropertyList--Don't show list of items and
                            properties at the start of each project build.
                        ShowCommandLine--Show TaskCommandLineEvent messages
                        ShowTimestamp--Display the Timestamp as a prefix to any
                        ShowEventId--Show eventId for started events, finished
                            events, and messages
                        ForceNoAlign--Does not align the text to the size of
                            the console buffer
                        DisableConsoleColor--Use the default console colors
                            for all logging messages.
                        DisableMPLogging-- Disable the multiprocessor
                            logging style of output when running in
                            non-multiprocessor mode.
                        EnableMPLogging--Enable the multiprocessor logging
                            style even when running in non-multiprocessor
                            mode. This logging style is on by default.
                        Verbosity--overrides the /verbosity setting for this

  /noconsolelogger   Disable the default console logger and do not log events
                     to the console. (Short form: /noconlog)

  /fileLogger[n]     Logs the build output to a file. By default
                     the file is in the current directory and named
                     "msbuild[n].log". Events from all nodes are combined into
                     a single log. The location of the file and other
                     parameters for the fileLogger can be specified through
                     the addition of the "/fileLoggerParameters[n]" switch.
                     "n" if present can be a digit from 1-9, allowing up to
                     10 file loggers to be attached. (Short form: /fl[n])

                     Provides any extra parameters for file loggers.
                     The presence of this switch implies the
                     corresponding /filelogger[n] switch.
                     "n" if present can be a digit from 1-9.
                     /fileloggerparameters is also used by any distributed
                     file logger, see description of /distributedFileLogger.
                     (Short form: /flp[n])
                     The same parameters listed for the console logger are
                     available. Some additional available parameters are:
                        LogFile--path to the log file into which the
                            build log will be written.
                        Append--determines if the build log will be appended
                            to or overwrite the log file. Setting the
                            switch appends the build log to the log file;
                            Not setting the switch overwrites the
                            contents of an existing log file.
                            The default is not to append to the log file.
                        Encoding--specifies the encoding for the file,
                            for example, UTF-8, Unicode, or ASCII
                     Default verbosity is Detailed.


  /distributedlogger:<central logger>*<forwarding logger>
                     Use this logger to log events from MSBuild, attaching a
                     different logger instance to each node. To specify
                     multiple loggers, specify each logger separately.
                     (Short form /dl)
                     The <logger> syntax is:
                       [<logger class>,]<logger assembly>[;<logger parameters>]
                     The <logger class> syntax is:
                       [<partial or full namespace>.]<logger class name>
                     The <logger assembly> syntax is:
                       {<assembly name>[,<strong name>] | <assembly file>}
                     The <logger parameters> are optional, and are passed
                     to the logger exactly as you typed them. (Short form: /l)

                     Logs the build output to multiple log files, one log file
                     per MSBuild node. The initial location for these files is
                     the current directory. By default the files are called
                     "MSBuild<nodeid>.log". The location of the files and
                     other parameters for the fileLogger can be specified
                     with the addition of the "/fileLoggerParameters" switch.

                     If a log file name is set through the fileLoggerParameters
                     switch the distributed logger will use the fileName as a
                     template and append the node id to this fileName to
                     create a log file for each node.

  /logger:<logger>   Use this logger to log events from MSBuild. To specify
                     multiple loggers, specify each logger separately.
                     The <logger> syntax is:
                       [<logger class>,]<logger assembly>[;<logger parameters>]
                     The <logger class> syntax is:
                       [<partial or full namespace>.]<logger class name>
                     The <logger assembly> syntax is:
                       {<assembly name>[,<strong name>] | <assembly file>}
                     The <logger parameters> are optional, and are passed
                     to the logger exactly as you typed them. (Short form: /l)

  /validate          Validate the project against the default schema. (Short
                     form: /val)

  /validate:<schema> Validate the project against the specified schema. (Short
                     form: /val)

                     List of extensions to ignore when determining which
                     project file to build. Use a semicolon or a comma
                     to separate multiple extensions.
                     (Short form: /ignore)

                     Enables or Disables the reuse of MSBuild nodes.
                     The parameters are:
                     True --Nodes will remain after the build completes
                            and will be reused by subsequent builds (default)
                     False--Nodes will not remain after the build completes
                     (Short form: /nr)

                     Creates a single, aggregated project file by
                     inlining all the files that would be imported during a
                     build, with their boundaries marked. This can be
                     useful for figuring out what files are being imported
                     and from where, and what they will contribute to
                     the build. By default the output is written to
                     the console window. If the path to an output file
                     is provided that will be used instead.
                     (Short form: /pp)

                     Shows detailed information at the end of the build
                     about the configurations built and how they were
                     scheduled to nodes.
                     (Short form: /ds)

  @<file>            Insert command-line settings from a text file. To specify
                     multiple response files, specify each response file

  /noautoresponse    Do not auto-include the MSBuild.rsp file. (Short form:

  /nologo            Do not display the startup banner and copyright message.

  /version           Display version information only. (Short form: /ver)

  /help              Display this usage message. (Short form: /? or /h)


        MSBuild MyApp.sln /t:Rebuild /p:Configuration=Release
        MSBuild MyApp.csproj /t:Clean

Wednesday, December 5, 2012

Download sites for old, free MS compilers and IDEs

[UPDATE 2015-03-13] I do not recommend installing VS2010 Express (or Professional). This program is very difficult to remove and has become obsolete, with VS2013. If you are stuck with this on your computer you may be able to remove it by following these steps.
  1. Make sure you have copies of the VS2010 SP1 installer. The web installer is fine, but the iso may come in useful. Don't bother creating a manual system restore point. Even when you roll back changes, the issues that cause the majority of problems are not resolved. However I guess it can't hurt either. If you really want to cover your bases, the best thing would be to make a recovery disk and an disk image. This way you can roll back changes if they go awry.
  2. If you have installed VS2010 SP1, you must remove this first. You should be able to remove it from add/remove programs in the control panel. If not try
    • C:\ProgramData\VS\vs2010sp1\SetupCache\setup.exe /uninstall /force
    • Download the installer and run VS10sp1-KB983509.exe /uninstall /force
    • Download and extract the iso image and run setup.exe /uninstall /force
    • Reinstall SP1 and then try uninstalling it from add/remove programs in control panel. You may need to reinstall VS2010 VC10 or VCS10 before SP1.
  3. Remove VS2010 Tools for Office Runtime and VS2010 ADO.NET Entity Framework. You may need to remove these before SP1.
  4. Use the VS2010 uninstall utility documented first here then later here. Make sure you use the options in the post /full to remove everything!
Note this took many iterations to actually get all of VS2010 off of my system, so be prepared for frustration and irritation, but persevere and it can be done. Just make sure you have copies of the installer so that you can reinstall and try to remove everything in the correct order again! You may have an issue removing or reinstalling Windows SDKs. If so see Upgrading to Visual Studio 2013.

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

[UPDATE 2013-10-03, 2014-04-16] Sad to report that clicking on #1 MS Visual Studio 2008 Express states that it has been retired. Oh, yes! VS2008 Express didn't allow targeting x64 platforms, without some serious hoop-jumping, but VS2010 Express does, and VS2010 also lets you select the V90 toolset, so building shared objects for Python-2.7 is no problem. Also, SDK7 has the exact same compilers as V90, and it is still supported so they are more current. Ditto for SDK7.1 & V100. Oh no! Does this mean we finally have to switch to Python 3? If you really want VS2008, the web installer does however still work. Grab it from my dropbox.

  1. Microsoft Visual Studio 2008 Express Editions with SP1: This one is necessary for python 2.7.3 extensions. You can't get it from the Visual Studio page anymore, now that they've moved on to 2012, but download links for vcsetup.exe which install MSVC90 Express are in the Microsoft Download Center, as are vcssetup (C#), vbsetup.exe (vbs) and vwsetup.exe (web). You also get the opportunity to install SQL Server 2008 and Silverlight SDK. Visual Studio 2008 Express has been discontinued, and is obsolete because Visual Studio 2010 Express includes the V90 toolset. See update note at top of page. Not sure if Visual Studio 2008 SP1 download will install for free, but it may. VS2008 is not required to build Python-2.7 extensions, instead use SDK7 which replaces the v90 compilers.
  2. Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1: You cannot build 64-bit applications with the free version of VC90 unless you have Windows SDK 7. Note this *not* 7.1, which is VC100, also note that .NET Framework 3.5 is part of Windows 7, so you don't have to download it separately, and of course make sure that you get all updates before installing. Specifically you must have SP1. The paths in vcvarsall.bat are incorrect, you will need to fix them to use pip with v90. Otherwise use sdk7 shell and follow the directions in the post on installing Python x64 extensions with pip
  3. Visual Studio 2010 Express is still available from the main VS site. They have 2012 and 2010, and all of the flavors.
  4. Microsoft SDK 7.1 has some issues with VC100. Basically you should follow this procedures:
    1. Visual Studio 2010 RTM
    2. Windows SDK 7.1
    3. Visual Studio 2010 SP1
    4. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1
    See these posts:
    Setup & Install by Heath Stewart
    Visual C++ Team Blog
    FIX: Visual C++ compilers are removed ...
  5. Microsoft Windows SDK for Windows 7 and .NET Framework 4
  6. Beware here, you might run into this error, which is very confusing, the installer quits with the message "Installation of the “Microsoft Windows SDKfor Windows 7” product has reported the following error: Please refer to Samples\Setup\HTML\ConfigDetails.htm document for further information." Totally useless, but further examination of the log file or googling around you might see the real source of the error, it's not SP1 but the redistributable that is too new: "C:\Program Files\Microsoft SDKs\Windows\v7.1\Setup\SFX\vcredist_x64.exe installation failed with return code 5100". The solution is to remove the redistributables and then reinstall them later. See this knowledge base article:
    Windows SDK Fails to Install with Return Code 5100
  7. Upgrade to Microsoft Visual Studio 2010 SP1. See this link for what's in SP1.
    Description of Visual Studio 2010 Service Pack 1
  8. Download the update after you follow the procedures here:
    Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1

Wednesday, November 28, 2012

Eclipse in ubuntu

There are so many posts out there on installing eclipse on Ubuntu, that I decided to take the plunge. I had been troubled by having to manage two repos, but since I mainly use PyDev and Android SDK, neither of which are in the Ubuntu repository, so it was either deal or try to go it alone. I'm happy to report that eclipse 4.2, aka juno, is running great in Quantal, no errors so far!

With Oneiric and Precise I had successfully used both Ubuntu and eclipse repositories, but with each upgrade there were always snafus; I would have to re-install some plugin and end up losing my workspace settings. Playing the repo shuffle, you never know when Ubuntu might suddenly decide to revoke or add a package, and there are so many eclipse plugins it just makes sense to me to let one system manage the dependencies. Reading up a bit here and there I began to realize that this is an issue eclipse and Linux users have addressed and that eclipse would work out of the box on Ubuntu.

So the first thing I did was completely remove eclipse from my system. I had some false starts, first off Synaptic may be a great tool but it does not remove certain dependencies the same way as the Ubuntu Software Center (USC). Same goes for `apt-get`. Too bad because using USC is slow and tedious because you can only install/remove one package at a time. But unless you want to leave a lot of detritus on disk USC does a better job at killing off orphans. That made me realize another reason for installing eclipse as add-on software, since using the Ubuntu repo installed over 100 packages! I'm trying to keep my machine uncluttered and organized so that's just ridiculous.

I did a few things different than the vast number of sites posting instructions.
  1. I created a folder called "/opt" in my home directory and untarrred my eclipse download there. I chose the Java EE version for 32-bit Linux because it has the most complete selection of useful plugin preinstalled. I decided on ~/opt instead of /opt because I'm the only user and I didn't want to deal with ownership and permissions. This is a nice compromise that doesn't clutter your system. However, in general I would recommend copying the entire package to /opt and changing ownership to root:root.
  2. I created a soft symlink to ../opt/eclipse/eclipse called "eclipse" in ~/bin. This folder is already present and is always first on your path. If I had moved eclipse to /opt, then it would be available to all users, and then each user could create a symlink from there to their private ~/bin folder. I didn't have to change the permissions, because it's already 777.
  3. This next step is the major difference between what I did and what I've seen posted everywhere. I created a desktop file in ~/.local/share/applicatons called "eclipse.desktop" with the following. I looked into both gnome-panel and alacarte (ne main menu), but I ruled them out because the desktop file is so easy to create, and both of these tools are for true gnome desktops not Unity, and also seemed outdated. There is ample help on putting applications in the desktop. A couple of notes: ~ is not expanded, so use your complete path, X-Ayatana is the name of the Unity group, if you want to add extra commands and I had to export the eclipse xpm icon as a png file because it didn't work in the Unity dash.

    [Desktop Entry]

  4. I started eclipse from the dash and pinned it to the launcher. Then followed the customary procedures to update and install packages from within eclpse. From the menu select help > check for updates, and then help > install new software. Note: I have started help using the internal browser (default) and it works fine. My system already has both libwebkitgtk-1.0.0 and libwebkitgtk-3.0.0 so I didn't have to install or remove anything; eclipse help just worked out of the box. I have read numerous reports regarding this issue (eclipse crashes when help window opened), so evidently it has been solved.
So there you go. If you are on the fence, and tired of managing two repositories, I hope I've convinced you that it's okay to install eclipse as an add-on and it will work fine, and not break your system.

Monday, November 26, 2012

Install Add On Software and Create GTK+ Desktop File

Unfortunately not all of the software that you want will be in your linux distro. The most common examples of these are the awesome Sublime Text 2 editor and the bloated but ubiquitous eclipse IDE. Actually eclipse is in most linux distro repositories, but only a limited set of plugins are available, and eclipse uses its own repository system to update and install plugins which is in principle incompatible with your distro's packaging system. For example, if you install eclipse 3.8 from the Ubuntu Quantal Quetzal repo, you can not update it from the eclipse repository, and if you install the PyDev plugin using their repository through the eclipse packaging system, when your upgrade to the next Ubuntu version, your plugins will not work and you will have to reinstall them. If you are okay with this, then you may be alright, but you will have some extra work to do come upgrade time, and you will need to keep track of what is installed via the official repo and what is being maintained by eclipse's repos, because neither can keep track of the other, and they are all for the most part dumped in the /home/<username>/.eclipse/ directory.

Linux actually has a protocol for installing software add-ons, if you decide that what's in your distro's repository is not enough for you. There are a few options.

  1. Find a PPA (Personal Package Archives for Ubuntu), add it and then install the package with apt-get. This will most likely install the package in /usr/, /usr/bin/, /usr/lib/, ... and take care of configuration and creating a desktop file so that you the application will appear in the unity dash (on Ubuntu) and you can add it to the launcher or launch files from Nautilus. This works for Sublime Text 2. PPA's are often listed on Launchpad.
  2. Build the code from source using autotools and it should install in /usr/local/, /usr/local/bin/, /usr/local/lib/, /usr/local/include/, ... but of course you will need to make sure that the application's dependencies are all satisfied and do not conflict with your distro. This can be a tough row to hoe.
  3. Find a pre-compiled package and drop it into your system's /opt directory, which is designated specifically for add-on software not provided by the system's package repositories. An alternative often used and actually my preference is to extract the new package into your home directory.  This works, but to avoid cluttering $HOME I add a folder called ~/opt and drop the add-ons in there. Again, it is up to you to make sure that the software's dependencies are satisfied. For example, eclipse requires some jdk, whether it is from Oracle or the openJDK project, and it will need to be a version within the range specified by the add-on software, e.g. jdk >= 7.0 in the case of eclipse.
Now you need to integrate the add-on software into your system. Again you have some choices and it seems that there is some debate about what is the best course to take.
  1. The easiest thing to do is to run the application from the console, and don't integrate it into your desktop (Gnome) or file system browser (Nautilus), by opening a terminal window and typing ./<executable>. You may need to change the permissions first by using sudo chmod 755 <executable>.
  2. Another option that doesn't require you to edit your desktop files is to add the folder to your path, however, this may cause problems if there are filename conflicts, e.g. your system already has a libz.a but your new software does too. One safe way to deal with this is to add  the new directory to the end of your path, or create a special shell for it. This way your default system files will always be used.
  3. Create a symlink to /home/<username>/bin/ which is always first on your path. This is my preferred method for several reasons. First it will survive an update because your personal profile is not altered during upgrades. Second, it only affects you, and not other users if you share your machine. Of course if your intent is to make the application usable for all users, then either you need to add it to all profiles or make a symlink to /usr/bin/. However, links in /usr/bin/ will probably not survive an upgrade because this is system file territory
  4. Create a gtk desktop file in /home/<username>/.local/share/applications/, as described in Desktop files: putting your application in the desktop menus, which again is my preferred method. I wrote a post on eclipse in Ubuntu that has an example of a desktop file. The reasons are the same as above. First it survives an upgrade and second it only affects you. You can customize the desktop file to have as many actions as you want, for example to have an option to open a new window by right clicking directly from the launcher icon, or to list all of your current eclipse profiles so that you can launch them directly. The desktop file is very flexible. You should now be able to find your application in the Unity dash (on Ubuntu) and once it runs you can lock it to the launcher. Also if you add the %f option after the location of the executable in the desktop file, then it will be available to Nautilus as well. Then you can right click and see other applications that can open a file associated with your new application. See this Ask Ubuntu answer on how to install the Sublime Text 2 editor. A lot of people have recommended editing the system wide desktop files, which are in /usr/share/applications/ but I don't recommend this as it will not survive an upgrade.
In general, I tend to be more conservative when it comes to editing system files and directories, preferring to keep my customization localized to my profile, and only deviating from that when absolutely necessary. But you may have a different use case which will necessitate a different approach.

Monday, November 19, 2012

PyGTK with Christoph Gohlke and GTK all in one bundle for x64 systems

This is very straightforward.

  1. Download and install PyGTK, Py2Cairo and PyGObjects from Christoph Gohlke's website. Note, do not install pycairo, as Gohlke clearly states it is not compatible with PyGTK.
  2. Create a folder called "runtime" in `C:\Python27\Lib\site-packages\gtk-2.0`.
  3. Download and install the GTK all in one bundle for 64-bit systems from Gnome. Make sure that it is the same version as PyGTK that you installed from Christoph Gohlke's site, which was version 2.22 upon writing.
  4. Extract the GTK files to the "runtime" folder created in step 2.
  5. Add the runtime folder to your python path, and/or the path environment variable of any shells that you want to have access to GTK
Note: It is not strictly necessary for the location of the runtime folder in step 2 to be in the Python27 file system, it could be in your` /home/bin`, `/local/bin`or `C:\` directory or where ever you want it, as long as it is on your path.

Python x64 Package Extensions with pip, MSVC 2008 Express & SDK 7

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

[UPDATED 2012-02-04, 2014-04-18] You can skip this if you fix your vcvarsall.bat to point at the correct paths.

If you try `user@machine ~$ pip install package` from a bash shell or even `C:\Users\user>C:\Python27\Scripts\pip.exe install package` on a 64-bit MS Windows machine with the 64-bit version of Python installed, then you are likely to get a traceback from `` that `vcvarsall.bat` only returned `path` but it was also looking for `lib`, `libpath` and `includes`. Oh well.

Because you know that your version of Python 2.7 was compiled with MS Visual C++ 2008 (aka VC90), you might try to use pip in the VC90 command prompt, since it loads the variables for you, but then you will get the following linker error:
fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'
In a virgin cmd.exe shell you might try to run vcvarsall.bat amd64 to see if it works, but then you will see that it doesn't, when you get this discouraging stifling message:
The specified configuration type is missing.  The tools for the configuration might not be installed.
Then you might put this into Google and eventually you will find that MVSC 2008 Express doesn't support 64-bit compilation without Windows SDK 7 (see this post to find out where you can download this). Luckily, that has it's own nifty environment, run it from the start menu and change to 64-bit compilers and release mode by typing ...
> setenv /x64 /release
... and watching the screen font color change from yellow to green. Fun! Distutils would also like you to tell it not to bother looking around for vcvarsall, and just use the SDK environment by setting two environment variables:
> set MSSdk=1
Also, you will need to have both Python27 and Python27\Scripts on your path before you run the SDK 7 CMD shell, or distutils will still fail. Unfortunately you can't add this by editing the path after starting the shell. The easiest way to do this is to right click on the My Computer icon on the desktop, select the Advanced tab and then click the Environment Variables button near the bottom. This will bring up the Environment Variables window. Add a new user variable for your personal profile by clicking the New button in the upper pane, and then type PATH in the Variable name: box and C:\Python27;C:\Python27\Scripts in the Variable value: box. Note the semicolon in between the 2 paths, if you've never edited your path before, this is the path separator on MS Windows. Then click OK, OK and OK. The changes are immediate for new CMD shells. Your local PATH is appended to the end of the system PATH.

Now try `pip install package` again and if you're lucky, voila, pip says, "Sucessfully installed package". Thanks pip! I tried this with dulwich, and it installed no problem.

There is some nifty info in the Python distutils docs and the Cython folks also have some nice 64-bit extensions help too.

Friday, November 16, 2012

Google drive hog

Hard to believe but every time Google drive windows app syncs it creates a 34mb temp directory, with a partial python library! It quickly adds up. Screw that, uninstall! How come dropbox has no issues?

Friday, November 9, 2012

Shell binaries needed for virtualenvwrapper

[UPDATED 2013-02-07]

You will need fmt from coreutils, make and mktemp. You also need iconv and intl libraries. Your best bet in Git bash is to install base MSYS* (part of MinGW) and then copy or link those files from /c/mingw/msys/1.0/bin/ to your Git bash home bin folder ~/bin since it will always be first on its PATH.

Do not use GnuWin32 because mktemp is broken (see bug), you don't need all 99 coreutils binaries (just fmt) and ideally you want binaries compiled for a POSIX environment (Git bash is built with MSYS),  although the gnuwin32 make binary does work in Git bash, and so does fmt. See my post for the differences between MinGW, MinGW-w64 and GnuWin32.

Coincidentally, make will also be useful for building HTML and pdftex from Sphinx rst files.

*Since, like many other MS Windows users, I've been migrated to Windows7 x64, I now recommend using MinGW-w64 instead of MinGW (aka mingw32). See my post on setting up MSYS home for virtualenvwrapper for more info.

Wednesday, November 7, 2012

Sphinx make latexpdf for MS Windows

Sphinx is an amazing documentation tool. I was up and running with gorgeous html in minutes! But generating a pdf took a little longer to figure out because IMHO on Windows using any open source is always a challenge.

  1. Install Sphinx (1.1.3) - just use pip, easy_install, or setup, whatever you prefer.
    • user@user ~$ pip install sphinx
  2. This will probably also install Pygments and Jinja-2, I can't remember.
  3. Install MikTex package repository binary. This takes care of downloading and updating your LaTeX binaries. Alternately install TeX Live and Lua TeX Windows binaries. I haven't tested this so not sure how well it works, but essentially should be the same as MikTeX since it's the same source.
  4. You will need a working make binary. The easiest one is from GNUwin32, but I would recommend installing MSYS either from MinGW or MinGW-w64. Alternately you could use Cygwin, but that's a different ballgame.
  5. After you've run sphinx-quickstart from the tutorial, type make latexpdf and voila. If you used MikTeX, it will ask you to install several LaTeX packages.
Another option is to go straight to pdf with rst2pdf. First install reportlab binary from the cheese factory. Then use pip to install rst2pdf.

Tuesday, October 9, 2012

Compile binutils on your phone with C4Droid

1. Make your sdcard executable by remounting it as rw, and therefore removing noexec flag.
2. After wget binutils-2.22.tar.gz and extracting it, change shebangs in configure, missing, install-sh and mkinstalldirs to #!/system/bin/sh.
3. I made a shell script in /data/local/bin/ that creates TEMP and TMP environmental variables equal to GCCROOT/tmpdir and then runs arm-linux-androideabi-gcc with any args passed. You could also use a symbolic link or add GCCROOT to your path. Either way do the same for make, ar, as, ld and strip.
4. Run ./configure --prefix=/data/local/ SHELL=/system/bin/sh MAKE_SHELL=/system/bin/sh CFLAGS=-fno-short-enums
5. Forgot to mention patches. (a) unistd.h getpagesize macro isn't detected, so comment out redundant definition in getpagesize.c (b) time returns time_t pointer so cast it from int in archive.c (c) bucomm.c calls mkdtempwhich is never declared in bionic so add extern char * mkdtemp(char * path) to bucomm.h (d) there's one other type mismatch but I can't remember where.
6. Run make and make install and Viola!

How to remove noexec flag from your sdcard

Run mount -o remount, rw /mnt/sdcard/. This will also work on any external storage,  Even an internal mmc card. It won't survive a restart. You could try to modify your vold fstab file. Or add the commands to your .bashrc or profile and reverse it in .bashlogout. You may need to use /storage/sdcard0 or sdcard1 depending what ROM you are running.

Note this will not work on a fuse file system, which unfortunately is the new norm on many new smartphones (EG: Motorola RAZR HD).

Do Not Format Android External Storage

Anything but vfat! If you do Android will not mount any of your external storage and consequently any apps on York sdcard will be unusable until you reformat the offending card, which could actually be internal, back to vfat fs. Unfortunately busybox mkdosfs is broken so the only way to do this without advanced is to use newfs_msdos which just works. This means that you will never be able to make hard or symbolic links from or on your sdcard.

Tuesday, September 18, 2012

SL4A dialogCreateInput vs dialogGetInput

It should seem totally obvious but the main differences between these two are summarized below:


  1. dialogCreateInput also requires dialogGetResponse to get the response inputted into the dialog box. The result attribute of dialogCreateInput is always None, and the actual response is in the result attribute of dialogGetResponse.
  2. dialogCreateInput lets you select the input type: "text", "number", etc. which will also determine the type of soft keyboard used. A number pad is shown for "numbers"
  3. dialogCreateInput does not have any buttons, so the user must use the back arrow or home key to escape if no code is used to dismiss the dialog. The obvious solution would be to use the return key.


  1. dialogGetInput returns the input in result attribute. the window
  2. dialogGetInput does not you select the type of input.
  3. dialogGetInput has OK and Cancel buttons to close
SL4A has a UI help page which mentions the difference between these two, but it still might not be immediately obvious. There is some detail under the heading: some notes right before the API references table.
In addition to the above functions, dialogGetInput and dialogGetPassword are convenience functions that create, display and return the relevant dialogs in one call.
There is only ever one instance of a dialog. Any dialogCreate call will cause the existing dialog to be destroyed.

Wednesday, August 22, 2012

Use deep copy to make cheap copies of objects


There is a feature in MATLAB called "cheap copies" that makes expanding an array of different objects of the same class a snap. When I say different objects I mean that in the case of handle class objects, each object references a unique object, so that a change in one of its properties affects only it, and is not simultaneously changed in all of the other objects. That's what would happen if all of the objects were really references to the exact same object.

For example assume you have the following MATLAB handle class.
classdef MyClass < handle
        function obj = MyClass(a)
            obj.a = a;
Then ...
myClassArray(3) = MyClass(5)
...creates an array of three MyClass objects which are unique.


In Python the way to do this is with deepcopy from the builtin copy package.

For example assume you have the following Python class.
from copy import deepcopy

class MyClass(object):
    def __init__(self, a=5):
        self.a = a

# make 3 copies, but all point to the same instance

myClassArray = [MyClass(a=5)] * 3

# now make deep copies of each object in the array except the first
# one, so that they are now all unique instances

myClassArray[1:] = [deepcopy(dummy) for dummy in myClassArray[1:]]
Enjoy! For objects that are computationally expensive, this is a very fast way to build a bunch of identical but unique "copies" of objects of the same class.

Monday, August 20, 2012

Python Primer

[UPDATED 2016-08-04] add links to online training and references under learning curve
[UPDATED 2015-07-10] add links to Carl Kleffner's Anaconda/Binstar Repo
[UPDATED 2015-05-18] add Mac OSX section and site.USER_BASE usage
[UPDATED 2015-04-04] add tl;dr for
[UPDATED 2015-04-04] pip and setuptools are part of official Python since 2.7.9.
[UPDATED 2014-04-18] add anaconda distro
[UPDATED 2013-08-01] Setuptools has been updated and has merged with Distribute. 
[UPDATED 2013-04-18] I've added some better tutorial resources and the new Enthough Canopy distribution. I also removed the link URLs, sine they are embedded in the text, and rearranged the heading styles, since they were unreadable (yuk!). Also I couldn't resist linking xkcd antigravity post.


Can't be bothered? For any platform (Windows, Mac OSX and Linux) just download and install Anaconda Python by Continuum IO. No admin rights required! Start by opening the Anaconda Launcher to start Spyder a full featured editor. Anaconda already includes most of the packages you'll need for science, engineering, math and data analysis. That's it. Enjoy!

Getting Started

I put this together for my coworkers, but it could be applicable to anyone. Python is really easy to learn. Got Mac OSX? Then you already have Python-2.7.6. Got Windows? Just download the installer for your system. Some people prefer to install a Python distribution, which has stable versions of the most popular Python packages bundled together in a single installer, often together with a package manager and an interactive development environment (IDE). Official Python, Continuum Analytics Anaconda, WinPython, Enthought Canopy and ActivePython Community Edition offer installers for both 32-bit and 64-bit versions of Python on Windows and Macintosh. If you have a 32-bit computer you must use the 32-bit version, however, if you have a 64-bit version, you can use either. Using a 64-bit version allows you to access more than 2GB of RAM, but you may encounter some hurdles building packages that include extensions. The distributions have already compiled the packages for your version, but if you go with the official Python, then you will probably find yourself at Cristoph Gohlke's website. Mac OSX actually already has Python 2.X built in. IMHO do not install official Python on a Mac. Either use the Python version already installed or use Homebrew to install the desired version.

Python 2 or 3?

Only download 2.7.x, not 3.2.x, since no one uses Python 3 yet. Everything is Python 2. Some exaggeration here, but you get the point.

Windows x64

Using a 64-bit version of Python will let you access more than 2GB of data which may be necessary for many large scientific or engineering analyses. Many packages are distributed for Windows-x64 as "wheels", but for those that are not, you can find them on Christoph Gohlke’s Python extensions page. Package distributions for Windows x64 often are marked as win_amd64. If a package is not distributed as wheel anywhere, then, in order for pip to build any C/C++ extensions in the package from source, you will need to install the Microsoft Visual C++ Compiler for Python-2.7, which is free and does not require administrative rights. See the sections on installing packages using pip and wheels below for more information.

The exception to this rule are packages like NumPy and SciPy that require compiling FORTRAN libraries like BLAS and LAPACK. These libraries are part of the Intel Math Kernel Library (MKL) which is what Professor Gohlke uses, however there are several open source versions such as OpenBLAS, GotoBLAS and ATLAS. Unfortunately, you will not be able to install/compile these packages from source using pip and MS VC compilers for Python-2.7. You must use binary wheels from either Professor Gohlke's Python Extensions website or you can get the from Carl Kleffner's Anaconda/Binstar PyPI Repo or his BitBucket downloads.


Mac OSX already has Python-2.X installed. Mavericks and Yosemite both have Python-2.7, whereas older versions like Snow Leopard and Lion have Python-2.6. Mac OSX Python also has NumPy and Scipy already installed as well, although it might be quite old. Luckily, there is considerable support for Mac. For example there are precompiled NumPy wheels on PyPI and precompiled SciPy wheels as well. Mac OSX Python sets site.USER_BASE to ~/Library/Python/2.X a folder owned by you in your own profile, which allows you to install modules and packages without sudo using the --user scheme. First install pip from PyPI and add it to your path by running the following from a terminal ...

~ $ cd Downloads # into Downloads folder
~/Downloads $ curl -Ok # download pip package
~/Downloads $ python install --user # install into site.USER_BASE/lib/python/site-packages
~/Downloads $ cd # back to $HOME
~/ $ echo "# add PYTHONUSERBASE to path" >> .bash_profile
~/ $ echo 'export PATH=~/Library/Python/2.7/bin:${PATH}' >> .bash_profile # use single quotes to delay variable expansion

... now test pip by opening a new terminal and using pip to update setuptools ...

~/Downloads $ pip install --user -U setuptools # 

... so that packages can be installed into your local Python library with pip using the --user option as well. You don't need to add it to your PYTHONPATH, Python adds PYTHONUSERBASE by default. You will probably need to use Homebrew to install Qt, but for everything else pip will work fine. You will need to get XCode from Apple Developer Program to compile extensions. It should be free. Finally mark my words: Beware of sudo!. Don't ever use it on Mac OSX.

Learning Curve

Start by going through the official tutorial. It has some boring parts but it gets to the meat pretty quick. After that you will probably find yourself consulting the standard reference library often. Also Google has a great Python reference site. There are several super fun interactive sites like Philip Guo's python tutor and the ever popular Code Academy. For a more comprehensive treatment, there are two great online books: The first is Learning Python the Hard Way and the other is Kenneth Reitz's Python Guide. SciPy has a NumPy for MATLAB users primer in their wiki. For scientific computing, you can view/download the entire Primer on Scientific Programming with Python by Hans Petter Langtangen. Rockstar Fredrik Lundh provides great tips on eff-bot. Another rockstar Doug Hellman is also a resource. There are also about a million blogs on Python.

Here's a list of links to mostly free online Python tutorials in no particular order:

Getting Set Up

There are individual references for each Python packages (what Python toolboxes are called). You can find links to the references and download the packages from PyPI (aka "the Cheese Shop") or from their individual sites, often on or Github. Note that pip will automatically download packages from the Cheese Shop first unless you specify a file. The key packages you must have to get started are in the following sections.

Update Setuptools and Pip

Use pip from a command prompt to install new packages. Since Python-2.7.9, pip and setuptools, a dependency, are bundled with official Python but you need to update them to the newest versions.

  1. Install Python first and make sure you select the option to add python.exe and the Scripts folder to your PATH environment variable.
  2. Open a Windows Command Prompt and type pip install -U setuptools to update setuptools to the latest version.
  3. To update pip on Windows you must use python.exe -m pip install -U pip. If you try to update pip using the script it will fail due because you will be denied access to the pip script itself since windows will be using it.

installing packages

There are some interesting, perhaps updated sections on Installing Python Modules in the official Python docs. Most modules are pure Python and can be simply installed using pip like this:

user@computer ~ $ pip install wheel

Usually pip finds the package at the Cheese Shop, downloads the package and its dependencies, and then installs them for you. If there are any issues, pip is nice enough to roll everything back, so using pip is the safest way to install packages. Setuptools also manages packages with the easy_install command, but I really don't recommend using it because it isn't nearly as nice as pip. Some packages are distributed as "wheels" because they have platform dependent binaries. You can also install these using pip after downloading the wheel. EG: Say you download NumPy-MKL from UC Irvine Professor Christoph Gohlke Python Extension Packages website; use the following to install it.

user@computer ~/downloads $ pip install numpy‑1.9.2+mkl‑cp27‑none‑win_amd64.whl

Many packages are simply distributed as a tarball or zip-file; pip installs these too, exactly the same way it installs wheels, although you may need the Microsoft Visual C++ Compiler for Python-2.7 for Windows or XCode for Macintosh if the source has any C/C++ extensions, which will be automatically compiled by pip. Very rarely you may need to edit setup configuration to point to shared libraries, which is well beyond the scope of this primer.

Wheels & eggs

Binary distributions of packages can be distributed as "wheels", "eggs" and Windows installers. You can install a wheel using pip, eggs can be installed using easy_install and just double click a Windows installer. Wheels although the newest type of package distribution, are definitely the preferred method. You don't have to install the wheel package to use wheels, but the wheel package can convert other distributions to wheels so it might be useful.

user@computer ~ $ pip install wheel

EG: Say you download from SourceForge then you can convert it to a wheel like this.

user@computer ~/downloads $ wheel convert


Mathematics and scientific libraries essential for numerical computing, engineering analysis and mathematics and scientific research. As discussed in the Windows x64 section these packages use FORTRAN libraries and therefore can't be compiled by MS VC Compilers for Python-27, therefore you must install binaries, either wheels or bdist_wininst distributions. Download individual binary installers from sourceforge for Windows (x86) and Mac OS X. For Windows (x64) download Numpy MKL on Christoph Gohlke's site and Scipy on Christoph Gohlke's site or from Carl Kleffner's Anaconda/Binstar PyPI repo.


Statistics and data analysis. Think Excel or R. Downloads for all platforms.


Matplotlib is a plotting library that makes beautiful graphics and will be very familiar to MATLAB users. There are downloads for Windows (x86), Windows (x64) and Mac OS.


Ipython is about as slick as an interpreter gets, with color coding, tab completion and lots of magic. It also can create notebooks using tornado, work in a terminal or a custom Qt shell. It requires Setuptools, PyZMQ, PyQt and Tornado, and on Windows also requires Pyreadline and PyWin32). Download installers from Github for Windows (x86), Windows (x64) and Mac OS X.


Python Distributions

Also if you can’t be bothered to hunt down and install these separately, the you can install a distro. There are several major ones; here are the scientific/engineering/math ones. They vary in size, packages included, and level of customization. I don’t particularly recommend them, although the convenience may get you over the initial barrier to getting started, Python is so easy to use and learn that you will quickly be hampered by the limits imposed by the distros. Also you may find the user experience burdensome EG: having to start a launcher to access your Python apps will quickly become annoying. Also there may be bugs with user supported software that may not be quickly addressed because the smaller community is just a subset of the much larger Python community that address issues in the most frequently used Python source. In particular I have found Python (x,y) to particularly buggy. Finally, any proprietary software limited to a distribution may be a limit to deployment to other user akin to trying to deploy a MATLAB app. You have been warned.

Portable Python

WinPython and Portable Python are Python environments that run from a USB stick or CD/DVD. These are very cool and can aid in deploying Python apps as stand alone applications!

Development Environments

A good development environment (IDE) will have syntax highlighting, autocompletion and debugging built in.

  • Eclipse + Pydev is the best in my opinion, it comes with debugging, a console, git built in, autocomplete and indent, it is very similar to Visual Studio but in some ways way better. Eclipse is a generic IDE that can be used for nearly every programming language.
  • Spyder is a very nice Python specific IDE that will be very familiar to MATLAB users.  It has a built-in console, variable viewer/editor, help/documentation viewer and debugger! It comes stand alone or bundled with Python (x,y). The developers are very active and responsive.
  • PyCharms by JetBrains is a great IDE. I've tried it, and it's as good or better than eclipse. In fact many people prefer it, since its much smaller, focusing only on Python. JetBrains is also responsible for IntelliJ IDEA, the Java IDE that is super popular.
  • Sublime Text 2 is a pretty good lightweight IDE written completely in Python, the trial is free but to get rid of the occasional popup cost $60. This has become my personal favorite! It also does syntax highlighting for most major computer languages, and is extensible through plugins that can be managed easily with Package Control.
  • Notepad++ also has syntax highlighting for python as well as other languages.
  • Vim, a very lightweight but extremely popular terminal editor, also has syntax highlighting for python and other langauges, but it can be tricky to learn. Generally used from a terminal and standard in most POSIX environments, like MsysGit.
  • Geany is a lightweight multipurpose IDE.
  • Aptana Studio 3 is a preconfigured variant of eclipse + Pydev (I haven’t tried it)
  • Atom is a free editor from GitHub. It's available for all platforms. It is similar to Sublime Text 2/3, but it's free, more recently maintained and has many features.
  • Ninja-IDE, Ninja-IDE Is Not Just Another IDE, is the latest free newcomer.
  • Python Tools for Visual Studio (PTVS), is a mature free extension for Visual Studio 2013, 2012 and 2010. It works with both community edition and desktop express.
  • IDLE, part of standard Python, is a basic IDE with syntax highlighting, autocompletion, debugging and more.
  • LightTable is a free editor with syntax highlighting, some autocompletion, plugin management and and code evaluation. It looks promising and is the framework for Juno, the Julia editor.

Consoles, Terminals and Shells

A console is a place to enter command lines. Think windows CMD. It can be a convenient place to set up environmental variables such as custom paths. I recommend using ConEmu from Maximus5 GitHub Releases. Another excellent, yet older and apparently unmaintained terminal emulator is Console (v2.00) by Bozho available at

Git Version Control

Version control is essential when writing computer code. Inevitably you will make mistakes and need to go back, you may need to work with others, you may have new ideas that you want to flesh out, or your computer will crash and take your hard drive with it. Theses are issues that version control solves. Git has emerged as the go to version control tool. For Windows msysgit is conveniently packaged with MSYS, a posix environment with many BASH and posix tools that compliments Python nicely. If you need a graphical client TortoiseGit will suffice.

Virtual Environments

I can't conscientiously write a Python primer without at least mentioning virtual environments. Occasionally a project may depend on specific Python packages, and due to threats, real or imagined, of backwards incompatibilities, a Python virtual environment is used to freeze the project dependencies. The virtualenv package accomplishes this task by creating a Python sandbox that contains only the specific packages desired. A virtual environment can also be used to test out development distributions, or other untested packages that you don't want to install into your system site-packages folder. In particular, Max OSX and Linux users may prefer to use virtual environments since they don't require root. There is also a very convenient wrapper called virtualenvwrapper that makes managing virtual environments a cinch. Anaconda users should use conda instead of pip and virtualenv.

Fork me on GitHub