Monday, January 7, 2013

Purereadline

I have been playing with SL4A (the Android scripting layer) and PY4A (Python for Android) and I wanted to see if I could get the readline package working. Readline is based on GNU's libreadline. So I was able to compile libreadline.so, but I was too lazy to build Python properly for bionic (androids version of libc) in order to get a working Python header file. This got me thinking that a standalone pure Python version of readline could be useful for anyone without the Python headers (python.h and pyconfig.h).

Download a tarball of libreadline.so and libhistory.so for bionic from my public dropbox.

FYI: I copied these shared object libraries into /data/data/com.googlecode.pythonforandroid/files/python/lib, made symlinks to remove the version number, changed the permissions to 755 and the ownership to PY4A's username and group, which I represent below as app_XX. Alternately, if you don't have root permission, you still have permissions to use sl4a's folder when using either py4a or their bash shell, so put your shared objects there instead.

    /data/data/com.googlecode.androidscripting/files/python/lib

~$ wget https://dl.dropbox.com/u/19049582/rl-6.2_arm_linux_androideabi.tar
~$ tar -xf rl-6.2_arm_linux_androideabi.tar
~$ cp rl-6.2_arm_linux_androideabi.tar/libreadline.so.6.2 /data/data/com.googlecode.pythonforandroid/files/python/lib
~$ cp rl-6.2_arm_linux_androideabi.tar/libhistory.so.6.2 /data/data/com.googlecode.pythonforandroid/files/python/lib
~$ cd /data/data/com.googlecode.pythonforandroid/files/python/lib
/data/data/com.googlecode.pythonforandroid/files/python/lib$ ln libreadline.so.6.2 libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ ln libhistory.so.6.2 libhistory.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chmod 755 libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chmod 755 libhistory.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chown app_XX:app_XX libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chown app_XX:app_XX libhistory.so

You can test them in the SL4A interpreter by running the following script:

>>> from ctypes import *
>>> rl = CDLL("libreadline.so")
>>> rl.rl_initialize()
>>>  rl_completer_word_break_characters = c_char_p.in_dll(rl, " rl_completer_word_break_characters")
>>> print rl_completer_word_break_characters
" \t\n\"\\'`@$><=;|&{("

Therefore I present purereadline, an all Python wrapper for libreadline using ctypes. Uh ... it's still in beta, and I don't even know if it will work, but I can't think of any reason why it wouldn't. I can use ctypes fine with the shared object libraries to run functions and get exported variables. The shared object libraries compile very nicely using c4droid although I had to patch setpwent.

See issue #6 on py4a for more information and links.

dulwich porcelain

dulwich is an pure python implementation of git plumbing. It comes with a bin/scripts folder but it is incomplete - although I'm sure the developers would welcome additions. I have been working on a package called dulwich-porcelain, which facilitates adding features like checkout, fetch, push and status. Uh ... it's still beta, but contributions are welcome.

I also started a fork of dulwich with binpatches, porcelain and android branches. Android was actually the reason I origin started working on this. With just three commands it's easy to use your android device to retrieve your repos from a remote server anywhere you have 3g.

~$ dulwich clone <remote> <repo-name>
~$ cd <repo-name>
~/<repo-name>$ fetch_refs
~/<repo-name>$ checkout

n.b.: the functions also work as Python commands.

 You also need sl4a and py4a and some ssh authentication such as dropbear which is built into Cyanogenmod 7, openSSH is now used on CM9. Paramiko would be another option. I should make a post about this, but basically, I created an .ssh folder on my sdcard and created keys there. Cyanogenmod has a dropbear wiki and there is also a blogpost by TK, but both refer to creating an ssh server to communicate with your phone from a remote device, not creating keys to communicate with a remote device from your phone. Something like this should work.

~$ mkdir .ssh
~$ cd .ssh
~/.ssh$ dropbearkey -t rsa -f id_rsa
~/.ssh$ dropbearkey -y -f id_rsa > id_rsa_pub

Now you need to copy the public key, id_rsa_pub, to your remote Git server. You also need to make a small change to dulwich SSHvendor in client.py to make dropbear work. You can find the relevant changes in the android branch of my dulwich fork; basically append this class ...

class DropbearSSHVendor(object):
    def connect_ssh(self, host, command, username=None, port=None):
        import subprocess
        #FIXME: This has no way to deal with passwords..
        args = ['ssh', '-i', '/mnt/sdcard/.ssh/id_rsa']
        if port is not None:
            args.extend(['-p', str(port)])
        if username is not None:
            host = '%s@%s' % (username, host)
        args.append(host)
        proc = subprocess.Popen(args + command,
                                stdin=subprocess.PIPE,
                                stdout=subprocess.PIPE)
        return SubprocessWrapper(proc)

to the client.py file and then set get_ssh_vendor to the new vendor instead of the default one.

# Can be overridden by users
# get_ssh_vendor = SSHVendor
get_ssh_vendor = DropbearSSHVendor

Known hosts also got created automatically by dropbear in my .ssh folder on my sdcard. Dropbear doesn't let you use passwords, which is fine because dulwich has no way to deal with them. You could use keyring to get around that.

Now you can copy, send via email/bluetooth/wifi and/or edit your repo files on-the-go without needing to use a public computer or worry about ssh keys or passwords.

Thursday, January 3, 2013

converting Ubuntu to Xubuntu

[UPDATED 2012-05-13] The real fix for the apport-checkresume bug is to purge, clean and reinstall.

I have an old Compaq/HP laptop that I've been running Ubuntu on since Oneiric Ocelot (11.10). Unfortunately a pop up informed me that upgrading to Quantal Quetzal (12.10) wasn't recommended - Unity 2D isn't supported going forward, and the drivers for the ATI graphics in the Compaq/HP laptop don't support 3D acceleration.

Since Xubuntu doesn't use Unity, and I had no problem upgrading a VM on a similarly old Dell laptop also with an ATI graphics card, I decided to convert my machine to Xubuntu. For those unfamiliar with  Xubuntu, it is Canonical Ubuntu with an XFCE desktop, based on X11.

What follows are the steps, blunders, fixes and resources I used/followed to convert my system from Unity to xfce.
  1. Install xubuntu-desktop. I did this from the Ubuntu-Software-Center, but you could easily type sudo apt-get install xubuntu-desktop in a terminal.
  2. Remove ubuntu-desktop and unity. You will get some ugly warnings.
  3. If you restart now, you may get the dreaded "low graphics mode". So while you still can, edit /etc/lightdm/lightdm.conf so that the greeter is lightdm-gtk-greeter. If you do get stuck in "low graphics mode" never fear, just hit CTRL-ALT-F2 to get a terminal and type your username and password. Then you can use sudo nano /etc/lightdm/lightdm.conf to edit the config file and then  use sudo reboot and all should be well.
  4. Remove any remain unity packages. Here are some posts to follow:

Epilogue: So I was able to completely convert my old laptop successfully and upgrade to Quantal, but I did have to run these commands to push through the installation.
sudo dpkg --configure -a
sudo apt-get -f upgrade
You make also experience issues/crashes when your system resumes from suspend. Examining the detail may show the offender as apportcheckresume [1, 2, 3, 4], which is ironic, because this is the bug reporting app, and it is causing the bug, because it can't resume from suspend. I can confirm that purging apport, cleaning your repo database and reinstalling apport fixes this issue.

sudo apt-get purge apport
sudo apt-get autoremove
sudo apt-get clean
sudo apt-get install apport

Good luck and enjoy!

P.S., I was able to compare my converted Xubuntu setup with an iso installed Xubuntu, which gave me a way to check as I went along which packages should be removed.

P.P.S., synaptic and deborphan are good tools to use to clean up orphans and broken dependencies, sudo apt-get autoremove will do something similar. Although synaptic seems to resolve dependencies differently than Ubuntu-Software-Center, it has saved my ass more than once.

P.P.P.S., I suppose as an alternate to lightdm you could install gdm, but it seems like more work, and it is not the default display manager for Xubuntu.

[1] http://askubuntu.com/questions/230670/apportcheckresume-recurring-error-and-gnome-shell-fixations
[2] http://askubuntu.com/questions/203745/suspend-breaks-on-12-10s-kernel
[3] http://askubuntu.com/questions/141671/keep-getting-messages-about-internal-system-errors
[4] http://ubuntuforums.org/showthread.php?t=2073983


Friday, December 28, 2012

more msvc express cheats

To build the INSTALL project you can use devenv which is really just a shortcut to msvc.

Visual Studio C++ 2008 Express (MSVC90)

VCEXPRESS = DEVENV

On express devenv is renamed vcexpress, but does it have the same functionality? Here is the help.

Use:
vcexpress  [solutionfile | projectfile | anyfile.ext]  [switches]

The first argument for devenv is usually a solution file or project file.
You can also use any other file as the first argument if you want to have the
file open automatically in an editor. When you enter a project file, the IDE
looks for an .sln file with the same base name as the project file in the
parent directory for the project file. If no such .sln file exists, then the
IDE looks for a single .sln file that references the project. If no such single
.sln file exists, then the IDE creates an unsaved solution with a default .sln
file name that has the same base name as the project file.

Command line builds:
devenv solutionfile.sln /build [ solutionconfig ] [ /project projectnameorfile [ /projectconfig name ] ]
Available command line switches:

/Log Logs IDE activity to the specified file for troubleshooting.
/ResetSettings Restores the IDE's default settings, optionally resets to
the specified VSSettings file.
/SafeMode Launches the IDE in safe mode loading minimal windows.

Product-specific switches:

/debugexe Open the specified executable to be debugged. The
remainder of the command line is passed to this
executable as its arguments.
/useenv Use PATH, INCLUDE, LIBPATH, and LIB environment variables
instead of IDE paths for VC++ builds.

To attach the debugger from the command line, use:
VsJITDebugger.exe -p <pid>

VCBUILD

VCBUILD is also available, but I don't know exactly what the difference is.
http://msdn.microsoft.com/en-us/library/hw9dzw3c(v=vs.90)


C:\Program Files\Microsoft Visual Studio 9.0\VC>vcbuild /?
Microsoft (R) Visual C++ Project Builder - Command Line Version 9.00.30729
Copyright (C) Microsoft Corporation. All rights reserved.

Usage: vcbuild [options] [project|solution] [config|$ALL]

Options:
/clean (/c)     Clean build outputs only
/error:<str>    Prefix to add to error lines on output to stderr
/errfile:<file> Log all errors to the file specified
/htmllog:<file> Logs output to html file specified
                (default: $(IntDir)\BuildLog.htm)
/implib         Makes an import library for a DLL configuration (does not link)
/info:<str>     Prefix to add to information lines on output to stdout
/link (/l)      Performs a link without building sources
/logcommands    Prints commands and response files to screen
/logfile:<file> Log all output and build information to the file specified
/override:<file>Override the project settings with the settings in the given
                property sheet file

/M<number>      Specifies the number of concurrent builds to run, if possible
/msbuild:<opt>  Pass <opt> to msbuild.exe

/nocolor        Do not output error and warning messages in color
/nohtmllog      Do not write an html build log file

/noimplib       Does not generate an import library.
                NOTE: this option overrides /implib.
/nologo         Suppress version and copyright message
/nondefmsbuild  Do not use the copy of msbuild.exe located in the .NET
                Frameworks installation
/platform:<str> Build only configurations for the given platform
/implibobjs:<s> Additional dependencies for the librarian
/rebuild (/r)   Clean build outputs and perform a build
/forcelink      Forces a link without building sources
/showenv        Show environment in the html build log
/time           Times the build from start to finish
/upgrade        Upgrades the project file to the latest format supported
                NOTE: the upgrade switch does not perform a build
                NOTE: this option is ignored for solution files
/useenv (/u)    Use environment variables for INCLUDE and LIB paths
/overrideRefVer When upgrading, assume .NET Framework Version 3.5 for any assemb
ly references.
                Requires /upgrade switch.
/wrnfile:<file> Log all warnings to the file specified
/warning:<str>  Prefix to add to warning lines on output to stdout
@<file>         Read options from the specified response file

Default Behavior:
===============================================================================
If no project is specified, and there is only a single .vcproj file in the
directory, that project will be built.
If no configuration is specified, and the VCBUILD_DEFAULT_CFG environment
variable is set, the configuration it specifies will be built. If it is not
set, then all configurations will be built.

The default action is to build the specified configurations without cleaning.

Options will also be read from the VCBUILD_DEFAULT_OPTIONS environment
variable.
===============================================================================

Visual Studio C++ 2010 Express (MSVC100)

vcexpress = devenv

On express devenv is renamed vcexpress, but does it have the same functionality? Here is the help.

Use:
vcexpress  [solutionfile | projectfile | anyfile.ext]  [switches]

The first argument for vcexpress is usually a solution file or project file.
You can also use any other file as the first argument if you want to have the
file open automatically in an editor. When you enter a project file, the IDE
looks for an .sln file with the same base name as the project file in the
parent directory for the project file. If no such .sln file exists, then the
IDE looks for a single .sln file that references the project. If no such single
.sln file exists, then the IDE creates an unsaved solution with a default .sln
file name that has the same base name as the project file.

Command line builds:
vcexpress solutionfile.sln /build [ solutionconfig ] [ /project projectnameorfile [ /projectconfig name ] ]
Available command line switches:

/Log Logs IDE activity to the specified file for troubleshooting.
/ResetSettings Restores the IDE's default settings, optionally resets to
the specified VSSettings file.
/SafeMode Launches the IDE in safe mode loading minimal windows.

Product-specific switches:

/debugexe Open the specified executable to be debugged. The
remainder of the command line is passed to this
executable as its arguments.
/useenv Use PATH, INCLUDE, LIBPATH, and LIB environment variables
instead of IDE paths for VC++ builds.

To attach the debugger from the command line, use:
VsJITDebugger.exe -p <pid>

MSBUILD

http://msdn.microsoft.com/en-us/library/dd293626

On MSVC100, VCBUILD seems to be renamed to MSBUILD, which is what actually runs even on mcvs90, when you run VCBUILD anyway.


Microsoft (R) Build Engine Version 4.0.30319.1
[Microsoft .NET Framework, Version 4.0.30319.269]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Syntax:              MSBuild.exe [options] [project file]

Description:         Builds the specified targets in the project file. If
                     a project file is not specified, MSBuild searches the
                     current working directory for a file that has a file
                     extension that ends in "proj" and uses that file.

Switches:

  /target:<targets>  Build these targets in this project. Use a semicolon or a
                     comma to separate multiple targets, or specify each
                     target separately. (Short form: /t)
                     Example:
                       /target:Resources;Compile

  /property:<n>=<v>  Set or override these project-level properties. <n> is
                     the property name, and <v> is the property value. Use a
                     semicolon or a comma to separate multiple properties, or
                     specify each property separately. (Short form: /p)
                     Example:
                       /property:WarningLevel=2;OutDir=bin\Debug\

  /maxcpucount[:n]   Specifies the maximum number of concurrent processes to
                     build with. If the switch is not used, the default
                     value used is 1. If the switch is used without a value
                     MSBuild will use up to the number of processors on the
                     computer. (Short form: /m[:n])

  /toolsversion:<version>
                     The version of the MSBuild Toolset (tasks, targets, etc.)
                     to use during build. This version will override the
                     versions specified by individual projects. (Short form:
                     /tv)
                     Example:
                       /toolsversion:3.5

  /verbosity:<level> Display this amount of information in the event log.
                     The available verbosity levels are: q[uiet], m[inimal],
                     n[ormal], d[etailed], and diag[nostic]. (Short form: /v)
                     Example:
                       /verbosity:quiet

  /consoleloggerparameters:<parameters>
                     Parameters to console logger. (Short form: /clp)
                     The available parameters are:
                        PerformanceSummary--Show time spent in tasks, targets
                            and projects.
                        Summary--Show error and warning summary at the end.
                        NoSummary--Don't show error and warning summary at the
                            end.
                        ErrorsOnly--Show only errors.
                        WarningsOnly--Show only warnings.
                        NoItemAndPropertyList--Don't show list of items and
                            properties at the start of each project build.
                        ShowCommandLine--Show TaskCommandLineEvent messages
                        ShowTimestamp--Display the Timestamp as a prefix to any
                            message.
                        ShowEventId--Show eventId for started events, finished
                            events, and messages
                        ForceNoAlign--Does not align the text to the size of
                            the console buffer
                        DisableConsoleColor--Use the default console colors
                            for all logging messages.
                        DisableMPLogging-- Disable the multiprocessor
                            logging style of output when running in
                            non-multiprocessor mode.
                        EnableMPLogging--Enable the multiprocessor logging
                            style even when running in non-multiprocessor
                            mode. This logging style is on by default.
                        Verbosity--overrides the /verbosity setting for this
                            logger.
                     Example:
                        /consoleloggerparameters:PerformanceSummary;NoSummary;
                                                 Verbosity=minimal

  /noconsolelogger   Disable the default console logger and do not log events
                     to the console. (Short form: /noconlog)

  /fileLogger[n]     Logs the build output to a file. By default
                     the file is in the current directory and named
                     "msbuild[n].log". Events from all nodes are combined into
                     a single log. The location of the file and other
                     parameters for the fileLogger can be specified through
                     the addition of the "/fileLoggerParameters[n]" switch.
                     "n" if present can be a digit from 1-9, allowing up to
                     10 file loggers to be attached. (Short form: /fl[n])

  /fileloggerparameters[n]:<parameters>
                     Provides any extra parameters for file loggers.
                     The presence of this switch implies the
                     corresponding /filelogger[n] switch.
                     "n" if present can be a digit from 1-9.
                     /fileloggerparameters is also used by any distributed
                     file logger, see description of /distributedFileLogger.
                     (Short form: /flp[n])
                     The same parameters listed for the console logger are
                     available. Some additional available parameters are:
                        LogFile--path to the log file into which the
                            build log will be written.
                        Append--determines if the build log will be appended
                            to or overwrite the log file. Setting the
                            switch appends the build log to the log file;
                            Not setting the switch overwrites the
                            contents of an existing log file.
                            The default is not to append to the log file.
                        Encoding--specifies the encoding for the file,
                            for example, UTF-8, Unicode, or ASCII
                     Default verbosity is Detailed.
                     Examples:
                       /fileLoggerParameters:LogFile=MyLog.log;Append;
                                           Verbosity=diagnostic;Encoding=UTF-8

                       /flp:Summary;Verbosity=minimal;LogFile=msbuild.sum
                       /flp1:warningsonly;logfile=msbuild.wrn
                       /flp2:errorsonly;logfile=msbuild.err

  /distributedlogger:<central logger>*<forwarding logger>
                     Use this logger to log events from MSBuild, attaching a
                     different logger instance to each node. To specify
                     multiple loggers, specify each logger separately.
                     (Short form /dl)
                     The <logger> syntax is:
                       [<logger class>,]<logger assembly>[;<logger parameters>]
                     The <logger class> syntax is:
                       [<partial or full namespace>.]<logger class name>
                     The <logger assembly> syntax is:
                       {<assembly name>[,<strong name>] | <assembly file>}
                     The <logger parameters> are optional, and are passed
                     to the logger exactly as you typed them. (Short form: /l)
                     Examples:
                       /dl:XMLLogger,MyLogger,Version=1.0.2,Culture=neutral
                       /dl:MyLogger,C:\My.dll*ForwardingLogger,C:\Logger.dll

  /distributedFileLogger
                     Logs the build output to multiple log files, one log file
                     per MSBuild node. The initial location for these files is
                     the current directory. By default the files are called
                     "MSBuild<nodeid>.log". The location of the files and
                     other parameters for the fileLogger can be specified
                     with the addition of the "/fileLoggerParameters" switch.

                     If a log file name is set through the fileLoggerParameters
                     switch the distributed logger will use the fileName as a
                     template and append the node id to this fileName to
                     create a log file for each node.

  /logger:<logger>   Use this logger to log events from MSBuild. To specify
                     multiple loggers, specify each logger separately.
                     The <logger> syntax is:
                       [<logger class>,]<logger assembly>[;<logger parameters>]
                     The <logger class> syntax is:
                       [<partial or full namespace>.]<logger class name>
                     The <logger assembly> syntax is:
                       {<assembly name>[,<strong name>] | <assembly file>}
                     The <logger parameters> are optional, and are passed
                     to the logger exactly as you typed them. (Short form: /l)
                     Examples:
                       /logger:XMLLogger,MyLogger,Version=1.0.2,Culture=neutral
                       /logger:XMLLogger,C:\Loggers\MyLogger.dll;OutputAsHTML

  /validate          Validate the project against the default schema. (Short
                     form: /val)

  /validate:<schema> Validate the project against the specified schema. (Short
                     form: /val)
                     Example:
                       /validate:MyExtendedBuildSchema.xsd

  /ignoreprojectextensions:<extensions>
                     List of extensions to ignore when determining which
                     project file to build. Use a semicolon or a comma
                     to separate multiple extensions.
                     (Short form: /ignore)
                     Example:
                       /ignoreprojectextensions:.sln

  /nodeReuse:<parameters>
                     Enables or Disables the reuse of MSBuild nodes.
                     The parameters are:
                     True --Nodes will remain after the build completes
                            and will be reused by subsequent builds (default)
                     False--Nodes will not remain after the build completes
                     (Short form: /nr)
                     Example:
                       /nr:true

  /preprocess[:file]
                     Creates a single, aggregated project file by
                     inlining all the files that would be imported during a
                     build, with their boundaries marked. This can be
                     useful for figuring out what files are being imported
                     and from where, and what they will contribute to
                     the build. By default the output is written to
                     the console window. If the path to an output file
                     is provided that will be used instead.
                     (Short form: /pp)
                     Example:
                       /pp:out.txt

  /detailedsummary
                     Shows detailed information at the end of the build
                     about the configurations built and how they were
                     scheduled to nodes.
                     (Short form: /ds)

  @<file>            Insert command-line settings from a text file. To specify
                     multiple response files, specify each response file
                     separately.

  /noautoresponse    Do not auto-include the MSBuild.rsp file. (Short form:
                     /noautorsp)

  /nologo            Do not display the startup banner and copyright message.

  /version           Display version information only. (Short form: /ver)

  /help              Display this usage message. (Short form: /? or /h)

Examples:

        MSBuild MyApp.sln /t:Rebuild /p:Configuration=Release
        MSBuild MyApp.csproj /t:Clean
                             /p:Configuration=Debug;TargetFrameworkVersion=v3.5

Wednesday, December 5, 2012

Download sites for old, free MS compilers and IDEs

[UPDATE 2015-03-13] I do not recommend installing VS2010 Express (or Professional). This program is very difficult to remove and has become obsolete, with VS2013. If you are stuck with this on your computer you may be able to remove it by following these steps.
  1. Make sure you have copies of the VS2010 SP1 installer. The web installer is fine, but the iso may come in useful. Don't bother creating a manual system restore point. Even when you roll back changes, the issues that cause the majority of problems are not resolved. However I guess it can't hurt either. If you really want to cover your bases, the best thing would be to make a recovery disk and an disk image. This way you can roll back changes if they go awry.
  2. If you have installed VS2010 SP1, you must remove this first. You should be able to remove it from add/remove programs in the control panel. If not try
    • C:\ProgramData\VS\vs2010sp1\SetupCache\setup.exe /uninstall /force
    • Download the installer and run VS10sp1-KB983509.exe /uninstall /force
    • Download and extract the iso image and run setup.exe /uninstall /force
    • Reinstall SP1 and then try uninstalling it from add/remove programs in control panel. You may need to reinstall VS2010 VC10 or VCS10 before SP1.
  3. Remove VS2010 Tools for Office Runtime and VS2010 ADO.NET Entity Framework. You may need to remove these before SP1.
  4. Use the VS2010 uninstall utility documented first here then later here. Make sure you use the options in the post /full to remove everything!
Note this took many iterations to actually get all of VS2010 off of my system, so be prepared for frustration and irritation, but persevere and it can be done. Just make sure you have copies of the installer so that you can reinstall and try to remove everything in the correct order again! You may have an issue removing or reinstalling Windows SDKs. If so see Upgrading to Visual Studio 2013.

[UPDATE 2014-11-13] Free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft. The free VC90 compilers can be used to install package source that contains c-extensions using either pip, setuptools or distutils. For example, pip install dulwich will build and install the Python Git implementation which contains several speedups as c-extensions.

[UPDATE 2013-10-03, 2014-04-16] Sad to report that clicking on #1 MS Visual Studio 2008 Express states that it has been retired. Oh, yes! VS2008 Express didn't allow targeting x64 platforms, without some serious hoop-jumping, but VS2010 Express does, and VS2010 also lets you select the V90 toolset, so building shared objects for Python-2.7 is no problem. Also, SDK7 has the exact same compilers as V90, and it is still supported so they are more current. Ditto for SDK7.1 & V100. Oh no! Does this mean we finally have to switch to Python 3? If you really want VS2008, the web installer does however still work. Grab it from my dropbox.

  1. Microsoft Visual Studio 2008 Express Editions with SP1: This one is necessary for python 2.7.3 extensions. You can't get it from the Visual Studio page anymore, now that they've moved on to 2012, but download links for vcsetup.exe which install MSVC90 Express are in the Microsoft Download Center, as are vcssetup (C#), vbsetup.exe (vbs) and vwsetup.exe (web). You also get the opportunity to install SQL Server 2008 and Silverlight SDK. Visual Studio 2008 Express has been discontinued, and is obsolete because Visual Studio 2010 Express includes the V90 toolset. See update note at top of page. Not sure if Visual Studio 2008 SP1 download will install for free, but it may. VS2008 is not required to build Python-2.7 extensions, instead use SDK7 which replaces the v90 compilers.
  2. Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1: You cannot build 64-bit applications with the free version of VC90 unless you have Windows SDK 7. Note this *not* 7.1, which is VC100, also note that .NET Framework 3.5 is part of Windows 7, so you don't have to download it separately, and of course make sure that you get all updates before installing. Specifically you must have SP1. The paths in vcvarsall.bat are incorrect, you will need to fix them to use pip with v90. Otherwise use sdk7 shell and follow the directions in the post on installing Python x64 extensions with pip. 
  3. Visual Studio 2010 Express is still available from the main VS site. They have 2012 and 2010, and all of the flavors.
  4. Microsoft SDK 7.1 has some issues with VC100. Basically you should follow this procedures:
    1. Visual Studio 2010 RTM
    2. Windows SDK 7.1
    3. Visual Studio 2010 SP1
    4. Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1
    See these posts:
    Setup & Install by Heath Stewart
    Visual C++ Team Blog
    FIX: Visual C++ compilers are removed ...
  5. Microsoft Windows SDK for Windows 7 and .NET Framework 4
  6. Beware here, you might run into this error, which is very confusing, the installer quits with the message "Installation of the “Microsoft Windows SDKfor Windows 7” product has reported the following error: Please refer to Samples\Setup\HTML\ConfigDetails.htm document for further information." Totally useless, but further examination of the log file or googling around you might see the real source of the error, it's not SP1 but the redistributable that is too new: "C:\Program Files\Microsoft SDKs\Windows\v7.1\Setup\SFX\vcredist_x64.exe installation failed with return code 5100". The solution is to remove the redistributables and then reinstall them later. See this knowledge base article:
    Windows SDK Fails to Install with Return Code 5100
  7. Upgrade to Microsoft Visual Studio 2010 SP1. See this link for what's in SP1.
    Description of Visual Studio 2010 Service Pack 1
  8. Download the update after you follow the procedures here:
    Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1


Wednesday, November 28, 2012

Eclipse in ubuntu

There are so many posts out there on installing eclipse on Ubuntu, that I decided to take the plunge. I had been troubled by having to manage two repos, but since I mainly use PyDev and Android SDK, neither of which are in the Ubuntu repository, so it was either deal or try to go it alone. I'm happy to report that eclipse 4.2, aka juno, is running great in Quantal, no errors so far!

With Oneiric and Precise I had successfully used both Ubuntu and eclipse repositories, but with each upgrade there were always snafus; I would have to re-install some plugin and end up losing my workspace settings. Playing the repo shuffle, you never know when Ubuntu might suddenly decide to revoke or add a package, and there are so many eclipse plugins it just makes sense to me to let one system manage the dependencies. Reading up a bit here and there I began to realize that this is an issue eclipse and Linux users have addressed and that eclipse would work out of the box on Ubuntu.

So the first thing I did was completely remove eclipse from my system. I had some false starts, first off Synaptic may be a great tool but it does not remove certain dependencies the same way as the Ubuntu Software Center (USC). Same goes for `apt-get`. Too bad because using USC is slow and tedious because you can only install/remove one package at a time. But unless you want to leave a lot of detritus on disk USC does a better job at killing off orphans. That made me realize another reason for installing eclipse as add-on software, since using the Ubuntu repo installed over 100 packages! I'm trying to keep my machine uncluttered and organized so that's just ridiculous.

I did a few things different than the vast number of sites posting instructions.
  1. I created a folder called "/opt" in my home directory and untarrred my eclipse download there. I chose the Java EE version for 32-bit Linux because it has the most complete selection of useful plugin preinstalled. I decided on ~/opt instead of /opt because I'm the only user and I didn't want to deal with ownership and permissions. This is a nice compromise that doesn't clutter your system. However, in general I would recommend copying the entire package to /opt and changing ownership to root:root.
  2. I created a soft symlink to ../opt/eclipse/eclipse called "eclipse" in ~/bin. This folder is already present and is always first on your path. If I had moved eclipse to /opt, then it would be available to all users, and then each user could create a symlink from there to their private ~/bin folder. I didn't have to change the permissions, because it's already 777.
  3. This next step is the major difference between what I did and what I've seen posted everywhere. I created a desktop file in ~/.local/share/applicatons called "eclipse.desktop" with the following. I looked into both gnome-panel and alacarte (ne main menu), but I ruled them out because the desktop file is so easy to create, and both of these tools are for true gnome desktops not Unity, and also seemed outdated. There is ample help on putting applications in the desktop. A couple of notes: ~ is not expanded, so use your complete path, X-Ayatana is the name of the Unity group, if you want to add extra commands and I had to export the eclipse xpm icon as a png file because it didn't work in the Unity dash.

    [Desktop Entry]
    Version=1.0
    Name=Eclipse
    GenericName=IDE
    Exec=/home/marko/bin/eclipse
    Terminal=false
    Icon=/home/marko/opt/eclipse/icon.png
    Type=Application
    Categories=Development;IDE

  4. I started eclipse from the dash and pinned it to the launcher. Then followed the customary procedures to update and install packages from within eclpse. From the menu select help > check for updates, and then help > install new software. Note: I have started help using the internal browser (default) and it works fine. My system already has both libwebkitgtk-1.0.0 and libwebkitgtk-3.0.0 so I didn't have to install or remove anything; eclipse help just worked out of the box. I have read numerous reports regarding this issue (eclipse crashes when help window opened), so evidently it has been solved.
So there you go. If you are on the fence, and tired of managing two repositories, I hope I've convinced you that it's okay to install eclipse as an add-on and it will work fine, and not break your system.

Monday, November 26, 2012

Install Add On Software and Create GTK+ Desktop File

Unfortunately not all of the software that you want will be in your linux distro. The most common examples of these are the awesome Sublime Text 2 editor and the bloated but ubiquitous eclipse IDE. Actually eclipse is in most linux distro repositories, but only a limited set of plugins are available, and eclipse uses its own repository system to update and install plugins which is in principle incompatible with your distro's packaging system. For example, if you install eclipse 3.8 from the Ubuntu Quantal Quetzal repo, you can not update it from the eclipse repository, and if you install the PyDev plugin using their repository through the eclipse packaging system, when your upgrade to the next Ubuntu version, your plugins will not work and you will have to reinstall them. If you are okay with this, then you may be alright, but you will have some extra work to do come upgrade time, and you will need to keep track of what is installed via the official repo and what is being maintained by eclipse's repos, because neither can keep track of the other, and they are all for the most part dumped in the /home/<username>/.eclipse/ directory.

Linux actually has a protocol for installing software add-ons, if you decide that what's in your distro's repository is not enough for you. There are a few options.

  1. Find a PPA (Personal Package Archives for Ubuntu), add it and then install the package with apt-get. This will most likely install the package in /usr/, /usr/bin/, /usr/lib/, ... and take care of configuration and creating a desktop file so that you the application will appear in the unity dash (on Ubuntu) and you can add it to the launcher or launch files from Nautilus. This works for Sublime Text 2. PPA's are often listed on Launchpad.
  2. Build the code from source using autotools and it should install in /usr/local/, /usr/local/bin/, /usr/local/lib/, /usr/local/include/, ... but of course you will need to make sure that the application's dependencies are all satisfied and do not conflict with your distro. This can be a tough row to hoe.
  3. Find a pre-compiled package and drop it into your system's /opt directory, which is designated specifically for add-on software not provided by the system's package repositories. An alternative often used and actually my preference is to extract the new package into your home directory.  This works, but to avoid cluttering $HOME I add a folder called ~/opt and drop the add-ons in there. Again, it is up to you to make sure that the software's dependencies are satisfied. For example, eclipse requires some jdk, whether it is from Oracle or the openJDK project, and it will need to be a version within the range specified by the add-on software, e.g. jdk >= 7.0 in the case of eclipse.
Now you need to integrate the add-on software into your system. Again you have some choices and it seems that there is some debate about what is the best course to take.
  1. The easiest thing to do is to run the application from the console, and don't integrate it into your desktop (Gnome) or file system browser (Nautilus), by opening a terminal window and typing ./<executable>. You may need to change the permissions first by using sudo chmod 755 <executable>.
  2. Another option that doesn't require you to edit your desktop files is to add the folder to your path, however, this may cause problems if there are filename conflicts, e.g. your system already has a libz.a but your new software does too. One safe way to deal with this is to add  the new directory to the end of your path, or create a special shell for it. This way your default system files will always be used.
  3. Create a symlink to /home/<username>/bin/ which is always first on your path. This is my preferred method for several reasons. First it will survive an update because your personal profile is not altered during upgrades. Second, it only affects you, and not other users if you share your machine. Of course if your intent is to make the application usable for all users, then either you need to add it to all profiles or make a symlink to /usr/bin/. However, links in /usr/bin/ will probably not survive an upgrade because this is system file territory
  4. Create a gtk desktop file in /home/<username>/.local/share/applications/, as described in Desktop files: putting your application in the desktop menus, which again is my preferred method. I wrote a post on eclipse in Ubuntu that has an example of a desktop file. The reasons are the same as above. First it survives an upgrade and second it only affects you. You can customize the desktop file to have as many actions as you want, for example to have an option to open a new window by right clicking directly from the launcher icon, or to list all of your current eclipse profiles so that you can launch them directly. The desktop file is very flexible. You should now be able to find your application in the Unity dash (on Ubuntu) and once it runs you can lock it to the launcher. Also if you add the %f option after the location of the executable in the desktop file, then it will be available to Nautilus as well. Then you can right click and see other applications that can open a file associated with your new application. See this Ask Ubuntu answer on how to install the Sublime Text 2 editor. A lot of people have recommended editing the system wide desktop files, which are in /usr/share/applications/ but I don't recommend this as it will not survive an upgrade.
In general, I tend to be more conservative when it comes to editing system files and directories, preferring to keep my customization localized to my profile, and only deviating from that when absolutely necessary. But you may have a different use case which will necessitate a different approach.
Fork me on GitHub