setuptools detritus

Pre pip-1.5.1, you used to have to install setuptools prior to Pip. Especially after the distribute/setuptools merger, this potentially left you with a mishmash of setuptools and distribute folders and egg files, as easy-install and pip install packages differently.

If you used the recommended ez-setup.py provided by setuptools, this would always download a tarball or zipfile in whatever directory ez-setup.py was in, probably downloads, but in the case of packages bundled with it, it could be anywhere, never to be deleted, completely undetected by you, and if you never installed pip, and never found the easy way to update your packages, then everytime you used easy-install or distutils (and probably pip too, since ez-setup.py specifies the version of setuptools required before setup.py runs) to install a package bundled with setuptools' ez-setup.py it would leave another piece of detritus.

Then mix in the pip way of doing things, which is indifferent to egg-files. If you use pip to update setuptools, (pip install -U setuptools) which is a great way, because once you have cleaned up all of the random pth and egg leftovers, pip always takes care of cleaning up its own mess. However as mentioned previously, unless that mess is an egg file. Pip doesn't touch these files, which leads to two installs of setuptools, an egg file, which for those wondering is like a wheel, or a zip file, essentially a binary, platform specific installation, and the pip way, which is an uncompressed folder with python source code, byte compiled code and any extensions, similar to what distutils would do, if it weren't monkeypatched.

Well no longer. Pip now takes care of everything. So start from scratch, delete all instances of easy-install, setuptools(, distribute if you still have it) and sure what the heck pip, everywhere, in scripts(/bin [1]) and lib. Then use get-pip.py to install both at the same time. Then periodically update setuptools using pip install -U setuptools. Ah. All better.

[1] Note: that this is really only a Windows or Mac issue, not Linux, because Python packages are included in each distro's repository. It can be an issue for virtualenvs anywhere, even Linux, so it's good to understand in principle. On a Linux share, without root access, you might have this issue with packages installed in .local. On Mac if you are using an official binary installation of Python, then you will find your scripts in /Library/Frameworks/Python.framework/Versions/2.7/bin. In a virtualenv you should find scripts in .virtualenvs/name-of-venv/bin and .virtualenvs\name-of-venv\scripts for Mac/Linux and Windows systems respectively. For non-root local installs on Linux shares the scripts folder is .local/bin.

Trouble updating to virtualenvwrapper 4.2?

I must be the only one, because I haven't found anything online, but using pip, easy-install or distutils fails, even after cleaning up all of my random setuptools pth and egg leftovers, unless I install pbr first. The traceback I get is something like "file not found, warning LoadManifest -c could not find standard file". Doug Hellman's pbr package is supposed to be installed as part of either stevedore or virtualenvwrapper, but something is not working somewhere. No time to troubleshoot, but workaround is just as easy.

git subtree merge strategy in your everyday life

Here is yet another post on the super useful git feature called the subtree merge strategy. Not to be confused with the newish git-subtree command, which uses this strategy, but may or may not be completely ready for general consumption. Imagine a use case where you have a package, for example hg-fast-export, that you want to use in your project. You want to deploy it with your project, so you consider using git submodule. An alternative would be to use the git subtree merge strategy. There are numerous sites that explain the pros and cons of subtree merge vs submodules.

~/myproj (master)$ git remote add hg-fast-export https://github.com/frej/fast-export.git
~/myproj (master)$ git fetch hg-fast-export master
~/myproj (master)$ git read-tree --prefix=hg-fast-export/ -u remotes/hg-fast-export/master
~/myproj (master)$ git pull --squash -s subtree --no-commit hg-fast-export master
~/myproj (master)$ git pull --squash -s recursive -Xsubtree=hg-fast-export/ --no-commit hg-fast-export master
~/myproj (master)$ git diff-tree hg-fast-export master

1. Add the remote that hosts the sub-package.
2. Fetch the branch from the remote that you want to use in your project.
3. Read the remote branch into its own folder tree in the working copy. The name of the new directory is provided by the prefix if you append a slash at the end.
4. Merge updates from the sub-package into the hg-fast-export tree, squash them all into a single commit, but don't auto-commit. This gives you a chance to check the merge, before commiting. Merge prepopulates the message with a summary of the commit messages from the sub-package's remote branch.
5. Explicitly specify trees to merge if Git subtree merge strategy can't determine by itself.
6. Get diff between two trees.

I think if you want to push from your sub-package back to the remote, you would want to create an orphan branch, and push from there. This is covered in the alternate posts linked to from here.

Pushing to gh-pages for Github pages is a perfect use of subtree merge strategy - I provide an example in this SO answer.

Turn off Objective-C for m-files, Turn on MATLAB

I just posted about how cool Sublime Text 2 editor is, in Markdown builder for Sublime Text 2, but it's not all ice cream and apple pie. One thing that's a drag is that it always chooses Objective-C syntax for MATLAB. It's a pretty quick fix. Opening up the MATLAB language file found under packages in the preferences folder there's this comment re: file type

 <key>fileTypes</key>
 <array>
  <!-- Actually, it's generally .m, but that's taken by Objective-C. It needs an entry to show up in the syntax list. -->
  <string>matlab</string>
 </array>

Look in the Objective-C language file and you see this is indeed true, m-files are treated as Objective-C instead of MATLAB.

 <key>fileTypes</key>
 <array>
  <string>m</string>
  <string>h</string>
 </array>

Cut and paste <string>m</string> from the Objective-C file to the MATLAB file, and viola, it's all ice cream and apple pie!

Markdown builder for Sublime Text 2

By far the coolest editor around is Sublime Text 2. Set it to build Markdown using PyMarkdown

{
    "cmd": ["markdown_py.bat", "$file", "-f", "$file.html"],
    "selector": "text.html.markdown"
}

Save it in your Sublime preferences folder under Packages/User/Markdown.sublime-build. Now write some Markdown, hit ctrl-B and voila, HTML generated from your Markdown.

Are MATLAB containers.Map more efficient?

UPDATE 2015-02-10 I learned something interesting recently that sheds some light on the answer to this question. I believe that MATLAB's containers.Map is a wrapper around Java's java.util.HashMap. A hash map or hash table is a zeroth order look-up table that uses a hash of the look-up key or index for tables as the address in memory that contains the corresponding value. Therefore it doesn't matter how big the hash map or hash table is, MATLAB (or Java) can immediately return the contents given the key. Since the hash is guaranteed to be unique for any given index or key, the memory address of the value will also be unique. So the answer to the question, "Are MATLAB containers.Map more efficient?" is absolutely and without any doubt a loud and resounding, "Yes!" QED

MATLAB's containers.Map's are similar to a Java HashMap or a Python dictionary. They have been around at least since 2008b believe it or not. They were billed as

Fast Key Lookup Provided with New Map Data Structure

when they were introduced. See Programming Fundamentals in 2008b Release Notes. How much memory do they realy take up? MATLAB doesn't say. It seems like they're smaller, but you'll soon see that every containers.Map is always 112 Bytes. How much faster can they really be? My guess is they are really only more efficient for really large data, that are only nested with other containers.Map's of primitive types (IE integers, doubles, &c.).

They have some limitations in subassignment and indexing.

For example:

• If the value of map('k1') is also a containers.Map,

map('k1')('k2')
Error: ()-indexing must appear last in an index expression.

then this is invalid because MATLAB only allows one pair of parentheses which must appear last. So you'll have to split this into two commands. This also allows assignment into the nested containers.Map ...

val_is_map = map('k1') % make a temp handle to the top level containers.Map
val_is_map('k2') = new_value % assignment using the handle also changes the containers.Map it points to
test = map('k1') % make another handle to test the original containers.Map was updated
test('k2') == new_value % true

... because containers.Map's are handles (IE pointers) so changing the value of a copied containers.Map changes the source.
• If the value of map('k1') is a structure,

map('k1').field = 'foo'
Error using containers.Map/subsasgn
Only one level of indexing is supported by a containers.Map.

then this is also not valid because only the top level of containers.Map can be assigned.
• Finally if map('k1') is a cell array, then good luck trying to index it.

>> dict = containers.Map({'all','amend','author','msg'}, ...
       {{'-a','--all',true}, ...
       {[],'--amend',true}, ...
       {[],'--author',true}, ...
       {'-m','--message',false}})

dict = 

  Map with properties:

        Count: 4
      KeyType: char
    ValueType: any

>> s = struct('all',{'-a','--all',true}, ...
       'amend',{[],'--amend',true}, ...
       'author',{[],'--author',true}, ...
       'msg',{'-m','--message',false})

s = 

1x3 struct array with fields:

    all
    amend
    author
    msg

>> c1 = {{'all','-a','--all',true}, ...
       {'amend',[],'--amend',true}, ...
       {'author',[],'--author',false}, ...
       {'msg','-m','--message',false}}

c1 = 

    {1x4 cell}    {1x4 cell}    {1x4 cell}    {1x4 cell}

>> c2 = {'all','-a','--all',true; ...
       'amend',[],'--amend',true; ...
       'author',[],'--author',false; ...
       'msg','-m','--message',false}

c2 = 

    'all'       '-a'    '--all'        [1]
    'amend'       []    '--amend'      [1]
    'author'      []    '--author'     [0]
    'msg'       '-m'    '--message'    [0]

>> whos
  Name      Size            Bytes  Class

  dict      4x1               112  containers.Map
  s         1x3              1670  struct
  c1        1x4              2344  cell
  c2        4x4              1896  cell

MATLAB syntax for Java inner objects

MATLAB is a Java (and .NET) interpreter, yay!
using-java-libraries-in-matlab
But calling inner objects can be tricky in MATLAB. Use the `javaMethod` and `javaObject` builtins. All the examples are from org.eclipse.jgit
javamethod
javaobject

Constants

These are the easiest of all. Though not technically an inner anything, a constant could be confusing, but it is called exactly as it would be in Java or MATLAB.

    filesystem = org.eclipse.jgit.util.FS.DETECTED

FS is a the filesystem class in org.eclipse.jgit.util pacakge. Its constant DETECTED can be accessed using regular dot notation.
org/eclipse/jgit/util/FS

Enumeration of an inner class

This is where it starts to get tricky. A nested or inner class is created in a separate class file preceded with a dollar sign. MATLAB uses the same notation, but only as a string in the javaMethod command.

    NOTRACK = javaMethod('valueOf', ...
        'org.eclipse.jgit.api.CreateBranchCommand$SetupUpstreamMode', ...
        'NOTRACK')

The `CreateBranchCommand has a nested class called `SetupUpstreamMode`. Access it in MATLAB with a dollar symbol, "$", instead of dot notation, but access it using `javaMethod`. For example it has several enumerations. `NOTRACK` is an enumeration of `SetupUpstreamMode`. Calling the `valueOf()` method of `SetupUpstreamMode` and passing it the string, "NOTRACK" inside the MATLAB builtin `javaMethod` does the trick.
org/eclipse/jgit/api/CreateBranchCommand
org/eclipse/jgit/api/CreateBranchCommand.SetupUpstreamMode
org/eclipse/jgit/api/CreateBranchCommand.SetupUpstreamMode.html#NOTRACK

Construct an inner class object

This is also easy.

    user = javaObject('org.eclipse.jgit.transport.CredentialItem$Username')

Username is an static nested class of CredentialItem. Access it using the dollar sign instead of dot notation in a call to `javaObject`.
org/eclipse/jgit/transport/CredentialItem.Username

And that's pretty much that. There are some other Java tools, like javaArray, javaMethodEDT & javaObjectEDT. I'll update this more later. Promise.

Credit for MATLAB brush: Will Schleter. Thanks!