Thursday, May 21, 2015

Dr. Horrible's Sing Along Blog

Love struck super villain (Neil Patrick Harris) loses to super hero (Nathan Fillion) musical. This just never get’s old. Almost as good as the official Star Wars trailer.

Thursday, May 14, 2015

[MATLAB] Do *not* use `obj.empty` to preallocate object array

FYI: You do not need to use `obj.empty` to preallocate an object array.

In fact as soon as you assign a value to any element in the object array it grows the array to that size, which allocates (or reallocates) RAM for the new object array, therefore defeating the point of preallocating space.

From Empty Arrays section of OOP documentation:

“If you make an assignment to a property value, MATLAB calls the SimpleClass constructor to grow the array to the require size:”

Instead if you want to preallocate space for an object array, grow the array once by assigning the last object first. This requires the class to have a no-arg constructor. Each time you grow your array you will reallocate RAM for it, wasting time and space, so do it once with the max expected size of the array. See Initialize Object Arrays and Initializing Arrays of Handle Objects in the OOP documentation.

  >> S(max_size) = MyClass(args)

Another option is to preallocate any other container like a cell array (best IMHO), structure or containers.Map and then fill in the class objects as they are created. An advantage to this is you don’t have to subclass matlab.mixin.Heterogeneous to group different classes together.

  >> S = cell(max_size); args = {1,2,3;4,5,6;7,8,9};
  >> for x = 1:size(args,1), S(x) = MyClass(args{x,:});end

The only time to use an empty object is if you want it as a default for the situation where nothing gets instantiated, and you need the it be an instance of the class. Of course any empty array will do this, IE: '', [] and {} are also empty.

  >> S = MyClass.empty
  >> if blah,S = MyClass(args);end
  >> if isa(S, 'MyClass') && isempty(S),do stuff; end

Another reason might be to clear defaults if the constructor is called recursively, although obj.delete will do the same thing.

I hope this helps someone; it definitely helped me understand the odd nature of MATLAB. This behavior is because everything in MATLAB is an array, even a scalar is a <1x1 double> read the C-API mxArray for external references and mwArray for compiled/deployed MATLAB for more info.

MATLAB = Matrix Laboratory
Class definitions didn’t appear until 2008. Other languages like C++, Java, Python and Ruby are object first. So the empty method is meant to duplicate the ability to be empty similar to other MATLAB datatypes such as double, cell, struct, etc. IMO outside of MATLAB it's a very artificial and somewhat meaningless construct.

Wednesday, April 22, 2015

Git big media on Windows

[UPDATE 2015-06-09] I have switched to git-fat-0.5.0 [2015-05-06]; the fork on PyPI maintained on GitHub by Alan Braithwaite of Cyan Inc. Unfortunately, I could not figure out how to clone a git-media repo, therefore git-media sucks.

Git-Fat - just works

Basically this comes with everything you need to work on Windows, Mac or Linux. It uses rsync for transport, and the rest is mostly written in Python but does depend on some libraries that are standard in Linux and have mature ports in Windows and Mac. One of the major benefits of git-fat over git-media is that it uses a .gitfat config file which updates your .git/config when git fat init is run. This is similar to git submodules and makes repos portable. In general there's more functionality and features than git-media. For example, you can list the files managed by git-fat, check for orphans and pull data from or push data to your remote storage. The only catch is that the wheel file at PyPI has metatags for win32 not amd64. This is easy to fix, but I think there are a couple of use cases that might differ from how the distribution was implemented.

Bootstrap

If you look at a Linux install, the repository has a symlink to git_fat.py in bin called git-fat. Why not just bootstrap git-fat if we only really need one file. Just dump everything in a single folder, change the file name to git-fat and make sure there's a shebang that uses #! /usr/bin/env python which git seems to prefer, then stick it on your path. This works for both msys-git and Windows cmd.

MSYS-Git

msysgit comes with Git Bash, a posix shell which includes many Linux libraries ported to Windows, such as gawk and ssh. Unfortunately it does not come with rsync, however you can get rsync from the msys source either from mingw-w64 (that's where I got it), from msys2, from the original mingw project, from mingw builds and from lots of places. You could even get it from cygwin. I usually stick files like this in my local bin folder which is always first on my path in git bash. You'll need to also grab the iconv, intl, lzma and popt msys libraries which rsync depends on. Anyway, since you have these libraries, you don't need the ones bundled in the wheel, however, git-fat is written to look for those bundled files if it detects that your platform is Windows, so just comment out those lines. You will need to change awk to gawk, since awk is a shell script that calls gawk. Again you can bootstrap this file, ie: put it in your local bin folder or install it into your Python site-packages and/or scripts folder.

__main__

This is the way I ended up using it. I put the windows libraries into the site-packages git-fat folder instead of in scripts and then in the git-fat script, added the site-packages git-fat folder to the shell's path. Then I called the git-fat module as a script by adding a __main__.py file to it which basically imports git_fat.py and calls it using Python with the -m option, but you could just as easily call the module as a script. This just keeps these extra libraries bundled together rather than dumping them into the scripts folder with everything else. Also since I mostly use git bash it doesn't put git-fat's libraries ahead of git's since they both use gawk and ssh.

Usage

Usage is extremely easy compared to git-media, which is a plus! Note these instructions are for msysgit git bash. For Windows cmd window replace git fat with git-fat everywhere. Both methods should work fine.

  1. Clone a repo that uses git-fat: git clone my://remote/repo.git
  2. At this point there are only placeholders for your files with the same names, but just sha numbers that tell git-fat which file to grab from your remote storage
  3. Run git fat init which sets up the filters and smudges that tell your local repo how to use git-fat with the .gitattributes file which is part of the repo already.
  4. Run git fat pull which downloads your files from the remote storage specified in the .gitfat, which is also already in the repo
  5. Run git fat list to see a list of managed files
  6. Run git fat status to see a list of orphans waiting to be pulled/pushed?

Creating a repo and setting it up to use git-fat is also easy. There is great help in the readme at PyPI and the readme at GitHub.

  1. Create a .gitfat file that specifies where rsync should store files. Note there are no indents. A windows UNC path seems to work fine.
    [rsync]
    remote = //server/share/repo/fat
    
  2. Create a .gitattributes file to specify which files to store at the remote
  3. Commit the .gitfat and .gitattribute files
  4. Run git fat init to set up your local .git/config
  5. Hack, commit, push, etc.
  6. Run git fat push to send stuff to your remote

Git-Media - sucky

Finally time to install Ruby. You're going to need it if you want to use git-media which let's you mix big biinary files within your git repo, but store them in some remote host, which could be google-drive, amazon s3, another server via ssh/scp or a network share. Why don't you want to store big binary files in your git repo? Since Git stores each revision instead of deltas, that means that it will quickly blow up as you make new commits.

RubyInstaller

Super easy, they recommend 2.1, no admin rights required, unzips into c:\ just like python, I checked all of the options: tk/tcl, add ruby to path and what was the last option? Then I ran gem update.

git-media gem

gem install git-media trollup right_aws

right_aws OpenSSL::Digest issue

There is a tiny issue with right_aws where it outputs the message:

Digest::Digest is deprecated; use Digest
which is easily fixed by following the comments in git-media issue #3 or right_aws pull request #176.

git-media setup

The readme on the github overview page has everything you need to know.

other large file storage

Wednesday, April 15, 2015

Recommended Python Project Layout

[UPDATE 2015-06-04] Create top level package to bundle all sub-packages and package-data together for install.

Been looking for a good, comprehensive, credible guide:

  1. Pretty good links in this SO Q&A:
    1. What is the best project structure for a Python application?
    2. Especially this one:
      1. Open Sourcing a Python Project the Right Way
  2. And maybe, maybe theses ones:
    1. Learn Python The Hard Way Exercise 46: A Project Skeleton
    2. The Hitchhiker’s Guide to Python! Structuring Your Project by Kenneth Reitz
      1. Repository Structure and Python also by Kenneth Reitz
    3. How to Package Your Python Code: Minimal Structure by Scott Torborg
    4. Interesting Things, Largely Python and Twisted Related: Filesystem structure of a Python project by Jean-Paul Calderone
  3. Of course understanding Python Modules and Packages
    1. The Python Tutorial: 6. Modules
  4. An understanding of how to install packages, and roughly I guess how pip and setuptools interact with distutils is good
    1. Python Documentation: Installing Python Modules
  5. Way later down the line it helps to understand distutils and setuptools for deploying packages
    1. Python Packaging User Guide
    2. Setuptools
    3. Python Documentation: Distributing Python Modules
    4. How to Package Your Python Code by Scott Torborg
    5. The Hitchhiker’s Guide to Packaging
  6. There is also a package that will create a boiler plate project layout for you but I don’t recommend it
    1. PyPI: Python Boilerplate Template

It's hard to pin a standard style down. Here’s mine:

MyProject/ <- git repository
|
+- .gitignore <- *.pyc, IDE files, venv/, build/, dist/, doc/_build, etc.
|
+- requirements.txt <- to install into a virtualenv
|
+- setup.py <- use setuptools, include packages, extensions, scripts and data 
|
+- MANIFEST.in <- files to include in or exclude from sdist
|
+- readme.rst <- incorporate into setup.py and docs
|
+- changes.rst <- release notes, incorporate into setup.py and docs
|
+- myproject_script.py <- script to run myproject from command line, use Python
|                         argparse for command line arguments put shebang
|                         `#! /usr/bin/env python` on 1st line and end with a
|                         `if __name__ == "__main__":` section, include in
|                         setup.py scripts section for install
|
+- any_other_scripts.py <- scripts for configuration, documentation generation
|                          or downloading assets, etc., include in setup.py
|
+- venv/ <- virtual environment to run tests, validate setup.py, development
|
+- myproject/ <- top level package keeps sub-packages and package-data together
   |             for install
   |
   +- __init__.py <- contains __version__, an API by importing key modules,
   |                 classes, functions and constants, __all__ for easy import
   |
   +- docs/ <- use Sphinx to auto-generate documentation
   |
   +- tests/ <- use nose to perform unit tests
   |
   +- other_package_data/ <- images, data files, include in setup.py
   |
   +- mypoject_lib/ <- main source code for myproject
   |  |
   |  +- __init__.py <- necessary to make mypoject_lib a sub-package
   |  |
   |  +- … <- the rest of the folders and files in myproject
   |
   +- related_project/ <- a GUI library that uses myproject_lib or tools that
      |                   myproject_lib depends on that's bundled together, etc.
      |
      +- __init__.py <- necessary to make related_project a sub-package
      |
      +- … <- the rest of the folders and files in your the related project

Thursday, March 26, 2015

Building Python x64 on Windows 7 with SDK 7.0

[UPDATE 2015-07-02] Check out Python Bootstrap a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.

I know I was just ranting about the inability to distribute Python27 w/o admin rights, but surprise! This is a piece of cake, thanks to the whoever the amazing Python developers are who maintain the PCbuild and external buildbot tools for Windows. There are also a few sites out there that have similar information on building Python for windows, but honestly everything you need is in PCbuild/readme.txt. Read it, then read it again. Seriously. Also check out the Python docs developer's guide. Hmm, thinking of setting up an AppVeyor buildbot for this.

OK, let's do this:
  1. Get Python and install it on your system. You may need a working binary to bootstrap the amd64 build.
  2. Get a working version of Microsoft SDK for Windows 7 (7.0). AFAIK Visual Studio 2013 Express Desktop or Community editions include both SDK 7.0 and 7.1, so alternately install that. Make sure that you include the redistributables in when installing the SDK because you will need them to distribute your Python build. See upgrade to vs2013 for fixes to some issues you may encounter especially if you have some other VC components already installed.
  3. Get a working svn binary for windows and put it on your path. I use CollabNet SubVersion commandline binaries. The easiest way to get and build all of the external libraries (bzip2, sqlite3, tk/tcl, etc.) is to use the Tools/buildbot batch scripts which call svn.exe.
  4. Either download the gzipped source tarball from python.org or clone the tag v2.7.10 of the cpython mercurial repository. Archives of the Hg repo are also conveniently available from the repo viewer.
  5. Read the PCbuild/readme.txt again
  6. Open the SDK command shell from the Startmenu. Change the target to Release x86, the default is Debug x64, by typing the following:
  7. C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /Release /x86
  8. Change to the directory where the source tarball is extracted.
  9. Patch the Tools/buildbot/externals batch script exactly as described in the PCbuild readme. I added the release build immediately after the debug fields.
  10. if not exist tcltk\bin\tcl85.dll (
        @rem all and install need to be separate invocations, otherwise nmakehlp is not found on install
        cd tcl-8.5.15.0\win
        nmake -f makefile.vc INSTALLDIR=..\..\tcltk clean all
        nmake -f makefile.vc INSTALLDIR=..\..\tcltk install
        cd ..\..
    )
    
    if not exist tcltk\bin\tk85.dll (
        cd tk-8.5.15.0\win
        nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 clean
        nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 all
        nmake -f makefile.vc INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.15.0 install
        cd ..\..
    )
    
    if not exist tcltk\lib\tix8.4.3\tix84.dll (
        cd tix-8.4.3.5\win
        nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk clean
        nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk all
        nmake -f python.mak DEBUG=0 MACHINE=IX86 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk install
        cd ..\..
    )
    
  11. From the archive root (Python-2.7.9) call the externals batch script. It will copy and build all of the externals from svn.python.org in a folder called externals/.
  12. Now cd to PCbuild and call build.bat. Voila, python.exe for x86.
  13. Almost there. go back up to the root of the extracted tarball and rename externals to externals-x86.
  14. Change the target to Release x64 by typing the following:
  15. C:\Users\myname\downloads\Python-2.7.9>setenv /Release /x64
  16. Set an environment variable HOST_PYTHON=C:\Python27\python.exe. You may not need this at all or you might be able to use the 32-bit version just built.
  17. Patch the buildbot externals-amd64 batch script just like the x86 script.
  18. if not exist tcltk64\bin\tcl85.dll (
        cd tcl-8.5.15.0\win
        nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 clean all
        nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 install
        cd ..\..
    )
    
    if not exist tcltk64\bin\tk85.dll (
        cd tk-8.5.15.0\win
        nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 clean
        nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 all
        nmake -f makefile.vc MACHINE=AMD64 INSTALLDIR=..\..\tcltk64 TCLDIR=..\..\tcl-8.5.15.0 install
        cd ..\..
    )
    
    if not exist tcltk64\lib\tix8.4.3\tix84.dll (
        cd tix-8.4.3.5\win
        nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 clean
        nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 all
        nmake -f python.mak DEBUG=0 MACHINE=AMD64 TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0 INSTALL_DIR=..\..\tcltk64 install
        cd ..\..
    )
    
  19. From the archive root (Python-2.7.9) call the externals-amd64 batch script
  20. Finally cd back to PCbuild and call build.bat -p x64. Voila, python.exe for x64.
  21. Add externals/tcltk to your path and run the tests
  22. C:\Users\myname\downloads\Python-2.7.9\PCbuild>set PATH=C:\Users\myname\downloads\Python-2.7.9\externals;%PATH%
    C:\Users\myname\downloads\Python-2.7.9\PCbuild>rt
    
  23. Copy the VC runtime from Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86\Microsoft.VC90.CRT\to PCbuild folder, and Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\amd64\Microsoft.VC90.CRT\to PCbuild\amd64 folder.
  24. To distribute create a similar file structure for both archtypes and copy the files into the folders
  25. Python27
    |
    +-python.exe
    |
    +-pythonw.exe
    |
    +-python27.dll
    |
    +-msvcr90.dll <- from VC/redist/MICROSOFT.VC90.CRT
    |
    +-msvcp90.dll <- from VC/redist/MICROSOFT.VC90.CRT
    |
    +-msvcm90.dll <- from VC/redist/MICROSOFT.VC90.CRT
    |
    +-MICROSOFT.VC90.CRT.manifest <- from VC/redist/MICROSOFT.VC90.CRT
    |
    +-DLLs <- all externals/tcltk/bin, PCbuild/*.dll & PCbuild/*.pyd files
    |         & PC/py.ico, PC/pyc.ico & PC/pycon.ico.
    |
    +-Lib <- same as source archive except all *.pyc files, all plat-* folders
    |        & ensurepip folder
    |
    +-libs <- all PCbuild/*.lib files
    |
    +-include <- same as source archive + PC/pyconfig.h
    |
    +-tcl <- everything in tcltk/lib
    |
    +-Scripts <- PCbuild/idle.bat
    |
    +-Doc <- set %PYTHON%=python.exe and build Sphinx docs with Sphinx if you have it
    

Note: wish85.exe and tclsh85.exe won't work with this Python installed file structure although it will work in the externals bin folder because they look for the tcl85.dll in ../lib. Also note that idle.bat needs some fixin. And also it's very important to know that most executables in Scripts created by installer packages have the path to python.exe hardwired, IE: they are initially not portable, however check out this blog for a few quick tricks to fix them.

You can download my x64 build and x86 build from dropbox. Congratulations!

Thursday, March 19, 2015

Python issue 22516: Administrator rights required for local installation

[UPDATE 2015-07-02] Check out Python Bootstrap a continuously integrated build of Python-2.7 for Windows that can be installed without admin rights.

This issue starts simple enough, but then unravels to reveal some very interesting insights. Evidently creating a Python for Windows installer that does not depend on administrator rights is not as easy as it seems. The question comes down to what we really need. Steve Dover from Microsoft breaks it down like this:

  1. Python as application
  2. This is when someone wants to use python to write scripts, do analysis, etc. Python is an application on their windows machine just like MATLAB or Excel. This could be installed per-user and possible without administrative rights.
  3. Python as library
  4. This version of Python could be used to embed Python in an application. Something similar to what pyexe and other python-freezing packages do. This could be a zip file.
  5. Python as-if system
  6. This version would be installed on a system, possibly by system admins, in a custom windows build, similar to the way that Python is integrated into Mac and Linux. It would require admin rights, and could be used by system admins to add custom functionality to the corporate OS.

Let me say 1st off that I think that #3 is absurd. I can't imagine Windows system admins ever using Python in this way. Most of them have never even heard of Python. And there is an entire .NET infrastructure to do exactly this. Why would you use Python instead of C#? Windows will never, ever be like Linux. It is not a POSIX environment, it does not use GNU tools and it does not need Python.

I think that #1 and #2 could serve the same purpose and should really be the only option available. Users who want to use Python on Windows should unzip the Python Windows distro to their System Root (ie: C:\Python27) and use it. No admin rights required. End of story. It should contain the msvcrt90 redistributable and all libraries it needs. There should be no other dependencies.

There is also a 0th option - do not distribute a binary for Windows at all. Let Windows users build from source themselves, or recommend alternate distribution, which Python.org already does on its alternative distribution page.. This is what Ruby does, and perhaps it's the best way to satisfy everyone. But the fact that official Python is available for windows is a very nice thing. Althoght Enthought and ActiveState have been around for a long time, they are private and could go out of business. Nevertheless, this does seem to be the path people are taking.

Anaconda, from Continuum Analytics, a relative newcomer, founded by Peter Wang and Travis Oliphant formerly from Enthought, seems to have become, almost overnight, the most popular source of Python on windows. It's baby sister, Miniconda, is less know but merely installs the conda package/environment manager and Python-2.7 which can be used to install more Python version and packages, whereas Anaconda preinstalls most sci-data packages. The only major concern for me with Anaconda is that it is closed source. Is it the python.org version built out of the box?WinPython on the other hand is open source on github and offers both 32 and 64 bit versions that do not require admin rights to instal. Enthought is also closed source and PortablePython only offers an older out of date 32bit version. There is also PythonXY but for me it seemed buggy.

Not sure what the future of Python on Windows will look like. If you are interested in shaping that future, I suggest you contact one of the Python devs and let them know what your use case is.

Fork me on GitHub