Friday, April 18, 2014

Django debug server Windows service

I hijacked from Trac-Hacks to do the same for Django. It's long and rambling but gets the job done.
It's in this Gist: It was working on my Windows-7 machine, but I changed trac --> django everywhere in the source, so I might have broken something, so please let me know if you find any problems.

Thursday, April 17, 2014

Fix vcvarsall.bat to install Python-2.7 x64 extensions with v90 instead of sdk7

Previously in order to install Python-2.7 packages with extensions on my 64-bit Windows-7 machine, I have been using the Windows SDK-7 shell and setting the target and some environmental variables manually. What a drag! Then a few days ago, I discovered that the reason vcvarsall.bat has been complaining that it can't find my x64 compilers, even tho they are right there in my Visual Studio 2008 VC folder is because the path the the batch file that sets up the environmental variables for the x64 compiler was wrong. In fact all of the paths were wrong except for vcvars32.bat.

I posted a patch in this Gist:

Now to install packages that include extensions, EG: Dulwich, I just use pip install -U dulwich and easy peasy!

Thursday, April 3, 2014

My fears about MATLAB

Someone just sent me a funny email about funny MATLAB bugs. When I say "funny" I mean in the geeky-(in fact I didn't even get why they were so funny)-way. I probably was targeted for this funny email thread because I have suggested switching from MATLAB to Python, but I think my reasons for suggesting this switch have been misunderstood. It doesn't have anything to do with bugs or the quality of MATLAB software - IMHO MATLAB is outstanding. My complaint is more practical. I am afraid of the consequences of the facts that MATLAB is closed-source and TMW is privately held. I hope that this perspective is useful, however what I express below is my opinion and based on my experiences not necessarily on facts, so please forgive me if there are some inaccuracies.

Here are my fears about MATLAB enumerated, and why I suggest switching to Python:

  1. If MATLAB goes out of business, your code is useless, and they could decide to close shop any day, because they’re not publicly traded.
  2. You can’t use it without a license! In my case I have to VPN to work to use it, this makes me not want to work from home because using vpn is just a smidge slower and another hurdle to getting started and of course forget working on airplanes or anywhere where there’s no internet.
  3. The $$$ for the toolboxes is obscene! Especially considering they are repackaging what is already free in the public domain in C/C++, FORTRAN and Python. It’s a racket. Ostensibly you could argue that the money is for support, but there are similar paid support options for Python (Anaconda, Enthought, ActiveState, etc.) that are substantially cheaper, and rival MATLAB support options to varying degrees of success.
  4. MATLAB is not portable, unless of course you use the MATLAB compiler toolbox and one of its toolbox add-ons like MATLAB builder-NE, builder-JA or builder-XL. That right there floors me as a scam, you can’t port it to someone unless you buy not one but 2 extra $$$$$ toolboxes and they are by far the most expensive toolboxes! And that’s the way it is with everything they sell. EG parallel toolbox is useless without the extra distributed computing toolbox, both of which are $$$$$! And you pay per number of machines in the computing cluster. Back to portability. If you want to share your MATLAB code with someone they will have to purchase MATLAB to use it.
  5. There is not enough consideration for backwards compatibility. I believe that TMW tries to minimize backwards incompatibility, but they still do it anyway, and in their business model, where they only make money if you maintain your license, they have a strong incentive to introduce backwards incompatibility. Other languages have the opposite incentive, in order to maintain the code-base of external packages, libraries and toolboxes that would fail if backwards incompatibilities were introduced. IE you will never find backwards incompatibilities between any version of Python-2.7.x (or in between versions of Python-3.x.x, although Python-3 is not strictly backwards compatible with Python-2).
  6. It is very powerful, but it is not the most powerful coding platform out there. FORTRAN as ancient as it is, still exceeds MATLAB and the newer Julia is faster than them all! See the comparison chart where is says “High-Performance JIT Compiler”.

Other than these few issues, I revere MATLAB and have used it extensively for decades. I rarely have issues with it and it is clearly one of the best analysis tools for science and engineering, but it isn’t the only one and it’s debatable if it’s the best.

I adhere to the philosophy of "use the right tool for the right job" and that nothing is binary, even quanta come in different flavors.

Friday, March 21, 2014

Universal Scientific Prayer of Hope and Gratitude

Oh Great Universe!

I praise your infinite complexities,
your beautiful creations,
and above all your life giving ability.

[Please let our understanding of <insert whatever>
be sufficient for <insert whatever>.]


Tuesday, February 18, 2014

shared object files in MSVC, GCC and XCODE

Shared object libraries are what many MS Windows users know as dynamic-link library (DLL), a library that can be loaded at runtime allowing shared objects in the library to be used by other programs. The other option would be statically linking dependent libraries together. Both options have pros and cons, but I am going to write about how to compile shared object libraries for all three platforms: MS Windows, GNU Linux and Apple Macintosh.

MS Windows MSVC

Microsoft Visual C++ doesn't have a compiler option to create a shared object library, instead objects that are to be exported as DLLs and objects that need to be imported are declared in the source. It's convenient to define macros for the DLL export and import declarations. The following example defines the macros and then uses them to import the first object which is required by the second object which is also exported.
// define macros for import and export delcarations
#define DllImport   __declspec( dllimport )
#define DllExport   __declspec( dllexport )

// imported_object from dll
DllImport long imported_object( arg_type *arg_name );

// exported_object to dll
DllExport long exported_object( arg_type *arg_name );
// some code

// call imported object
imported_object( arg_name );

// more code
Ideally you should include a header file with the signature of the imported object, but MSVC won't complain about it, however Linux and Apple will. Also MSVC uses the command `nmake` instead of `make` to call Makefiles. Finally, you will need to use the `SetEnv.cmd` script from the version of Windows SDK used to set all of the MSVC variables that the compiler requires and set the release mode, either DEBUG or RELEASE and the target, either /x86 or /x64.
C:\> setenv /RELEASE /X64
When it's time to compile the libraries they need to be linked. You don't need to do anything to tell MSVC where the files are located as long as their relative positions stay the same, you can use the following in a Makefile, and call it using `NMAKE` in your SDK environment after calling `SetEnv.cmd` with your release mode and target.
cl /LD /Wall /Fo output\folder /Fe output\folder\exported_library.dll \
exported_library_src1.c exported_library_src2.c exported_library_src3.c  \
/link output\folder\imported_library.lib
The compiler command is `cl`. The `/LD` flag indicates to build a DLL not an EXE. The `/Fo` flag is used to specify the output folder, and the `/Fe` flag is the desired name of the output object. All of the source files should be listed and the `/link` flag pass the shared object library to be imported at runtime to the linker, so there's no need to call the linker separately. Note that the linked library is not the DLL but the library, `lib` object also created. The `/Wall` flag enables all compiler warnings, which can be helpful.
There is another way to compile a shared library with Microsoft compilers. Pass a definitions .def file that contains the names of the exports to the linker. It's fine to use both declarations and a definitions file.
Also beware, if you are using a C++ compiler or using the .cpp extension for your source, your exports may get mangled. Use the extern "C" declaration before your DllImport or DllExport declarations. You may also need to turn off the generation of debugging info.

Linux GCC

The GNU Compiler Collection uses the `-shared` flag to indicate that the file should be a shared objected. That's it. But if you want the library to be relocatable relative to its dependencies, use the `-rpath` linker flag and set it to $ORIGIN which expands to the current location of the library, wherever it is. Because variables may get expanded this is wrapped in quotes, and in a Makefile it will be necessary to use an extra `$`. Make sure to include a header that contains the signatures for all of the linked objects that are called in your source or the compiler will complain. This is also true for Darwin.
cc -Lpath/to/links -Wl,-rpath='$${ORIGIN}' -shared -fPIC -Wall \
exported_library_src1.c exported_library_src2.c exported_library_src3.c \
-o output/folder/ -limport_library
The compiler here is `cc`, `-L` is the flag used to specify where the linked libraries are, -Wl,X,Y is a flag that passes arguments X=Y to the linker. Here we set the linker flag `-rpath` to the linker variable $ORIGIN but with some extra quotes and an extra `$` since this is in a Makefile. Later using either `objdump` or `readelf` you can see what RPATH is set to. The position independent flag, `-fPIC` is always necessary, although I honestly don't know why. The outputs are specified by the `-o` flag and any linked files are specified by the `-l` flag, so calling the linker separately is not necessary. Note that the output name includes both the "lib" prefix and the ".so" extension, but the library references by the linker omits both!

Macintosh XCode

The compiler on Macintosh is called XCode, and it can be downloaded for free, after registering for a free Apple Developers account. The commands are very similar to GCC, since it is GCC ported to Darwin, except that XCode's linker doesn't have an $ORIGIN variable, instead it has `@loader_path` and `@rpath`. Also even though Darwin will look for both *.so and *.dylib files, I use *.dylib extension for Apple's shared objects because IMO this makes it easy to distinguish between Linux and Macintosh. Finally, set the libraries name using the `-install_name` flag with the `@rpath` variable so that it will look for other libraries relative to itself.
cc -Lpath/to/links -Wl,-rpath,'@loader_path' -shared -fPIC -Wall \
exported_library_src1.c exported_library_src2.c exported_library_src3.c \
-o output/folder/libexported_library.dylib -limport_library \
-install_name @rpath/libexported_library.dylib
So now the loader path is set, and the name of the output file was set by the compiler to use the relative path variable `@rpath` instead of the absolute path where it was compiled which makes it relocatable anywhere, because it will look for its dependencies using the relative path of where it's actually located instead of where it was when it was first built. The result is identical to using `$ORIGIN` on a linux box. Now when `libexported_library.dylib` calls `libimport_library.dylib` it will use a relative path. You can see the library's name and path by using the `otool` command with `-l` flag.

What if you need to change the compiled name of a dynamic library that was already compiled? You can also use (deprecated) `install_name_tool` with the `-id` flag to change the path to `@rpath` after compilation and linking. How would you know what the library's compiled name is? Use the (deprecated) `otool -l` command to read the load commands which will show he name of the library and its RPATH.
$ install_name_tool -id @rpath/libexported_library.dylib output/folder/libexported_library.dylib
$ otool -l libexported_library.dylib
[UPDATED 2014-02-26] use -install_name @rpath/libname.dylib to the compiler as an option instead of calling install_name_tool after compilation. Also unfortunately, Apple has updated their manpages, so there are no longer any links to otool or install_name_tool.


[1] DYLD (1)
[2] Run-Path Dependent Libraries
[3] Dynamic Libraries - Introduction

Tuesday, January 28, 2014

setuptools detritus

Pre pip-1.5.1, you used to have to install setuptools prior to Pip. Especially after the distribute/setuptools merger, this potentially left you with a mishmash of setuptools and distribute folders and egg files, as easy-install and pip install packages differently.

If you used the recommended provided by setuptools, this would always download a tarball or zipfile in whatever directory was in, probably downloads, but in the case of packages bundled with it, it could be anywhere, never to be deleted, completely undetected by you, and if you never installed pip, and never found the easy way to update your packages, then everytime you used easy-install or distutils (and probably pip too, since specifies the version of setuptools required before runs) to install a package bundled with setuptools' it would leave another piece of detritus.

Then mix in the pip way of doing things, which is indifferent to egg-files. If you use pip to update setuptools, (pip install -U setuptools) which is a great way, because once you have cleaned up all of the random pth and egg leftovers, pip always takes care of cleaning up its own mess. However as mentioned previously, unless that mess is an egg file. Pip doesn't touch these files, which leads to two installs of setuptools, an egg file, which for those wondering is like a wheel, or a zip file, essentially a binary, platform specific installation, and the pip way, which is an uncompressed folder with python source code, byte compiled code and any extensions, similar to what distutils would do, if it weren't monkeypatched.

Well no longer. Pip now takes care of everything. So start from scratch, delete all instances of easy-install, setuptools(, distribute if you still have it) and sure what the heck pip, everywhere, in scripts(/bin [1]) and lib. Then use to install both at the same time. Then periodically update setuptools using pip install -U setuptools. Ah. All better.

[1] Note: that this is really only a Windows or Mac issue, not Linux, because Python packages are included in each distro's repository. It can be an issue for virtualenvs anywhere, even Linux, so it's good to understand in principle. On a Linux share, without root access, you might have this issue with packages installed in .local. On Mac if you are using an official binary installation of Python, then you will find your scripts in /Library/Frameworks/Python.framework/Versions/2.7/bin. In a virtualenv you should find scripts in .virtualenvs/name-of-venv/bin and .virtualenvs\name-of-venv\scripts for Mac/Linux and Windows systems respectively. For non-root local installs on Linux shares the scripts folder is .local/bin.
Fork me on GitHub Creative Commons License
poquitopicante by Mark Mikofski is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at
Permissions beyond the scope of this license may be available at