## Friday, September 26, 2014

### Resize Guest Linux VM

This has happened to me several times, I thought that 20gb would be enough for my Linux virtual machine, but I was so wrong. No problem, just expand the hard drive from the VM manager and then resize the partitions using GParted. Right? Wrong!

As M1k3y would say:

"When in danger, when in doubt, run in circles scream and shout!"

Relax! It is actually quite simple.
2. Change VM disk settings so that the iso is in the CD drive and the CD drive is connected at power on.
3. Use the VM manager to first defragment the drive then expand it to the desired size.
4. During BIOS screen use ESC key to select the boot device as the CD drive.
5. Select all defaults as Debian boots off of the CD, then double click on the GParted icon to start.
6. If your unallocated portion is immediately after your main root/boot partition then you're in luck! Click the partition, select resize & move, and drag the right side to fill up the unallocated portion.
7. If however there is a partition between your old /root and your unallocated portion, you will have to move some stuff around. This is where it gets tricky. If you move /root anywhere, then your computer will boot into grub rescue> and you will have to boot the kernel manually and then reinstall grub.

## Moving Stuff Around

All of these steps will cause GParted to warn you that your system will probably not boot when you restart. In fact, it definitely won't start because GRUB will have no idea where to find your /root partition. This will cause it to boot into grub rescue, which has just enough commands for you to find and set the root partition and manually boot the kernel. Then you can reinstall grub to make the changes permanent.
1. To move partitions around, you can copy and delete a partition to move it from the left side to the right side.
2. For nested partitions, like and extended linux-swap, you can select resize & move to drag the boundary of the extended partition as far into neighboring unallocated portion as you can, then select resize & move again for the linux-swap partition and move it as far into the extended partition as possible and finally drag the opposite side of the extended partition against the linux-swap so that it is the same size as it was originally, but in a new location on the drive.
3. Finally, now that your unallocated portion is next to the /root partition, resize it to fill the entire drive.

## Booting Kernel Manually from GRUB

There are several good references, including several askUbuntu and *Nix StackExchange, but this article by Carla Schroder from the Linux Foundation is superb. The GRUB 2.00 manual is also actually quite helpful if you know where to look. Try section 4.3.
1. You can use ls to see what drives GRUB sees.
grub rescue> ls
(hd0) (hd0,msdos3) (hd0,msdos5) (fd0)

GRUB tells you it sees your hard drive, 2 partitions: /dev/sda3 and /dev/sda5 and a floppy drive.
2. List the contents of each drive to find the one with /boot. You can drop off the msdos prefix and just use the partition number.
grub rescue> ls (hd0,3)/
./ ../ boot/ etc/ bin/ sbin/ vmlinuz initrd.img ...

You just repartitioned your drive so you already new you moved your root from sda1 to sda3 (for example), so this is just validation. Note the trailing slash /, without it GRUB tells you what type of filesystem it is, eg ext4.
3. List the contents of /boot and you will see all of the kernels available.
grub rescue> ls (hd0,3)/boot
./ ../ grub/ vmlinuz-3.13.0-30-generic initrd.img-3.13.0-30-generic abi-3.13.0-30-generic System.map-3.13.0-30-generic vmlinuz-3.13.0-36-generic ...

Some Linux distros put a symlink to the linux kernel and ramdisk (initrd) in root which you can use to load them.
4. Set the root and prefix variables and load the normal module.
grub rescue> set root=(hd0,3)
grub rescue> set prefix=(hd0,3)/boot/grub
grub rescue> insmod normal
grub rescue> set
root=hd0,msdos3
prefix=(hd0,msdos3)/boot/grub

5. Load the linux kernel specifying the location of root, load the ramdisk image initrd.img and boot.
grub rescue> insmod linux
grub rescue> linux /vmlinuz root=/dev/sda3
grub rescue> initrd /initrd.img
grub rescue> boot

Your machine should boot up. If it boots up but crashes with can't find /etc/fstab, /dev, /sys, /proc and /sbin/init errors and ends up in BusyBox with a (initrmfs) prompt, then you forgot to specify the location of root!
6. Now for the crucial step, as if everything else wasn't enough, right? Everything you do in GRUB is temporary. In order to fix GRUB you will have to reinstall it. Open a Bash terminal and execute update-grub and then grub-install and you are good to go.
$update-grub Generating grub configuration file ... Found background: /usr/share/images/grub/Apollo_17_The_Last_Moon_Shot_Edit1.tga Found background image: /usr/share/images/grub/Apollo_17_The_Last_Moon_Shot_Edit1.tga Found linux image: /boot/vmlinuz-3.13.0-29-generic Found initrd image: /boot/initrd.img-3.13.0-29-generic Found linux image: /boot/vmlinuz-3.13.0-27-generic Found initrd image: /boot/initrd.img-3.13.0-27-generic Found linux image: /boot/vmlinuz-3.13.0-24-generic Found initrd image: /boot/initrd.img-3.13.0-24-generic Found memtest86+ image: /boot/memtest86+.elf Found memtest86+ image: /boot/memtest86+.bin done$ grub-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.

Note that you are installing GRUB on the hardrive, /dev/sda, not into any partition, so don't use a partition number. Reboot again with your fingers crossed, but you should be back in familiar territory.
Hope this helps! Good luck!

## Thursday, September 11, 2014

### Cory Doctorow's Homeland - Free (as in beer) YA fiction

If for some bizarre reason you find yourself looking for a YA dystopian novel set in SF in modern times, then you can read Cory Doctorow's Homeland, and its prequel Little Brother for free (if you don't mind plugs by the author for independent brick'n'mortar bookshops nationwide between chapters).

For my teaching friends, there is a study guide for high school students and another one for Little Brother.

All of Cory Doctorow's (of Boing Boing and xkcd fame) novels are available free for download on his craphound website.

I used the epub file in the Nook reader app on my Android phone. I just dropped the file in the Nook's "My Documents folder". The epub file is unfortunately incorrectly formatted for Google books. For an Amazon Kindle compatible version scroll to the bottom for the mobi file.

Maybe enjoy? (I did, but beware the themes hit very close to home. That's good right?)

# Introduction

memcached is great for sending objects from process to process. I was able to use memcached to set and retrieve objects from two different instances of MATLAB. Here I used both Java and .NET libraries.

I compiled Windows binaries of memcached-client libraries for both the Java and .NET using Java-1.7 and Visual Studio 2010 and zipped them up here.

The memcached server was compiled by Trond Norbye from Couchbase (previously Membase previously Northscale) for  32-bit and 64-bit Windows OS. See below how to run it as a service using the Python PyWin32 package.

# .NET

## Details

Memcached server version 1.4.5 patched for 64-bit Windows by Trond Norbye from Couchbase (previously Membase previously Northscale).
Memcached .NET client: Memcached.ClientLibrary port of com.meetup.memcached from sourceforge.

## Build

Microsoft Visual Studio 2010 C# - Release/AnyCPU configuration. Since the original project files were for MSVC-8, MSVC-10 started an conversion wizard that first backed up the previous project files and then created a new MSVC-10 project files including a new solution file. Then I only build the clientlib_2.0 project from the MSVC-10 IDE GUI using build from the menu, and voila a new dll.

## Usage

1. Start memcached server
2. C:\> C:\path\to\memcached.exe -vv

3. Do this on both MATLAB instances
4. >> asm = NET.addAssembly('C:\full\path\to\Memcached.ClientLibrary.dll')
>> pool = Memcached.ClientLibrary.SockIOPool.GetInstance()
>> pool.SetServers({'127.0.0.1:11211'})
>> pool.Initialize
>> mc = Memcached.ClientLibrary.MemcachedClient()

5. MATLAB instance #1
6. >> mc.Set('name','mark')
1

7. MATLAB instance #2
8. >> mc.Get('name')
mark


# Java

MATLAB and memcached also works perfectly with the original meetup.com Java version that the .NET version was ported from. The commands are exactly the same as the .NET version.

## Source:

com.meetup.memcached

## Build:

There are two branches in Greg Whalin's GitHub repository: master and performance. Performance is more recent and uses a Schooner client which they benchmark against the older meetup.com (Whalin) client as well as spymemcached (Couchbase/Membase) and xmemcache (see below), and on the wiki pages and in a powerpoint presentation in the doc folder they report that the performance branch is allegedly faster.

You can build the performance branch either from the repo, or download a zip file of the Memcached-Java-Client-3.0.0 release from the downloads page. There are several required libraries to make this run in the lib folder of the performance branch also included in the zip-file or tar-ball.

I was able to build this using jdk-7 by ...

1. extracting the file,
2. making a new bin folder at the same level as lib and src,
3. then navigating to src\main\java
4. and executing the following:
5. C:\> "C:\Program Files\Java\jdk1.7.0_55\bin\javac.exe" -verbose -cp ..\..\..\lib\commons-pool-1.5.6.jar;..\..\..\lib\slf4j-api-1.6.1.jar;..\..\..\lib\slf4j-simple-1.6.1.jar -sourcepath com\schooner\MemCached\ -source 1.6 -target 1.6 -d ..\..\..\bin com\whalin\MemCached\*.java com\schooner\MemCached\*.java com\schooner\MemCached\command\*.java

6. Then back up to the new bin folder an executing this:
7. c:\> "C:\Program Files\java\jdk1.7.0_55\bin\jar.exe" -cvf Memcached-Java-Client.jar com\

There are also newer jar files (3.0.2) from the Maven repository. All of the newer performance branch versions (3.0.0 and 3.0.2) work almost exactly like the old meetup.com client, but the package is named com.whalin.MemCached instead. Also now you will also have to add commons-pool-1.5.6.jar, slf4j-api-1.6.1.jar and slf4j-simple-1.6.1.jar to your MATLAB Java classpath using javaaddpath(). Finally the logger output is cleaned up so that you don't have to set up any logger properties as in the master branch. In fact if you do want to set up logging properties, the new performance branch doesn't use apache.org.log4j it uses slf4j simple logger.
>> javaaddpath('C:\full\path\to\commons-pool-1.5.6.jar')
>> import com.whalin.MemCached.MemCachedClient
>> import com.whalin.MemCached.SockIOPool
>> pool = SockIOPool.getInstance;
>> pool.setServers({'127.0.0.1:11211'})
>> pool.initialize
>> mc = MemCachedClient;
>> pool.getAliveCheck
1
>> mc.set('name','mark')
1
>> mc.get('name')
mark
>> pool.shutDown
>> pool.getAliveCheck
0

You can also build the master branch from the Github source.
$/c/Program\ Files/Java/jdk1.7.0_55/bin/javac.exe -verbose -classpath ../lib/log4j.jar -sourcepath com/meetup/memcached/ -source 1.6 -target 1.6 -d ../bin/ com/meetup/memcached/*.java$ /c/Program\ Files/Java/jdk1.7.0_55/bin/jar.exe -cvf MeetupMemcached.jar com/

In the master branch, the Apache logger is not cleaned up for the non-bench/test sources (with isErrorEnabled(), etc. see jlog4 documentaion), so you always get this error message:
log4j:WARN No appenders could be found for logger (com.meetup.memcached.SockIOPool).
log4j:WARN Please initialize the log4j system properly.
Because there are log calls in the MemcachedClient and SockIOPool files that never have BasicConfiguration.configure() or set any jlog4.<properties> like the appender hence the error. Of course they never meant for log messages to be displayed when deployed, so wrapping with isErrorEnabled() like the .NET version does would be better. In order to get rid of that message and output logging messages to a file configure the Apache.org log4j logger yourself by creating a FileAppender with the default layout and then configuring the BasicConfigurator.
>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
fileAppender_withDefaultLayout =
org.apache.log4j.FileAppender@60f585a2
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

An optional boolean 3rd argument appends or overwrites the log file. Relative paths seem to be working so far, but full absolute path might be better.

## Usage:

First start memcached server, then start sock IO pools and memcached clients on both matlab instances.
>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)
>> pool = com.meetup.memcached.SockIOPool.getInstance()
>> pool.setServers({'127.0.0.1:11211'})
>> pool.initialize
>> mc = com.meetup.memcached.MemcachedClient
>> mc.set('name','mark') % on 1st instance/process
1
>> mc.get('name') % on 2nd instance/process
mark


# Other clients

• xmemcached - Github:killme2008, Google Code project and Maven repo
• The Xmemcached client works well. It is the most recently updated. There are jar files on the releases page of the GitHub repo. Here is an example from the 2.0.0 release from this April. The example is from the google.code wiki User Guide in English.
>> javaaddpath('C:\full\path\to\xmemcached-2.0.0.jar')
[localhost/127.0.0.1:11211]
builder =
net.rubyeye.xmemcached.XMemcachedClientBuilder@7ef6a26
>> mc = builder.build()
log4j:WARN No appenders could be found for logger (net.rubyeye.xmemcached.XMemcachedClient).
log4j:WARN Please initialize the log4j system properly.
mc =
net.rubyeye.xmemcached.XMemcachedClient@275e6ce5
>> mc.set('name',0,'mark')
ans =
1
>> mc.get('name')
ans =
mark

To get rid of the logger message you will have to download SLF4J and add slf4j-api-X.Y.Z.jar, slf4j-simple-X.Y.Z.jar and slf4j-log4j-X.Y.Z.jar to your path first, where X.Y.Z is the SLF4J version, 1.7.7 as of 2014-08-27. Then configure log4j as above for com.meetup.memcached.
>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
fileAppender_withDefaultLayout =
org.apache.log4j.FileAppender@60f585a2
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

• Enyim - NuGet and Github:enyim
• I couldn't get this assembly to load as built, I always got an "Strong Name Validation Failed" error, perhaps because I am on a 64-bit machine, but using MSVC express.
• spymemcached - Github:dustin, Google Code project and Maven repo
• I also couldn't get this to work. Even though I could make an array of Java socket objects using InetSocket the connection was always refused even though the memcached server was operating on the specified port.

# memcached server as Windows service

This is pretty easy using Python PyWin32 win32service.

## Thursday, July 24, 2014

### Django from development to production: Apache and PostgreSQL

Perhaps like many, I start my Django projects with the default settings. That means that my database backend is SQLite and I use the simple HTTP server provided by Django for debugging. (See this post for a hack to get the debug server to run as a Windows service.) Now it's time for production, and that means using a real HTTP server like Apache and a perhaps a more flexible database like PostgreSQL. Here are some notes on the steps I took, and a couple of missteps as well. There are many other blogs with similar notes, eg Salty Crane. By the way I am using Django-1.6.5 and South-1.0. The Django How-To Deployment and Installation guides both recommend using Apache with mod-wsgi. The Install FAQs recommend PostgreSQL with psycopg2.

# Apache HTTP Server

1. Download Apache from ApacheLounge. I chose the 64-bit Windows binary built with VC10 because my system is 64-bit and I have VC10. Also it matches the mod-wsgi binary available for Windows-x64. I also chose Apache-2.4 version instead of the older 2.2.
2. The ApacheLounge zip file has instructions, but it's simple, just extract it to c:\Apahce24. I also made a shortcut to ApacheMonitor.exe in my Startup folder. This nifty app runs in the system tray giving you the server status, and lets you restart, stop or start the server or open services. Finally I changed ownership of the folder recursively to SYSTEM.
3. Download mod_wsgi from Christoph Gohlke's Python Extension Packages for Windows. How would we survive without this site? Extract the library, and copy it to your Apache/modules folder.
4. edit httpd.conf to load the mod_wsgi module. See the Django documentation on how to Use Apache with mod_wsgi and the mod_wsgi quick installation guide for configuration. Specifically add the line LoadModule wsgi_module modules/mod_wsgi.so - I added it to the end of the list of modules. Note comments are preceded by # (aka: the hash symbol).
5. Provide the mod_wsgi parameters that allow the server to serve the Django folder:
WSGIScriptAlias / /path/to/mysite.com/mysite/wsgi.py
WSGIPythonPath /path/to/mysite.com

<Directory /path/to/mysite.com/mysite>
<Files wsgi.py>
Require all granted
</Files>
</Directory>

6. Alias and allow the static and media folders:
Alias /static/ /path/to/mysite.com/STATIC_ROOT/
Alias /media/ /path/to/mysite.com/MEDIA_ROOT/
<Directory /path/to/mysite.com/STATIC_ROOT>
Require all granted
</Directory>
<Directory /path/to/mysite.com/MEDIA_ROOT>
Require all granted
</Directory>

I put these and the preceding lines in the section of httpd.conf where it says to specify which folders to allow access.
7. Use manage.py collectstatic to make all admin files and other css/js files available to server. Make sure STATIC_ROOT is set to the same folder that is aliased in the httpd.conf file, and that it is empty because it will be overwritten. I keep all of my bootstrap and tablesorter files as well as images and icons in an assets folder that is on my STATIC_DIRS list.
8. Turn DEBUG and TEMPLATE_DEBUG off.
9. You must specify ALLOWED_HOSTS, e.g. mydomain.com, or you will get a 500 server error.
10. Install Apache as service, from admin COM window, navigate to C:\Apache24\bin and type httpd.exe -k install, now go to your site and see if it works? You may need to start the service. You can use the ApacheMonitor to start it or open services. If you are not an admin, it will prompt you for admin creds.

# Now for Postgre

1. first make a copy of your db.sqlite3 file.
2. Make a mental check list of all of the apps and models you have and use manage.py dumpdata --natural myapp1 myapp2 myapp3.mymodel auth.user etc. > myfixtures.json to save them all to a JSON file. Note: only select your models, or you may get integrity or other errors when loading fixtures, and don't forget auth.user or auth.groups if you are using them. Specify models of apps using dot notation. You may need the --natural option for auth objects, not sure, but it doesn't hurt.
3. Download PostgreSQL from EnterpriseDB and install it. I also made a file in my profile's bin folder with the following code
C:\PROGRA~1\PostgreSQL\9.3\bin\psql.exe %*
so that I can use manage.py dbshell in my Git Bash shell which has %USERPROFILE%\bin (aka ~/bin) on my path by default, and I made another script for Bash
#! /bin/sh
/c/Program\ Files/9.3/bin/psql.exe "$@"  so that I can use psql -U -h localhost -p 5432  in my Git Bash shell. 4. Download the Python PostgreSQL binding psycopg (aka psycopg2) from Stickpeople Project another extraordinary good Samaritan service like Christoph Gohlke's Python Extension Packages for Windows, who also has another version of psycopg2 build from PostgreSQL-9.3 available for download. 5. Use the pgAdmin3 panel to connect to the server and create a new user and set the password, e.g. django. Just right click on Postgre-9.3 (localhost:5432), select Connect and enter your password. Then right click on Databases and select New Database. For new users right click on Login Roles and select New Login Role. 6. Create a new database for you Django project and set the new user as the owner. See the PostgreSQL notes in the Django Database documentation. 7. Update your settings for the new database. A PostgreSQL example is given in the Settings documentation for DATABASE. Set the ENGINE key to postgresql_psycopg2, NAME to the name you gave your Django project's database, USER to the owner you set for the new database, and PASSWORD to the database owner's password. HOST is probably 'localhost' and PORT is probably 5432. 8. Check one last time that you've backed up the old database and used dumpdata to save fixtures of your apps and models, including auth.users and auth.groups in a JSON file. Then use manage.py syncdb to install your Django project in the new database. Say yes or no when it asks to create a superuser, because when you load the fixtures it will overwrite any rows in your tables. 9. Use South to migrate the databases, eg manage.py migrate <app>, etc. for all apps in your project. 10. Now use manage.py loaddata myfixtures.json to load your app and model data into the new database. You should be able to load them with one file containing all of the fixtures, because Django will reference the fixture for tables or rows not yet created. Do not include any extra Django fixtures, or you will raise exceptions. Specifically do not use manage.py dumpdata without specifying any of your apps or models or Django saves extra duplicate info such as content_types, which can not be loaded into the new database because they already exist. Finally, does your app work? Test the data. Hopefully everything is hunky dory. ## Wednesday, July 16, 2014 ### WinMerge with Git WinMerge is a fine diff tool for Windows platforms. I downloaded a portable version and extracted into my root folder as C:\WinMerge\. Then I used it as a Git difftool by executing these lines in my Git Bash shell. $ git config --global difftool.winmerge.cmd '/c/winmerge/winmergeu.exe -e -u -x -wl -wr -dl LOCAL -dr REMOTE "${LOCAL}" "${REMOTE}"'
$git config --global mergetool.winmerge.cmd '/c/winmerge/winmergeu.exe -e -u -x -dl LOCAL -dr REMOTE "${LOCAL}" "${REMOTE}" "${MERGED}"'

The command line options are explained in the WinMerge Documentation. Now you can call it as your diff or merge tool.
\$ git difftool -t winmerge

## Friday, June 27, 2014

### using South to migrate a Django sqlite3 database with unique_together

This has been fixed in Django-1.7, which is still in development. So this post apples to Django-1.6.5 (or older perhaps).

If you add a new field to a Django model with the Meta option tag 'unique_together' in a project that uses a backend sqlite3 database, then you will get the following error from South when you try to migrate it:
"object reserved for internal use"
And also that South can't roll back the changes to a sqlite3 database. So unless you had a backup you're hosed.

Luckily you backed up your database before applying the changes right? So restore it and now try this:
1. Remove the offending meta option tag.
2. Migrate the change to the model.
3. Restore the meta tag option and migrate again.
Did it work? If it didn't guess you'll have to wait for Django-1.7 or switch to a more robust backend database. It did work for me, this time at least, but who knows about next time. Can't wait for Django-1.7. Can't wait for Sphinx-1.3 too, for that matter. Check out Napoleon! No more bizarre doc strings. Numpy/Google style here we come . . .

### dynamically upload file to Django model FileField

This post was inspired by this SO Q&A which is for an ImageField but which I adapted for an FileField.

Does your app generate some content that is too large or not appropriate for a database? You can store it as a FileField which points to a file in your MEDIAROOT folder. How do you upload the generated content to the FileField? Creation is a bit similar to ManyToManyField if you are familiar with using that.
1. Add the FileField to your model and set blank=True, null=True so that you can instantiate it without setting the FileField.
2. Create the object leaving off the FileField. Save the instance.
3. When you retrieve the FileField from your model you get a FieldFile (note the word order swap) which allows you to interact with the File object (a Django version of a Python file object). You could save the content to disk then call the FieldFile.save() method, but you can skip this unnecessary step. Let's assume the content can be serialized as a JSON object. The following code will upload the content to Django. Also note that FieldFile.save() content must be a django.core.files.File object not a Python file object.
    from StringIO import StringIO
import json
from django.core.files import File
f = StringIO(json.dumps(my_content, indent=2, sort_keys=True))
try:
my_model_object.my_file_field.save('my_file_name.json', File(f))
# FieldFile.save() saves the instance of the model by default
finally:
f.close()  # StringIO has no __exit__ method to use with

Setting the FileField to null=True and blank=True is only necessary if you want to upload a file object, otherwise you can pass file name as the FileField when you construct the database object. EG:
my_model_object(my_char_field='some other model fields', my_file_field='my_file_field.json')

This will upload 'my_file_field.json' from disk if it is a valid path.