Thursday, January 24, 2013

Console 2 by bozho

[UPDATE 2015-04-27]: After trying ConEmu by Maximus5, there is no turning back. ConEmu is so obviously superior, it has full 256-color xterm support, properly resizes windows like normal windows when maximized and as if that wasn't enough, comes with all of the most common shells like git bash and various windows compiler environments preconfigured, including admin versions. Seriously! And I'm not alone; other Console2 users have professed their conversion to ConEmu as well.

I just went to Maximus5 GitHub release page for ConEmu, downloaded the 7z archive of the most current release and unzipped it into C:\ConEmu and awesome console emulator installed. Make a short cut of ConEmu (either 32 or 64 bit versions executable to your desktop then start - first time it asks for some simple setup questions, like where to store config file, what shells to use (select from preconfigured) and then starts. It just works. Better than Windows CMD, mintty, rxvt, powershell and yest even the revered Console2. Thanks Bozho, but it's time to move on.



The console terminal for MS Windows kicks ass compared to mintty, rxvtcmd or powershell.


Why? Let me enumerate:

  1. Resizable windows
  2. Tabs
  3. Customizable shells.
  4. Transparent background setting.
  5. Customizable copy/paste commands.
I could go on and on. This is like 7zip, just get it and start being more productive right now!

Monday, January 21, 2013

nano wrap lines

[UPDATED 2013-05-30] This post was originally intended for Linux, but since then I've also found a great set of nanorc's for mac. Surprised I never wrote a post on that, although I did pen this blip on vim. Point is, macs come with nano, but no nanorc files at all!

I admit that vim is slowly wining me back. But nano can be tweaked if you care to try. I made a few quick changes to the default .nanorc - I found a few gitcommit.nanorc online. I also added line wrapping, autoindent and tabs-to-spaces - and now nano is much better for me.

Since there is some bug in nano-2.2.6 that prevents it from reading both /etc/nanorc and ~/.nanorc copy nanorc to your home folder and edit that file.

    me@linux ~$ cp /etc/nanorc ~/.nanorc

I ended up going with this gist by keithamus because it was simple and sufficient for what I need.
But there were a few others:
Burried in a Github repo called dotfile by nicinabox is this nearly identical nanorc with a few additions. Another gist by koraktor is a more complex, but you can start to see common bits and pieces. There are a couple posts that might have some issues. Milian Wolff's site has a long nanorc, and void on Github is also very similar.

Append the gicommit.nanorc to the end of the nanorc file.

    me@linux ~$ echo include ~/.local/share/nano/gitcommit.nanorc >> ~/.nanorc

Then as you can see copy the new gitcommit.nanorc file to ~/.local/share/nano/gitcommit.nanorc.

Autoindent, mouse and tabstospaces are really nice features, but what I really needed was line-wrapping which I added by commenting out the line that says, "set nowrap." Voila, nearly as good as vim.

Here is a patch with my changes:

--- /etc/nanorc
+++ ~/.nanorc
@@ -16,7 +16,7 @@
 ## ", ', ), >, ], and }.

 ## Use auto-indentation.
-# set autoindent
+set autoindent

 ## Backup files to filename~.
 # set backup
@@ -51,7 +51,7 @@
 ## If fill is 0 or less, the line length will be the screen width less
 ## this number.
 ##
-# set fill -8
+set fill -8

 ## Enable ~/.nano_history for saving and reading search/replace strings.
 set historylog
@@ -70,7 +70,7 @@
 ## double click), and execute shortcuts.  The mouse will work in the X
 ## Window System, and on the console when gpm is running.
 ##
-# set mouse
+set mouse

 ## Allow multiple file buffers (inserting a file will put it into a
 ## separate buffer).  You must have configured with --enable-multibuffer
@@ -91,7 +91,7 @@
 # set nonewlines

 ## Don't wrap text at all.
-set nowrap
+# set nowrap

 ## Set operating directory.  nano will not read or write files outside
 ## this directory and its subdirectories.  Also, the current directory
@@ -150,10 +150,10 @@
 set suspend

 ## Use this tab size instead of the default; it must be greater than 0.
-# set tabsize 8
+set tabsize 4

 ## Convert typed tabs to spaces.
-# set tabstospaces
+set tabstospaces

 ## Save automatically on exit, don't prompt.
 # set tempfile
@@ -174,7 +174,7 @@
 # set undo

 ## Enable soft line wrapping (AKA full line display)
-# set softwrap
+set softwrap


 ## Color setup
@@ -297,3 +297,6 @@

 ## XML-type files
 include "/usr/share/nano/xml.nanorc"
+
+## git commit msg
+include /home/marko/.local/share/nano/gitcommit.nanorc

Thursday, January 17, 2013

mac vim syntax on

vim is often the default editor. It is surprisingly better than nano, although sometimes nano is the default (e.g. Ubuntu). One nice feature of vim is that it already contains several nice vimrc files for git (in addition to many, many code syntax files) e.g.: gitconfig.vimrc, gitcommit.vimrc, etc. But, vim's default syntax setting on mac is surprisingly off! So turn it on. But first start the mac terminal.

    dude@mac ~$ echo syntax on >> ~/.vimrc

in case you already have a .vimrc file, use the double greater-than sign to append to it.

Ah, now isn't that a lot nicer?

P.S. If you dare to try nano, check out these nanorc on Github, since you won't find them on your mac. (ha!)

colorful git

On Ubuntu, by default git color.ui is not set to true. But it's so nice! Not sure why this isn't the default setting? hm ...

    dude@box ~$ git config --global color.ui true

Wednesday, January 16, 2013

Django, PyDev, Virtualenv Setup

[UPDATED 2013-01-16]

Django integration in PyDev for eclipse is sweet, and contrary to what my previous post here said, it is not tricky. Although there must have been some reason I made this post right? The only gotcha for me, is that in addition to selecting your virtualenv as the Python interpreter, you must also select the standard Python lib folder so that threading.py and a few other packages that PyDev uses will be available. Luckily, PyDev gives you a nice warning message when you select a virtualenv that says exactly that.

Now selecting this folder should be straightforward but no real guidance is given because each system is a little different. For example on Ubuntu and system installed Python you would add the /usr/lib/python/ while on Windows using the official Python Windows installer, you would use C:\Python27\lib\. And of course you could use any local version of Python that you built yourself, where ever that might be. The easiest way to locate the correct folder is to find the standard Python libraries, threading.py in particular since PyDev specifically mentions it.

Now originally I also had posted an issue with django-admin.py being confused between the system installed version and the version in a virtualenv, but I have not seen that issue with the latest version of PyDev, and I am also using the global site packages in the virtualenv, which also has a different version of Django than the base install. So ... ?

To sum up, assuming you already have eclipse with PyDev installed:

  1. Install virtualenvwrapper: pip install virtualenvwrapper. This will also install the latest version of virtualenv.
  2. Make your virtualenv: mkvirtualenv myDjangoProject. This will also switch on your virtualenv.
  3. Install django in your virtuenv: pip install django. Also install anything else you want, and/or use toggleglobalsitepackages to turn on/off sitepackages.
  4. In eclipse under the Window menu option select preferences and then from the list of preference settings on the right, select PyDev > Interpreter - Python. Now in the Python Interpreters window, click New, type in a name, then browse to ~/.virtualenv/myDjangoProject/bin/python or %USERPROFILE%\.virtualenvs\be\Scripts\python.exe on Windows, click the open button then OK.
  5. Generally I don't alter or add any folders or built-ins at this next step; I just accept what PyDev tells me I need. However, as discussed above, when using a virtualenv, the standard python library folder must also be added, so do it now. Check the box for either /usr/lib/python/ or C:\Python27\lib depending on your system. If successful, you should be returned to the Python Interpreters window, and you should see your new interpreter listed with the default one. Voila.
Voila! Now you should be able to use the Django integrated submenu options with your PyDev Django projects. You can start an app, run any custom manage.py command, start a Django shell and sync your database. I hope it works for you. Enjoy and happy coding!

Tuesday, January 8, 2013

distribute/setuptools for PY4A

[UPDATE 2013-05-29] I have to say, don't bother with distribute or setuptools. This is what I do.

distribute/setuptools on python for android

**Beware this may break SL4A Python interpreter, but you will still be able to use standalone shell script Python. Also you will need root unless you have Python headers.**

[UPDATED 2013-04-06]
  1. add .pydistutils.cfg in $HOME (usually /data). If you have root access, you can make sdcard executable or write to /data/local/ (similar to /usr/local/ on Linux) and/or /data/data/com.googlecode.pythonforandroid/ (the PY4A directory). I don't necessarily recommend the install locations below, but you will need to put something so that setuptools knows where to install, since PY4A doesn't use the traditional Python file-structure. Also if bin isn't on your path, then you will need to use full pathnames for your shell commands. I used EXTERNAL_STORAGE=/mnt/sdcard/ on my VZW HTC Droid Incredible (adr6300) but on my VZW Motorola Droid RAZR HD (XT926) I use EXTERNAL_STORAGE=/storage/sdcard0/, so the location of your external storage may change.

  2. [install]
    install-purelib=/mnt/sdcard/com.googlecode.pythonforandroid/extras/python/site-packages
    install-platlib=/data/local/lib/python2.6/site-packages
    install-scripts=/data/local/bin


  3. add sitecustomize.py to any folder in $PYTHONPATH to process .pth files. I put this in /mnt/sdcard/com.googlecode.pythonforandroid/extras/python/ and it seemed to work fine.

  4. import site
    SITEPKGS = "/mnt/sdcard/com.googlecode.pythonforandroid/extras/python/site-packages"
    site.addsitedir(SITEPKGS)


  5. add site-packages to python shell script (highlighted yellow). You need both step 2 and step 3. [UPDATE 2013-04-06] I don't think that you have to do both - either use a sitecustomize.py file or add the new site-packages folders to the path. Do both for each install folder in step 1.

    #! /system/bin/sh

    export EXTERNAL_STORAGE=/mnt/sdcard/
    PYTHONPATH=${PYTHONPATH}:${EXTERNAL_STORAGE}com.googlecode.pythonforandroid/extras/python

    PYTHONPATH=${PYTHONPATH}:${EXTERNAL_STORAGE}com.googlecode.pythonforandroid/extras/python/site-packages
    PYTHONPATH=${PYTHONPATH}:/data/data/com.googlecode.pythonforandroid/files/python/lib/python2.6/lib-dynload
    export PYTHONPATH
    export TEMP=${EXTERNAL_STORAGE}com.googlecode.pythonforandroid/extras/python/tmp
    export PYTHON_EGG_CACHE=$TEMP
    export PYTHONHOME=/data/data/com.googlecode.pythonforandroid/files/python
    export LD_LIBRARY_PATH=/data/data/com.googlecode.pythonforandroid/files/python/lib
    /data/data/com.googlecode.pythonforandroid/files/python/bin/python "$@"


  6. create lib, include, bin structure in Python directory if not already existing
    • these steps only necessary if you don't already have Python headers.
    • I use C4Droid on my device but you're better off using ndk.
    • download python 2.6.2 tarball, extract and run ./configure. Do not patch.
    • add lib/python2.6/config/ and put Makefile from python installation in it
    • add pyconfig.h and all Python headers to include/python2.6/
  7. download, extract and install distribute
    • run (as su) from one level up so that site.py doesn’t interfere since path always adds current folder to path first!

    # python ./distribute-0.6.28/ setup.py install

  8. Now you can run easy_install requests and it will install Requests. But, libc from bionic is crazy, and most code does not run without patches, so I recommend using only pure python packages. In that case you could just copy the pure python modules to your site-packages folder and create a sitecustomize.py.

Monday, January 7, 2013

Fedora 17 open-vm-tools


[UPDATE 2013-04-24]
This blog post here is now obsolete as open-vm-tools-9.2.2 with Fedora specific scripts is now available (via yum) from Fedora for both Fedora-17Fedora-18 and soon to be Fedora-19 too.
Also a new stable open-vm-tools (open-vm-tools-9.2.3-1031360) has been released today 2013-04-24. A pseudo list of fixes is on Dmitry Torokhov sf.net page.


[UPDATE 2013-04-08]
Starting in Fedora 18, procps has been replaced by procps-ng. I used a soft symlink to point to libprocps: $ sudo ln -s /usr/lib/libprocps.so /usr/lib/libproc.so You will need to install the dev packages from Fedora.
Also you will have to build using CFLAGS=-Wno-deprecated-declarations to avoid errors from deprecated glib code.
AFAIK neither Fedora 16 nor 17 come with open-vm-tools, the open-source version of vmware-tools provided for guest systems by vmware. Ubuntu and OpenSUSE both have this essential tool in their repositories. In their instructions specifically for Fedora 16/17, vmware says to use the version of vmware-tools that comes with vmware-player, however, if your host machine is too old to run the latest version, then you are SOL because you can only update vmware-tools by updating vmware-player, which is a catch-22. Perhaps fortunately now, you can download current version of vmware-tools if you can figure out which version to use, because only red-hat enterprise is listed.

Fortunately you can build open-vm-tools from source, just make sure you have the dev files from the following packages:

  • gtk
  • pam
  • ... <need to finish this list>
You can install them using `sudo yum install <package>-dev`. As you are building open-vm-tools you will be prompted which dev-packages are missing, so just install and restart.

To build you follow the familiar autotools pattern: `./configure`, `make` and `make install` which will automatically put it in /usr/local/... which is probably the best place for it. I don't recommend ever installing anything into /usr/bin/ or /usr/lib/ because they will probably be overwritten on any updates/upgrades and distribution updates/upgrades. The other option would be either /opt or ~/opt but that is usually reserved for precompiled binaries that are ready to be installed, and placing an add-on software package in ~/opt will only allow the user who installs it to use it, which might be your intent - but I digress.

I had no problems installing the dev packages or building open-vm-tools for my Fedora 17 vm, but when I restarted, the guest tools service isn't running! Of course, because I need to edit my /etc/init.d folder to add open-vm-tools as a service. I'm still working on that part, there are some clues on Fedora's init scripts man page and I've also copied the init.d scripts from Ubuntu's and openSUSE's open-vm-tool scripts. And of course there is also vmware-config-tools.pl which should also provide some more clues. One thought I had was to download one of the newer vmware-tools tarballs for RHEL and see what's in that version vmware-config-tools.pl script.

I was able to start the service manually by executing vmware-user-suid-wrapper but it complains that some of the modules haven't been loaded before logging in. You can load the modules manually too, but the init.d scripts are the right way to go. Even without the modules I can get drag and drop and auto-screen resizing to work just by running vmtoolsd or the aforementioned vmware-user.suid-wrapper. I can also mount my host shared folders using vmware-mounter. If anyone wants to help out, then we will have a very nice packaged version of open-vm-tools for Fedora.

The best online help I've found for open-vm-tools and vmware-tools for linux guests actually has been on the arch-linux site. According to the first few paragraphs you can enable open-vm-tools at boot by using `systemctl start vmtoolsd` and `systemctl enable vmtoolsd`. Trying either of these commands returns `vmtoolsd.service not found`. Bummer. Googling gives these fedora links:
The last of these 3 links seems to confirm that indeed systemctl will automatically add the service for the next boot. However I don't think this takes care of all of the scripts, there are some in /etc/sysconfig. Anyway, it's obvious that you have to have the /etc/init.d or rc.d and /sysconfig scripts to start and enable the service. The vmware-config-tools.pl file from my version of vmware was too old to be relevant. There are rpm's of more current vmware-config-tools, so maybe I'll have a look in there.

Purereadline

I have been playing with SL4A (the Android scripting layer) and PY4A (Python for Android) and I wanted to see if I could get the readline package working. Readline is based on GNU's libreadline. So I was able to compile libreadline.so, but I was too lazy to build Python properly for bionic (androids version of libc) in order to get a working Python header file. This got me thinking that a standalone pure Python version of readline could be useful for anyone without the Python headers (python.h and pyconfig.h).

Download a tarball of libreadline.so and libhistory.so for bionic from my public dropbox.

FYI: I copied these shared object libraries into /data/data/com.googlecode.pythonforandroid/files/python/lib, made symlinks to remove the version number, changed the permissions to 755 and the ownership to PY4A's username and group, which I represent below as app_XX. Alternately, if you don't have root permission, you still have permissions to use sl4a's folder when using either py4a or their bash shell, so put your shared objects there instead.

    /data/data/com.googlecode.androidscripting/files/python/lib

~$ wget https://dl.dropbox.com/u/19049582/rl-6.2_arm_linux_androideabi.tar
~$ tar -xf rl-6.2_arm_linux_androideabi.tar
~$ cp rl-6.2_arm_linux_androideabi.tar/libreadline.so.6.2 /data/data/com.googlecode.pythonforandroid/files/python/lib
~$ cp rl-6.2_arm_linux_androideabi.tar/libhistory.so.6.2 /data/data/com.googlecode.pythonforandroid/files/python/lib
~$ cd /data/data/com.googlecode.pythonforandroid/files/python/lib
/data/data/com.googlecode.pythonforandroid/files/python/lib$ ln libreadline.so.6.2 libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ ln libhistory.so.6.2 libhistory.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chmod 755 libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chmod 755 libhistory.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chown app_XX:app_XX libreadline.so
/data/data/com.googlecode.pythonforandroid/files/python/lib$ chown app_XX:app_XX libhistory.so

You can test them in the SL4A interpreter by running the following script:

>>> from ctypes import *
>>> rl = CDLL("libreadline.so")
>>> rl.rl_initialize()
>>>  rl_completer_word_break_characters = c_char_p.in_dll(rl, " rl_completer_word_break_characters")
>>> print rl_completer_word_break_characters
" \t\n\"\\'`@$><=;|&{("

Therefore I present purereadline, an all Python wrapper for libreadline using ctypes. Uh ... it's still in beta, and I don't even know if it will work, but I can't think of any reason why it wouldn't. I can use ctypes fine with the shared object libraries to run functions and get exported variables. The shared object libraries compile very nicely using c4droid although I had to patch setpwent.

See issue #6 on py4a for more information and links.

dulwich porcelain

dulwich is an pure python implementation of git plumbing. It comes with a bin/scripts folder but it is incomplete - although I'm sure the developers would welcome additions. I have been working on a package called dulwich-porcelain, which facilitates adding features like checkout, fetch, push and status. Uh ... it's still beta, but contributions are welcome.

I also started a fork of dulwich with binpatches, porcelain and android branches. Android was actually the reason I origin started working on this. With just three commands it's easy to use your android device to retrieve your repos from a remote server anywhere you have 3g.

~$ dulwich clone <remote> <repo-name>
~$ cd <repo-name>
~/<repo-name>$ fetch_refs
~/<repo-name>$ checkout

n.b.: the functions also work as Python commands.

 You also need sl4a and py4a and some ssh authentication such as dropbear which is built into Cyanogenmod 7, openSSH is now used on CM9. Paramiko would be another option. I should make a post about this, but basically, I created an .ssh folder on my sdcard and created keys there. Cyanogenmod has a dropbear wiki and there is also a blogpost by TK, but both refer to creating an ssh server to communicate with your phone from a remote device, not creating keys to communicate with a remote device from your phone. Something like this should work.

~$ mkdir .ssh
~$ cd .ssh
~/.ssh$ dropbearkey -t rsa -f id_rsa
~/.ssh$ dropbearkey -y -f id_rsa > id_rsa_pub

Now you need to copy the public key, id_rsa_pub, to your remote Git server. You also need to make a small change to dulwich SSHvendor in client.py to make dropbear work. You can find the relevant changes in the android branch of my dulwich fork; basically append this class ...

class DropbearSSHVendor(object):
    def connect_ssh(self, host, command, username=None, port=None):
        import subprocess
        #FIXME: This has no way to deal with passwords..
        args = ['ssh', '-i', '/mnt/sdcard/.ssh/id_rsa']
        if port is not None:
            args.extend(['-p', str(port)])
        if username is not None:
            host = '%s@%s' % (username, host)
        args.append(host)
        proc = subprocess.Popen(args + command,
                                stdin=subprocess.PIPE,
                                stdout=subprocess.PIPE)
        return SubprocessWrapper(proc)

to the client.py file and then set get_ssh_vendor to the new vendor instead of the default one.

# Can be overridden by users
# get_ssh_vendor = SSHVendor
get_ssh_vendor = DropbearSSHVendor

Known hosts also got created automatically by dropbear in my .ssh folder on my sdcard. Dropbear doesn't let you use passwords, which is fine because dulwich has no way to deal with them. You could use keyring to get around that.

Now you can copy, send via email/bluetooth/wifi and/or edit your repo files on-the-go without needing to use a public computer or worry about ssh keys or passwords.

Thursday, January 3, 2013

converting Ubuntu to Xubuntu

[UPDATED 2012-05-13] The real fix for the apport-checkresume bug is to purge, clean and reinstall.

I have an old Compaq/HP laptop that I've been running Ubuntu on since Oneiric Ocelot (11.10). Unfortunately a pop up informed me that upgrading to Quantal Quetzal (12.10) wasn't recommended - Unity 2D isn't supported going forward, and the drivers for the ATI graphics in the Compaq/HP laptop don't support 3D acceleration.

Since Xubuntu doesn't use Unity, and I had no problem upgrading a VM on a similarly old Dell laptop also with an ATI graphics card, I decided to convert my machine to Xubuntu. For those unfamiliar with  Xubuntu, it is Canonical Ubuntu with an XFCE desktop, based on X11.

What follows are the steps, blunders, fixes and resources I used/followed to convert my system from Unity to xfce.
  1. Install xubuntu-desktop. I did this from the Ubuntu-Software-Center, but you could easily type sudo apt-get install xubuntu-desktop in a terminal.
  2. Remove ubuntu-desktop and unity. You will get some ugly warnings.
  3. If you restart now, you may get the dreaded "low graphics mode". So while you still can, edit /etc/lightdm/lightdm.conf so that the greeter is lightdm-gtk-greeter. If you do get stuck in "low graphics mode" never fear, just hit CTRL-ALT-F2 to get a terminal and type your username and password. Then you can use sudo nano /etc/lightdm/lightdm.conf to edit the config file and then  use sudo reboot and all should be well.
  4. Remove any remain unity packages. Here are some posts to follow:

Epilogue: So I was able to completely convert my old laptop successfully and upgrade to Quantal, but I did have to run these commands to push through the installation.
sudo dpkg --configure -a
sudo apt-get -f upgrade
You make also experience issues/crashes when your system resumes from suspend. Examining the detail may show the offender as apportcheckresume [1, 2, 3, 4], which is ironic, because this is the bug reporting app, and it is causing the bug, because it can't resume from suspend. I can confirm that purging apport, cleaning your repo database and reinstalling apport fixes this issue.

sudo apt-get purge apport
sudo apt-get autoremove
sudo apt-get clean
sudo apt-get install apport

Good luck and enjoy!

P.S., I was able to compare my converted Xubuntu setup with an iso installed Xubuntu, which gave me a way to check as I went along which packages should be removed.

P.P.S., synaptic and deborphan are good tools to use to clean up orphans and broken dependencies, sudo apt-get autoremove will do something similar. Although synaptic seems to resolve dependencies differently than Ubuntu-Software-Center, it has saved my ass more than once.

P.P.P.S., I suppose as an alternate to lightdm you could install gdm, but it seems like more work, and it is not the default display manager for Xubuntu.

[1] http://askubuntu.com/questions/230670/apportcheckresume-recurring-error-and-gnome-shell-fixations
[2] http://askubuntu.com/questions/203745/suspend-breaks-on-12-10s-kernel
[3] http://askubuntu.com/questions/141671/keep-getting-messages-about-internal-system-errors
[4] http://ubuntuforums.org/showthread.php?t=2073983


Fork me on GitHub