Shared object libraries are what many MS Windows users know as dynamic-link library (DLL), a library that can be loaded at runtime allowing shared objects in the library to be used by other programs. The other option would be statically linking dependent libraries together. Both options have pros and cons, but I am going to write about how to compile shared object libraries for all three platforms: MS Windows, GNU Linux and Apple Macintosh.
Microsoft Visual C++ doesn't have a compiler option to create a shared object library, instead objects that are to be exported as DLLs and objects that need to be imported are declared in the source. It's convenient to define macros for the DLL export and import declarations. The following example defines the macros and then uses them to import the first object which is required by the second object which is also exported.
// define macros for import and export delcarations
#define DllImport __declspec( dllimport )
#define DllExport __declspec( dllexport )
// imported_object from dll
DllImport long imported_object( arg_type *arg_name );
// exported_object to dll
DllExport long exported_object( arg_type *arg_name );
{
// some code
// call imported object
imported_object( arg_name );
// more code
}
Ideally you should include a header file with the signature of the imported object, but MSVC won't complain about it, however Linux and Apple will. Also MSVC uses the command `nmake` instead of `make` to call Makefiles. Finally, you will need to use the `SetEnv.cmd` script from the version of Windows SDK used to set all of the MSVC variables that the compiler requires and set the release mode, either DEBUG or RELEASE and the target, either /x86 or /x64.
C:\> setenv /RELEASE /X64
When it's time to compile the libraries they need to be linked. You don't need to do anything to tell MSVC where the files are located as long as their relative positions stay the same, you can use the following in a Makefile, and call it using `NMAKE` in your SDK environment after calling `SetEnv.cmd` with your release mode and target.
The compiler command is `cl`. The `/LD` flag indicates to build a DLL not an EXE. The `/Fo` flag is used to specify the output folder, and the `/Fe` flag is the desired name of the output object. All of the source files should be listed and the `/link` flag pass the shared object library to be imported at runtime to the linker, so there's no need to call the linker separately. Note that the linked library is not the DLL but the library, `lib` object also created. The `/Wall` flag enables all compiler warnings, which can be helpful.
There is another way to compile a shared library with Microsoft compilers. Pass a definitions .def file that contains the names of the exports to the linker. It's fine to use both declarations and a definitions file.
Also beware, if you are using a C++ compiler or using the .cpp extension for your source, your exports may get mangled. Use the extern "C" declaration before your DllImport or DllExport declarations. You may also need to turn off the generation of debugging info.
Linux GCC
The GNU Compiler Collection uses the `-shared` flag to indicate that the file should be a shared objected. That's it. But if you want the library to be relocatable relative to its dependencies, use the `-rpath` linker flag and set it to $ORIGIN which expands to the current location of the library, wherever it is. Because variables may get expanded this is wrapped in quotes, and in a Makefile it will be necessary to use an extra `$`. Make sure to include a header that contains the signatures for all of the linked objects that are called in your source or the compiler will complain. This is also true for Darwin.
The compiler here is `cc`, `-L` is the flag used to specify where the linked libraries are, -Wl,X,Y is a flag that passes arguments X=Y to the linker. Here we set the linker flag `-rpath` to the linker variable $ORIGIN but with some extra quotes and an extra `$` since this is in a Makefile. Later using either `objdump` or `readelf` you can see what RPATH is set to. The position independent flag, `-fPIC` is always necessary, although I honestly don't know why. The outputs are specified by the `-o` flag and any linked files are specified by the `-l` flag, so calling the linker separately is not necessary. Note that the output name includes both the "lib" prefix and the ".so" extension, but the library references by the linker omits both!
The compiler on Macintosh is called XCode, and it can be downloaded for free, after registering for a free Apple Developers account. The commands are very similar to GCC, since it is GCC ported to Darwin, except that XCode's linker doesn't have an $ORIGIN variable, instead it has `@loader_path` and `@rpath`. Also even though Darwin will look for both *.so and *.dylib files, I use *.dylib extension for Apple's shared objects because IMO this makes it easy to distinguish between Linux and Macintosh. Finally, set the libraries name using the `-install_name` flag with the `@rpath` variable so that it will look for other libraries relative to itself.
So now the loader path is set, and the name of the output file was set by the compiler to use the relative path variable `@rpath` instead of the absolute path where it was compiled which makes it relocatable anywhere, because it will look for its dependencies using the relative path of where it's actually located instead of where it was when it was first built. The result is identical to using `$ORIGIN` on a linux box. Now when `libexported_library.dylib` calls `libimport_library.dylib` it will use a relative path. You can see the library's name and path by using the `otool` command with `-l` flag.
What if you need to change the compiled name of a dynamic library that was already compiled? You can also use (deprecated) `install_name_tool` with the `-id` flag to change the path to `@rpath` after compilation and linking. How would you know what the library's compiled name is? Use the (deprecated) `otool -l` command to read the load commands which will show he name of the library and its RPATH.
[UPDATED 2014-02-26] use -install_name @rpath/libname.dylib to the compiler as an option instead of calling install_name_tool after compilation. Also unfortunately, Apple has updated their manpages, so there are no longer any links to otool or install_name_tool.
Pre pip-1.5.1, you used to have to install setuptools prior to Pip. Especially after the distribute/setuptools merger, this potentially left you with a mishmash of setuptools and distribute folders and egg files, as easy-install and pip install packages differently.
If you used the recommended ez-setup.py provided by setuptools, this would always download a tarball or zipfile in whatever directory ez-setup.py was in, probably downloads, but in the case of packages bundled with it, it could be anywhere, never to be deleted, completely undetected by you, and if you never installed pip, and never found the easy way to update your packages, then everytime you used easy-install or distutils (and probably pip too, since ez-setup.py specifies the version of setuptools required before setup.py runs) to install a package bundled with setuptools' ez-setup.py it would leave another piece of detritus.
Then mix in the pip way of doing things, which is indifferent to egg-files. If you use pip to update setuptools, (pip install -U setuptools) which is a great way, because once you have cleaned up all of the random pth and egg leftovers, pip always takes care of cleaning up its own mess. However as mentioned previously, unless that mess is an egg file. Pip doesn't touch these files, which leads to two installs of setuptools, an egg file, which for those wondering is like a wheel, or a zip file, essentially a binary, platform specific installation, and the pip way, which is an uncompressed folder with python source code, byte compiled code and any extensions, similar to what distutils would do, if it weren't monkeypatched.
Well no longer. Pip now takes care of everything. So start from scratch, delete all instances of easy-install, setuptools(, distribute if you still have it) and sure what the heck pip, everywhere, in scripts(/bin [1]) and lib. Then use get-pip.py to install both at the same time. Then periodically update setuptools using pip install -U setuptools. Ah. All better.
[1] Note: that this is really only a Windows or Mac issue, not Linux, because Python packages are included in each distro's repository. It can be an issue for virtualenvs anywhere, even Linux, so it's good to understand in principle. On a Linux share, without root access, you might have this issue with packages installed in .local. On Mac if you are using an official binary installation of Python, then you will find your scripts in /Library/Frameworks/Python.framework/Versions/2.7/bin. In a virtualenv you should find scripts in .virtualenvs/name-of-venv/bin and .virtualenvs\name-of-venv\scripts for Mac/Linux and Windows systems respectively. For non-root local installs on Linux shares the scripts folder is .local/bin.
Fetch the branch from the remote that you want to use in your project.
Read the remote branch into its own folder tree in the working copy. The name of the new directory is provided by the prefix if you append a slash at the end.
Merge updates from the sub-package into the hg-fast-export tree, squash them all into a single commit, but don't auto-commit. This gives you a chance to check the merge, before commiting. Merge prepopulates the message with a summary of the commit messages from the sub-package's remote branch.
Explicitly specify trees to merge if Git subtree merge strategy can't determine by itself.
Get diff between two trees.
I think if you want to push from your sub-package back to the remote, you would want to create an orphan branch, and push from there. This is covered in the alternate posts linked to from here.
Pushing to gh-pages for Github pages is a perfect use of subtree merge strategy - I provide an example in this SO answer.
vim is often the default editor. It is surprisingly better than nano, although sometimes nano is the default (e.g. Ubuntu). One nice feature of vim is that it already contains several nice vimrc files for git (in addition to many, many code syntax files) e.g.: gitconfig.vimrc, gitcommit.vimrc, etc. But, vim's default syntax setting on mac is surprisingly off! So turn it on. But first start the mac terminal.
dude@mac ~$ echo syntax on >> ~/.vimrc
in case you already have a .vimrc file, use the double greater-than sign to append to it.
Ah, now isn't that a lot nicer?
P.S. If you dare to try nano, check out these nanorc on Github, since you won't find them on your mac. (ha!)
[UPDATED 2018-11-14] recommend Python 3, cross out old recommendation, link to Kenneth Reitz's recommendation and Python-2 countdown
[UPDATED 2016-08-04] add links to online training and references under learning curve
[UPDATED 2015-07-10] add links to Carl Kleffner's Anaconda/Binstar Repo
[UPDATED 2015-05-18] add Mac OSX section and site.USER_BASE usage
[UPDATED 2015-04-04] add tl;dr for Python-2.7.9-x64_scidata.zip
[UPDATED 2015-04-04] pip and setuptools are part of official Python since 2.7.9.
[UPDATED 2014-04-18] add anaconda distro
[UPDATED 2013-08-01] Setuptools has been updated and has merged with Distribute.
[UPDATED 2013-04-18] I've added some better tutorial resources and the new Enthough Canopy distribution. I also removed the link URLs, sine they are embedded in the text, and rearranged the heading styles, since they were unreadable (yuk!). Also I couldn't resist linking xkcd antigravity post.
I put this together for my coworkers, but it could be applicable to anyone. Python is really easy to learn. Got Mac OSX? Then you already have Python-2.7.6. Got Windows? Just
download the installer for your system. Some people prefer to install a Python distribution, which has stable versions of the most popular Python packages bundled together in a single installer, often together with a package manager and an interactive development environment (IDE). Official Python, Continuum Analytics Anaconda, WinPython, Enthought Canopy and ActivePython Community Edition offer installers for both 32-bit and 64-bit versions of Python on Windows and Macintosh. If you have a 32-bit computer you must use the 32-bit version, however, if you have a 64-bit version, you can use either. Using a 64-bit version allows you to access more than 2GB of RAM, but you may encounter some hurdles building packages that include extensions. The distributions have already compiled the packages for your version, but if you go with the official Python, then you will probably find yourself at Cristoph Gohlke's website. Mac OSX actually already has Python 2.X built in. IMHO do not install official Python on a Mac. Either use the Python version already installed or use Homebrew to install the desired version.
Python 2 or 3?
Only download 2.7.x, not 3.2.x, since no one uses Python 3 yet. Everything is Python 2. Some exaggeration here, but you get the point.
Using a 64-bit version of Python will let you access more than 2GB of data which may be necessary for many large scientific or engineering analyses. Many packages are distributed for Windows-x64 as "wheels", but for those that are not, you can find them on Christoph Gohlke’s Python extensions page. Package distributions for Windows x64 often are marked as win_amd64. If a package is not distributed as wheel anywhere, then, in order for pip to build any C/C++ extensions in the package from source, you will need to install the Microsoft Visual C++ Compiler for Python-2.7, which is free and does not require administrative rights. See the sections on installing packages using pip and wheels below for more information.
Mac OSX already has Python-2.X installed. Mavericks and Yosemite both have Python-2.7, whereas older versions like Snow Leopard and Lion have Python-2.6. Mac OSX Python also has NumPy and Scipy already installed as well, although it might be quite old. Luckily, there is considerable support for Mac. For example there are precompiled NumPy wheels on PyPI and precompiled SciPy wheels as well. Mac OSX Python sets site.USER_BASE to ~/Library/Python/2.X a folder owned by you in your own profile, which allows you to install modules and packages without sudo using the --user scheme. First install pip from PyPI and add it to your path by running the following from a terminal ...
~ $ cd Downloads # into Downloads folder
~/Downloads $ curl -Ok https://pypi.python.org/packages/source/p/pip/pip-6.1.1.tar.gz # download pip package
~/Downloads $ python setup.py install --user # install into site.USER_BASE/lib/python/site-packages
~/Downloads $ cd # back to $HOME
~/ $ echo "# add PYTHONUSERBASE to path" >> .bash_profile
~/ $ echo 'export PATH=~/Library/Python/2.7/bin:${PATH}' >> .bash_profile # use single quotes to delay variable expansion
... now test pip by opening a new terminal and using pip to update setuptools ...
~/Downloads $ pip install --user -U setuptools #
... so that packages can be installed into your local Python library with pip using the --user option as well. You don't need to add it to your PYTHONPATH, Python adds PYTHONUSERBASE by default. You will probably need to use Homebrew to install Qt, but for everything else pip will work fine. You will need to get XCode from Apple Developer Program to compile extensions. It should be free. Finally mark my words: Beware of sudo!. Don't ever use it on Mac OSX.
Use pip from a command prompt to install new packages. Since Python-2.7.9, pip and setuptools, a dependency, are bundled with official Python but you need to update them to the newest versions.
Install Python first and make sure you select the option to add python.exe and the Scripts folder to your PATH environment variable.
Open a Windows Command Prompt and type pip install -U setuptools to update setuptools to the latest version.
To update pip on Windows you must use python.exe -m pip install -U pip. If you try to update pip using the script it will fail due because you will be denied access to the pip script itself since windows will be using it.
installing packages
There are some interesting, perhaps updated sections on Installing Python Modules in the official Python docs. Most modules are pure Python and can be simply installed using pip like this:
user@computer ~ $ pip install wheel
Usually pip finds the package at the Cheese Shop, downloads the package and its dependencies, and then installs them for you. If there are any issues, pip is nice enough to roll everything back, so using pip is the safest way to install packages. Setuptools also manages packages with the easy_install command, but I really don't recommend using it because it isn't nearly as nice as pip. Some packages are distributed as "wheels" because they have platform dependent binaries. You can also install these using pip after downloading the wheel. EG: Say you download NumPy-MKL from UC Irvine Professor Christoph Gohlke Python Extension Packages website; use the following to install it.
Many packages are simply distributed as a tarball or zip-file; pip installs these too, exactly the same way it installs wheels, although you may need the Microsoft Visual C++ Compiler for Python-2.7 for Windows or XCode for Macintosh if the source has any C/C++ extensions, which will be automatically compiled by pip. Very rarely you may need to edit setup configuration to point to shared libraries, which is well beyond the scope of this primer.
Binary distributions of packages can be distributed as "wheels", "eggs" and Windows installers. You can install a wheel using pip, eggs can be installed using easy_install and just double click a Windows installer. Wheels although the newest type of package distribution, are definitely the preferred method. You don't have to install the wheel package to use wheels, but the wheel package can convert other distributions to wheels so it might be useful.
Matplotlib is a plotting library that makes beautiful graphics and will be very familiar to MATLAB users. There are downloads for Windows (x86), Windows (x64) and Mac OS.
Ipython is about as slick as an interpreter gets, with color coding, tab completion and lots of magic. It also can create notebooks using tornado, work in a terminal or a custom Qt shell. It requires Setuptools, PyZMQ, PyQt and Tornado, and on Windows also requires Pyreadline and PyWin32). Download installers from Github for Windows (x86), Windows (x64) andMac OS X.
PyZMQ is installed using wheels via pip. See the section below on pip and then use it to install wheels. There are installation directions on the ZMQ website. Alternately install the Windows SDK7 for Windows 7 and build PyZMQ from source. It is also available (as are many of these packages) from Christoph Gohlke's website.
Python Distributions
Also if you can’t be bothered to hunt down and install these separately, the you can install a distro. There are several major ones; here are the scientific/engineering/math ones. They vary in size, packages included, and level of customization. I don’t particularly recommend them, although the convenience may get you over the initial barrier to getting started, Python is so easy to use and learn that you will quickly be hampered by the limits imposed by the distros. Also you may find the user experience burdensome EG: having to start a launcher to access your Python apps will quickly become annoying. Also there may be bugs with user supported software that may not be quickly addressed because the smaller community is just a subset of the much larger Python community that address issues in the most frequently used Python source. In particular I have found Python (x,y) to particularly buggy. Finally, any proprietary software limited to a distribution may be a limit to deployment to other user akin to trying to deploy a MATLAB app. You have been warned.
WinPython and Portable Python are Python environments that run from a USB stick or CD/DVD. These are very cool and can aid in deploying Python apps as stand alone applications!
Development Environments
A good development environment (IDE) will have syntax highlighting, autocompletion and debugging built in.
Eclipse + Pydev is the best in my opinion, it comes with debugging, a console, git built in, autocomplete and indent, it is very similar to Visual Studio but in some ways way better. Eclipse is a generic IDE that can be used for nearly every programming language.
Spyder is a very nice Python specific IDE that will be very familiar to MATLAB users. It has a built-in console, variable viewer/editor, help/documentation viewer and debugger! It comes stand alone or bundled with Python (x,y). The developers are very active and responsive.
PyCharms by JetBrains is a great IDE. I've tried it, and it's as good or better than eclipse. In fact many people prefer it, since its much smaller, focusing only on Python. JetBrains is also responsible for IntelliJ IDEA, the Java IDE that is super popular.
Sublime Text 2 is a pretty good lightweight IDE written completely in Python, the trial is free but to get rid of the occasional popup cost $60. This has become my personal favorite! It also does syntax highlighting for most major computer languages, and is extensible through plugins that can be managed easily with Package Control.
Notepad++ also has syntax highlighting for python as well as other languages.
Vim, a very lightweight but extremely popular terminal editor, also has syntax highlighting for python and other langauges, but it can be tricky to learn. Generally used from a terminal and standard in most POSIX environments, like MsysGit.
Aptana Studio 3 is a preconfigured variant of eclipse + Pydev (I haven’t tried it)
Atom is a free editor from GitHub. It's available for all platforms. It is similar to Sublime Text 2/3, but it's free, more recently maintained and has many features.
Ninja-IDE, Ninja-IDE Is Not Just Another IDE, is the latest free newcomer.
IDLE, part of standard Python, is a basic IDE with syntax highlighting, autocompletion, debugging and more.
LightTable is a free editor with syntax highlighting, some autocompletion, plugin management and and code evaluation. It looks promising and is the framework for Juno, the Julia editor.
Consoles, Terminals and Shells
A console is a place to enter command lines. Think windows CMD. It can be a convenient place to set up environmental variables such as custom paths. I recommend using ConEmu from Maximus5 GitHub Releases. Another excellent, yet older and apparently unmaintained terminal emulator is Console (v2.00) by Bozho available at Sourceforge.net.
Git Version Control
Version control is essential when writing computer code. Inevitably you will make mistakes and need to go back, you may need to work with others, you may have new ideas that you want to flesh out, or your computer will crash and take your hard drive with it. Theses are issues that version control solves. Git has emerged as the go to version control tool. For Windows msysgit is conveniently packaged with MSYS, a posix environment with many BASH and posix tools that compliments Python nicely. If you need a graphical client TortoiseGit will suffice.
Virtual Environments
I can't conscientiously write a Python primer without at least mentioning virtual environments. Occasionally a project may depend on specific Python packages, and due to threats, real or imagined, of backwards incompatibilities, a Python virtual environment is used to freeze the project dependencies. The virtualenv package accomplishes this task by creating a Python sandbox that contains only the specific packages desired. A virtual environment can also be used to test out development distributions, or other untested packages that you don't want to install into your system site-packages folder. In particular, Max OSX and Linux users may prefer to use virtual environments since they don't require root. There is also a very convenient wrapper called virtualenvwrapper that makes managing virtual environments a cinch. Anaconda users should use conda instead of pip and virtualenv.
Yes, you are so sleek, ouch you cost so much money, wow Facetime, but really, you can't copy an audio CD, or even a data CD for that matter, without jumping through hoops? Oh don't worry, apple's infinitely better by 50MMM in profits lossless format, which is only 50MB per song, so much smaller than the raw format, is what I should rip all of my music to so I can have like 6 songs on my, oh only works on Ipod or Ipad, Ihole. Point is, I have to go through 3 ridiculous steps to do something that should only take one step.
Number #1: Can't duplicate an audio CD.
P.S. It goes without saying that of course Ubuntu flavored Linux does this piece of cake - all I had to do was put it in my machine and Ubuntu asks, "Want to copy CD? Play it? Want a martini?" Why yes and all of the above, and yet you do this with zero profit? How so? I will love Linux til the day I die.
Brasero is the Linux app that does this, thanks to the thankless folks at GNOME.
I can't wait for Google to start making their own hardware. Or for Android machines to start taking over a larger market share. I get Android. I don't get Apple. For all the hype, so far not a single thing has worked hitch free. Ah, but the gadgets are sexy. The macbookpro is so sleek and cool, the iPad so convenient.
