Emacs IPython Notebook and the shaving of a Yak

It was this week during the project pitch exercise here at the Data Science For Social Good that I fell down a rabbit hole.  I wanted to get summary statistics on foreclosures and land values for each of Chicago’s 50 wards.  Of course I was not doing that when the well known data scientist and volunteer mentor Max Shron approached me I was fiddling with my editor. He politely introduced me to the concept of a “Yak Shave.”  As the definitive source of programming slang, the Jargon file defines it:


[MIT AI Lab, after 2000: orig. probably from a Ren & Stimpy episode.] Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working on.

Now there is some disagreement over whether this is a term of derision. Wikitionary includes an alternate meaning:

The actually useless activity you do that appears important when you are consciously or unconsciously procrastinating about a larger problem.

I thought I’d get more work done if I just fixed a problem with my .emacs file, but then I spent the whole afternoon yak shaving.


This was what Max was gently chiding me for.  After all, I am a PhD student our lives are devoted to the idea of Yak Shaving, even if we don’t have a name for it.  We all want to make our projects work without admitting to our advisers that we are stuck on step 3 of our weekly 50 part research assignment.  So I put down my fiddling and went to the meeting but I did not forget about it.  The culture of our group is nothing if not polite and friendly.

Now the truth is that this piece of out is slightly over 1 GB and I could have done all of my data cleaning in R.  However we all know that Python and Pandas are the better tools and we are trying to come up to speed quickly.  (For those of us on twitter, John Myles White, has been working on the next interpreted language to enter the speed wars, Julia). This idea of yak-shaving had me giggling for an hour.  I am a recent convert to gnu/linux and  the gnu part of that partnership is FREE Software with deep collectivist roots and installation procedures reminiscent of Dostoevsky novel if it works or years in Gulag if they don’t.  Their GNU mascot looks like a close relative of the Yak.

IpythonNotebookInEmacsEven the Wikitionary entry on useless yak shaving mentions the notoriously arcane .emacs file that needs to be constantly configured. These days may be coming to an end.  Not that I did not spend the better part of a sick day fiddling with it to get two pieces of canonical free software virtuosity, Fernando Perez‘s IPython and Richard Stallman‘s Emacs to play together well.  First, I found the brilliant ein library by Takafumi Arakaki.  But that alone did not shave the Yak.  I had to abandon my ad-hoc plugins for emacs and come to terms with Emacs’ three package managers.  It was MELPA tutorial from the indefatigable Xah Lee that worked for me.  Details will follow but here is a screen shot so you know that it is possible you to shave this Yak! …And in a lot less time than it took me.


Emacs-IPython-Notebook Installation Tutorial

The Emacs package system is far from perfect.  The most proficient users of Emacs are unaffected by this flaw.  Many users of Emacs are experts who live at the bleeding edge of the linux kernel and gcc compiler.  This guide is for the mere mortals who have used Emacs for either its superb integration with R through ESS or the Carsten Dominick’s ubelieveable org-mode that threatens to make even PhD students productive. Basic Emacs is extraordinarily powerful and you can add a few packages with minimal knowledge.  Vincent Goulet has helped thousands of frantic stats students with his Modified Emacs for Windows/Mac OSX.  However as you want to move past that you have to add packages yourself.

Gods vs Mortals

All packages can be downloaded as source.  This can be very tricky as many packages depend on other packages which can be hard to configure for us mere mortals.  When possible it is advisable to avoid this and use a trusted repository.  A repository pools the effort and when possible automates the effort involved in keeping up to date.  This is important as bugs and security flaws in all software are discovered over time.  In this tutorial, I am going to install such a package. Another amazing piece of scientific computing is Fernando Perez’s IPython.  See my other blogpost about setting up a virtual environment for IPython.  The notebook whose developed was led by Brian Granger and Min Reagan Kelly revolutionizes both interactive computing and computer language pedagogy.  No single blog is long enough to defend such grandiose claims, but I am pretty amazed.  I just hate editing in the browser.

The Package Systems

The best blog post I found on the emacs package system was from Xah Lee.  I will work hard to add something here. There are six package systems in emacs 24.x. They are:


The first is the official system.  I am not going to cover tromey, marmalade-repo or DELPS.  I just don’t know them yet.  I was able to install other packages successfully in el-get.  It did not work for me with the Emacs-IPython-Notebook.

Let’s get started

So you may not have a .emacs file.  This file loads all of your customization files into emacs. Create it if you don’t.

$ touch .emacs

Now find it C-x C-f ~/.emacs (The capital ‘C’ means control.)

Add the following lines:

(setq package-archives ‘((“gnu” . “http://elpa.gnu.org/packages/”)

(when (>= emacs-major-version 24)
(require ‘package)
(add-to-list ‘package-archives ‘(“melpa” . “http://melpa.milkbox.net/packages/”) t)

This adds melpa to your repositories. You also need to add the line

(load-theme ‘zenburn t)

to get the zeburn theme (better colors).

package-menuNow to list all the available packages. Type M-x package-list-packages.  (M means Meta on most keyboards that is the Alt key. Also use tab completion if possible, it helps!)  We are going to take two packages.  Takafumi Arakaki’s brilliant ein and the zenburn theme colors. Type C-s to search for ein in the package list, not the github repo.  As of now you have to look for the second one in the file. Go to the beginning of the line to type ‘i’ which marks the package for installation and x which will signal emacs to actually install it.  Repeat the same for the zenburn package.

Load it into Emacs to see the change

But for any of this to work you have to re-run the .emacs file.  Type M-x eval-buffer.

newColors If everything works the colors will change.  You can examine repository for this post including a working .emacs file (and my personal .emacs file) at my github repo which is linked here.

Now to start the notebook.  Go to the directory with a notebook or where you want to keep them and open a new shell. Type

$ ipython notebook –pylab=inline

Back in emacs type.

The pay-off

M-x ein:notebooklist-open

Click on open new notebook and your IPython notebook is in your buffer where it always belonged.

ein I will cover el-get in the next blog….I promise!