Ubuntu: Set up a virtual environment with IPython, numpy and pandas

So this is a much needed update for an earlier post on setting up a
virtual environment. The original post was based on one I found
http://technomilk.wordpress.com/2011/07/27/setting-up-our-django-site-environment-with-pythonbrew-and-virtualenv/. My prior post is https://mofj.commons.gc.cuny.edu/2013/06/25/setting-up-a-virtual-environment-with-ipython-numpy-and-pandas/. Most of the time you read about setting up virtual environments, it is in the context of web development. But the same benefits hold for analysis and research software. You want to be able to reproduce results. It also increases security not to be adding all
the unverified libraries with root level privileges. This post is a
minor modification of the outstanding tutorial I have been using for
the last few months. There are three reasons why this needs to be updated:
– there is another version of python
– it does not cover IPython
– pythonbrew which managed the versions of python is longer maintained
I will repeat the steps here. First install the c libraries that python needs to function.

I use apt-get in ubuntu so type

$ cd ~

$ sudo apt-get install libsqlite3-dev libbz2-dev libxml2-dev libxslt-dev curl

Get a non-system version of python

Then install the pyenv scripts from source. Here is the link for pyenv https://github.com/yyuu/pyenv#basic-github-checkout. Pyenv is in many ways more
sophsticated than pythonbrew. It is written in http://en.wikipedia.org/wiki/Bash_shell not any particular version of python. The advantage is that it is not
dependent on anything in the language itself. The disadvantage is that it is much harder
to read the code and understand the nature of a bug. As my advisor always used to tell
me there was nothing more brain-dead than a shell, I have spent most of my time avoiding them. I use the Bourne Again Shell that comes with Ubuntu. The syntax is tricky because everything is a string. Variable definitions can’t have any spaces. There are some good tutorials which I will include later. For now, here are three links that can tell you
the differences between bash and an interpreted language like Python.
– http://askubuntu.com/questions/110907/python-compared-to-bash
– http://superuser.com/questions/414965/when-to-use-bash-and-when-to-use-perl-python-ruby
– http://stackoverflow.com/questions/209470/can-i-use-python-as-a-bash-replacement

I am assuming you have git installed. If not, https://www.digitalocean.com/community/articles/how-to-install-git-on-ubuntu-12-04 is a good tutorial for installing git.

$ cd ~
$ git clone git://github.com/yyuu/pyenv.git .pyenv

Define environment variable PYENV_ROOT to point to the path where
pyenv repo is cloned and add $PYENV_ROOT/bin to your $PATH for access
to the pyenv command-line utility.

$ echo ‘export PYENV_ROOT=”$HOME/.pyenv”‘ >> ~/.bashrc
$ echo ‘export PATH=”$PYENV_ROOT/bin:$PATH”‘ >> ~/.bashrc

Add pyenv init to your shell to enable shims and autocompletion. Shims and binstubs are worth knowing about.  You can read up on them https://github.com/yyuu/pyenv#understanding-shims.

$ echo ‘eval “$(pyenv init -)”‘ >> ~/.bashrc

Restart your shell so the path changes take effect. You can now begin using pyenv.

$ exec $SHELL

Install Python versions into $PYENV_ROOT/versions. For example, to install Python 2.7.5, download and unpack the source, then run:

$ pyenv install 2.7.5
$ pyenv rehash

And now we have to tell the system to use this new version of python

$ pyenv local 2.7.5

Install virtualenv

We are going to do two tricky things we are going to install
virtualenv in the version of python AND install the pyenv plugin virtualenv.

$ pip install virtualenv

and to install the pyenv plugin.

$ git clone git://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

You are now ready to create a virtual environment in a non-system version of python. I don’t understand why this will not work if you are anywhere else.

$ cd ~/.pyenv/versions/2.7.5
$ pyenv virtualenv

We can list all of the virtual environments. Change directory to the
one you want to work in and in my case the virtual environment is
no-more-drug-war:

$ pyenv shell no-more-drug-war:

We can list the virtualenvs:

$ pyenv virtualenvs
dssg (created from /usr)
lc (created from /usr)
* no-more-drug-war (created from /usr)
scrp (created from /usr)
seek (created from /usr)

We can activate the virtual environment with the following command.

$ pyenv activate no-more-drug-war

You can deactivate the activate’d virtualenv by pyenv deactivate.

$ pyenv deactivate

So, in order to know what packages we have installed at any time, we install yolk.

$ pip install yolk

Do not type sudo! To see what it installed at any time:

$ yolk -l

A list of further packages for IPython are available here. Type these individually and they each may take a few minutes to install.

$ pip install jinja2

$ pip install pyzmq

$ pip install pygments

$ pip install tornado

$ pip install nose

$ pip install numpy

$ pip install scipy

$ pip install matplotlib

$ pip install pandas

$ pip install ipython

Turning it on and off

Now to get out of your virtual environment, just type

$ pyenv deactivate

To get back in, type:

$ pyenv activate no-more-drug-war

Good luck!

I will try to send a pull request to add some of this to pyenv and correct my question on stack overlfow.

Emacs IPython Notebook and the shaving of a Yak

It was this week during the project pitch exercise here at the Data Science For Social Good that I fell down a rabbit hole.  I wanted to get summary statistics on foreclosures and land values for each of Chicago’s 50 wards.  Of course I was not doing that when the well known data scientist and volunteer mentor Max Shron approached me I was fiddling with my editor. He politely introduced me to the concept of a “Yak Shave.”  As the definitive source of programming slang, the Jargon file defines it:

yak

[MIT AI Lab, after 2000: orig. probably from a Ren & Stimpy episode.] Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working on.

Now there is some disagreement over whether this is a term of derision. Wikitionary includes an alternate meaning:

The actually useless activity you do that appears important when you are consciously or unconsciously procrastinating about a larger problem.

I thought I’d get more work done if I just fixed a problem with my .emacs file, but then I spent the whole afternoon yak shaving.

gerwinski-gnu-head

This was what Max was gently chiding me for.  After all, I am a PhD student our lives are devoted to the idea of Yak Shaving, even if we don’t have a name for it.  We all want to make our projects work without admitting to our advisers that we are stuck on step 3 of our weekly 50 part research assignment.  So I put down my fiddling and went to the meeting but I did not forget about it.  The culture of our group is nothing if not polite and friendly.

Now the truth is that this piece of out is slightly over 1 GB and I could have done all of my data cleaning in R.  However we all know that Python and Pandas are the better tools and we are trying to come up to speed quickly.  (For those of us on twitter, John Myles White, has been working on the next interpreted language to enter the speed wars, Julia). This idea of yak-shaving had me giggling for an hour.  I am a recent convert to gnu/linux and  the gnu part of that partnership is FREE Software with deep collectivist roots and installation procedures reminiscent of Dostoevsky novel if it works or years in Gulag if they don’t.  Their GNU mascot looks like a close relative of the Yak.

IpythonNotebookInEmacsEven the Wikitionary entry on useless yak shaving mentions the notoriously arcane .emacs file that needs to be constantly configured. These days may be coming to an end.  Not that I did not spend the better part of a sick day fiddling with it to get two pieces of canonical free software virtuosity, Fernando Perez‘s IPython and Richard Stallman‘s Emacs to play together well.  First, I found the brilliant ein library by Takafumi Arakaki.  But that alone did not shave the Yak.  I had to abandon my ad-hoc plugins for emacs and come to terms with Emacs’ three package managers.  It was MELPA tutorial from the indefatigable Xah Lee that worked for me.  Details will follow but here is a screen shot so you know that it is possible you to shave this Yak! …And in a lot less time than it took me.

 

Learning environments for data analysis software

Welcome to my blog

This is my first blog post using the iPython notebook. I am very excited about the things it can do. Here is what I want to cover:

  • Who I am
  • What the blog will cover
  • Why I named it Measure of Justice

Evan Misshula

I am a PhD student in Criminal Justice. I try to use social networks and data mining to help people make rational decisions about public safety. I care passionately about people that the world writes off. It is no shock. There have been many times when I have been written off.

Math, Computing, Causality, Networks, Security and Ethics

Early in my graduate career, I was struck that we spend a great deal of effort policing minority communities for drug use which has little effect on the non-involved but spend way less effort protecting the banking system from hackers. I also thought that there was a lot to learn about managing threats from inside by looking at both intrusion detection and counter- intelligence. Not suprisingly, I believe in second chances. Who gets those chances and when they come are an area of great interest.

What’s in a name?

When I studied Stochastic Control, Girsanov’s Theorem governed which measures
were deformable into each other. Two measures needed to have the same sets of measure zero, to equivilent. In other words it is what we think that is impossible, not unlikely that is important.

My favorite new toy

I am excited about blogging again because I can now put code and math in the blog. I have spent a lot of time in graduate school learning new tools. This blog will hopefully document some of the challenges and help others find their way. Others blogs have certainly helped me.

We can assign variables in the ipython notebook.

In [28]:

a=5
print a
5

In [30]:

a=5
b=9 a+b 

Out[30]:

But you can also reach into the operating system and execute bash commands.

In [31]:

pwd

Out[31]:

u'/home/evan/Documents/ipython/blog/blog'

In [32]:

ls
120907-Blogging with the IPython Notebook.ipynb EvanNB1.html old/
121120-Back from PyCon Canada 2012.ipynb EvanNB1.ipynb EvanNB1_header.html fig/

This is a markdown cell

You can italicize and use boldface. It allows us to comment code and create interactive presentations. You can build lists of your favorite tools. Here are mine.

  • linux
  • emacs
  • r statistical language
  • Emacs Speaks Statistics
  • Org-mode
  • LaTeX
  • Sweave
  • Ggplot

What is most important is to LaTeX support. My favorite math equation is $e^{i\pi}+1=0$. It can also render math numbered equations:
$$e^x=\sum_{j=0}^{\infty}\frac{x^j}{j!}$$

The browser displays

The program can display the numeric or character output of programs.

In [33]:

print "hi Doug"
x=3 
hi Doug

In [9]:

x

Out[9]:

3

It can also display graphs:

In [34]:

%pylab inline
plot(rand(100))
Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

Out[34]:

[Line2D(_line0)]

In [35]:

x = linspace(0, 3*pi)
plot(x, 0.5*sin(x), label=r'$\sin(x)$') plot(x, cos(x), 'ro', label=r'$\cos(x)$') title(r'Two familiar functions')
legend()

Out[35]:

Legend

Symbolic Manipulation

The ipython notebook can also make symbolic calculations and solve complex algebraic equations:

In [36]:

%load_ext sympyprinting import sympy as sym
from sympy import *
x, y, z = sym.symbols("x y z")
The sympyprinting extension is already loaded. To reload it, use: %reload_ext sympyprinting

In [37]:

Rational(3,2)*pi + exp(I*x) / (x**2 + y**2) 

Out[37]:

$$\frac{3}{2} \pi + \frac{e^{\mathbf{\imath} x}}{x^{2} + y^{2}}$$

In [38]:

eq = ((x+y)**3 * (x+3)) eq

Out[38]:

$$\left(x + 3\right) \left(x + y\right)^{3}$$

In [39]:

expand(eq) 

Out[39]:

$$x^{4} + 3 x^{3} y + 3 x^{3} + 3 x^{2} y^{2} + 9 x^{2} y + x y^{3} + 9 x y^{2} + 3 y^{3}$$

Ipython can even calculate the derivative!!

In [40]:

diff(cos(x**2)**2 / (1+x)**2, x)

Out[40]:

$$- 4 \frac{x \operatorname{sin}\left(x^{2}\right)
\operatorname{cos}\left(x^{2}\right)}{\left(x + 1\right)^{2}} – 2 \frac{\operatorname{cos}^{2}\left(x^{2}\right)}{\left(x +
1\right)^{3}}$$

It can also display pictures and videos…

In [19]:

from IPython.display import Image
Image(filename='/home/evan/Pictures/Evan.jpg')

Out[19]:

In [20]:

from IPython.display import YouTubeVideo
YouTubeVideo('ystkKXzt9Wk') 

Out[20]:

We can even use other languages (including R)!!

This is because ipython communicates between the kernel and the browser so it knows how to send data to
another interpreter.

In [41]:

%%ruby puts "Hello from Ruby #{RUBY_VERSION}"
Hello from Ruby 1.9.3

In [42]:

%%bash echo "hello from $BASH" 
hello from /bin/bash

In [23]:

import rpy2;
from rpy2 import robjects; robjects.r("version")

Out[23]:

_
platform x86_64-unknown-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 2
minor 15.2
year 2012
month 10
day 26
svn rev 61015
language R
version.string R version 2.15.2 (2012-10-26)
nickname Trick or Treat 

In [24]:

%load_ext rmagic
The rmagic extension is already loaded. To reload it, use: %reload_ext rmagic

In [25]:

X = np.array([0,1,2,3,4]) Y = np.array([3,5,4,6,7])

In [26]:

%%R -i X,Y -o XYcoef
XYlm = lm(Y~X)
XYcoef = coef(XYlm)
print(summary(XYlm))
par(mfrow=c(2,2))
plot(XYlm)
Call:
lm(formula = Y ~ X)

Residuals:
1 2 3 4 5
-0.2 0.9 -1.0 0.1 0.2

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.2000 0.6164 5.191 0.0139 *
X 0.9000 0.2517 3.576 0.0374 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7958 on 3 degrees of freedom
Multiple R-squared: 0.81,	Adjusted R-squared: 0.7467
F-statistic: 12.79 on 1 and 3 DF, p-value: 0.03739

In [27]:

XYcoef

Out[27]:

[ 3.2  0.9]

There is more to come. Ipython does d3 interactive graphs but I have not been able to get them to work. It also handles cython (python wrapped c-code) and
mpi parallel code. More later. It is time for bed.