Interactive Method Development with iPython and Vim: An Example

This morning, while reviewing the documentation for lists during exercise 38 in Learn Python 3 the Hard Way, I was inspired to attempt to create a method that would return the counts of each unique element in a list. At a future point I may try to expand it to work with dictionaries as well. While there are built in functions to perform these actions, I thought it would be educational and fun to try to write my own. I started with this idea and just started developing something interactively in iPython as I was alreadying working through parts of LP3THW in it. I thought that the process was interesting enough to document and so here we go.

Developing the Function

I worked interactively in iPython to quickly develop my functions as you can see below. The actual process required more iteration to get working just the way that I wanted, but after completing each part, I used Ctrl + to jump back through the iPython history and get the finished bits of code I wanted, giving me this:

I now had my code, but only inside of iPython. Additionaly I had been a bit sloppy about name variables while working interactively and things needed to be cleaned up a bit. Cleaning up formatting from copying and pasting from iPython and globally editing variable names are both tasks easy to accomplish in vim.

Creating Finished Script

To begin the work to finished script, I copied the definitions directly from iPython, opened a blank .py file with vim, and pasted the contents in. Now I have my code in a file, but it is messy:

screenshot_exp_1.png

Remove prompt characters

Let’s start by getting rid of the prompt characters with visual block. Go to the first line of the code and hit Ctrl + V to enter visual block. Then use the usual vim navigation to select all of the prompt characters(including the leading space) and then press x to remove them.

Cleanup Variables

Now I wanted to make all my variable names match. You can do this manually, but with vim you can easily search and replace multiple terms. The way I did this in this case was to use / to enter search, then use \ + v enable “very magic” search to enable use of | (“pipe”) to search for multiple terms. Then I used the substitute command with a blank first term to replace instances of the most recent search with the new variable name. In this case these were the commands for vim:

/\vmy_list|ml
:%s//counted_list/g
screenshot_exp_3.png

At this point I can add comments or additional functions from in vim and then save as normal. In this post I simply wanted to show the process of going from sketching an idea in iPython to moving to a complete script in vim quickly and efficiently. This write up took considerably longer to write up than it took to create the functions, bring them into vim, clean them up, and export the completed script.

Really Basic R Tips – Intro.

Aren’t there plenty of basic R tutorials out there? Yes, and they are great. However I wanted to share a few things that I wish I had understood a bit better when I first started working with R. I am sure they probably were covered in one of the many books, videos, blog posts, or tutorials that I consumed when beginning to learn R. Many of those resources are much more comprehensive and thorough than this series of posts is going to be. That is really great, and the excruciating detail IS important for a complete understanding and performing complex tasks. However, for anyone coming to R from outside of computer science(which is going to be most people, it definitely was me), you are more interested in simply getting stuff done than reveling in the minutia of what the computer is doing. This will come later. After a solid foundation is gained, these details provide the basis of much more powerful and elegant analyses. In the mean time, they can present a real “can’t see the forest for the trees” problem, as the shear number and precision of them can be overwhelming and make it hard to keep straight just what is applicable and important in any particular situation. This series will hopefully help bring clarity to some of the initial confusion of learning R, and in all likelihood, programming in general.

If you are coming to R from some field other than computer science, there is an important thing to keep in mind: coding is a tool. The word coding might bring to mind a scrawny, pale, young man, swaddled in a stretched and stained black hoody, scolioatically hunched over a keyboard, bathed in flickering green light from the screen inches from his face, typing furiously to produce text filled with arcane strings of symbols that seem to have no real meaning, surrounded by empty, crumpled Cheetos bags and piles of half empty energy drink cans. Try to forget this nerdy stereo type. For you, code, and in particular R, is simply a tool. In fact, it is tool that you already have experience with, though at first this may not seem obvious. I am making an assumption about the reader(not usually safe…but let me know if this does not apply to you, as I want to hear more about your life), that is: that you have used a calculator.

A calculator is really nothing more or less than a purpose built computer that is programmed using the standard “language” of arithmetic. Realizing this can be useful in a couple of ways. First, it helps break down the idea that you have to be a computer geek to benefit from being able to code. This is like saying an accountant has to have a fetish for building calculators from scratch to be able to use one to track the daily expenditures of a business. This is self evidently not true(though the two things are not mutually exclusive). A calculator is simply a tool and coding in R is simply a somewhat more advanced tool that lets you do MUCH more than a simple calculator.

The other important thing that thinking this way shows us, is that the output is *nearly* completely dependent on the input you provide. It may seem like “This stupid script is not doing what it is supposed to” but usually this really means “This script is not doing it what I want it to because what I want it to do is actually different from what I am telling it to do”. This often is the crux of an issue you may be having and the way through the difficulty is to step back, make sure you understand what you are trying to do, what you are telling the program to do, and what the output that it is giving you really means. The computer is simply a tool, not a wizard that can plumb the depths of your soul and use what it sees there to conjure the fulfillment of your deepest desires. You wouldn’t expect to push “x^2 + 7x + 10 = ” on a calculator and have it tell you that the intercepts are -2 and -5. The calculator is capable of helping you determine this, but you have to tell it how to do it in a way that it “understands”, specifically (-7 – sqrt(7^2 – 4*1*10))/(2*1) and then (-7 + sqrt(7^2 – 4*1*10))/(2*1) . Try to keep this in mind as you work to learn to program and realize that especially at first, what it actually means to learn to code is that you are learning how to provide the correct input to a very powerful calculator.

In the next post, the focus will be on getting setup to work with R and how to make your workflow easier. This will enable spending less time on fiddly nuts and bolts issues and instead make it easier to get real work done with R.

*I say nearly because due to the much more complex nature of computer programs and scripts, you can sometimes get errors from the software you are using itself. However, this is usually quite rare when using stable releases of common tools to perform common tasks.