Tuesday, July 5, 2011

Python: Beware of standard parameter values

I just spent an hour hunting for a strange bug in my Python code, and once I found it it made perfect sense (it often does when one is on that side of a bug hunt), but it was still obscure enough that I want to share it with you.

The bug occurred because of my (wrong) use of standard parameter values. Look at the following piece of code
class Foo(object):
def __init__(self, mylist = []):
self._mylist = mylist

def addSomethingToMyList(self, something):

def __str__(self):
return str(self._mylist)

if __name__ == '__main__':
f = Foo()
print "I just added 42 to f's mylist, now it contains", f
f2 = Foo()
print "I just added 117 to f2's mylist, now it contains", f2

Just by looking at it, one could expect that this would print:
I just added 42 to f's mylist, now it contains [42]
I just added 117 to f2's mylist, now it contains [117]

It doesn't though. What it prints is this:
I just added 42 to f's mylist, now it contains [42]
I just added 117 to f2's mylist, now it contains [42, 117]

Now why is that, you may ask? Well it has something to do with when standard parameter values are instantiated - which must be on class definition time. So if you use a standard parameter value that is a reference type (objects, list, dicts, sets, etc) it is that one instance that is passed on to all instances of your class.

The lesson therefore is: never use reference types in standard parameter values!

Tuesday, March 22, 2011

Building the Scavenger Python library

This is the second "building scavenger" post -- see the post immediately before this one for a description of how to build the Scavenger daemon. In this post I will describe how you can install the Scavenger Python library on a client device, so that you may start offloading tasks to your Scavenger daemon instance.

When installing the Scavenger library you designate a Python interpreter that you wish to install the Python modules into, so the first dependency is of course Python. Secondly, the script building the library uses git to fetch the source code from GitHub, so git is a dependency too.

Once you have these two dependencies installed, building and installing the Scavenger library (on a Mac or Linux machine) is as simple as 1) downloading this script, and 2) running the script with the path to the chosen Python interpreter as an argument.
$ wget https://github.com/madsdk/scavenger-library/raw/master/build/build_and_install.sh
$ sh build_and_install.sh /usr/bin/python
If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Building the Scavenger daemon

I recently moved the development of Scavenger onto GitHub. In that process I created two build scripts: One for building the Scavenger daemon on a Linux or Mac machine, and another for building and installing the Scavenger library on the same platforms. In this post I will describe the installation of the Scavenger daemon.

In order to build the Scavenger daemon, you need to install a couple of dependencies. You need git, the gcc compiler and, in order to compile the Presence daemon used for discovery, you also need to install the Qt SDK. If you are on a Mac, getting gcc installed means that you should install Xcode and getting git installed can e.g., be done using MacPorts.

After having installed these dependencies building the Scavenger daemon is as simple of downloading and running the Mac script or the Linux script. The script should be downloaded to and run from the folder in which you would like to build the Scavenger daemon. When run it will download and build Stackless Python, checkout and build the relevant GitHub projects, and configure a Scavenger daemon instance.

After having successfully run the build script, you can start a Scavenger daemon instance by entering the scavenger_daemon directory and running the start_daemon.sh script.

If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Wednesday, March 2, 2011

Building PyGTK for Mac

In the past couple of months a number of people have asked me to provide new native Mac PyGTK builds. I do not use PyGTK anymore, so I don't have time to maintain these builds - sorry. Out of curiosity I recently built PyGTK on my Snow Leopard (10.6.6) machine, just to see if the process has gotten any easier, and I must say that it has. I was able to build the entire thing without any hacking. So, to those of you asking me to provide new builds, here you have a complete HOWTO of how to build a native Mac PyGTK:

  1. Create a new user account (to make sure that no MacPorts/Fink/other libraries are available in the environment)

  2. Install git (http://code.google.com/p/git-osx-installer/downloads/list?can=3)

  3. Log in to the new account (I ssh'ed in from a Terminal)

  4. Download the build script (https://github.com/jralls/gtk-osx-build/raw/master/gtk-osx-build-setup.sh)

  5. Add ~/.local/bin to PATH

  6. Edit ~/.jhbuildrc-custom setting setup_sdk like so: setup_sdk(target="10.6", sdk_version="10.6", architectures=["i386"])

  7. Run these build commands:

    1. $ jhbuild bootstrap

    2. $ jhbuild build meta-gtk-osx-bootstrap

    3. $ jhbuild build meta-gtk-osx-core

    4. $ jhbuild build pygtk

  8. Download the PyGTK hello world script (http://www.pygtk.org/pygtk2tutorial/examples/helloworld.py)

  9. Update your PYTHONPATH: export export PYTHONPATH=$PYTHONPATH:~/gtk/inst/lib/python2.6/site-packages

  10. Make sure that 32-bit Python is used: export VERSIONER_PYTHON_PREFER_32_BIT=yes

  11. Test your build by running the hello world script: python helloworld.py

Note that to perform the last step you need to be properly logged in (not just through SSH).

I hope that this helps some of the people that have asked me about PyGTK builds. Happy hacking ;-)

Tuesday, January 11, 2011

Testing Restlet webservices from telnet

For the upcoming P2P course at Aarhus University (which I have been co-teaching for the past four years) I am currently playing around with the Restlet REST API. We want the students to learn something about peer-to-peer algorithms, protocols, and techniques, and seeing as we like a hands-on approach we always ask them to code a small P2P system as an ongoing exercise throughout the course. In past years we have asked them to do this programming using regular sockets, but that is about to change... While learning about the socket abstraction may be instrumental in teaching a coming computer scientist about the underlying constructs of the network stack, it may be time to teach them this stuff only cursory and then skip ahead to a higher level of abstraction. I mean, who (besides myself) actually works with sockets anymore?

We have therefore decided to teach the students about REST and let them use the Restlet API to create their own P2P system. But before throwing a new API at my students, I like to experiment with it myself, and I therefore dived into a little bit of Restlet programming. In the process I needed to be able to test a HTTP PUT call to my Restlet, which can not (afaik) be done easily from a webbrowser. I therefore went to my old friend telnet for help. So, if you need to test a HTTP PUT call to a webserver from telnet do the following:
telnet localhost 8182
PUT /peerlist HTTP/1.1
Host: localhost
Content-Length: 13

Hello, World!
Substitute host, URL, and content to your hearts contend.

Wednesday, January 5, 2011

Scavenger and Presence are moving to GitHub

I am currently re-booting my work on Scavenger (and therefore also on the Presence service discovery framework) as I need it for some upcoming research. In the process I have decided to move the source code from the Google Code pages and onto GitHub instead. In the last couple of months I have become increasingly fond of git---especially as a collaborative tool where other developers can freely fork my repositories, play around with the source on their own, and then send me a pull request when they are satisfied with their changes. This way of working has worked out extremely well in my small LaTeX YaSnippet project. I hope that switching to GitHub means that more people will consider adding to both Scavenger and Presence.

While moving the code I have also done quite a bit of refactoing and "spring cleaning". This means that the code has now been split into six (more or less) independent projects. All of these can be found here https://github.com/madsdk/

Installing the Scavenger client library or the Scavenger daemon has also been simplified quite a bit. Both the lib and the daemon now have build scripts that work on Mac and Linux. If I e.g., would like a Scavenger daemon running om my laptop, I would simply download and run this script. The script must be placed in the folder where you would like to install the Scavenger daemon, and it will then download and build all dependencies (Stackless Python, Presence daemon, diverse Python modules). Notice that in order to build the Presence daemon you need to have the Qt development tools installed.

Happy hacking!

Monday, December 6, 2010

Building Stackless Python on Snow Leopard

I have had a hard time debugging this problem I have had with running Stackless Python on my Snow Leopard Mac - that is until I stumbled upon this post. It seems that the problem is with using Stackless on a 64-bit Mac OS, and the solution is thus to build Stackless in a 32-bit version. So, if you're having problems with Stackless Python crashing with a "Bus error", do this when you configure during the build process:
CC="gcc -arch i386" ./configure --enable-stacklessfewerregisters
That seems to have solved the problem for me anyway ;-)